Sep 11 03:38:18.845819 ip-10-0-138-121 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 11 03:38:18.845832 ip-10-0-138-121 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 11 03:38:18.845841 ip-10-0-138-121 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 11 03:38:18.846142 ip-10-0-138-121 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 11 03:38:28.995790 ip-10-0-138-121 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 11 03:38:28.995806 ip-10-0-138-121 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 24d6f83f4c70417894f6a85514096ba9 --
Sep 11 03:40:10.581815 ip-10-0-138-121 systemd[1]: Starting Kubernetes Kubelet...
Sep 11 03:40:11.031135 ip-10-0-138-121 kubenswrapper[2580]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 11 03:40:11.031135 ip-10-0-138-121 kubenswrapper[2580]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 11 03:40:11.031135 ip-10-0-138-121 kubenswrapper[2580]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 11 03:40:11.031135 ip-10-0-138-121 kubenswrapper[2580]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 11 03:40:11.031135 ip-10-0-138-121 kubenswrapper[2580]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 11 03:40:11.033120 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.033035    2580 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 11 03:40:11.037207 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.037191    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 03:40:11.037207 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.037208    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 03:40:11.037276 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.037212    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 03:40:11.037276 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.037215    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 03:40:11.037276 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.037218    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 03:40:11.037276 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.037221    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 03:40:11.037276 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.037225    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 03:40:11.037276 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.037228    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 03:40:11.037276 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.037231    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 03:40:11.037276 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.037233    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 03:40:11.037276 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.037237    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 03:40:11.037276 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.037240    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 03:40:11.037276 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.037243    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 03:40:11.037276 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.037245    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 03:40:11.037276 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.037248    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 03:40:11.037276 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.037251    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 03:40:11.037276 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.037254    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 03:40:11.037276 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.037256    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 03:40:11.037276 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.037259    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 03:40:11.037276 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.037261    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 03:40:11.037276 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.037264    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 03:40:11.037276 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.037270    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 03:40:11.037757 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.037273    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 03:40:11.037757 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.037275    2580 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 03:40:11.037757 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.037278    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 03:40:11.037757 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.037281    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 03:40:11.037757 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.037284    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 03:40:11.037757 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.037287    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 03:40:11.037757 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.037290    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 03:40:11.037757 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.037293    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 03:40:11.037757 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.037295    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 03:40:11.037757 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.037298    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 03:40:11.037757 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.037302    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 03:40:11.037757 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.037306    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 03:40:11.037757 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.037309    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 03:40:11.037757 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.037312    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 03:40:11.037757 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.037314    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 03:40:11.037757 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.037317    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 03:40:11.037757 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.037320    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 03:40:11.037757 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.037322    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 03:40:11.037757 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.037325    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 03:40:11.038222 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.037327    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 03:40:11.038222 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.037330    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 03:40:11.038222 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.037332    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 03:40:11.038222 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.037335    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 03:40:11.038222 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.037337    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 03:40:11.038222 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.037340    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 03:40:11.038222 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.037342    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 03:40:11.038222 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.037345    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 03:40:11.038222 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.037348    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 03:40:11.038222 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.037351    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 03:40:11.038222 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.037353    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 03:40:11.038222 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.037356    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 03:40:11.038222 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.037359    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 03:40:11.038222 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.037361    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 03:40:11.038222 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.037365    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 03:40:11.038222 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.037368    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 03:40:11.038222 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.037370    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 03:40:11.038222 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.037373    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 03:40:11.038222 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.037375    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 03:40:11.038222 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.037378    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 03:40:11.038725 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.037381    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 03:40:11.038725 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.037383    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 03:40:11.038725 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.037386    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 03:40:11.038725 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.037388    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 03:40:11.038725 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.037391    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 03:40:11.038725 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.037393    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 03:40:11.038725 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.037396    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 03:40:11.038725 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.037398    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 03:40:11.038725 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.037401    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 03:40:11.038725 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.037405    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 03:40:11.038725 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.037408    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 03:40:11.038725 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.037412    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 03:40:11.038725 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.037415    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 03:40:11.038725 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.037418    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 03:40:11.038725 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.037422    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 03:40:11.038725 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.037424    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 03:40:11.038725 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.037427    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 03:40:11.038725 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.037430    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 03:40:11.038725 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.037432    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 03:40:11.038725 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.037435    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 03:40:11.039224 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.037438    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 03:40:11.039224 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.037440    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 03:40:11.039224 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.037443    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 03:40:11.039224 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.037445    2580 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 03:40:11.039224 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.037448    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 03:40:11.039224 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.037451    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 03:40:11.039224 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.037453    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 03:40:11.039224 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.037456    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 03:40:11.039224 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.037461    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 03:40:11.039224 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.037465    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 03:40:11.039224 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.037468    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 03:40:11.039224 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.037471    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 03:40:11.039224 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.037474    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 03:40:11.039224 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.037477    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 03:40:11.039224 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.037480    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 03:40:11.039224 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.037482    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 03:40:11.039224 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.037485    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 03:40:11.039224 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.037487    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 03:40:11.039224 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.038189    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 03:40:11.039699 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.038195    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 03:40:11.039699 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.038198    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 03:40:11.039699 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.038200    2580 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 03:40:11.039699 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.038203    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 03:40:11.039699 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.038206    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 03:40:11.039699 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.038208    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 03:40:11.039699 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.038211    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 03:40:11.039699 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.038215    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 03:40:11.039699 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.038218    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 03:40:11.039699 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.038221    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 03:40:11.039699 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.038224    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 03:40:11.039699 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.038226    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 03:40:11.039699 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.038229    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 03:40:11.039699 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.038232    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 03:40:11.039699 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.038234    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 03:40:11.039699 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.038237    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 03:40:11.039699 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.038240    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 03:40:11.039699 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.038242    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 03:40:11.039699 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.038245    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 03:40:11.039699 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.038248    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 03:40:11.040211 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.038250    2580 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 03:40:11.040211 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.038253    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 03:40:11.040211 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.038256    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 03:40:11.040211 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.038258    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 03:40:11.040211 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.038261    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 03:40:11.040211 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.038264    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 03:40:11.040211 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.038267    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 03:40:11.040211 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.038269    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 03:40:11.040211 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.038272    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 03:40:11.040211 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.038274    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 03:40:11.040211 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.038277    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 03:40:11.040211 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.038280    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 03:40:11.040211 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.038283    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 03:40:11.040211 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.038286    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 03:40:11.040211 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.038288    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 03:40:11.040211 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.038290    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 03:40:11.040211 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.038295    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 03:40:11.040211 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.038299    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 03:40:11.040211 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.038302    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 03:40:11.040697 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.038305    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 03:40:11.040697 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.038308    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 03:40:11.040697 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.038312    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 03:40:11.040697 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.038315    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 03:40:11.040697 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.038318    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 03:40:11.040697 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.038320    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 03:40:11.040697 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.038323    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 03:40:11.040697 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.038326    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 03:40:11.040697 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.038329    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 03:40:11.040697 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.038331    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 03:40:11.040697 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.038333    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 03:40:11.040697 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.038337    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 03:40:11.040697 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.038340    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 03:40:11.040697 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.038342    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 03:40:11.040697 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.038345    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 03:40:11.040697 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.038348    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 03:40:11.040697 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.038350    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 03:40:11.040697 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.038353    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 03:40:11.040697 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.038355    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 03:40:11.040697 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.038358    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 03:40:11.041198 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.038361    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 03:40:11.041198 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.038363    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 03:40:11.041198 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.038365    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 03:40:11.041198 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.038368    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 03:40:11.041198 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.038372    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 03:40:11.041198 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.038375    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 03:40:11.041198 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.038377    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 03:40:11.041198 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.038380    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 03:40:11.041198 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.038382    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 03:40:11.041198 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.038385    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 03:40:11.041198 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.038388    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 03:40:11.041198 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.038390    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 03:40:11.041198 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.038393    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 03:40:11.041198 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.038395    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 03:40:11.041198 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.038398    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 03:40:11.041198 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.038402    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 03:40:11.041198 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.038405    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 03:40:11.041198 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.038407    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 03:40:11.041198 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.038409    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 03:40:11.041198 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.038412    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 03:40:11.041741 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.038415    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 03:40:11.041741 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.038417    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 03:40:11.041741 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.038420    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 03:40:11.041741 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.038422    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 03:40:11.041741 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.038425    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 03:40:11.041741 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.038428    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 03:40:11.041741 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.038430    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 03:40:11.041741 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.038437    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 03:40:11.041741 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.038441    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 03:40:11.041741 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.038444    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 03:40:11.041741 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.038446    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 03:40:11.041741 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.038449    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 03:40:11.041741 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.038453    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 03:40:11.041741 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.038456    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 03:40:11.041741 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.038460    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 03:40:11.041741 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.038463    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 03:40:11.041741 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.038466    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 03:40:11.041741 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.038469    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 03:40:11.041741 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.038472    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 03:40:11.042291 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.038859    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 03:40:11.042291 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.038866    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 03:40:11.042291 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.038869    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 03:40:11.042291 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.038873    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 03:40:11.042291 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.038876    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 03:40:11.042291 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.038879    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 03:40:11.042291 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.038881    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 03:40:11.042291 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.038884    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 03:40:11.042291 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.038887    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 03:40:11.042291 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.038890    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 03:40:11.042291 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.038893    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 03:40:11.042291 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.038896    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 03:40:11.042291 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.038898    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 03:40:11.042291 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.038901    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 03:40:11.042291 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.038903    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 03:40:11.042291 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.038905    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 03:40:11.042291 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.038908    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 03:40:11.042291 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.038912    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 03:40:11.042291 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.038915    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 03:40:11.042291 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.038917    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 03:40:11.042801 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.038922    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 03:40:11.042801 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.038925    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 03:40:11.042801 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.038927    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 03:40:11.042801 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.038930    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 03:40:11.042801 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.038932    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 03:40:11.042801 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.038935    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 03:40:11.042801 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.038937    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 03:40:11.042801 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.038940    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 03:40:11.042801 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.038942    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 03:40:11.042801 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.038945    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 03:40:11.042801 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.038947    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 03:40:11.042801 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.038949    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 03:40:11.042801 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.038952    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 03:40:11.042801 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.038954    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 03:40:11.042801 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.038957    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 03:40:11.042801 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.038959    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 03:40:11.042801 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.038962    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 03:40:11.042801 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.038965    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 03:40:11.042801 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.038967    2580 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 03:40:11.042801 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.038970    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 03:40:11.043330 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.038972    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 03:40:11.043330 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.038975    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 03:40:11.043330 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.038978    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 03:40:11.043330 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.038980    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 03:40:11.043330 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.038982    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 03:40:11.043330 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.038985    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 03:40:11.043330 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.038987    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 03:40:11.043330 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.038990    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 03:40:11.043330 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.038992    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 03:40:11.043330 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.038995    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 03:40:11.043330 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.038998    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 03:40:11.043330 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.039000    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 03:40:11.043330 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.039004    2580 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 03:40:11.043330 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.039007    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 03:40:11.043330 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.039009    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 03:40:11.043330 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.039011    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 03:40:11.043330 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.039014    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 03:40:11.043330 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.039016    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 03:40:11.043330 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.039019    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 03:40:11.043330 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.039021    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 03:40:11.043830 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.039024    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 03:40:11.043830 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.039026    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 03:40:11.043830 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.039029    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 03:40:11.043830 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.039031    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 03:40:11.043830 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.039033    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 03:40:11.043830 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.039036    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 03:40:11.043830 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.039039    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 03:40:11.043830 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.039043    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 03:40:11.043830 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.039046    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 03:40:11.043830 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.039049    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 03:40:11.043830 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.039052    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 03:40:11.043830 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.039055    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 03:40:11.043830 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.039057    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 03:40:11.043830 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.039060    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 03:40:11.043830 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.039063    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 03:40:11.043830 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.039065    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 03:40:11.043830 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.039068    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 03:40:11.043830 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.039070    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 03:40:11.043830 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.039072    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 03:40:11.044338 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.039075    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 03:40:11.044338 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.039080    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 03:40:11.044338 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.039083    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 03:40:11.044338 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.039086    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 03:40:11.044338 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.039090    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 03:40:11.044338 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.039095    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 03:40:11.044338 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.039098    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 03:40:11.044338 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.039100    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 03:40:11.044338 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.039103    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 03:40:11.044338 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.039106    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 03:40:11.044338 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.039108    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 03:40:11.044338 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.039111    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 03:40:11.044338 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.039113    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 03:40:11.044338 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.039116    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 03:40:11.044338 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.039118    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 03:40:11.044338 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.039120    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 03:40:11.044338 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.039123    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 03:40:11.044338 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.039125    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 03:40:11.044338 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.039128    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 03:40:11.044828 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.039130    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 03:40:11.044828 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.039205    2580 flags.go:64] FLAG: --address="0.0.0.0"
Sep 11 03:40:11.044828 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.039237    2580 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 11 03:40:11.044828 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.039243    2580 flags.go:64] FLAG: --anonymous-auth="true"
Sep 11 03:40:11.044828 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.039247    2580 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 11 03:40:11.044828 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.039252    2580 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 11 03:40:11.044828 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.039255    2580 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 11 03:40:11.044828 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.039259    2580 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 11 03:40:11.044828 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.039263    2580 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 11 03:40:11.044828 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.039268    2580 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 11 03:40:11.044828 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.039271    2580 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 11 03:40:11.044828 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.039274    2580 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 11 03:40:11.044828 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.039277    2580 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 11 03:40:11.044828 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.039280    2580 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 11 03:40:11.044828 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.039284    2580 flags.go:64] FLAG: --cgroup-root=""
Sep 11 03:40:11.044828 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.039287    2580 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 11 03:40:11.044828 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.039290    2580 flags.go:64] FLAG: --client-ca-file=""
Sep 11 03:40:11.044828 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.039292    2580 flags.go:64] FLAG: --cloud-provider="external"
Sep 11 03:40:11.044828 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.039297    2580 flags.go:64] FLAG: --cluster-dns="[]"
Sep 11 03:40:11.044828 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.039302    2580 flags.go:64] FLAG: --cluster-domain=""
Sep 11 03:40:11.044828 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.039305    2580 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 11 03:40:11.044828 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.039308    2580 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 11 03:40:11.044828 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.039311    2580 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 11 03:40:11.044828 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.039314    2580 flags.go:64] FLAG: --container-log-max-files="5"
Sep 11 03:40:11.045416 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.039318    2580 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 11 03:40:11.045416 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.039345    2580 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 11 03:40:11.045416 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.039349    2580 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 11 03:40:11.045416 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.039352    2580 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 11 03:40:11.045416 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.039356    2580 flags.go:64] FLAG: --contention-profiling="false"
Sep 11 03:40:11.045416 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.039358    2580 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 11 03:40:11.045416 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.039361    2580 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 11 03:40:11.045416 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.039365    2580 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 11 03:40:11.045416 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.039368    2580 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 11 03:40:11.045416 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.039372    2580 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 11 03:40:11.045416 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.039375    2580 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 11 03:40:11.045416 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.039378    2580 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 11 03:40:11.045416 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.039380    2580 flags.go:64] FLAG: --enable-load-reader="false"
Sep 11 03:40:11.045416 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.039383    2580 flags.go:64] FLAG: --enable-server="true"
Sep 11 03:40:11.045416 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.039387    2580 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 11 03:40:11.045416 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.039391    2580 flags.go:64] FLAG: --event-burst="100"
Sep 11 03:40:11.045416 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.039394    2580 flags.go:64] FLAG: --event-qps="50"
Sep 11 03:40:11.045416 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.039397    2580 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 11 03:40:11.045416 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.039400    2580 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 11 03:40:11.045416 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.039403    2580 flags.go:64] FLAG: --eviction-hard=""
Sep 11 03:40:11.045416 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.039409    2580 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 11 03:40:11.045416 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.039412    2580 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 11 03:40:11.045416 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.039415    2580 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 11 03:40:11.045416 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.039418    2580 flags.go:64] FLAG: --eviction-soft=""
Sep 11 03:40:11.045416 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.039421    2580 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 11 03:40:11.046038 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.039424    2580 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 11 03:40:11.046038 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.039428    2580 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 11 03:40:11.046038 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.039431    2580 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 11 03:40:11.046038 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.039434    2580 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 11 03:40:11.046038 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.039437    2580 flags.go:64] FLAG: --fail-swap-on="true"
Sep 11 03:40:11.046038 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.039440    2580 flags.go:64] FLAG: --feature-gates=""
Sep 11 03:40:11.046038 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.039444    2580 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 11 03:40:11.046038 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.039447    2580 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 11 03:40:11.046038 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.039450    2580 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 11 03:40:11.046038 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.039453    2580 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 11 03:40:11.046038 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.039456    2580 flags.go:64] FLAG: --healthz-port="10248"
Sep 11 03:40:11.046038 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.039459    2580 flags.go:64] FLAG: --help="false"
Sep 11 03:40:11.046038 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.039462    2580 flags.go:64] FLAG: --hostname-override="ip-10-0-138-121.ec2.internal"
Sep 11 03:40:11.046038 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.039465    2580 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 11 03:40:11.046038 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.039469    2580 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 11 03:40:11.046038 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.039471    2580 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 11 03:40:11.046038 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.039475    2580 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 11 03:40:11.046038 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.039478    2580 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 11 03:40:11.046038 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.039481    2580 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 11 03:40:11.046038 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.039484    2580 flags.go:64] FLAG: --image-service-endpoint=""
Sep 11 03:40:11.046038 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.039486    2580 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 11 03:40:11.046038 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.039489    2580 flags.go:64] FLAG: --kube-api-burst="100"
Sep 11 03:40:11.046038 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.039492    2580 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 11 03:40:11.046651 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.039496    2580 flags.go:64] FLAG: --kube-api-qps="50"
Sep 11 03:40:11.046651 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.039499    2580 flags.go:64] FLAG: --kube-reserved=""
Sep 11 03:40:11.046651 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.039502    2580 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 11 03:40:11.046651 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.039505    2580 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 11 03:40:11.046651 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.039510    2580 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 11 03:40:11.046651 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.039513    2580 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 11 03:40:11.046651 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.039516    2580 flags.go:64] FLAG: --lock-file=""
Sep 11 03:40:11.046651 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.039519    2580 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 11 03:40:11.046651 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.039522    2580 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 11 03:40:11.046651 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.039525    2580 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 11 03:40:11.046651 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.039543    2580 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 11 03:40:11.046651 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.039546    2580 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 11 03:40:11.046651 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.039549    2580 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 11 03:40:11.046651 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.039552    2580 flags.go:64] FLAG: --logging-format="text"
Sep 11 03:40:11.046651 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.039555    2580 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 11 03:40:11.046651 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.039558    2580 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 11 03:40:11.046651 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.039561    2580 flags.go:64] FLAG: --manifest-url=""
Sep 11 03:40:11.046651 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.039564    2580 flags.go:64] FLAG: --manifest-url-header=""
Sep 11 03:40:11.046651 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.039569    2580 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 11 03:40:11.046651 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.039572    2580 flags.go:64] FLAG: --max-open-files="1000000"
Sep 11 03:40:11.046651 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.039576    2580 flags.go:64] FLAG: --max-pods="110"
Sep 11 03:40:11.046651 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.039580    2580 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 11 03:40:11.046651 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.039583    2580 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 11 03:40:11.046651 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.039586    2580 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 11 03:40:11.046651 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.039589    2580 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 11 03:40:11.047281 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.039592    2580 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 11 03:40:11.047281 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.039595    2580 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 11 03:40:11.047281 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.039598    2580 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 11 03:40:11.047281 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.039606    2580 flags.go:64] FLAG: --node-status-max-images="50"
Sep 11 03:40:11.047281 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.039609    2580 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 11 03:40:11.047281 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.039612    2580 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 11 03:40:11.047281 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.039615    2580 flags.go:64] FLAG: --pod-cidr=""
Sep 11 03:40:11.047281 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.039618    2580 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10"
Sep 11 03:40:11.047281 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.039624    2580 flags.go:64] FLAG: --pod-manifest-path=""
Sep 11 03:40:11.047281 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.039627    2580 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 11 03:40:11.047281 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.039630    2580 flags.go:64] FLAG: --pods-per-core="0"
Sep 11 03:40:11.047281 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.039633    2580 flags.go:64] FLAG: --port="10250"
Sep 11 03:40:11.047281 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.039638    2580 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 11 03:40:11.047281 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.039642    2580 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-060a5e4c88a71f8ae"
Sep 11 03:40:11.047281 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.039645    2580 flags.go:64] FLAG: --qos-reserved=""
Sep 11 03:40:11.047281 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.039648    2580 flags.go:64] FLAG: --read-only-port="10255"
Sep 11 03:40:11.047281 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.039651    2580 flags.go:64] FLAG: --register-node="true"
Sep 11 03:40:11.047281 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.039655    2580 flags.go:64] FLAG: --register-with-taints=""
Sep 11 03:40:11.047281 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.039659    2580 flags.go:64] FLAG: --registry-burst="10"
Sep 11 03:40:11.047281 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.039662    2580 flags.go:64] FLAG: --registry-qps="5"
Sep 11 03:40:11.047281 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.039665    2580 flags.go:64] FLAG: --reserved-cpus=""
Sep 11 03:40:11.047281 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.039668    2580 flags.go:64] FLAG: --reserved-memory=""
Sep 11 03:40:11.047281 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.039671    2580 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 11 03:40:11.047281 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.039675    2580 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 11 03:40:11.047281 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.039678    2580 flags.go:64] FLAG: --rotate-certificates="false"
Sep 11 03:40:11.047938 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.039681    2580 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 11 03:40:11.047938 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.039684    2580 flags.go:64] FLAG: --runonce="false"
Sep 11 03:40:11.047938 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.039687    2580 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 11 03:40:11.047938 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.039690    2580 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 11 03:40:11.047938 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.039693    2580 flags.go:64] FLAG: --seccomp-default="false"
Sep 11 03:40:11.047938 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.039696    2580 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 11 03:40:11.047938 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.039699    2580 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 11 03:40:11.047938 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.039702    2580 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 11 03:40:11.047938 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.039705    2580 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 11 03:40:11.047938 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.039709    2580 flags.go:64] FLAG: --storage-driver-password="root"
Sep 11 03:40:11.047938 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.039712    2580 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 11 03:40:11.047938 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.039715    2580 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 11 03:40:11.047938 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.039718    2580 flags.go:64] FLAG: --storage-driver-user="root"
Sep 11 03:40:11.047938 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.039721    2580 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 11 03:40:11.047938 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.039724    2580 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 11 03:40:11.047938 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.039727    2580 flags.go:64] FLAG: --system-cgroups=""
Sep 11 03:40:11.047938 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.039730    2580 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 11 03:40:11.047938 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.039735    2580 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 11 03:40:11.047938 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.039738    2580 flags.go:64] FLAG: --tls-cert-file=""
Sep 11 03:40:11.047938 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.039743    2580 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 11 03:40:11.047938 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.039748    2580 flags.go:64] FLAG: --tls-min-version=""
Sep 11 03:40:11.047938 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.039751    2580 flags.go:64] FLAG: --tls-private-key-file=""
Sep 11 03:40:11.047938 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.039754    2580 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 11 03:40:11.047938 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.039757    2580 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 11 03:40:11.047938 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.039761    2580 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 11 03:40:11.048568 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.039764    2580 flags.go:64] FLAG: --v="2"
Sep 11 03:40:11.048568 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.039768    2580 flags.go:64] FLAG: --version="false"
Sep 11 03:40:11.048568 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.039772    2580 flags.go:64] FLAG: --vmodule=""
Sep 11 03:40:11.048568 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.039776    2580 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 11 03:40:11.048568 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.039779    2580 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 11 03:40:11.048568 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.039892    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 03:40:11.048568 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.039895    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 03:40:11.048568 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.039899    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 03:40:11.048568 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.039901    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 03:40:11.048568 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.039904    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 03:40:11.048568 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.039906    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 03:40:11.048568 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.039910    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 03:40:11.048568 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.039912    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 03:40:11.048568 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.039915    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 03:40:11.048568 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.039917    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 03:40:11.048568 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.039920    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 03:40:11.048568 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.039922    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 03:40:11.048568 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.039925    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 03:40:11.048568 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.039928    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 03:40:11.048568 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.039931    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 03:40:11.048568 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.039934    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 03:40:11.049105 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.039937    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 03:40:11.049105 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.039939    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 03:40:11.049105 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.039942    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 03:40:11.049105 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.039944    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 03:40:11.049105 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.039947    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 03:40:11.049105 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.039951    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 03:40:11.049105 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.039955    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 03:40:11.049105 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.039957    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 03:40:11.049105 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.039960    2580 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 03:40:11.049105 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.039962    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 03:40:11.049105 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.039965    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 03:40:11.049105 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.039967    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 03:40:11.049105 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.039970    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 03:40:11.049105 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.039974    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 03:40:11.049105 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.039978    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 03:40:11.049105 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.039981    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 03:40:11.049105 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.039984    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 03:40:11.049105 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.039986    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 03:40:11.049105 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.039990    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 03:40:11.049579 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.039992    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 03:40:11.049579 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.039995    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 03:40:11.049579 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.039998    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 03:40:11.049579 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.040001    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 03:40:11.049579 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.040003    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 03:40:11.049579 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.040006    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 03:40:11.049579 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.040008    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 03:40:11.049579 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.040011    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 03:40:11.049579 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.040013    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 03:40:11.049579 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.040016    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 03:40:11.049579 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.040018    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 03:40:11.049579 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.040021    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 03:40:11.049579 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.040024    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 03:40:11.049579 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.040026    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 03:40:11.049579 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.040029    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 03:40:11.049579 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.040031    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 03:40:11.049579 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.040034    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 03:40:11.049579 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.040036    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 03:40:11.049579 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.040042    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 03:40:11.050043 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.040045    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 03:40:11.050043 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.040049    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 03:40:11.050043 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.040052    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 03:40:11.050043 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.040054    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 03:40:11.050043 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.040057    2580 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 03:40:11.050043 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.040060    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 03:40:11.050043 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.040062    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 03:40:11.050043 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.040065    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 03:40:11.050043 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.040068    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 03:40:11.050043 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.040070    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 03:40:11.050043 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.040073    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 03:40:11.050043 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.040075    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 03:40:11.050043 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.040078    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 03:40:11.050043 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.040080    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 03:40:11.050043 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.040083    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 03:40:11.050043 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.040085    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 03:40:11.050043 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.040088    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 03:40:11.050043 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.040090    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 03:40:11.050043 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.040093    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 03:40:11.050043 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.040096    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 03:40:11.050525 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.040098    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 03:40:11.050525 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.040101    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 03:40:11.050525 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.040103    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 03:40:11.050525 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.040107    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 03:40:11.050525 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.040111    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 03:40:11.050525 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.040114    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 03:40:11.050525 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.040117    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 03:40:11.050525 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.040119    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 03:40:11.050525 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.040122    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 03:40:11.050525 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.040124    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 03:40:11.050525 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.040127    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 03:40:11.050525 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.040131    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 03:40:11.050525 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.040134    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 03:40:11.050525 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.040136    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 03:40:11.050525 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.040145    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 03:40:11.050525 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.040148    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 03:40:11.050525 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.040151    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 03:40:11.050525 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.040153    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 03:40:11.050525 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.040156    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 03:40:11.050525 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.040158    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 03:40:11.051030 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.040161    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 03:40:11.051030 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.040164    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 03:40:11.051030 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.040166    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 03:40:11.051030 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.040169    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 03:40:11.051030 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.040171    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 03:40:11.051030 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.040179    2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 11 03:40:11.051030 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.047860    2580 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 11 03:40:11.051030 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.047876    2580 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 11 03:40:11.051030 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.047894    2580 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 11 03:40:11.051030 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.047903    2580 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 11 03:40:11.051030 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.047959    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 03:40:11.051030 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.047964    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 03:40:11.051030 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.047968    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 03:40:11.051030 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.047972    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 03:40:11.051030 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.047975    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 03:40:11.051446 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.047978    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 03:40:11.051446 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.047980    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 03:40:11.051446 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.047983    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 03:40:11.051446 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.047986    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 03:40:11.051446 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.047990    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 03:40:11.051446 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.047992    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 03:40:11.051446 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.047996    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 03:40:11.051446 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.047998    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 03:40:11.051446 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.048002    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 03:40:11.051446 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.048005    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 03:40:11.051446 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.048008    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 03:40:11.051446 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.048011    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 03:40:11.051446 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.048014    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 03:40:11.051446 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.048017    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 03:40:11.051446 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.048020    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 03:40:11.051446 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.048022    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 03:40:11.051446 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.048025    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 03:40:11.051446 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.048028    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 03:40:11.051446 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.048030    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 03:40:11.051446 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.048033    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 03:40:11.051957 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.048037    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 03:40:11.051957 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.048042    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 03:40:11.051957 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.048046    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 03:40:11.051957 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.048049    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 03:40:11.051957 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.048053    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 03:40:11.051957 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.048056    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 03:40:11.051957 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.048059    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 03:40:11.051957 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.048062    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 03:40:11.051957 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.048064    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 03:40:11.051957 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.048067    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 03:40:11.051957 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.048070    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 03:40:11.051957 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.048073    2580 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 03:40:11.051957 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.048075    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 03:40:11.051957 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.048078    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 03:40:11.051957 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.048080    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 03:40:11.051957 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.048083    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 03:40:11.051957 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.048086    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 03:40:11.051957 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.048089    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 03:40:11.051957 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.048091    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 03:40:11.052479 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.048094    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 03:40:11.052479 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.048096    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 03:40:11.052479 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.048099    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 03:40:11.052479 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.048102    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 03:40:11.052479 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.048105    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 03:40:11.052479 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.048107    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 03:40:11.052479 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.048110    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 03:40:11.052479 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.048113    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 03:40:11.052479 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.048115    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 03:40:11.052479 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.048118    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 03:40:11.052479 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.048120    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 03:40:11.052479 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.048122    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 03:40:11.052479 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.048125    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 03:40:11.052479 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.048128    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 03:40:11.052479 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.048131    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 03:40:11.052479 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.048133    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 03:40:11.052479 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.048136    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 03:40:11.052479 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.048139    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 03:40:11.052479 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.048142    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 03:40:11.052479 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.048145    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 03:40:11.053071 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.048147    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 03:40:11.053071 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.048150    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 03:40:11.053071 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.048153    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 03:40:11.053071 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.048155    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 03:40:11.053071 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.048158    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 03:40:11.053071 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.048160    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 03:40:11.053071 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.048163    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 03:40:11.053071 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.048165    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 03:40:11.053071 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.048168    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 03:40:11.053071 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.048171    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 03:40:11.053071 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.048173    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 03:40:11.053071 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.048176    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 03:40:11.053071 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.048178    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 03:40:11.053071 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.048181    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 03:40:11.053071 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.048183    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 03:40:11.053071 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.048186    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 03:40:11.053071 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.048188    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 03:40:11.053071 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.048191    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 03:40:11.053071 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.048193    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 03:40:11.053071 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.048196    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 03:40:11.053645 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.048199    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 03:40:11.053645 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.048201    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 03:40:11.053645 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.048204    2580 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 03:40:11.053645 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.048206    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 03:40:11.053645 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.048209    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 03:40:11.053645 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.048212    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 03:40:11.053645 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.048214    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 03:40:11.053645 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.048217    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 03:40:11.053645 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.048220    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 03:40:11.053645 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.048223    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 03:40:11.053645 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.048226    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 03:40:11.053645 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.048229    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 03:40:11.053645 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.048232    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 03:40:11.053645 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.048236    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 03:40:11.053645 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.048240    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 03:40:11.054010 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.048245    2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 11 03:40:11.054010 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.048357    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 03:40:11.054010 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.048362    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 03:40:11.054010 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.048366    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 03:40:11.054010 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.048368    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 03:40:11.054010 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.048372    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 03:40:11.054010 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.048375    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 03:40:11.054010 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.048377    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 03:40:11.054010 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.048380    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 03:40:11.054010 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.048383    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 03:40:11.054010 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.048386    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 03:40:11.054010 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.048389    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 03:40:11.054010 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.048392    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 03:40:11.054010 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.048394    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 03:40:11.054010 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.048397    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 03:40:11.054380 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.048400    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 03:40:11.054380 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.048403    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 03:40:11.054380 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.048405    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 03:40:11.054380 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.048408    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 03:40:11.054380 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.048410    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 03:40:11.054380 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.048413    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 03:40:11.054380 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.048415    2580 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 03:40:11.054380 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.048418    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 03:40:11.054380 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.048420    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 03:40:11.054380 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.048423    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 03:40:11.054380 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.048426    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 03:40:11.054380 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.048429    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 03:40:11.054380 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.048432    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 03:40:11.054380 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.048434    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 03:40:11.054380 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.048438    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 03:40:11.054380 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.048440    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 03:40:11.054380 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.048443    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 03:40:11.054380 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.048446    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 03:40:11.054380 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.048448    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 03:40:11.054380 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.048451    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 03:40:11.054896 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.048453    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 03:40:11.054896 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.048456    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 03:40:11.054896 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.048458    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 03:40:11.054896 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.048461    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 03:40:11.054896 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.048464    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 03:40:11.054896 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.048466    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 03:40:11.054896 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.048469    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 03:40:11.054896 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.048471    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 03:40:11.054896 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.048474    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 03:40:11.054896 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.048476    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 03:40:11.054896 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.048479    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 03:40:11.054896 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.048482    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 03:40:11.054896 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.048485    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 03:40:11.054896 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.048487    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 03:40:11.054896 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.048490    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 03:40:11.054896 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.048492    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 03:40:11.054896 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.048495    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 03:40:11.054896 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.048497    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 03:40:11.054896 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.048501    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 03:40:11.055359 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.048504    2580 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 03:40:11.055359 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.048507    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 03:40:11.055359 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.048510    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 03:40:11.055359 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.048513    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 03:40:11.055359 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.048517    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 03:40:11.055359 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.048520    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 03:40:11.055359 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.048523    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 03:40:11.055359 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.048525    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 03:40:11.055359 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.048529    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 03:40:11.055359 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.048548    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 03:40:11.055359 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.048551    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 03:40:11.055359 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.048554    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 03:40:11.055359 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.048557    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 03:40:11.055359 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.048559    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 03:40:11.055359 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.048562    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 03:40:11.055359 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.048565    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 03:40:11.055359 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.048568    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 03:40:11.055359 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.048571    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 03:40:11.055359 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.048573    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 03:40:11.055359 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.048576    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 03:40:11.055956 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.048579    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 03:40:11.055956 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.048582    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 03:40:11.055956 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.048585    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 03:40:11.055956 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.048588    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 03:40:11.055956 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.048591    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 03:40:11.055956 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.048595    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 03:40:11.055956 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.048598    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 03:40:11.055956 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.048601    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 03:40:11.055956 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.048604    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 03:40:11.055956 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.048606    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 03:40:11.055956 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.048608    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 03:40:11.055956 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.048611    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 03:40:11.055956 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.048613    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 03:40:11.055956 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.048616    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 03:40:11.055956 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.048618    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 03:40:11.055956 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.048621    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 03:40:11.055956 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.048624    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 03:40:11.055956 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.048626    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 03:40:11.055956 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.048629    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 03:40:11.056437 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.048633    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 03:40:11.056437 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.048637    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 03:40:11.056437 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.048640    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 03:40:11.056437 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.048644    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 03:40:11.056437 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.048647    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 03:40:11.056437 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.048650    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 03:40:11.056437 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:11.048653    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 03:40:11.056437 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.048658    2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 11 03:40:11.056437 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.048768    2580 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 11 03:40:11.056437 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.051167    2580 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 11 03:40:11.056437 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.052283    2580 server.go:1019] "Starting client certificate rotation"
Sep 11 03:40:11.056437 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.052398    2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 11 03:40:11.056437 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.053313    2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 11 03:40:11.056880 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.055011    2580 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 11 03:40:11.056880 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.056116    2580 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 11 03:40:11.073771 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.073748    2580 log.go:25] "Validated CRI v1 runtime API"
Sep 11 03:40:11.078021 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.078004    2580 log.go:25] "Validated CRI v1 image API"
Sep 11 03:40:11.078922 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.078906    2580 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 11 03:40:11.082279 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.082264    2580 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 11 03:40:11.085197 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.085179    2580 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 7dbacb8f-2d07-4df6-bcac-175bd594d496:/dev/nvme0n1p3 f26ba054-0f90-4084-90a1-2bfda9e8d9ca:/dev/nvme0n1p4]
Sep 11 03:40:11.085249 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.085197    2580 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 11 03:40:11.091370 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.091253    2580 manager.go:216] Machine: {Timestamp:2026-09-11 03:40:11.089887784 +0000 UTC m=+0.380502551 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3122677 MemoryCapacity:33164472320 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec24f99f6fa744d84b26eb071a01fb44 SystemUUID:ec24f99f-6fa7-44d8-4b26-eb071a01fb44 BootID:24d6f83f-4c70-4178-94f6-a85514096ba9 Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632894464 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582238208 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582234112 Type:vfs Inodes:4048397 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:25:2d:49:a1:bd Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:25:2d:49:a1:bd Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ca:40:e1:7d:3a:08 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164472320 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 11 03:40:11.091370 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.091366    2580 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 11 03:40:11.091509 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.091453    2580 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 11 03:40:11.095269 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.095240    2580 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 11 03:40:11.095434 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.095272    2580 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-138-121.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 11 03:40:11.096160 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.096149    2580 topology_manager.go:138] "Creating topology manager with none policy"
Sep 11 03:40:11.096213 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.096167    2580 container_manager_linux.go:309] "Creating device plugin manager"
Sep 11 03:40:11.096213 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.096178    2580 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 11 03:40:11.096213 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.096201    2580 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 11 03:40:11.097529 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.097516    2580 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 11 03:40:11.102017 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.101995    2580 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-qrwp8"
Sep 11 03:40:11.102390 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.102377    2580 state_mem.go:36] "Initialized new in-memory state store"
Sep 11 03:40:11.102528 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.102517    2580 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 11 03:40:11.102597 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.102569    2580 kubelet.go:486] "Attempting to sync node with API server"
Sep 11 03:40:11.102597 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.102594    2580 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 11 03:40:11.102699 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.102615    2580 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 11 03:40:11.102699 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.102630    2580 kubelet.go:398] "Adding apiserver pod source"
Sep 11 03:40:11.102699 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.102642    2580 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 11 03:40:11.103044 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.103030    2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 11 03:40:11.103106 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.103053    2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 11 03:40:11.106837 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.106696    2580 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 11 03:40:11.109746 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.109729    2580 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 11 03:40:11.109802 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.109763    2580 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 11 03:40:11.110110 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.110092    2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-qrwp8"
Sep 11 03:40:11.112578 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:40:11.112550    2580 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-138-121.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 11 03:40:11.112578 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:40:11.112554    2580 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 11 03:40:11.112745 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.112656    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 11 03:40:11.112745 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.112673    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 11 03:40:11.112745 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.112683    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 11 03:40:11.112745 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.112692    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 11 03:40:11.112745 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.112705    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 11 03:40:11.112745 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.112713    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 11 03:40:11.112745 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.112732    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 11 03:40:11.112745 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.112740    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 11 03:40:11.113061 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.112757    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 11 03:40:11.113061 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.112765    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 11 03:40:11.113061 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.112778    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 11 03:40:11.113061 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.112786    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 11 03:40:11.113734 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.113717    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 11 03:40:11.113810 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.113741    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 11 03:40:11.116467 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.116450    2580 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-121.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 11 03:40:11.117750 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.117736    2580 server.go:1268] "Started kubelet"
Sep 11 03:40:11.117841 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.117804    2580 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 11 03:40:11.117890 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.117809    2580 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 11 03:40:11.117890 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.117877    2580 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 11 03:40:11.118754 ip-10-0-138-121 systemd[1]: Started Kubernetes Kubelet.
Sep 11 03:40:11.119261 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.119248    2580 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 11 03:40:11.120484 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.120467    2580 server.go:310] "Adding debug handlers to kubelet server"
Sep 11 03:40:11.124775 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:40:11.124748    2580 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 11 03:40:11.127013 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.126886    2580 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 11 03:40:11.127083 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.126914    2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 11 03:40:11.127243 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:40:11.127225    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-121.ec2.internal\" not found"
Sep 11 03:40:11.127848 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.127828    2580 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 11 03:40:11.127923 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.127866    2580 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 11 03:40:11.127923 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.127881    2580 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 11 03:40:11.127923 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.127895    2580 factory.go:55] Registering systemd factory
Sep 11 03:40:11.127923 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.127917    2580 factory.go:223] Registration of the systemd container factory successfully
Sep 11 03:40:11.128105 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.127956    2580 reconstruct.go:97] "Volume reconstruction finished"
Sep 11 03:40:11.128105 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.127963    2580 reconciler.go:29] "Reconciler: start to sync state"
Sep 11 03:40:11.128184 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.128170    2580 factory.go:153] Registering CRI-O factory
Sep 11 03:40:11.128184 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.128179    2580 factory.go:223] Registration of the crio container factory successfully
Sep 11 03:40:11.128259 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.128243    2580 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 11 03:40:11.128298 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.128262    2580 factory.go:103] Registering Raw factory
Sep 11 03:40:11.128298 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.128279    2580 manager.go:1196] Started watching for new ooms in manager
Sep 11 03:40:11.128721 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.128707    2580 manager.go:349] Starting recovery of all containers
Sep 11 03:40:11.129704 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.129683    2580 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 03:40:11.133225 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:40:11.133201    2580 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-138-121.ec2.internal\" not found" node="ip-10-0-138-121.ec2.internal"
Sep 11 03:40:11.140278 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.140255    2580 manager.go:354] Recovery completed
Sep 11 03:40:11.146746 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.146731    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 03:40:11.148469 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.148448    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-121.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 03:40:11.148581 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.148484    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-121.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 03:40:11.148581 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.148499    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-121.ec2.internal" event="NodeHasSufficientPID"
Sep 11 03:40:11.149077 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.149063    2580 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 11 03:40:11.149077 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.149075    2580 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 11 03:40:11.149205 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.149092    2580 state_mem.go:36] "Initialized new in-memory state store"
Sep 11 03:40:11.149205 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.149109    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 03:40:11.149952 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.149933    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-121.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 03:40:11.150026 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.149967    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-121.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 03:40:11.150026 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.149987    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-121.ec2.internal" event="NodeHasSufficientPID"
Sep 11 03:40:11.151374 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.151362    2580 policy_none.go:49] "None policy: Start"
Sep 11 03:40:11.151415 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.151379    2580 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 11 03:40:11.151415 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.151389    2580 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 11 03:40:11.153386 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.153375    2580 policy_none.go:47] "Start"
Sep 11 03:40:11.187931 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.187915    2580 manager.go:337] "Starting Device Plugin manager"
Sep 11 03:40:11.188020 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:40:11.187955    2580 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 11 03:40:11.188020 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.187969    2580 server.go:87] "Starting device plugin registration server"
Sep 11 03:40:11.188327 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.188305    2580 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 11 03:40:11.188400 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.188319    2580 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 11 03:40:11.188476 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.188446    2580 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 11 03:40:11.188546 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.188520    2580 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 11 03:40:11.188596 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.188529    2580 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 11 03:40:11.190565 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:40:11.190528    2580 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 11 03:40:11.190646 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:40:11.190602    2580 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-138-121.ec2.internal\" not found"
Sep 11 03:40:11.215503 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.215477    2580 container_manager_linux.go:676] "Failed to ensure state" containerName="/system.slice" err="failed to move PID 2610 into the system container \"/system.slice\": <nil>"
Sep 11 03:40:11.289220 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.289125    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 03:40:11.290758 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.290743    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-121.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 03:40:11.290835 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.290777    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-121.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 03:40:11.290835 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.290791    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-121.ec2.internal" event="NodeHasSufficientPID"
Sep 11 03:40:11.290835 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.290824    2580 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-138-121.ec2.internal"
Sep 11 03:40:11.300029 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.300002    2580 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-138-121.ec2.internal"
Sep 11 03:40:11.300029 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:40:11.300030    2580 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-138-121.ec2.internal\": node \"ip-10-0-138-121.ec2.internal\" not found"
Sep 11 03:40:11.324090 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.324050    2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 11 03:40:11.327295 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.325115    2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 11 03:40:11.327295 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.325159    2580 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 11 03:40:11.327295 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.325179    2580 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 11 03:40:11.327295 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:40:11.325220    2580 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 11 03:40:11.332275 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:40:11.332259    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-121.ec2.internal\" not found"
Sep 11 03:40:11.333140 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.333127    2580 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 03:40:11.425908 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.425875    2580 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-121.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-138-121.ec2.internal"]
Sep 11 03:40:11.426070 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.425975    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 03:40:11.426980 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.426964    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-121.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 03:40:11.427067 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.426992    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-121.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 03:40:11.427067 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.427002    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-121.ec2.internal" event="NodeHasSufficientPID"
Sep 11 03:40:11.429360 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.429348    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 03:40:11.429516 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.429504    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-121.ec2.internal"
Sep 11 03:40:11.429575 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.429549    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 03:40:11.429863 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.429847    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/99e85cb9783bfb509f2d3b998351c555-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-121.ec2.internal\" (UID: \"99e85cb9783bfb509f2d3b998351c555\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-121.ec2.internal"
Sep 11 03:40:11.429934 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.429869    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/99e85cb9783bfb509f2d3b998351c555-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-121.ec2.internal\" (UID: \"99e85cb9783bfb509f2d3b998351c555\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-121.ec2.internal"
Sep 11 03:40:11.430119 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.430106    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-121.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 03:40:11.430170 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.430129    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-121.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 03:40:11.430170 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.430138    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-121.ec2.internal" event="NodeHasSufficientPID"
Sep 11 03:40:11.430232 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.430106    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-121.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 03:40:11.430232 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.430208    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-121.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 03:40:11.430232 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.430220    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-121.ec2.internal" event="NodeHasSufficientPID"
Sep 11 03:40:11.432330 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.432316    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-121.ec2.internal"
Sep 11 03:40:11.432411 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.432339    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 03:40:11.432411 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:40:11.432380    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-121.ec2.internal\" not found"
Sep 11 03:40:11.433098 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.433083    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-121.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 03:40:11.433164 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.433114    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-121.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 03:40:11.433164 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.433129    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-121.ec2.internal" event="NodeHasSufficientPID"
Sep 11 03:40:11.460126 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:40:11.460108    2580 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-121.ec2.internal\" not found" node="ip-10-0-138-121.ec2.internal"
Sep 11 03:40:11.464492 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:40:11.464477    2580 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-121.ec2.internal\" not found" node="ip-10-0-138-121.ec2.internal"
Sep 11 03:40:11.530984 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.530958    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/99e85cb9783bfb509f2d3b998351c555-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-121.ec2.internal\" (UID: \"99e85cb9783bfb509f2d3b998351c555\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-121.ec2.internal"
Sep 11 03:40:11.530984 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.530987    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/99e85cb9783bfb509f2d3b998351c555-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-121.ec2.internal\" (UID: \"99e85cb9783bfb509f2d3b998351c555\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-121.ec2.internal"
Sep 11 03:40:11.531180 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.531006    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/355aa2f2ac5d4d26d63779c83ede8936-config\") pod \"kube-apiserver-proxy-ip-10-0-138-121.ec2.internal\" (UID: \"355aa2f2ac5d4d26d63779c83ede8936\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-121.ec2.internal"
Sep 11 03:40:11.531180 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.531059    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/99e85cb9783bfb509f2d3b998351c555-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-121.ec2.internal\" (UID: \"99e85cb9783bfb509f2d3b998351c555\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-121.ec2.internal"
Sep 11 03:40:11.531180 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.531118    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/99e85cb9783bfb509f2d3b998351c555-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-121.ec2.internal\" (UID: \"99e85cb9783bfb509f2d3b998351c555\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-121.ec2.internal"
Sep 11 03:40:11.533090 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:40:11.533065    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-121.ec2.internal\" not found"
Sep 11 03:40:11.631736 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.631677    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/355aa2f2ac5d4d26d63779c83ede8936-config\") pod \"kube-apiserver-proxy-ip-10-0-138-121.ec2.internal\" (UID: \"355aa2f2ac5d4d26d63779c83ede8936\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-121.ec2.internal"
Sep 11 03:40:11.631736 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.631713    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/355aa2f2ac5d4d26d63779c83ede8936-config\") pod \"kube-apiserver-proxy-ip-10-0-138-121.ec2.internal\" (UID: \"355aa2f2ac5d4d26d63779c83ede8936\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-121.ec2.internal"
Sep 11 03:40:11.633802 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:40:11.633786    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-121.ec2.internal\" not found"
Sep 11 03:40:11.734309 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:40:11.734272    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-121.ec2.internal\" not found"
Sep 11 03:40:11.764465 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.764429    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-121.ec2.internal"
Sep 11 03:40:11.768182 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:11.768159    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-121.ec2.internal"
Sep 11 03:40:12.967055 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:40:12.967026    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-121.ec2.internal\" not found"
Sep 11 03:40:13.067538 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:40:13.067479    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-121.ec2.internal\" not found"
Sep 11 03:40:13.168049 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:40:13.168018    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-121.ec2.internal\" not found"
Sep 11 03:40:13.184205 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:13.184185    2580 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 11 03:40:13.184328 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:13.184313    2580 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 11 03:40:13.184390 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:13.184344    2580 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 11 03:40:13.248656 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:13.248614    2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-10 03:35:12 +0000 UTC" deadline="2028-04-26 21:35:25.407370184 +0000 UTC"
Sep 11 03:40:13.248656 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:13.248649    2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14249h55m12.158725897s"
Sep 11 03:40:13.268223 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:40:13.268196    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-121.ec2.internal\" not found"
Sep 11 03:40:13.368861 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:40:13.368828    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-121.ec2.internal\" not found"
Sep 11 03:40:13.397850 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:13.397819    2580 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 03:40:13.444228 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:13.444204    2580 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 03:40:13.460107 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:13.460082    2580 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-121.ec2.internal"
Sep 11 03:40:13.460759 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:13.460733    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod355aa2f2ac5d4d26d63779c83ede8936.slice/crio-0fc58a3ad46260608d52980a74bab478d4414f7040e90254ba6cab38e9de72cc WatchSource:0}: Error finding container 0fc58a3ad46260608d52980a74bab478d4414f7040e90254ba6cab38e9de72cc: Status 404 returned error can't find the container with id 0fc58a3ad46260608d52980a74bab478d4414f7040e90254ba6cab38e9de72cc
Sep 11 03:40:13.461228 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:13.461210    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99e85cb9783bfb509f2d3b998351c555.slice/crio-ed8cd884d78d5a2e07d01ad4c665adcc77918d038d4ea242c92d5d25d6139a27 WatchSource:0}: Error finding container ed8cd884d78d5a2e07d01ad4c665adcc77918d038d4ea242c92d5d25d6139a27: Status 404 returned error can't find the container with id ed8cd884d78d5a2e07d01ad4c665adcc77918d038d4ea242c92d5d25d6139a27
Sep 11 03:40:13.465460 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:13.465453    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 03:40:13.471827 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:13.471805    2580 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 11 03:40:13.472906 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:13.472889    2580 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-121.ec2.internal"
Sep 11 03:40:13.479973 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:13.479956    2580 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 11 03:40:14.235946 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.235902    2580 apiserver.go:52] "Watching apiserver"
Sep 11 03:40:14.242424 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.242397    2580 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 11 03:40:14.242866 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.242839    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-d9t58","openshift-network-diagnostics/network-check-target-f7hng","openshift-network-operator/iptables-alerter-gqqk2","kube-system/global-pull-secret-syncer-8knjw","kube-system/konnectivity-agent-tm9tp","kube-system/kube-apiserver-proxy-ip-10-0-138-121.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l42ph","openshift-multus/multus-additional-cni-plugins-tr4l2","openshift-multus/network-metrics-daemon-qzhz6","openshift-ovn-kubernetes/ovnkube-node-j4pgw","openshift-cluster-node-tuning-operator/tuned-5gs25","openshift-dns/node-resolver-wkqmr","openshift-image-registry/node-ca-mlcbg","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-121.ec2.internal"]
Sep 11 03:40:14.245465 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.245442    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l42ph"
Sep 11 03:40:14.247997 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.247964    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 11 03:40:14.248300 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.248260    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 11 03:40:14.248300 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.248291    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 11 03:40:14.248470 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.248341    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-mxbvh\""
Sep 11 03:40:14.248931 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.248639    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-mlcbg"
Sep 11 03:40:14.250889 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.250868    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 11 03:40:14.251032 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.251014    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 11 03:40:14.251132 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.251117    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-5c4gv\""
Sep 11 03:40:14.251305 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.251285    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-gqqk2"
Sep 11 03:40:14.251377 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.251305    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 11 03:40:14.254372 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.254032    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 11 03:40:14.254372 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.254064    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 11 03:40:14.254372 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.254070    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8knjw"
Sep 11 03:40:14.254372 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:40:14.254182    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8knjw" podUID="f6fa8b71-4ef9-4c91-b7ee-7ef235532ea0"
Sep 11 03:40:14.254372 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.254203    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 11 03:40:14.254372 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.254275    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-jshf8\""
Sep 11 03:40:14.255371 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.255351    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-tm9tp"
Sep 11 03:40:14.255816 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.255757    2580 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 03:40:14.257471 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.257400    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 11 03:40:14.257577 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.257472    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 11 03:40:14.257814 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.257718    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-tr4l2"
Sep 11 03:40:14.257814 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.257740    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-5vd9x\""
Sep 11 03:40:14.259623 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.259575    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 11 03:40:14.259777 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.259761    2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 11 03:40:14.260950 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.260099    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 11 03:40:14.260950 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.260284    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 11 03:40:14.260950 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.260346    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 11 03:40:14.260950 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.260492    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-m74qq\""
Sep 11 03:40:14.260950 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.260572    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 11 03:40:14.261248 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.261054    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qzhz6"
Sep 11 03:40:14.261248 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:40:14.261117    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qzhz6" podUID="eb00790b-1a90-4786-8b81-cbfc706f9c46"
Sep 11 03:40:14.265696 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.263369    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-d9t58"
Sep 11 03:40:14.265696 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.265207    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 11 03:40:14.265837 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.265741    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-vx8ml\""
Sep 11 03:40:14.265837 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.265818    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f7hng"
Sep 11 03:40:14.266053 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:40:14.265935    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f7hng" podUID="ad569c0f-e4f1-42f2-aa2b-48f0a4f16a57"
Sep 11 03:40:14.269148 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.269121    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-j4pgw"
Sep 11 03:40:14.271500 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.271249    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 11 03:40:14.271607 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.271532    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bc0076f9-9364-41f4-aadc-7e4cef25221a-system-cni-dir\") pod \"multus-d9t58\" (UID: \"bc0076f9-9364-41f4-aadc-7e4cef25221a\") " pod="openshift-multus/multus-d9t58"
Sep 11 03:40:14.271607 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.271560    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bc0076f9-9364-41f4-aadc-7e4cef25221a-multus-cni-dir\") pod \"multus-d9t58\" (UID: \"bc0076f9-9364-41f4-aadc-7e4cef25221a\") " pod="openshift-multus/multus-d9t58"
Sep 11 03:40:14.271607 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.271587    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-fmqrc\""
Sep 11 03:40:14.271801 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.271585    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/bc0076f9-9364-41f4-aadc-7e4cef25221a-host-var-lib-cni-multus\") pod \"multus-d9t58\" (UID: \"bc0076f9-9364-41f4-aadc-7e4cef25221a\") " pod="openshift-multus/multus-d9t58"
Sep 11 03:40:14.271801 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.271652    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/bc0076f9-9364-41f4-aadc-7e4cef25221a-multus-daemon-config\") pod \"multus-d9t58\" (UID: \"bc0076f9-9364-41f4-aadc-7e4cef25221a\") " pod="openshift-multus/multus-d9t58"
Sep 11 03:40:14.271801 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.271692    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/bc0076f9-9364-41f4-aadc-7e4cef25221a-host-run-multus-certs\") pod \"multus-d9t58\" (UID: \"bc0076f9-9364-41f4-aadc-7e4cef25221a\") " pod="openshift-multus/multus-d9t58"
Sep 11 03:40:14.271801 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.271714    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a91c03cc-9ab6-45df-a99a-3f197542c920-tuning-conf-dir\") pod \"multus-additional-cni-plugins-tr4l2\" (UID: \"a91c03cc-9ab6-45df-a99a-3f197542c920\") " pod="openshift-multus/multus-additional-cni-plugins-tr4l2"
Sep 11 03:40:14.271801 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.271738    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9t2dx\" (UniqueName: \"kubernetes.io/projected/a91c03cc-9ab6-45df-a99a-3f197542c920-kube-api-access-9t2dx\") pod \"multus-additional-cni-plugins-tr4l2\" (UID: \"a91c03cc-9ab6-45df-a99a-3f197542c920\") " pod="openshift-multus/multus-additional-cni-plugins-tr4l2"
Sep 11 03:40:14.271801 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.271765    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a7954230-378f-4cf1-b574-3c3b9f0ec4fb-konnectivity-ca\") pod \"konnectivity-agent-tm9tp\" (UID: \"a7954230-378f-4cf1-b574-3c3b9f0ec4fb\") " pod="kube-system/konnectivity-agent-tm9tp"
Sep 11 03:40:14.272113 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.271804    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb00790b-1a90-4786-8b81-cbfc706f9c46-metrics-certs\") pod \"network-metrics-daemon-qzhz6\" (UID: \"eb00790b-1a90-4786-8b81-cbfc706f9c46\") " pod="openshift-multus/network-metrics-daemon-qzhz6"
Sep 11 03:40:14.272113 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.271832    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bc0076f9-9364-41f4-aadc-7e4cef25221a-host-run-netns\") pod \"multus-d9t58\" (UID: \"bc0076f9-9364-41f4-aadc-7e4cef25221a\") " pod="openshift-multus/multus-d9t58"
Sep 11 03:40:14.272113 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.271852    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 11 03:40:14.272113 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.271855    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/bc0076f9-9364-41f4-aadc-7e4cef25221a-hostroot\") pod \"multus-d9t58\" (UID: \"bc0076f9-9364-41f4-aadc-7e4cef25221a\") " pod="openshift-multus/multus-d9t58"
Sep 11 03:40:14.272113 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.271877    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ng9t\" (UniqueName: \"kubernetes.io/projected/bc0076f9-9364-41f4-aadc-7e4cef25221a-kube-api-access-4ng9t\") pod \"multus-d9t58\" (UID: \"bc0076f9-9364-41f4-aadc-7e4cef25221a\") " pod="openshift-multus/multus-d9t58"
Sep 11 03:40:14.272113 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.271899    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/aae093a3-d3d4-4291-947e-3a59a9b4c84d-socket-dir\") pod \"aws-ebs-csi-driver-node-l42ph\" (UID: \"aae093a3-d3d4-4291-947e-3a59a9b4c84d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l42ph"
Sep 11 03:40:14.272113 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.271948    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cc4dd8ce-127a-4e14-9949-0b7075d1df37-host-slash\") pod \"iptables-alerter-gqqk2\" (UID: \"cc4dd8ce-127a-4e14-9949-0b7075d1df37\") " pod="openshift-network-operator/iptables-alerter-gqqk2"
Sep 11 03:40:14.272113 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.271971    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/bc0076f9-9364-41f4-aadc-7e4cef25221a-multus-socket-dir-parent\") pod \"multus-d9t58\" (UID: \"bc0076f9-9364-41f4-aadc-7e4cef25221a\") " pod="openshift-multus/multus-d9t58"
Sep 11 03:40:14.272113 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.271994    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/aae093a3-d3d4-4291-947e-3a59a9b4c84d-device-dir\") pod \"aws-ebs-csi-driver-node-l42ph\" (UID: \"aae093a3-d3d4-4291-947e-3a59a9b4c84d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l42ph"
Sep 11 03:40:14.272113 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.272019    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gftnx\" (UniqueName: \"kubernetes.io/projected/cc4dd8ce-127a-4e14-9949-0b7075d1df37-kube-api-access-gftnx\") pod \"iptables-alerter-gqqk2\" (UID: \"cc4dd8ce-127a-4e14-9949-0b7075d1df37\") " pod="openshift-network-operator/iptables-alerter-gqqk2"
Sep 11 03:40:14.272113 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.272041    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f6fa8b71-4ef9-4c91-b7ee-7ef235532ea0-kubelet-config\") pod \"global-pull-secret-syncer-8knjw\" (UID: \"f6fa8b71-4ef9-4c91-b7ee-7ef235532ea0\") " pod="kube-system/global-pull-secret-syncer-8knjw"
Sep 11 03:40:14.272113 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.272079    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a91c03cc-9ab6-45df-a99a-3f197542c920-cnibin\") pod \"multus-additional-cni-plugins-tr4l2\" (UID: \"a91c03cc-9ab6-45df-a99a-3f197542c920\") " pod="openshift-multus/multus-additional-cni-plugins-tr4l2"
Sep 11 03:40:14.272113 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.272105    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a91c03cc-9ab6-45df-a99a-3f197542c920-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-tr4l2\" (UID: \"a91c03cc-9ab6-45df-a99a-3f197542c920\") " pod="openshift-multus/multus-additional-cni-plugins-tr4l2"
Sep 11 03:40:14.272801 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.272130    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/84aade61-b056-43d4-bbcd-fd68eba2cb48-serviceca\") pod \"node-ca-mlcbg\" (UID: \"84aade61-b056-43d4-bbcd-fd68eba2cb48\") " pod="openshift-image-registry/node-ca-mlcbg"
Sep 11 03:40:14.272801 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.272152    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bc0076f9-9364-41f4-aadc-7e4cef25221a-multus-conf-dir\") pod \"multus-d9t58\" (UID: \"bc0076f9-9364-41f4-aadc-7e4cef25221a\") " pod="openshift-multus/multus-d9t58"
Sep 11 03:40:14.272801 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.272184    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bc0076f9-9364-41f4-aadc-7e4cef25221a-etc-kubernetes\") pod \"multus-d9t58\" (UID: \"bc0076f9-9364-41f4-aadc-7e4cef25221a\") " pod="openshift-multus/multus-d9t58"
Sep 11 03:40:14.272801 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.272223    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/aae093a3-d3d4-4291-947e-3a59a9b4c84d-sys-fs\") pod \"aws-ebs-csi-driver-node-l42ph\" (UID: \"aae093a3-d3d4-4291-947e-3a59a9b4c84d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l42ph"
Sep 11 03:40:14.272801 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.272266    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/84aade61-b056-43d4-bbcd-fd68eba2cb48-host\") pod \"node-ca-mlcbg\" (UID: \"84aade61-b056-43d4-bbcd-fd68eba2cb48\") " pod="openshift-image-registry/node-ca-mlcbg"
Sep 11 03:40:14.272801 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.272287    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-5gs25"
Sep 11 03:40:14.272801 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.272291    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/bc0076f9-9364-41f4-aadc-7e4cef25221a-host-run-k8s-cni-cncf-io\") pod \"multus-d9t58\" (UID: \"bc0076f9-9364-41f4-aadc-7e4cef25221a\") " pod="openshift-multus/multus-d9t58"
Sep 11 03:40:14.272801 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.272322    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/aae093a3-d3d4-4291-947e-3a59a9b4c84d-etc-selinux\") pod \"aws-ebs-csi-driver-node-l42ph\" (UID: \"aae093a3-d3d4-4291-947e-3a59a9b4c84d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l42ph"
Sep 11 03:40:14.272801 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.272345    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f6fa8b71-4ef9-4c91-b7ee-7ef235532ea0-original-pull-secret\") pod \"global-pull-secret-syncer-8knjw\" (UID: \"f6fa8b71-4ef9-4c91-b7ee-7ef235532ea0\") " pod="kube-system/global-pull-secret-syncer-8knjw"
Sep 11 03:40:14.272801 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.272390    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjbjx\" (UniqueName: \"kubernetes.io/projected/eb00790b-1a90-4786-8b81-cbfc706f9c46-kube-api-access-wjbjx\") pod \"network-metrics-daemon-qzhz6\" (UID: \"eb00790b-1a90-4786-8b81-cbfc706f9c46\") " pod="openshift-multus/network-metrics-daemon-qzhz6"
Sep 11 03:40:14.272801 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.272423    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bc0076f9-9364-41f4-aadc-7e4cef25221a-cnibin\") pod \"multus-d9t58\" (UID: \"bc0076f9-9364-41f4-aadc-7e4cef25221a\") " pod="openshift-multus/multus-d9t58"
Sep 11 03:40:14.272801 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.272447    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/cc4dd8ce-127a-4e14-9949-0b7075d1df37-iptables-alerter-script\") pod \"iptables-alerter-gqqk2\" (UID: \"cc4dd8ce-127a-4e14-9949-0b7075d1df37\") " pod="openshift-network-operator/iptables-alerter-gqqk2"
Sep 11 03:40:14.272801 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.272470    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f6fa8b71-4ef9-4c91-b7ee-7ef235532ea0-dbus\") pod \"global-pull-secret-syncer-8knjw\" (UID: \"f6fa8b71-4ef9-4c91-b7ee-7ef235532ea0\") " pod="kube-system/global-pull-secret-syncer-8knjw"
Sep 11 03:40:14.272801 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.272506    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a91c03cc-9ab6-45df-a99a-3f197542c920-cni-binary-copy\") pod \"multus-additional-cni-plugins-tr4l2\" (UID: \"a91c03cc-9ab6-45df-a99a-3f197542c920\") " pod="openshift-multus/multus-additional-cni-plugins-tr4l2"
Sep 11 03:40:14.272801 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.272532    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wm96l\" (UniqueName: \"kubernetes.io/projected/84aade61-b056-43d4-bbcd-fd68eba2cb48-kube-api-access-wm96l\") pod \"node-ca-mlcbg\" (UID: \"84aade61-b056-43d4-bbcd-fd68eba2cb48\") " pod="openshift-image-registry/node-ca-mlcbg"
Sep 11 03:40:14.272801 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.272555    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a7954230-378f-4cf1-b574-3c3b9f0ec4fb-agent-certs\") pod \"konnectivity-agent-tm9tp\" (UID: \"a7954230-378f-4cf1-b574-3c3b9f0ec4fb\") " pod="kube-system/konnectivity-agent-tm9tp"
Sep 11 03:40:14.272801 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.272590    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bc0076f9-9364-41f4-aadc-7e4cef25221a-cni-binary-copy\") pod \"multus-d9t58\" (UID: \"bc0076f9-9364-41f4-aadc-7e4cef25221a\") " pod="openshift-multus/multus-d9t58"
Sep 11 03:40:14.273614 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.272613    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bc0076f9-9364-41f4-aadc-7e4cef25221a-host-var-lib-cni-bin\") pod \"multus-d9t58\" (UID: \"bc0076f9-9364-41f4-aadc-7e4cef25221a\") " pod="openshift-multus/multus-d9t58"
Sep 11 03:40:14.273614 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.272637    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4btsj\" (UniqueName: \"kubernetes.io/projected/aae093a3-d3d4-4291-947e-3a59a9b4c84d-kube-api-access-4btsj\") pod \"aws-ebs-csi-driver-node-l42ph\" (UID: \"aae093a3-d3d4-4291-947e-3a59a9b4c84d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l42ph"
Sep 11 03:40:14.273614 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.272682    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a91c03cc-9ab6-45df-a99a-3f197542c920-system-cni-dir\") pod \"multus-additional-cni-plugins-tr4l2\" (UID: \"a91c03cc-9ab6-45df-a99a-3f197542c920\") " pod="openshift-multus/multus-additional-cni-plugins-tr4l2"
Sep 11 03:40:14.273614 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.272734    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 11 03:40:14.273614 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.272742    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bc0076f9-9364-41f4-aadc-7e4cef25221a-host-var-lib-kubelet\") pod \"multus-d9t58\" (UID: \"bc0076f9-9364-41f4-aadc-7e4cef25221a\") " pod="openshift-multus/multus-d9t58"
Sep 11 03:40:14.273614 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.272759    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 11 03:40:14.273614 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.272770    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/aae093a3-d3d4-4291-947e-3a59a9b4c84d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-l42ph\" (UID: \"aae093a3-d3d4-4291-947e-3a59a9b4c84d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l42ph"
Sep 11 03:40:14.273614 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.272776    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 11 03:40:14.273614 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.272795    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/aae093a3-d3d4-4291-947e-3a59a9b4c84d-registration-dir\") pod \"aws-ebs-csi-driver-node-l42ph\" (UID: \"aae093a3-d3d4-4291-947e-3a59a9b4c84d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l42ph"
Sep 11 03:40:14.273614 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.272820    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a91c03cc-9ab6-45df-a99a-3f197542c920-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-tr4l2\" (UID: \"a91c03cc-9ab6-45df-a99a-3f197542c920\") " pod="openshift-multus/multus-additional-cni-plugins-tr4l2"
Sep 11 03:40:14.273614 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.272993    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 11 03:40:14.274241 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.273715    2580 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 11 03:40:14.274431 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.274417    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-mqgmz\""
Sep 11 03:40:14.274524 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.274502    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-wkqmr"
Sep 11 03:40:14.274595 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.274417    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 11 03:40:14.274595 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.274549    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 11 03:40:14.276423 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.276408    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-4qfc6\""
Sep 11 03:40:14.276560 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.276515    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 11 03:40:14.276634 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.276562    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 11 03:40:14.291654 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.291635    2580 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-drbzq"
Sep 11 03:40:14.298217 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.298195    2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-drbzq"
Sep 11 03:40:14.362650 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.362624    2580 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 11 03:40:14.373671 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.373633    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bc0076f9-9364-41f4-aadc-7e4cef25221a-cnibin\") pod \"multus-d9t58\" (UID: \"bc0076f9-9364-41f4-aadc-7e4cef25221a\") " pod="openshift-multus/multus-d9t58"
Sep 11 03:40:14.373802 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.373681    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/cc4dd8ce-127a-4e14-9949-0b7075d1df37-iptables-alerter-script\") pod \"iptables-alerter-gqqk2\" (UID: \"cc4dd8ce-127a-4e14-9949-0b7075d1df37\") " pod="openshift-network-operator/iptables-alerter-gqqk2"
Sep 11 03:40:14.373802 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.373706    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a91c03cc-9ab6-45df-a99a-3f197542c920-cni-binary-copy\") pod \"multus-additional-cni-plugins-tr4l2\" (UID: \"a91c03cc-9ab6-45df-a99a-3f197542c920\") " pod="openshift-multus/multus-additional-cni-plugins-tr4l2"
Sep 11 03:40:14.373802 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.373730    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/28b30dae-a2d3-4251-99e8-04bf5153015b-host-kubelet\") pod \"ovnkube-node-j4pgw\" (UID: \"28b30dae-a2d3-4251-99e8-04bf5153015b\") " pod="openshift-ovn-kubernetes/ovnkube-node-j4pgw"
Sep 11 03:40:14.373802 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.373753    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/28b30dae-a2d3-4251-99e8-04bf5153015b-node-log\") pod \"ovnkube-node-j4pgw\" (UID: \"28b30dae-a2d3-4251-99e8-04bf5153015b\") " pod="openshift-ovn-kubernetes/ovnkube-node-j4pgw"
Sep 11 03:40:14.373802 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.373754    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bc0076f9-9364-41f4-aadc-7e4cef25221a-cnibin\") pod \"multus-d9t58\" (UID: \"bc0076f9-9364-41f4-aadc-7e4cef25221a\") " pod="openshift-multus/multus-d9t58"
Sep 11 03:40:14.373802 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.373775    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wm96l\" (UniqueName: \"kubernetes.io/projected/84aade61-b056-43d4-bbcd-fd68eba2cb48-kube-api-access-wm96l\") pod \"node-ca-mlcbg\" (UID: \"84aade61-b056-43d4-bbcd-fd68eba2cb48\") " pod="openshift-image-registry/node-ca-mlcbg"
Sep 11 03:40:14.373802 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.373797    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a7954230-378f-4cf1-b574-3c3b9f0ec4fb-agent-certs\") pod \"konnectivity-agent-tm9tp\" (UID: \"a7954230-378f-4cf1-b574-3c3b9f0ec4fb\") " pod="kube-system/konnectivity-agent-tm9tp"
Sep 11 03:40:14.374133 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.373820    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bc0076f9-9364-41f4-aadc-7e4cef25221a-cni-binary-copy\") pod \"multus-d9t58\" (UID: \"bc0076f9-9364-41f4-aadc-7e4cef25221a\") " pod="openshift-multus/multus-d9t58"
Sep 11 03:40:14.374133 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.373842    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bc0076f9-9364-41f4-aadc-7e4cef25221a-host-var-lib-cni-bin\") pod \"multus-d9t58\" (UID: \"bc0076f9-9364-41f4-aadc-7e4cef25221a\") " pod="openshift-multus/multus-d9t58"
Sep 11 03:40:14.374133 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.373865    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a91c03cc-9ab6-45df-a99a-3f197542c920-system-cni-dir\") pod \"multus-additional-cni-plugins-tr4l2\" (UID: \"a91c03cc-9ab6-45df-a99a-3f197542c920\") " pod="openshift-multus/multus-additional-cni-plugins-tr4l2"
Sep 11 03:40:14.374133 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.373891    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/28b30dae-a2d3-4251-99e8-04bf5153015b-run-ovn\") pod \"ovnkube-node-j4pgw\" (UID: \"28b30dae-a2d3-4251-99e8-04bf5153015b\") " pod="openshift-ovn-kubernetes/ovnkube-node-j4pgw"
Sep 11 03:40:14.374133 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.373912    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4e99df15-f68d-4668-a7bd-7f1da9aa9792-run\") pod \"tuned-5gs25\" (UID: \"4e99df15-f68d-4668-a7bd-7f1da9aa9792\") " pod="openshift-cluster-node-tuning-operator/tuned-5gs25"
Sep 11 03:40:14.374133 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.373937    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/aae093a3-d3d4-4291-947e-3a59a9b4c84d-registration-dir\") pod \"aws-ebs-csi-driver-node-l42ph\" (UID: \"aae093a3-d3d4-4291-947e-3a59a9b4c84d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l42ph"
Sep 11 03:40:14.374133 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.373960    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/28b30dae-a2d3-4251-99e8-04bf5153015b-run-systemd\") pod \"ovnkube-node-j4pgw\" (UID: \"28b30dae-a2d3-4251-99e8-04bf5153015b\") " pod="openshift-ovn-kubernetes/ovnkube-node-j4pgw"
Sep 11 03:40:14.374133 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.373981    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4e99df15-f68d-4668-a7bd-7f1da9aa9792-tmp\") pod \"tuned-5gs25\" (UID: \"4e99df15-f68d-4668-a7bd-7f1da9aa9792\") " pod="openshift-cluster-node-tuning-operator/tuned-5gs25"
Sep 11 03:40:14.374133 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.374004    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bc0076f9-9364-41f4-aadc-7e4cef25221a-system-cni-dir\") pod \"multus-d9t58\" (UID: \"bc0076f9-9364-41f4-aadc-7e4cef25221a\") " pod="openshift-multus/multus-d9t58"
Sep 11 03:40:14.374133 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.374028    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bc0076f9-9364-41f4-aadc-7e4cef25221a-multus-cni-dir\") pod \"multus-d9t58\" (UID: \"bc0076f9-9364-41f4-aadc-7e4cef25221a\") " pod="openshift-multus/multus-d9t58"
Sep 11 03:40:14.374133 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.374051    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/bc0076f9-9364-41f4-aadc-7e4cef25221a-host-var-lib-cni-multus\") pod \"multus-d9t58\" (UID: \"bc0076f9-9364-41f4-aadc-7e4cef25221a\") " pod="openshift-multus/multus-d9t58"
Sep 11 03:40:14.374133 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.374109    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/bc0076f9-9364-41f4-aadc-7e4cef25221a-multus-daemon-config\") pod \"multus-d9t58\" (UID: \"bc0076f9-9364-41f4-aadc-7e4cef25221a\") " pod="openshift-multus/multus-d9t58"
Sep 11 03:40:14.374133 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.374134    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a91c03cc-9ab6-45df-a99a-3f197542c920-tuning-conf-dir\") pod \"multus-additional-cni-plugins-tr4l2\" (UID: \"a91c03cc-9ab6-45df-a99a-3f197542c920\") " pod="openshift-multus/multus-additional-cni-plugins-tr4l2"
Sep 11 03:40:14.374768 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.374159    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9t2dx\" (UniqueName: \"kubernetes.io/projected/a91c03cc-9ab6-45df-a99a-3f197542c920-kube-api-access-9t2dx\") pod \"multus-additional-cni-plugins-tr4l2\" (UID: \"a91c03cc-9ab6-45df-a99a-3f197542c920\") " pod="openshift-multus/multus-additional-cni-plugins-tr4l2"
Sep 11 03:40:14.374768 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.374186    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/28b30dae-a2d3-4251-99e8-04bf5153015b-ovnkube-script-lib\") pod \"ovnkube-node-j4pgw\" (UID: \"28b30dae-a2d3-4251-99e8-04bf5153015b\") " pod="openshift-ovn-kubernetes/ovnkube-node-j4pgw"
Sep 11 03:40:14.374768 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.374210    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a7954230-378f-4cf1-b574-3c3b9f0ec4fb-konnectivity-ca\") pod \"konnectivity-agent-tm9tp\" (UID: \"a7954230-378f-4cf1-b574-3c3b9f0ec4fb\") " pod="kube-system/konnectivity-agent-tm9tp"
Sep 11 03:40:14.374768 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.374235    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb00790b-1a90-4786-8b81-cbfc706f9c46-metrics-certs\") pod \"network-metrics-daemon-qzhz6\" (UID: \"eb00790b-1a90-4786-8b81-cbfc706f9c46\") " pod="openshift-multus/network-metrics-daemon-qzhz6"
Sep 11 03:40:14.374768 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.374263    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bc0076f9-9364-41f4-aadc-7e4cef25221a-host-run-netns\") pod \"multus-d9t58\" (UID: \"bc0076f9-9364-41f4-aadc-7e4cef25221a\") " pod="openshift-multus/multus-d9t58"
Sep 11 03:40:14.374768 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.374288    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ng9t\" (UniqueName: \"kubernetes.io/projected/bc0076f9-9364-41f4-aadc-7e4cef25221a-kube-api-access-4ng9t\") pod \"multus-d9t58\" (UID: \"bc0076f9-9364-41f4-aadc-7e4cef25221a\") " pod="openshift-multus/multus-d9t58"
Sep 11 03:40:14.374768 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.374291    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bc0076f9-9364-41f4-aadc-7e4cef25221a-host-var-lib-cni-bin\") pod \"multus-d9t58\" (UID: \"bc0076f9-9364-41f4-aadc-7e4cef25221a\") " pod="openshift-multus/multus-d9t58"
Sep 11 03:40:14.374768 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.374311    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/aae093a3-d3d4-4291-947e-3a59a9b4c84d-socket-dir\") pod \"aws-ebs-csi-driver-node-l42ph\" (UID: \"aae093a3-d3d4-4291-947e-3a59a9b4c84d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l42ph"
Sep 11 03:40:14.374768 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.374337    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/28b30dae-a2d3-4251-99e8-04bf5153015b-var-lib-openvswitch\") pod \"ovnkube-node-j4pgw\" (UID: \"28b30dae-a2d3-4251-99e8-04bf5153015b\") " pod="openshift-ovn-kubernetes/ovnkube-node-j4pgw"
Sep 11 03:40:14.374768 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.374343    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a91c03cc-9ab6-45df-a99a-3f197542c920-system-cni-dir\") pod \"multus-additional-cni-plugins-tr4l2\" (UID: \"a91c03cc-9ab6-45df-a99a-3f197542c920\") " pod="openshift-multus/multus-additional-cni-plugins-tr4l2"
Sep 11 03:40:14.374768 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.374370    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4e99df15-f68d-4668-a7bd-7f1da9aa9792-etc-modprobe-d\") pod \"tuned-5gs25\" (UID: \"4e99df15-f68d-4668-a7bd-7f1da9aa9792\") " pod="openshift-cluster-node-tuning-operator/tuned-5gs25"
Sep 11 03:40:14.374768 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.374392    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4e99df15-f68d-4668-a7bd-7f1da9aa9792-etc-sysconfig\") pod \"tuned-5gs25\" (UID: \"4e99df15-f68d-4668-a7bd-7f1da9aa9792\") " pod="openshift-cluster-node-tuning-operator/tuned-5gs25"
Sep 11 03:40:14.374768 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.374419    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/aae093a3-d3d4-4291-947e-3a59a9b4c84d-device-dir\") pod \"aws-ebs-csi-driver-node-l42ph\" (UID: \"aae093a3-d3d4-4291-947e-3a59a9b4c84d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l42ph"
Sep 11 03:40:14.374768 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.374442    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gftnx\" (UniqueName: \"kubernetes.io/projected/cc4dd8ce-127a-4e14-9949-0b7075d1df37-kube-api-access-gftnx\") pod \"iptables-alerter-gqqk2\" (UID: \"cc4dd8ce-127a-4e14-9949-0b7075d1df37\") " pod="openshift-network-operator/iptables-alerter-gqqk2"
Sep 11 03:40:14.374768 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.374679    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/bc0076f9-9364-41f4-aadc-7e4cef25221a-host-var-lib-cni-multus\") pod \"multus-d9t58\" (UID: \"bc0076f9-9364-41f4-aadc-7e4cef25221a\") " pod="openshift-multus/multus-d9t58"
Sep 11 03:40:14.374768 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.374529    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a91c03cc-9ab6-45df-a99a-3f197542c920-cni-binary-copy\") pod \"multus-additional-cni-plugins-tr4l2\" (UID: \"a91c03cc-9ab6-45df-a99a-3f197542c920\") " pod="openshift-multus/multus-additional-cni-plugins-tr4l2"
Sep 11 03:40:14.374768 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.374728    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f6fa8b71-4ef9-4c91-b7ee-7ef235532ea0-kubelet-config\") pod \"global-pull-secret-syncer-8knjw\" (UID: \"f6fa8b71-4ef9-4c91-b7ee-7ef235532ea0\") " pod="kube-system/global-pull-secret-syncer-8knjw"
Sep 11 03:40:14.375595 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.374579    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bc0076f9-9364-41f4-aadc-7e4cef25221a-multus-cni-dir\") pod \"multus-d9t58\" (UID: \"bc0076f9-9364-41f4-aadc-7e4cef25221a\") " pod="openshift-multus/multus-d9t58"
Sep 11 03:40:14.375595 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.374619    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bc0076f9-9364-41f4-aadc-7e4cef25221a-cni-binary-copy\") pod \"multus-d9t58\" (UID: \"bc0076f9-9364-41f4-aadc-7e4cef25221a\") " pod="openshift-multus/multus-d9t58"
Sep 11 03:40:14.375595 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.374551    2580 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 11 03:40:14.375595 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.374684    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f6fa8b71-4ef9-4c91-b7ee-7ef235532ea0-kubelet-config\") pod \"global-pull-secret-syncer-8knjw\" (UID: \"f6fa8b71-4ef9-4c91-b7ee-7ef235532ea0\") " pod="kube-system/global-pull-secret-syncer-8knjw"
Sep 11 03:40:14.375595 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.374788    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a91c03cc-9ab6-45df-a99a-3f197542c920-cnibin\") pod \"multus-additional-cni-plugins-tr4l2\" (UID: \"a91c03cc-9ab6-45df-a99a-3f197542c920\") " pod="openshift-multus/multus-additional-cni-plugins-tr4l2"
Sep 11 03:40:14.375595 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.374794    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bc0076f9-9364-41f4-aadc-7e4cef25221a-host-run-netns\") pod \"multus-d9t58\" (UID: \"bc0076f9-9364-41f4-aadc-7e4cef25221a\") " pod="openshift-multus/multus-d9t58"
Sep 11 03:40:14.375595 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:40:14.374802    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 03:40:14.375595 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.374477    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bc0076f9-9364-41f4-aadc-7e4cef25221a-system-cni-dir\") pod \"multus-d9t58\" (UID: \"bc0076f9-9364-41f4-aadc-7e4cef25221a\") " pod="openshift-multus/multus-d9t58"
Sep 11 03:40:14.375595 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.374869    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a91c03cc-9ab6-45df-a99a-3f197542c920-cnibin\") pod \"multus-additional-cni-plugins-tr4l2\" (UID: \"a91c03cc-9ab6-45df-a99a-3f197542c920\") " pod="openshift-multus/multus-additional-cni-plugins-tr4l2"
Sep 11 03:40:14.375595 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:40:14.374875    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb00790b-1a90-4786-8b81-cbfc706f9c46-metrics-certs podName:eb00790b-1a90-4786-8b81-cbfc706f9c46 nodeName:}" failed. No retries permitted until 2026-09-11 03:40:14.874854518 +0000 UTC m=+3.033340844 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/eb00790b-1a90-4786-8b81-cbfc706f9c46-metrics-certs") pod "network-metrics-daemon-qzhz6" (UID: "eb00790b-1a90-4786-8b81-cbfc706f9c46") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 03:40:14.375595 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.374884    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/aae093a3-d3d4-4291-947e-3a59a9b4c84d-registration-dir\") pod \"aws-ebs-csi-driver-node-l42ph\" (UID: \"aae093a3-d3d4-4291-947e-3a59a9b4c84d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l42ph"
Sep 11 03:40:14.375595 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.374917    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/aae093a3-d3d4-4291-947e-3a59a9b4c84d-device-dir\") pod \"aws-ebs-csi-driver-node-l42ph\" (UID: \"aae093a3-d3d4-4291-947e-3a59a9b4c84d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l42ph"
Sep 11 03:40:14.375595 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.375122    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/cc4dd8ce-127a-4e14-9949-0b7075d1df37-iptables-alerter-script\") pod \"iptables-alerter-gqqk2\" (UID: \"cc4dd8ce-127a-4e14-9949-0b7075d1df37\") " pod="openshift-network-operator/iptables-alerter-gqqk2"
Sep 11 03:40:14.375595 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.375135    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4e99df15-f68d-4668-a7bd-7f1da9aa9792-etc-systemd\") pod \"tuned-5gs25\" (UID: \"4e99df15-f68d-4668-a7bd-7f1da9aa9792\") " pod="openshift-cluster-node-tuning-operator/tuned-5gs25"
Sep 11 03:40:14.375595 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.374755    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a91c03cc-9ab6-45df-a99a-3f197542c920-tuning-conf-dir\") pod \"multus-additional-cni-plugins-tr4l2\" (UID: \"a91c03cc-9ab6-45df-a99a-3f197542c920\") " pod="openshift-multus/multus-additional-cni-plugins-tr4l2"
Sep 11 03:40:14.375595 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.375221    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/bc0076f9-9364-41f4-aadc-7e4cef25221a-multus-daemon-config\") pod \"multus-d9t58\" (UID: \"bc0076f9-9364-41f4-aadc-7e4cef25221a\") " pod="openshift-multus/multus-d9t58"
Sep 11 03:40:14.375595 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.375229    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/aae093a3-d3d4-4291-947e-3a59a9b4c84d-socket-dir\") pod \"aws-ebs-csi-driver-node-l42ph\" (UID: \"aae093a3-d3d4-4291-947e-3a59a9b4c84d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l42ph"
Sep 11 03:40:14.376368 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.375250    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/84aade61-b056-43d4-bbcd-fd68eba2cb48-serviceca\") pod \"node-ca-mlcbg\" (UID: \"84aade61-b056-43d4-bbcd-fd68eba2cb48\") " pod="openshift-image-registry/node-ca-mlcbg"
Sep 11 03:40:14.376368 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.375333    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/aae093a3-d3d4-4291-947e-3a59a9b4c84d-sys-fs\") pod \"aws-ebs-csi-driver-node-l42ph\" (UID: \"aae093a3-d3d4-4291-947e-3a59a9b4c84d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l42ph"
Sep 11 03:40:14.376368 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.375352    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a7954230-378f-4cf1-b574-3c3b9f0ec4fb-konnectivity-ca\") pod \"konnectivity-agent-tm9tp\" (UID: \"a7954230-378f-4cf1-b574-3c3b9f0ec4fb\") " pod="kube-system/konnectivity-agent-tm9tp"
Sep 11 03:40:14.376368 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.375361    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/28b30dae-a2d3-4251-99e8-04bf5153015b-host-slash\") pod \"ovnkube-node-j4pgw\" (UID: \"28b30dae-a2d3-4251-99e8-04bf5153015b\") " pod="openshift-ovn-kubernetes/ovnkube-node-j4pgw"
Sep 11 03:40:14.376368 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.375399    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/28b30dae-a2d3-4251-99e8-04bf5153015b-run-openvswitch\") pod \"ovnkube-node-j4pgw\" (UID: \"28b30dae-a2d3-4251-99e8-04bf5153015b\") " pod="openshift-ovn-kubernetes/ovnkube-node-j4pgw"
Sep 11 03:40:14.376368 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.375406    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/aae093a3-d3d4-4291-947e-3a59a9b4c84d-sys-fs\") pod \"aws-ebs-csi-driver-node-l42ph\" (UID: \"aae093a3-d3d4-4291-947e-3a59a9b4c84d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l42ph"
Sep 11 03:40:14.376368 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.375423    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/28b30dae-a2d3-4251-99e8-04bf5153015b-log-socket\") pod \"ovnkube-node-j4pgw\" (UID: \"28b30dae-a2d3-4251-99e8-04bf5153015b\") " pod="openshift-ovn-kubernetes/ovnkube-node-j4pgw"
Sep 11 03:40:14.376368 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.375458    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4e99df15-f68d-4668-a7bd-7f1da9aa9792-etc-sysctl-conf\") pod \"tuned-5gs25\" (UID: \"4e99df15-f68d-4668-a7bd-7f1da9aa9792\") " pod="openshift-cluster-node-tuning-operator/tuned-5gs25"
Sep 11 03:40:14.376368 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.375481    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/84aade61-b056-43d4-bbcd-fd68eba2cb48-host\") pod \"node-ca-mlcbg\" (UID: \"84aade61-b056-43d4-bbcd-fd68eba2cb48\") " pod="openshift-image-registry/node-ca-mlcbg"
Sep 11 03:40:14.376368 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.375501    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/bc0076f9-9364-41f4-aadc-7e4cef25221a-host-run-k8s-cni-cncf-io\") pod \"multus-d9t58\" (UID: \"bc0076f9-9364-41f4-aadc-7e4cef25221a\") " pod="openshift-multus/multus-d9t58"
Sep 11 03:40:14.376368 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.375523    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/aae093a3-d3d4-4291-947e-3a59a9b4c84d-etc-selinux\") pod \"aws-ebs-csi-driver-node-l42ph\" (UID: \"aae093a3-d3d4-4291-947e-3a59a9b4c84d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l42ph"
Sep 11 03:40:14.376368 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.375540    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/84aade61-b056-43d4-bbcd-fd68eba2cb48-host\") pod \"node-ca-mlcbg\" (UID: \"84aade61-b056-43d4-bbcd-fd68eba2cb48\") " pod="openshift-image-registry/node-ca-mlcbg"
Sep 11 03:40:14.376368 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.375559    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/bc0076f9-9364-41f4-aadc-7e4cef25221a-host-run-k8s-cni-cncf-io\") pod \"multus-d9t58\" (UID: \"bc0076f9-9364-41f4-aadc-7e4cef25221a\") " pod="openshift-multus/multus-d9t58"
Sep 11 03:40:14.376368 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.375545    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f6fa8b71-4ef9-4c91-b7ee-7ef235532ea0-original-pull-secret\") pod \"global-pull-secret-syncer-8knjw\" (UID: \"f6fa8b71-4ef9-4c91-b7ee-7ef235532ea0\") " pod="kube-system/global-pull-secret-syncer-8knjw"
Sep 11 03:40:14.376368 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.375605    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4e99df15-f68d-4668-a7bd-7f1da9aa9792-etc-kubernetes\") pod \"tuned-5gs25\" (UID: \"4e99df15-f68d-4668-a7bd-7f1da9aa9792\") " pod="openshift-cluster-node-tuning-operator/tuned-5gs25"
Sep 11 03:40:14.376368 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.375614    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/84aade61-b056-43d4-bbcd-fd68eba2cb48-serviceca\") pod \"node-ca-mlcbg\" (UID: \"84aade61-b056-43d4-bbcd-fd68eba2cb48\") " pod="openshift-image-registry/node-ca-mlcbg"
Sep 11 03:40:14.376368 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.375619    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/aae093a3-d3d4-4291-947e-3a59a9b4c84d-etc-selinux\") pod \"aws-ebs-csi-driver-node-l42ph\" (UID: \"aae093a3-d3d4-4291-947e-3a59a9b4c84d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l42ph"
Sep 11 03:40:14.377117 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.375676    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4e99df15-f68d-4668-a7bd-7f1da9aa9792-sys\") pod \"tuned-5gs25\" (UID: \"4e99df15-f68d-4668-a7bd-7f1da9aa9792\") " pod="openshift-cluster-node-tuning-operator/tuned-5gs25"
Sep 11 03:40:14.377117 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:40:14.375692    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 03:40:14.377117 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.375702    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/896af27b-fde3-43f0-bdbb-5bdf0815873a-tmp-dir\") pod \"node-resolver-wkqmr\" (UID: \"896af27b-fde3-43f0-bdbb-5bdf0815873a\") " pod="openshift-dns/node-resolver-wkqmr"
Sep 11 03:40:14.377117 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.375725    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f6fa8b71-4ef9-4c91-b7ee-7ef235532ea0-dbus\") pod \"global-pull-secret-syncer-8knjw\" (UID: \"f6fa8b71-4ef9-4c91-b7ee-7ef235532ea0\") " pod="kube-system/global-pull-secret-syncer-8knjw"
Sep 11 03:40:14.377117 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.375746    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/28b30dae-a2d3-4251-99e8-04bf5153015b-etc-openvswitch\") pod \"ovnkube-node-j4pgw\" (UID: \"28b30dae-a2d3-4251-99e8-04bf5153015b\") " pod="openshift-ovn-kubernetes/ovnkube-node-j4pgw"
Sep 11 03:40:14.377117 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.375762    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/28b30dae-a2d3-4251-99e8-04bf5153015b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-j4pgw\" (UID: \"28b30dae-a2d3-4251-99e8-04bf5153015b\") " pod="openshift-ovn-kubernetes/ovnkube-node-j4pgw"
Sep 11 03:40:14.377117 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:40:14.375780    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f6fa8b71-4ef9-4c91-b7ee-7ef235532ea0-original-pull-secret podName:f6fa8b71-4ef9-4c91-b7ee-7ef235532ea0 nodeName:}" failed. No retries permitted until 2026-09-11 03:40:14.875761801 +0000 UTC m=+3.034248102 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f6fa8b71-4ef9-4c91-b7ee-7ef235532ea0-original-pull-secret") pod "global-pull-secret-syncer-8knjw" (UID: "f6fa8b71-4ef9-4c91-b7ee-7ef235532ea0") : object "kube-system"/"original-pull-secret" not registered
Sep 11 03:40:14.377117 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.375799    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f6fa8b71-4ef9-4c91-b7ee-7ef235532ea0-dbus\") pod \"global-pull-secret-syncer-8knjw\" (UID: \"f6fa8b71-4ef9-4c91-b7ee-7ef235532ea0\") " pod="kube-system/global-pull-secret-syncer-8knjw"
Sep 11 03:40:14.377117 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.375816    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/28b30dae-a2d3-4251-99e8-04bf5153015b-ovn-node-metrics-cert\") pod \"ovnkube-node-j4pgw\" (UID: \"28b30dae-a2d3-4251-99e8-04bf5153015b\") " pod="openshift-ovn-kubernetes/ovnkube-node-j4pgw"
Sep 11 03:40:14.377117 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.375850    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4btsj\" (UniqueName: \"kubernetes.io/projected/aae093a3-d3d4-4291-947e-3a59a9b4c84d-kube-api-access-4btsj\") pod \"aws-ebs-csi-driver-node-l42ph\" (UID: \"aae093a3-d3d4-4291-947e-3a59a9b4c84d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l42ph"
Sep 11 03:40:14.377117 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.375875    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4gtr\" (UniqueName: \"kubernetes.io/projected/ad569c0f-e4f1-42f2-aa2b-48f0a4f16a57-kube-api-access-s4gtr\") pod \"network-check-target-f7hng\" (UID: \"ad569c0f-e4f1-42f2-aa2b-48f0a4f16a57\") " pod="openshift-network-diagnostics/network-check-target-f7hng"
Sep 11 03:40:14.377117 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.375898    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/28b30dae-a2d3-4251-99e8-04bf5153015b-systemd-units\") pod \"ovnkube-node-j4pgw\" (UID: \"28b30dae-a2d3-4251-99e8-04bf5153015b\") " pod="openshift-ovn-kubernetes/ovnkube-node-j4pgw"
Sep 11 03:40:14.377117 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.375921    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/28b30dae-a2d3-4251-99e8-04bf5153015b-host-cni-netd\") pod \"ovnkube-node-j4pgw\" (UID: \"28b30dae-a2d3-4251-99e8-04bf5153015b\") " pod="openshift-ovn-kubernetes/ovnkube-node-j4pgw"
Sep 11 03:40:14.377117 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.375944    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4e99df15-f68d-4668-a7bd-7f1da9aa9792-etc-tuned\") pod \"tuned-5gs25\" (UID: \"4e99df15-f68d-4668-a7bd-7f1da9aa9792\") " pod="openshift-cluster-node-tuning-operator/tuned-5gs25"
Sep 11 03:40:14.377117 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.375969    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bc0076f9-9364-41f4-aadc-7e4cef25221a-host-var-lib-kubelet\") pod \"multus-d9t58\" (UID: \"bc0076f9-9364-41f4-aadc-7e4cef25221a\") " pod="openshift-multus/multus-d9t58"
Sep 11 03:40:14.377117 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.376002    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/aae093a3-d3d4-4291-947e-3a59a9b4c84d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-l42ph\" (UID: \"aae093a3-d3d4-4291-947e-3a59a9b4c84d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l42ph"
Sep 11 03:40:14.377631 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.376020    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bc0076f9-9364-41f4-aadc-7e4cef25221a-host-var-lib-kubelet\") pod \"multus-d9t58\" (UID: \"bc0076f9-9364-41f4-aadc-7e4cef25221a\") " pod="openshift-multus/multus-d9t58"
Sep 11 03:40:14.377631 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.376026    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a91c03cc-9ab6-45df-a99a-3f197542c920-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-tr4l2\" (UID: \"a91c03cc-9ab6-45df-a99a-3f197542c920\") " pod="openshift-multus/multus-additional-cni-plugins-tr4l2"
Sep 11 03:40:14.377631 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.376050    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4e99df15-f68d-4668-a7bd-7f1da9aa9792-lib-modules\") pod \"tuned-5gs25\" (UID: \"4e99df15-f68d-4668-a7bd-7f1da9aa9792\") " pod="openshift-cluster-node-tuning-operator/tuned-5gs25"
Sep 11 03:40:14.377631 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.376069    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/bc0076f9-9364-41f4-aadc-7e4cef25221a-host-run-multus-certs\") pod \"multus-d9t58\" (UID: \"bc0076f9-9364-41f4-aadc-7e4cef25221a\") " pod="openshift-multus/multus-d9t58"
Sep 11 03:40:14.377631 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.376075    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/aae093a3-d3d4-4291-947e-3a59a9b4c84d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-l42ph\" (UID: \"aae093a3-d3d4-4291-947e-3a59a9b4c84d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l42ph"
Sep 11 03:40:14.377631 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.376089    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/28b30dae-a2d3-4251-99e8-04bf5153015b-ovnkube-config\") pod \"ovnkube-node-j4pgw\" (UID: \"28b30dae-a2d3-4251-99e8-04bf5153015b\") " pod="openshift-ovn-kubernetes/ovnkube-node-j4pgw"
Sep 11 03:40:14.377631 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.376130    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/bc0076f9-9364-41f4-aadc-7e4cef25221a-hostroot\") pod \"multus-d9t58\" (UID: \"bc0076f9-9364-41f4-aadc-7e4cef25221a\") " pod="openshift-multus/multus-d9t58"
Sep 11 03:40:14.377631 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.376133    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/bc0076f9-9364-41f4-aadc-7e4cef25221a-host-run-multus-certs\") pod \"multus-d9t58\" (UID: \"bc0076f9-9364-41f4-aadc-7e4cef25221a\") " pod="openshift-multus/multus-d9t58"
Sep 11 03:40:14.377631 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.376235    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cc4dd8ce-127a-4e14-9949-0b7075d1df37-host-slash\") pod \"iptables-alerter-gqqk2\" (UID: \"cc4dd8ce-127a-4e14-9949-0b7075d1df37\") " pod="openshift-network-operator/iptables-alerter-gqqk2"
Sep 11 03:40:14.377631 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.376244    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/bc0076f9-9364-41f4-aadc-7e4cef25221a-hostroot\") pod \"multus-d9t58\" (UID: \"bc0076f9-9364-41f4-aadc-7e4cef25221a\") " pod="openshift-multus/multus-d9t58"
Sep 11 03:40:14.377631 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.376280    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a91c03cc-9ab6-45df-a99a-3f197542c920-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-tr4l2\" (UID: \"a91c03cc-9ab6-45df-a99a-3f197542c920\") " pod="openshift-multus/multus-additional-cni-plugins-tr4l2"
Sep 11 03:40:14.377631 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.376308    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/28b30dae-a2d3-4251-99e8-04bf5153015b-host-run-netns\") pod \"ovnkube-node-j4pgw\" (UID: \"28b30dae-a2d3-4251-99e8-04bf5153015b\") " pod="openshift-ovn-kubernetes/ovnkube-node-j4pgw"
Sep 11 03:40:14.377631 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.376331    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/28b30dae-a2d3-4251-99e8-04bf5153015b-env-overrides\") pod \"ovnkube-node-j4pgw\" (UID: \"28b30dae-a2d3-4251-99e8-04bf5153015b\") " pod="openshift-ovn-kubernetes/ovnkube-node-j4pgw"
Sep 11 03:40:14.377631 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.376353    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4e99df15-f68d-4668-a7bd-7f1da9aa9792-etc-sysctl-d\") pod \"tuned-5gs25\" (UID: \"4e99df15-f68d-4668-a7bd-7f1da9aa9792\") " pod="openshift-cluster-node-tuning-operator/tuned-5gs25"
Sep 11 03:40:14.377631 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.376376    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbw6x\" (UniqueName: \"kubernetes.io/projected/4e99df15-f68d-4668-a7bd-7f1da9aa9792-kube-api-access-rbw6x\") pod \"tuned-5gs25\" (UID: \"4e99df15-f68d-4668-a7bd-7f1da9aa9792\") " pod="openshift-cluster-node-tuning-operator/tuned-5gs25"
Sep 11 03:40:14.377631 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.376400    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/bc0076f9-9364-41f4-aadc-7e4cef25221a-multus-socket-dir-parent\") pod \"multus-d9t58\" (UID: \"bc0076f9-9364-41f4-aadc-7e4cef25221a\") " pod="openshift-multus/multus-d9t58"
Sep 11 03:40:14.377631 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.376427    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/28b30dae-a2d3-4251-99e8-04bf5153015b-host-cni-bin\") pod \"ovnkube-node-j4pgw\" (UID: \"28b30dae-a2d3-4251-99e8-04bf5153015b\") " pod="openshift-ovn-kubernetes/ovnkube-node-j4pgw"
Sep 11 03:40:14.378485 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.376450    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/896af27b-fde3-43f0-bdbb-5bdf0815873a-hosts-file\") pod \"node-resolver-wkqmr\" (UID: \"896af27b-fde3-43f0-bdbb-5bdf0815873a\") " pod="openshift-dns/node-resolver-wkqmr"
Sep 11 03:40:14.378485 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.376472    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bc0076f9-9364-41f4-aadc-7e4cef25221a-multus-conf-dir\") pod \"multus-d9t58\" (UID: \"bc0076f9-9364-41f4-aadc-7e4cef25221a\") " pod="openshift-multus/multus-d9t58"
Sep 11 03:40:14.378485 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.376499    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bc0076f9-9364-41f4-aadc-7e4cef25221a-etc-kubernetes\") pod \"multus-d9t58\" (UID: \"bc0076f9-9364-41f4-aadc-7e4cef25221a\") " pod="openshift-multus/multus-d9t58"
Sep 11 03:40:14.378485 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.376519    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4e99df15-f68d-4668-a7bd-7f1da9aa9792-host\") pod \"tuned-5gs25\" (UID: \"4e99df15-f68d-4668-a7bd-7f1da9aa9792\") " pod="openshift-cluster-node-tuning-operator/tuned-5gs25"
Sep 11 03:40:14.378485 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.376541    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/28b30dae-a2d3-4251-99e8-04bf5153015b-host-run-ovn-kubernetes\") pod \"ovnkube-node-j4pgw\" (UID: \"28b30dae-a2d3-4251-99e8-04bf5153015b\") " pod="openshift-ovn-kubernetes/ovnkube-node-j4pgw"
Sep 11 03:40:14.378485 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.376564    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzxwr\" (UniqueName: \"kubernetes.io/projected/28b30dae-a2d3-4251-99e8-04bf5153015b-kube-api-access-gzxwr\") pod \"ovnkube-node-j4pgw\" (UID: \"28b30dae-a2d3-4251-99e8-04bf5153015b\") " pod="openshift-ovn-kubernetes/ovnkube-node-j4pgw"
Sep 11 03:40:14.378485 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.376621    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4e99df15-f68d-4668-a7bd-7f1da9aa9792-var-lib-kubelet\") pod \"tuned-5gs25\" (UID: \"4e99df15-f68d-4668-a7bd-7f1da9aa9792\") " pod="openshift-cluster-node-tuning-operator/tuned-5gs25"
Sep 11 03:40:14.378485 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.376620    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/bc0076f9-9364-41f4-aadc-7e4cef25221a-multus-socket-dir-parent\") pod \"multus-d9t58\" (UID: \"bc0076f9-9364-41f4-aadc-7e4cef25221a\") " pod="openshift-multus/multus-d9t58"
Sep 11 03:40:14.378485 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.376282    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cc4dd8ce-127a-4e14-9949-0b7075d1df37-host-slash\") pod \"iptables-alerter-gqqk2\" (UID: \"cc4dd8ce-127a-4e14-9949-0b7075d1df37\") " pod="openshift-network-operator/iptables-alerter-gqqk2"
Sep 11 03:40:14.378485 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.376698    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a91c03cc-9ab6-45df-a99a-3f197542c920-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-tr4l2\" (UID: \"a91c03cc-9ab6-45df-a99a-3f197542c920\") " pod="openshift-multus/multus-additional-cni-plugins-tr4l2"
Sep 11 03:40:14.378485 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.376649    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4xhw\" (UniqueName: \"kubernetes.io/projected/896af27b-fde3-43f0-bdbb-5bdf0815873a-kube-api-access-s4xhw\") pod \"node-resolver-wkqmr\" (UID: \"896af27b-fde3-43f0-bdbb-5bdf0815873a\") " pod="openshift-dns/node-resolver-wkqmr"
Sep 11 03:40:14.378485 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.376563    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a91c03cc-9ab6-45df-a99a-3f197542c920-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-tr4l2\" (UID: \"a91c03cc-9ab6-45df-a99a-3f197542c920\") " pod="openshift-multus/multus-additional-cni-plugins-tr4l2"
Sep 11 03:40:14.378485 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.376757    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjbjx\" (UniqueName: \"kubernetes.io/projected/eb00790b-1a90-4786-8b81-cbfc706f9c46-kube-api-access-wjbjx\") pod \"network-metrics-daemon-qzhz6\" (UID: \"eb00790b-1a90-4786-8b81-cbfc706f9c46\") " pod="openshift-multus/network-metrics-daemon-qzhz6"
Sep 11 03:40:14.378485 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.376758    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bc0076f9-9364-41f4-aadc-7e4cef25221a-multus-conf-dir\") pod \"multus-d9t58\" (UID: \"bc0076f9-9364-41f4-aadc-7e4cef25221a\") " pod="openshift-multus/multus-d9t58"
Sep 11 03:40:14.378485 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.376840    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bc0076f9-9364-41f4-aadc-7e4cef25221a-etc-kubernetes\") pod \"multus-d9t58\" (UID: \"bc0076f9-9364-41f4-aadc-7e4cef25221a\") " pod="openshift-multus/multus-d9t58"
Sep 11 03:40:14.379470 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.379433    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a7954230-378f-4cf1-b574-3c3b9f0ec4fb-agent-certs\") pod \"konnectivity-agent-tm9tp\" (UID: \"a7954230-378f-4cf1-b574-3c3b9f0ec4fb\") " pod="kube-system/konnectivity-agent-tm9tp"
Sep 11 03:40:14.383639 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.383565    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ng9t\" (UniqueName: \"kubernetes.io/projected/bc0076f9-9364-41f4-aadc-7e4cef25221a-kube-api-access-4ng9t\") pod \"multus-d9t58\" (UID: \"bc0076f9-9364-41f4-aadc-7e4cef25221a\") " pod="openshift-multus/multus-d9t58"
Sep 11 03:40:14.384174 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.384117    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wm96l\" (UniqueName: \"kubernetes.io/projected/84aade61-b056-43d4-bbcd-fd68eba2cb48-kube-api-access-wm96l\") pod \"node-ca-mlcbg\" (UID: \"84aade61-b056-43d4-bbcd-fd68eba2cb48\") " pod="openshift-image-registry/node-ca-mlcbg"
Sep 11 03:40:14.384279 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.384254    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjbjx\" (UniqueName: \"kubernetes.io/projected/eb00790b-1a90-4786-8b81-cbfc706f9c46-kube-api-access-wjbjx\") pod \"network-metrics-daemon-qzhz6\" (UID: \"eb00790b-1a90-4786-8b81-cbfc706f9c46\") " pod="openshift-multus/network-metrics-daemon-qzhz6"
Sep 11 03:40:14.385782 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.385759    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4btsj\" (UniqueName: \"kubernetes.io/projected/aae093a3-d3d4-4291-947e-3a59a9b4c84d-kube-api-access-4btsj\") pod \"aws-ebs-csi-driver-node-l42ph\" (UID: \"aae093a3-d3d4-4291-947e-3a59a9b4c84d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l42ph"
Sep 11 03:40:14.385884 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.385822    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9t2dx\" (UniqueName: \"kubernetes.io/projected/a91c03cc-9ab6-45df-a99a-3f197542c920-kube-api-access-9t2dx\") pod \"multus-additional-cni-plugins-tr4l2\" (UID: \"a91c03cc-9ab6-45df-a99a-3f197542c920\") " pod="openshift-multus/multus-additional-cni-plugins-tr4l2"
Sep 11 03:40:14.386195 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.386168    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gftnx\" (UniqueName: \"kubernetes.io/projected/cc4dd8ce-127a-4e14-9949-0b7075d1df37-kube-api-access-gftnx\") pod \"iptables-alerter-gqqk2\" (UID: \"cc4dd8ce-127a-4e14-9949-0b7075d1df37\") " pod="openshift-network-operator/iptables-alerter-gqqk2"
Sep 11 03:40:14.462016 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.461970    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-121.ec2.internal" event={"ID":"99e85cb9783bfb509f2d3b998351c555","Type":"ContainerStarted","Data":"ed8cd884d78d5a2e07d01ad4c665adcc77918d038d4ea242c92d5d25d6139a27"}
Sep 11 03:40:14.463155 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.463129    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-121.ec2.internal" event={"ID":"355aa2f2ac5d4d26d63779c83ede8936","Type":"ContainerStarted","Data":"0fc58a3ad46260608d52980a74bab478d4414f7040e90254ba6cab38e9de72cc"}
Sep 11 03:40:14.477527 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.477500    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/28b30dae-a2d3-4251-99e8-04bf5153015b-ovnkube-config\") pod \"ovnkube-node-j4pgw\" (UID: \"28b30dae-a2d3-4251-99e8-04bf5153015b\") " pod="openshift-ovn-kubernetes/ovnkube-node-j4pgw"
Sep 11 03:40:14.477633 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.477543    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/28b30dae-a2d3-4251-99e8-04bf5153015b-host-run-netns\") pod \"ovnkube-node-j4pgw\" (UID: \"28b30dae-a2d3-4251-99e8-04bf5153015b\") " pod="openshift-ovn-kubernetes/ovnkube-node-j4pgw"
Sep 11 03:40:14.477633 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.477571    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/28b30dae-a2d3-4251-99e8-04bf5153015b-env-overrides\") pod \"ovnkube-node-j4pgw\" (UID: \"28b30dae-a2d3-4251-99e8-04bf5153015b\") " pod="openshift-ovn-kubernetes/ovnkube-node-j4pgw"
Sep 11 03:40:14.477633 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.477597    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4e99df15-f68d-4668-a7bd-7f1da9aa9792-etc-sysctl-d\") pod \"tuned-5gs25\" (UID: \"4e99df15-f68d-4668-a7bd-7f1da9aa9792\") " pod="openshift-cluster-node-tuning-operator/tuned-5gs25"
Sep 11 03:40:14.477633 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.477623    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbw6x\" (UniqueName: \"kubernetes.io/projected/4e99df15-f68d-4668-a7bd-7f1da9aa9792-kube-api-access-rbw6x\") pod \"tuned-5gs25\" (UID: \"4e99df15-f68d-4668-a7bd-7f1da9aa9792\") " pod="openshift-cluster-node-tuning-operator/tuned-5gs25"
Sep 11 03:40:14.477867 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.477648    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/28b30dae-a2d3-4251-99e8-04bf5153015b-host-cni-bin\") pod \"ovnkube-node-j4pgw\" (UID: \"28b30dae-a2d3-4251-99e8-04bf5153015b\") " pod="openshift-ovn-kubernetes/ovnkube-node-j4pgw"
Sep 11 03:40:14.477867 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.477621    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/28b30dae-a2d3-4251-99e8-04bf5153015b-host-run-netns\") pod \"ovnkube-node-j4pgw\" (UID: \"28b30dae-a2d3-4251-99e8-04bf5153015b\") " pod="openshift-ovn-kubernetes/ovnkube-node-j4pgw"
Sep 11 03:40:14.477867 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.477686    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/896af27b-fde3-43f0-bdbb-5bdf0815873a-hosts-file\") pod \"node-resolver-wkqmr\" (UID: \"896af27b-fde3-43f0-bdbb-5bdf0815873a\") " pod="openshift-dns/node-resolver-wkqmr"
Sep 11 03:40:14.477867 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.477735    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4e99df15-f68d-4668-a7bd-7f1da9aa9792-host\") pod \"tuned-5gs25\" (UID: \"4e99df15-f68d-4668-a7bd-7f1da9aa9792\") " pod="openshift-cluster-node-tuning-operator/tuned-5gs25"
Sep 11 03:40:14.477867 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.477743    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/896af27b-fde3-43f0-bdbb-5bdf0815873a-hosts-file\") pod \"node-resolver-wkqmr\" (UID: \"896af27b-fde3-43f0-bdbb-5bdf0815873a\") " pod="openshift-dns/node-resolver-wkqmr"
Sep 11 03:40:14.477867 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.477764    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4e99df15-f68d-4668-a7bd-7f1da9aa9792-etc-sysctl-d\") pod \"tuned-5gs25\" (UID: \"4e99df15-f68d-4668-a7bd-7f1da9aa9792\") " pod="openshift-cluster-node-tuning-operator/tuned-5gs25"
Sep 11 03:40:14.477867 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.477776    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/28b30dae-a2d3-4251-99e8-04bf5153015b-host-run-ovn-kubernetes\") pod \"ovnkube-node-j4pgw\" (UID: \"28b30dae-a2d3-4251-99e8-04bf5153015b\") " pod="openshift-ovn-kubernetes/ovnkube-node-j4pgw"
Sep 11 03:40:14.477867 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.477791    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4e99df15-f68d-4668-a7bd-7f1da9aa9792-host\") pod \"tuned-5gs25\" (UID: \"4e99df15-f68d-4668-a7bd-7f1da9aa9792\") " pod="openshift-cluster-node-tuning-operator/tuned-5gs25"
Sep 11 03:40:14.477867 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.477801    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzxwr\" (UniqueName: \"kubernetes.io/projected/28b30dae-a2d3-4251-99e8-04bf5153015b-kube-api-access-gzxwr\") pod \"ovnkube-node-j4pgw\" (UID: \"28b30dae-a2d3-4251-99e8-04bf5153015b\") " pod="openshift-ovn-kubernetes/ovnkube-node-j4pgw"
Sep 11 03:40:14.477867 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.477811    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/28b30dae-a2d3-4251-99e8-04bf5153015b-host-cni-bin\") pod \"ovnkube-node-j4pgw\" (UID: \"28b30dae-a2d3-4251-99e8-04bf5153015b\") " pod="openshift-ovn-kubernetes/ovnkube-node-j4pgw"
Sep 11 03:40:14.477867 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.477818    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4e99df15-f68d-4668-a7bd-7f1da9aa9792-var-lib-kubelet\") pod \"tuned-5gs25\" (UID: \"4e99df15-f68d-4668-a7bd-7f1da9aa9792\") " pod="openshift-cluster-node-tuning-operator/tuned-5gs25"
Sep 11 03:40:14.477867 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.477829    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/28b30dae-a2d3-4251-99e8-04bf5153015b-host-run-ovn-kubernetes\") pod \"ovnkube-node-j4pgw\" (UID: \"28b30dae-a2d3-4251-99e8-04bf5153015b\") " pod="openshift-ovn-kubernetes/ovnkube-node-j4pgw"
Sep 11 03:40:14.477867 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.477847    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4e99df15-f68d-4668-a7bd-7f1da9aa9792-var-lib-kubelet\") pod \"tuned-5gs25\" (UID: \"4e99df15-f68d-4668-a7bd-7f1da9aa9792\") " pod="openshift-cluster-node-tuning-operator/tuned-5gs25"
Sep 11 03:40:14.477867 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.477847    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4xhw\" (UniqueName: \"kubernetes.io/projected/896af27b-fde3-43f0-bdbb-5bdf0815873a-kube-api-access-s4xhw\") pod \"node-resolver-wkqmr\" (UID: \"896af27b-fde3-43f0-bdbb-5bdf0815873a\") " pod="openshift-dns/node-resolver-wkqmr"
Sep 11 03:40:14.478574 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.477888    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/28b30dae-a2d3-4251-99e8-04bf5153015b-host-kubelet\") pod \"ovnkube-node-j4pgw\" (UID: \"28b30dae-a2d3-4251-99e8-04bf5153015b\") " pod="openshift-ovn-kubernetes/ovnkube-node-j4pgw"
Sep 11 03:40:14.478574 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.477925    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/28b30dae-a2d3-4251-99e8-04bf5153015b-node-log\") pod \"ovnkube-node-j4pgw\" (UID: \"28b30dae-a2d3-4251-99e8-04bf5153015b\") " pod="openshift-ovn-kubernetes/ovnkube-node-j4pgw"
Sep 11 03:40:14.478574 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.477962    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/28b30dae-a2d3-4251-99e8-04bf5153015b-run-ovn\") pod \"ovnkube-node-j4pgw\" (UID: \"28b30dae-a2d3-4251-99e8-04bf5153015b\") " pod="openshift-ovn-kubernetes/ovnkube-node-j4pgw"
Sep 11 03:40:14.478574 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.477971    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/28b30dae-a2d3-4251-99e8-04bf5153015b-host-kubelet\") pod \"ovnkube-node-j4pgw\" (UID: \"28b30dae-a2d3-4251-99e8-04bf5153015b\") " pod="openshift-ovn-kubernetes/ovnkube-node-j4pgw"
Sep 11 03:40:14.478574 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.477991    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4e99df15-f68d-4668-a7bd-7f1da9aa9792-run\") pod \"tuned-5gs25\" (UID: \"4e99df15-f68d-4668-a7bd-7f1da9aa9792\") " pod="openshift-cluster-node-tuning-operator/tuned-5gs25"
Sep 11 03:40:14.478574 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.478019    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/28b30dae-a2d3-4251-99e8-04bf5153015b-node-log\") pod \"ovnkube-node-j4pgw\" (UID: \"28b30dae-a2d3-4251-99e8-04bf5153015b\") " pod="openshift-ovn-kubernetes/ovnkube-node-j4pgw"
Sep 11 03:40:14.478574 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.478022    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/28b30dae-a2d3-4251-99e8-04bf5153015b-run-ovn\") pod \"ovnkube-node-j4pgw\" (UID: \"28b30dae-a2d3-4251-99e8-04bf5153015b\") " pod="openshift-ovn-kubernetes/ovnkube-node-j4pgw"
Sep 11 03:40:14.478574 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.478055    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4e99df15-f68d-4668-a7bd-7f1da9aa9792-run\") pod \"tuned-5gs25\" (UID: \"4e99df15-f68d-4668-a7bd-7f1da9aa9792\") " pod="openshift-cluster-node-tuning-operator/tuned-5gs25"
Sep 11 03:40:14.478574 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.478033    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/28b30dae-a2d3-4251-99e8-04bf5153015b-run-systemd\") pod \"ovnkube-node-j4pgw\" (UID: \"28b30dae-a2d3-4251-99e8-04bf5153015b\") " pod="openshift-ovn-kubernetes/ovnkube-node-j4pgw"
Sep 11 03:40:14.478574 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.478084    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4e99df15-f68d-4668-a7bd-7f1da9aa9792-tmp\") pod \"tuned-5gs25\" (UID: \"4e99df15-f68d-4668-a7bd-7f1da9aa9792\") " pod="openshift-cluster-node-tuning-operator/tuned-5gs25"
Sep 11 03:40:14.478574 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.478080    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/28b30dae-a2d3-4251-99e8-04bf5153015b-run-systemd\") pod \"ovnkube-node-j4pgw\" (UID: \"28b30dae-a2d3-4251-99e8-04bf5153015b\") " pod="openshift-ovn-kubernetes/ovnkube-node-j4pgw"
Sep 11 03:40:14.478574 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.478127    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/28b30dae-a2d3-4251-99e8-04bf5153015b-ovnkube-script-lib\") pod \"ovnkube-node-j4pgw\" (UID: \"28b30dae-a2d3-4251-99e8-04bf5153015b\") " pod="openshift-ovn-kubernetes/ovnkube-node-j4pgw"
Sep 11 03:40:14.478574 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.478155    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/28b30dae-a2d3-4251-99e8-04bf5153015b-ovnkube-config\") pod \"ovnkube-node-j4pgw\" (UID: \"28b30dae-a2d3-4251-99e8-04bf5153015b\") " pod="openshift-ovn-kubernetes/ovnkube-node-j4pgw"
Sep 11 03:40:14.478574 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.478165    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/28b30dae-a2d3-4251-99e8-04bf5153015b-var-lib-openvswitch\") pod \"ovnkube-node-j4pgw\" (UID: \"28b30dae-a2d3-4251-99e8-04bf5153015b\") " pod="openshift-ovn-kubernetes/ovnkube-node-j4pgw"
Sep 11 03:40:14.478574 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.478188    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4e99df15-f68d-4668-a7bd-7f1da9aa9792-etc-modprobe-d\") pod \"tuned-5gs25\" (UID: \"4e99df15-f68d-4668-a7bd-7f1da9aa9792\") " pod="openshift-cluster-node-tuning-operator/tuned-5gs25"
Sep 11 03:40:14.478574 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.478210    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4e99df15-f68d-4668-a7bd-7f1da9aa9792-etc-sysconfig\") pod \"tuned-5gs25\" (UID: \"4e99df15-f68d-4668-a7bd-7f1da9aa9792\") " pod="openshift-cluster-node-tuning-operator/tuned-5gs25"
Sep 11 03:40:14.478574 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.478220    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/28b30dae-a2d3-4251-99e8-04bf5153015b-var-lib-openvswitch\") pod \"ovnkube-node-j4pgw\" (UID: \"28b30dae-a2d3-4251-99e8-04bf5153015b\") " pod="openshift-ovn-kubernetes/ovnkube-node-j4pgw"
Sep 11 03:40:14.478574 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.478236    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4e99df15-f68d-4668-a7bd-7f1da9aa9792-etc-systemd\") pod \"tuned-5gs25\" (UID: \"4e99df15-f68d-4668-a7bd-7f1da9aa9792\") " pod="openshift-cluster-node-tuning-operator/tuned-5gs25"
Sep 11 03:40:14.479341 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.478262    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/28b30dae-a2d3-4251-99e8-04bf5153015b-host-slash\") pod \"ovnkube-node-j4pgw\" (UID: \"28b30dae-a2d3-4251-99e8-04bf5153015b\") " pod="openshift-ovn-kubernetes/ovnkube-node-j4pgw"
Sep 11 03:40:14.479341 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.478287    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/28b30dae-a2d3-4251-99e8-04bf5153015b-run-openvswitch\") pod \"ovnkube-node-j4pgw\" (UID: \"28b30dae-a2d3-4251-99e8-04bf5153015b\") " pod="openshift-ovn-kubernetes/ovnkube-node-j4pgw"
Sep 11 03:40:14.479341 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.478308    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/28b30dae-a2d3-4251-99e8-04bf5153015b-log-socket\") pod \"ovnkube-node-j4pgw\" (UID: \"28b30dae-a2d3-4251-99e8-04bf5153015b\") " pod="openshift-ovn-kubernetes/ovnkube-node-j4pgw"
Sep 11 03:40:14.479341 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.478330    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4e99df15-f68d-4668-a7bd-7f1da9aa9792-etc-sysctl-conf\") pod \"tuned-5gs25\" (UID: \"4e99df15-f68d-4668-a7bd-7f1da9aa9792\") " pod="openshift-cluster-node-tuning-operator/tuned-5gs25"
Sep 11 03:40:14.479341 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.478331    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4e99df15-f68d-4668-a7bd-7f1da9aa9792-etc-modprobe-d\") pod \"tuned-5gs25\" (UID: \"4e99df15-f68d-4668-a7bd-7f1da9aa9792\") " pod="openshift-cluster-node-tuning-operator/tuned-5gs25"
Sep 11 03:40:14.479341 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.478374    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/28b30dae-a2d3-4251-99e8-04bf5153015b-host-slash\") pod \"ovnkube-node-j4pgw\" (UID: \"28b30dae-a2d3-4251-99e8-04bf5153015b\") " pod="openshift-ovn-kubernetes/ovnkube-node-j4pgw"
Sep 11 03:40:14.479341 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.478379    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4e99df15-f68d-4668-a7bd-7f1da9aa9792-etc-sysconfig\") pod \"tuned-5gs25\" (UID: \"4e99df15-f68d-4668-a7bd-7f1da9aa9792\") " pod="openshift-cluster-node-tuning-operator/tuned-5gs25"
Sep 11 03:40:14.479341 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.478416    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/28b30dae-a2d3-4251-99e8-04bf5153015b-env-overrides\") pod \"ovnkube-node-j4pgw\" (UID: \"28b30dae-a2d3-4251-99e8-04bf5153015b\") " pod="openshift-ovn-kubernetes/ovnkube-node-j4pgw"
Sep 11 03:40:14.479341 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.478426    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4e99df15-f68d-4668-a7bd-7f1da9aa9792-etc-systemd\") pod \"tuned-5gs25\" (UID: \"4e99df15-f68d-4668-a7bd-7f1da9aa9792\") " pod="openshift-cluster-node-tuning-operator/tuned-5gs25"
Sep 11 03:40:14.479341 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.478428    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4e99df15-f68d-4668-a7bd-7f1da9aa9792-etc-kubernetes\") pod \"tuned-5gs25\" (UID: \"4e99df15-f68d-4668-a7bd-7f1da9aa9792\") " pod="openshift-cluster-node-tuning-operator/tuned-5gs25"
Sep 11 03:40:14.479341 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.478456    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4e99df15-f68d-4668-a7bd-7f1da9aa9792-sys\") pod \"tuned-5gs25\" (UID: \"4e99df15-f68d-4668-a7bd-7f1da9aa9792\") " pod="openshift-cluster-node-tuning-operator/tuned-5gs25"
Sep 11 03:40:14.479341 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.478473    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4e99df15-f68d-4668-a7bd-7f1da9aa9792-etc-sysctl-conf\") pod \"tuned-5gs25\" (UID: \"4e99df15-f68d-4668-a7bd-7f1da9aa9792\") " pod="openshift-cluster-node-tuning-operator/tuned-5gs25"
Sep 11 03:40:14.479341 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.478473    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/28b30dae-a2d3-4251-99e8-04bf5153015b-log-socket\") pod \"ovnkube-node-j4pgw\" (UID: \"28b30dae-a2d3-4251-99e8-04bf5153015b\") " pod="openshift-ovn-kubernetes/ovnkube-node-j4pgw"
Sep 11 03:40:14.479341 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.478456    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/28b30dae-a2d3-4251-99e8-04bf5153015b-run-openvswitch\") pod \"ovnkube-node-j4pgw\" (UID: \"28b30dae-a2d3-4251-99e8-04bf5153015b\") " pod="openshift-ovn-kubernetes/ovnkube-node-j4pgw"
Sep 11 03:40:14.479341 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.478479    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/896af27b-fde3-43f0-bdbb-5bdf0815873a-tmp-dir\") pod \"node-resolver-wkqmr\" (UID: \"896af27b-fde3-43f0-bdbb-5bdf0815873a\") " pod="openshift-dns/node-resolver-wkqmr"
Sep 11 03:40:14.479341 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.478513    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4e99df15-f68d-4668-a7bd-7f1da9aa9792-sys\") pod \"tuned-5gs25\" (UID: \"4e99df15-f68d-4668-a7bd-7f1da9aa9792\") " pod="openshift-cluster-node-tuning-operator/tuned-5gs25"
Sep 11 03:40:14.479341 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.478513    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4e99df15-f68d-4668-a7bd-7f1da9aa9792-etc-kubernetes\") pod \"tuned-5gs25\" (UID: \"4e99df15-f68d-4668-a7bd-7f1da9aa9792\") " pod="openshift-cluster-node-tuning-operator/tuned-5gs25"
Sep 11 03:40:14.479341 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.478527    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/28b30dae-a2d3-4251-99e8-04bf5153015b-etc-openvswitch\") pod \"ovnkube-node-j4pgw\" (UID: \"28b30dae-a2d3-4251-99e8-04bf5153015b\") " pod="openshift-ovn-kubernetes/ovnkube-node-j4pgw"
Sep 11 03:40:14.480047 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.478560    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/28b30dae-a2d3-4251-99e8-04bf5153015b-etc-openvswitch\") pod \"ovnkube-node-j4pgw\" (UID: \"28b30dae-a2d3-4251-99e8-04bf5153015b\") " pod="openshift-ovn-kubernetes/ovnkube-node-j4pgw"
Sep 11 03:40:14.480047 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.478558    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/28b30dae-a2d3-4251-99e8-04bf5153015b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-j4pgw\" (UID: \"28b30dae-a2d3-4251-99e8-04bf5153015b\") " pod="openshift-ovn-kubernetes/ovnkube-node-j4pgw"
Sep 11 03:40:14.480047 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.478589    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/28b30dae-a2d3-4251-99e8-04bf5153015b-ovn-node-metrics-cert\") pod \"ovnkube-node-j4pgw\" (UID: \"28b30dae-a2d3-4251-99e8-04bf5153015b\") " pod="openshift-ovn-kubernetes/ovnkube-node-j4pgw"
Sep 11 03:40:14.480047 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.478591    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/28b30dae-a2d3-4251-99e8-04bf5153015b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-j4pgw\" (UID: \"28b30dae-a2d3-4251-99e8-04bf5153015b\") " pod="openshift-ovn-kubernetes/ovnkube-node-j4pgw"
Sep 11 03:40:14.480047 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.478608    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/28b30dae-a2d3-4251-99e8-04bf5153015b-ovnkube-script-lib\") pod \"ovnkube-node-j4pgw\" (UID: \"28b30dae-a2d3-4251-99e8-04bf5153015b\") " pod="openshift-ovn-kubernetes/ovnkube-node-j4pgw"
Sep 11 03:40:14.480047 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.478616    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4gtr\" (UniqueName: \"kubernetes.io/projected/ad569c0f-e4f1-42f2-aa2b-48f0a4f16a57-kube-api-access-s4gtr\") pod \"network-check-target-f7hng\" (UID: \"ad569c0f-e4f1-42f2-aa2b-48f0a4f16a57\") " pod="openshift-network-diagnostics/network-check-target-f7hng"
Sep 11 03:40:14.480047 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.478641    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/28b30dae-a2d3-4251-99e8-04bf5153015b-systemd-units\") pod \"ovnkube-node-j4pgw\" (UID: \"28b30dae-a2d3-4251-99e8-04bf5153015b\") " pod="openshift-ovn-kubernetes/ovnkube-node-j4pgw"
Sep 11 03:40:14.480047 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.478682    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/28b30dae-a2d3-4251-99e8-04bf5153015b-host-cni-netd\") pod \"ovnkube-node-j4pgw\" (UID: \"28b30dae-a2d3-4251-99e8-04bf5153015b\") " pod="openshift-ovn-kubernetes/ovnkube-node-j4pgw"
Sep 11 03:40:14.480047 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.478706    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4e99df15-f68d-4668-a7bd-7f1da9aa9792-etc-tuned\") pod \"tuned-5gs25\" (UID: \"4e99df15-f68d-4668-a7bd-7f1da9aa9792\") " pod="openshift-cluster-node-tuning-operator/tuned-5gs25"
Sep 11 03:40:14.480047 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.478708    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/28b30dae-a2d3-4251-99e8-04bf5153015b-systemd-units\") pod \"ovnkube-node-j4pgw\" (UID: \"28b30dae-a2d3-4251-99e8-04bf5153015b\") " pod="openshift-ovn-kubernetes/ovnkube-node-j4pgw"
Sep 11 03:40:14.480047 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.478731    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4e99df15-f68d-4668-a7bd-7f1da9aa9792-lib-modules\") pod \"tuned-5gs25\" (UID: \"4e99df15-f68d-4668-a7bd-7f1da9aa9792\") " pod="openshift-cluster-node-tuning-operator/tuned-5gs25"
Sep 11 03:40:14.480047 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.478757    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/28b30dae-a2d3-4251-99e8-04bf5153015b-host-cni-netd\") pod \"ovnkube-node-j4pgw\" (UID: \"28b30dae-a2d3-4251-99e8-04bf5153015b\") " pod="openshift-ovn-kubernetes/ovnkube-node-j4pgw"
Sep 11 03:40:14.480047 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.478828    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/896af27b-fde3-43f0-bdbb-5bdf0815873a-tmp-dir\") pod \"node-resolver-wkqmr\" (UID: \"896af27b-fde3-43f0-bdbb-5bdf0815873a\") " pod="openshift-dns/node-resolver-wkqmr"
Sep 11 03:40:14.480047 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.478848    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4e99df15-f68d-4668-a7bd-7f1da9aa9792-lib-modules\") pod \"tuned-5gs25\" (UID: \"4e99df15-f68d-4668-a7bd-7f1da9aa9792\") " pod="openshift-cluster-node-tuning-operator/tuned-5gs25"
Sep 11 03:40:14.480733 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.480712    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4e99df15-f68d-4668-a7bd-7f1da9aa9792-tmp\") pod \"tuned-5gs25\" (UID: \"4e99df15-f68d-4668-a7bd-7f1da9aa9792\") " pod="openshift-cluster-node-tuning-operator/tuned-5gs25"
Sep 11 03:40:14.481046 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.481025    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4e99df15-f68d-4668-a7bd-7f1da9aa9792-etc-tuned\") pod \"tuned-5gs25\" (UID: \"4e99df15-f68d-4668-a7bd-7f1da9aa9792\") " pod="openshift-cluster-node-tuning-operator/tuned-5gs25"
Sep 11 03:40:14.481234 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.481188    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/28b30dae-a2d3-4251-99e8-04bf5153015b-ovn-node-metrics-cert\") pod \"ovnkube-node-j4pgw\" (UID: \"28b30dae-a2d3-4251-99e8-04bf5153015b\") " pod="openshift-ovn-kubernetes/ovnkube-node-j4pgw"
Sep 11 03:40:14.481317 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.481282    2580 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 03:40:14.484229 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:40:14.484208    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 03:40:14.484229 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:40:14.484235    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 03:40:14.484395 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:40:14.484248    2580 projected.go:196] Error preparing data for projected volume kube-api-access-s4gtr for pod openshift-network-diagnostics/network-check-target-f7hng: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 03:40:14.484395 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:40:14.484305    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ad569c0f-e4f1-42f2-aa2b-48f0a4f16a57-kube-api-access-s4gtr podName:ad569c0f-e4f1-42f2-aa2b-48f0a4f16a57 nodeName:}" failed. No retries permitted until 2026-09-11 03:40:14.984288397 +0000 UTC m=+3.142774713 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-s4gtr" (UniqueName: "kubernetes.io/projected/ad569c0f-e4f1-42f2-aa2b-48f0a4f16a57-kube-api-access-s4gtr") pod "network-check-target-f7hng" (UID: "ad569c0f-e4f1-42f2-aa2b-48f0a4f16a57") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 03:40:14.486282 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.486236    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4xhw\" (UniqueName: \"kubernetes.io/projected/896af27b-fde3-43f0-bdbb-5bdf0815873a-kube-api-access-s4xhw\") pod \"node-resolver-wkqmr\" (UID: \"896af27b-fde3-43f0-bdbb-5bdf0815873a\") " pod="openshift-dns/node-resolver-wkqmr"
Sep 11 03:40:14.486282 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.486275    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbw6x\" (UniqueName: \"kubernetes.io/projected/4e99df15-f68d-4668-a7bd-7f1da9aa9792-kube-api-access-rbw6x\") pod \"tuned-5gs25\" (UID: \"4e99df15-f68d-4668-a7bd-7f1da9aa9792\") " pod="openshift-cluster-node-tuning-operator/tuned-5gs25"
Sep 11 03:40:14.486390 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.486277    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzxwr\" (UniqueName: \"kubernetes.io/projected/28b30dae-a2d3-4251-99e8-04bf5153015b-kube-api-access-gzxwr\") pod \"ovnkube-node-j4pgw\" (UID: \"28b30dae-a2d3-4251-99e8-04bf5153015b\") " pod="openshift-ovn-kubernetes/ovnkube-node-j4pgw"
Sep 11 03:40:14.557599 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.557565    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l42ph"
Sep 11 03:40:14.569104 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.569074    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-mlcbg"
Sep 11 03:40:14.580750 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.580725    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-gqqk2"
Sep 11 03:40:14.586304 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.586283    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-tm9tp"
Sep 11 03:40:14.598830 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.598808    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-tr4l2"
Sep 11 03:40:14.607326 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.607307    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-d9t58"
Sep 11 03:40:14.614034 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.614013    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-j4pgw"
Sep 11 03:40:14.620725 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.620697    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-5gs25"
Sep 11 03:40:14.626563 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.626542    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-wkqmr"
Sep 11 03:40:14.882087 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.882001    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f6fa8b71-4ef9-4c91-b7ee-7ef235532ea0-original-pull-secret\") pod \"global-pull-secret-syncer-8knjw\" (UID: \"f6fa8b71-4ef9-4c91-b7ee-7ef235532ea0\") " pod="kube-system/global-pull-secret-syncer-8knjw"
Sep 11 03:40:14.882087 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:14.882088    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb00790b-1a90-4786-8b81-cbfc706f9c46-metrics-certs\") pod \"network-metrics-daemon-qzhz6\" (UID: \"eb00790b-1a90-4786-8b81-cbfc706f9c46\") " pod="openshift-multus/network-metrics-daemon-qzhz6"
Sep 11 03:40:14.882307 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:40:14.882163    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 03:40:14.882307 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:40:14.882174    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 03:40:14.882307 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:40:14.882226    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f6fa8b71-4ef9-4c91-b7ee-7ef235532ea0-original-pull-secret podName:f6fa8b71-4ef9-4c91-b7ee-7ef235532ea0 nodeName:}" failed. No retries permitted until 2026-09-11 03:40:15.882210146 +0000 UTC m=+4.040696452 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f6fa8b71-4ef9-4c91-b7ee-7ef235532ea0-original-pull-secret") pod "global-pull-secret-syncer-8knjw" (UID: "f6fa8b71-4ef9-4c91-b7ee-7ef235532ea0") : object "kube-system"/"original-pull-secret" not registered
Sep 11 03:40:14.882307 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:40:14.882247    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb00790b-1a90-4786-8b81-cbfc706f9c46-metrics-certs podName:eb00790b-1a90-4786-8b81-cbfc706f9c46 nodeName:}" failed. No retries permitted until 2026-09-11 03:40:15.882236701 +0000 UTC m=+4.040723006 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/eb00790b-1a90-4786-8b81-cbfc706f9c46-metrics-certs") pod "network-metrics-daemon-qzhz6" (UID: "eb00790b-1a90-4786-8b81-cbfc706f9c46") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 03:40:15.083424 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:15.083390    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4gtr\" (UniqueName: \"kubernetes.io/projected/ad569c0f-e4f1-42f2-aa2b-48f0a4f16a57-kube-api-access-s4gtr\") pod \"network-check-target-f7hng\" (UID: \"ad569c0f-e4f1-42f2-aa2b-48f0a4f16a57\") " pod="openshift-network-diagnostics/network-check-target-f7hng"
Sep 11 03:40:15.083616 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:40:15.083594    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 03:40:15.083689 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:40:15.083627    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 03:40:15.083689 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:40:15.083641    2580 projected.go:196] Error preparing data for projected volume kube-api-access-s4gtr for pod openshift-network-diagnostics/network-check-target-f7hng: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 03:40:15.083795 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:40:15.083721    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ad569c0f-e4f1-42f2-aa2b-48f0a4f16a57-kube-api-access-s4gtr podName:ad569c0f-e4f1-42f2-aa2b-48f0a4f16a57 nodeName:}" failed. No retries permitted until 2026-09-11 03:40:16.083702305 +0000 UTC m=+4.242188620 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-s4gtr" (UniqueName: "kubernetes.io/projected/ad569c0f-e4f1-42f2-aa2b-48f0a4f16a57-kube-api-access-s4gtr") pod "network-check-target-f7hng" (UID: "ad569c0f-e4f1-42f2-aa2b-48f0a4f16a57") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 03:40:15.274885 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:15.274853    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc4dd8ce_127a_4e14_9949_0b7075d1df37.slice/crio-780786616dbaba898e00eac063b3a0806aa6beb2fedd6723aa5bc85a49fdd68f WatchSource:0}: Error finding container 780786616dbaba898e00eac063b3a0806aa6beb2fedd6723aa5bc85a49fdd68f: Status 404 returned error can't find the container with id 780786616dbaba898e00eac063b3a0806aa6beb2fedd6723aa5bc85a49fdd68f
Sep 11 03:40:15.276684 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:15.276650    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84aade61_b056_43d4_bbcd_fd68eba2cb48.slice/crio-112ef7ff0dbfabaf1f21d9dd5552040b435cada6089167d67267997c8e8bfdd8 WatchSource:0}: Error finding container 112ef7ff0dbfabaf1f21d9dd5552040b435cada6089167d67267997c8e8bfdd8: Status 404 returned error can't find the container with id 112ef7ff0dbfabaf1f21d9dd5552040b435cada6089167d67267997c8e8bfdd8
Sep 11 03:40:15.280305 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:15.280279    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc0076f9_9364_41f4_aadc_7e4cef25221a.slice/crio-e529fa4556723b4b2e7c4166c9e1287b4e5ae8ea0b3b114c14b8aa75715a09b9 WatchSource:0}: Error finding container e529fa4556723b4b2e7c4166c9e1287b4e5ae8ea0b3b114c14b8aa75715a09b9: Status 404 returned error can't find the container with id e529fa4556723b4b2e7c4166c9e1287b4e5ae8ea0b3b114c14b8aa75715a09b9
Sep 11 03:40:15.283511 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:15.283487    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28b30dae_a2d3_4251_99e8_04bf5153015b.slice/crio-2271385b1bb29fd69c758f49d60ab105c7ac024e447659306f2e84d92bf7e237 WatchSource:0}: Error finding container 2271385b1bb29fd69c758f49d60ab105c7ac024e447659306f2e84d92bf7e237: Status 404 returned error can't find the container with id 2271385b1bb29fd69c758f49d60ab105c7ac024e447659306f2e84d92bf7e237
Sep 11 03:40:15.284411 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:15.284369    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod896af27b_fde3_43f0_bdbb_5bdf0815873a.slice/crio-7499ae520154bbe033506571f674272400aa87b3441af6868df76f3e444e6032 WatchSource:0}: Error finding container 7499ae520154bbe033506571f674272400aa87b3441af6868df76f3e444e6032: Status 404 returned error can't find the container with id 7499ae520154bbe033506571f674272400aa87b3441af6868df76f3e444e6032
Sep 11 03:40:15.285789 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:15.285765    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda91c03cc_9ab6_45df_a99a_3f197542c920.slice/crio-dd6cb5d1516919519d7bfaea318cdd3a9cd0e3f52ad46e23a64418df76404c17 WatchSource:0}: Error finding container dd6cb5d1516919519d7bfaea318cdd3a9cd0e3f52ad46e23a64418df76404c17: Status 404 returned error can't find the container with id dd6cb5d1516919519d7bfaea318cdd3a9cd0e3f52ad46e23a64418df76404c17
Sep 11 03:40:15.287541 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:15.287481    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaae093a3_d3d4_4291_947e_3a59a9b4c84d.slice/crio-669ebafafa3e99e3f949aa0696d72e9aff80128b1d24085ee964434055bd3958 WatchSource:0}: Error finding container 669ebafafa3e99e3f949aa0696d72e9aff80128b1d24085ee964434055bd3958: Status 404 returned error can't find the container with id 669ebafafa3e99e3f949aa0696d72e9aff80128b1d24085ee964434055bd3958
Sep 11 03:40:15.287875 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:15.287850    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e99df15_f68d_4668_a7bd_7f1da9aa9792.slice/crio-429e2d0e7221a358a222569c20910d0273a0e8df316f55f057d9d53f3b24e79a WatchSource:0}: Error finding container 429e2d0e7221a358a222569c20910d0273a0e8df316f55f057d9d53f3b24e79a: Status 404 returned error can't find the container with id 429e2d0e7221a358a222569c20910d0273a0e8df316f55f057d9d53f3b24e79a
Sep 11 03:40:15.289422 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:15.289399    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7954230_378f_4cf1_b574_3c3b9f0ec4fb.slice/crio-20b39c29b935b7e3223e11de2f6bd7b0a9f39047010f3f3a517aaa3e7f6db8bc WatchSource:0}: Error finding container 20b39c29b935b7e3223e11de2f6bd7b0a9f39047010f3f3a517aaa3e7f6db8bc: Status 404 returned error can't find the container with id 20b39c29b935b7e3223e11de2f6bd7b0a9f39047010f3f3a517aaa3e7f6db8bc
Sep 11 03:40:15.301303 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:15.301253    2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-10 03:35:14 +0000 UTC" deadline="2028-05-19 13:03:40.91231449 +0000 UTC"
Sep 11 03:40:15.301413 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:15.301303    2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14793h23m25.611015268s"
Sep 11 03:40:15.457771 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:15.457693    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8knjw"
Sep 11 03:40:15.457771 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:15.457736    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f7hng"
Sep 11 03:40:15.457960 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:40:15.457836    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8knjw" podUID="f6fa8b71-4ef9-4c91-b7ee-7ef235532ea0"
Sep 11 03:40:15.458031 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:40:15.457983    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f7hng" podUID="ad569c0f-e4f1-42f2-aa2b-48f0a4f16a57"
Sep 11 03:40:15.466116 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:15.466093    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l42ph" event={"ID":"aae093a3-d3d4-4291-947e-3a59a9b4c84d","Type":"ContainerStarted","Data":"669ebafafa3e99e3f949aa0696d72e9aff80128b1d24085ee964434055bd3958"}
Sep 11 03:40:15.467249 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:15.467229    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-d9t58" event={"ID":"bc0076f9-9364-41f4-aadc-7e4cef25221a","Type":"ContainerStarted","Data":"e529fa4556723b4b2e7c4166c9e1287b4e5ae8ea0b3b114c14b8aa75715a09b9"}
Sep 11 03:40:15.468342 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:15.468324    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-mlcbg" event={"ID":"84aade61-b056-43d4-bbcd-fd68eba2cb48","Type":"ContainerStarted","Data":"112ef7ff0dbfabaf1f21d9dd5552040b435cada6089167d67267997c8e8bfdd8"}
Sep 11 03:40:15.470227 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:15.470207    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-121.ec2.internal" event={"ID":"355aa2f2ac5d4d26d63779c83ede8936","Type":"ContainerStarted","Data":"46559c0abd13f39f62a96f3ae356a89d9d748c1ba2dfaad7c5e2bddc2007c099"}
Sep 11 03:40:15.471402 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:15.471383    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tr4l2" event={"ID":"a91c03cc-9ab6-45df-a99a-3f197542c920","Type":"ContainerStarted","Data":"dd6cb5d1516919519d7bfaea318cdd3a9cd0e3f52ad46e23a64418df76404c17"}
Sep 11 03:40:15.472468 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:15.472443    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j4pgw" event={"ID":"28b30dae-a2d3-4251-99e8-04bf5153015b","Type":"ContainerStarted","Data":"2271385b1bb29fd69c758f49d60ab105c7ac024e447659306f2e84d92bf7e237"}
Sep 11 03:40:15.473813 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:15.473788    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-gqqk2" event={"ID":"cc4dd8ce-127a-4e14-9949-0b7075d1df37","Type":"ContainerStarted","Data":"780786616dbaba898e00eac063b3a0806aa6beb2fedd6723aa5bc85a49fdd68f"}
Sep 11 03:40:15.474680 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:15.474644    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-wkqmr" event={"ID":"896af27b-fde3-43f0-bdbb-5bdf0815873a","Type":"ContainerStarted","Data":"7499ae520154bbe033506571f674272400aa87b3441af6868df76f3e444e6032"}
Sep 11 03:40:15.475771 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:15.475754    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-tm9tp" event={"ID":"a7954230-378f-4cf1-b574-3c3b9f0ec4fb","Type":"ContainerStarted","Data":"20b39c29b935b7e3223e11de2f6bd7b0a9f39047010f3f3a517aaa3e7f6db8bc"}
Sep 11 03:40:15.479350 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:15.479330    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-5gs25" event={"ID":"4e99df15-f68d-4668-a7bd-7f1da9aa9792","Type":"ContainerStarted","Data":"429e2d0e7221a358a222569c20910d0273a0e8df316f55f057d9d53f3b24e79a"}
Sep 11 03:40:15.888981 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:15.888894    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb00790b-1a90-4786-8b81-cbfc706f9c46-metrics-certs\") pod \"network-metrics-daemon-qzhz6\" (UID: \"eb00790b-1a90-4786-8b81-cbfc706f9c46\") " pod="openshift-multus/network-metrics-daemon-qzhz6"
Sep 11 03:40:15.888981 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:15.888953    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f6fa8b71-4ef9-4c91-b7ee-7ef235532ea0-original-pull-secret\") pod \"global-pull-secret-syncer-8knjw\" (UID: \"f6fa8b71-4ef9-4c91-b7ee-7ef235532ea0\") " pod="kube-system/global-pull-secret-syncer-8knjw"
Sep 11 03:40:15.889211 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:40:15.889063    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 03:40:15.889211 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:40:15.889087    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 03:40:15.889211 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:40:15.889123    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f6fa8b71-4ef9-4c91-b7ee-7ef235532ea0-original-pull-secret podName:f6fa8b71-4ef9-4c91-b7ee-7ef235532ea0 nodeName:}" failed. No retries permitted until 2026-09-11 03:40:17.889105003 +0000 UTC m=+6.047591306 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f6fa8b71-4ef9-4c91-b7ee-7ef235532ea0-original-pull-secret") pod "global-pull-secret-syncer-8knjw" (UID: "f6fa8b71-4ef9-4c91-b7ee-7ef235532ea0") : object "kube-system"/"original-pull-secret" not registered
Sep 11 03:40:15.889211 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:40:15.889150    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb00790b-1a90-4786-8b81-cbfc706f9c46-metrics-certs podName:eb00790b-1a90-4786-8b81-cbfc706f9c46 nodeName:}" failed. No retries permitted until 2026-09-11 03:40:17.889133111 +0000 UTC m=+6.047619423 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/eb00790b-1a90-4786-8b81-cbfc706f9c46-metrics-certs") pod "network-metrics-daemon-qzhz6" (UID: "eb00790b-1a90-4786-8b81-cbfc706f9c46") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 03:40:16.091583 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:16.090941    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4gtr\" (UniqueName: \"kubernetes.io/projected/ad569c0f-e4f1-42f2-aa2b-48f0a4f16a57-kube-api-access-s4gtr\") pod \"network-check-target-f7hng\" (UID: \"ad569c0f-e4f1-42f2-aa2b-48f0a4f16a57\") " pod="openshift-network-diagnostics/network-check-target-f7hng"
Sep 11 03:40:16.091583 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:40:16.091136    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 03:40:16.091583 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:40:16.091158    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 03:40:16.091583 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:40:16.091171    2580 projected.go:196] Error preparing data for projected volume kube-api-access-s4gtr for pod openshift-network-diagnostics/network-check-target-f7hng: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 03:40:16.091583 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:40:16.091229    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ad569c0f-e4f1-42f2-aa2b-48f0a4f16a57-kube-api-access-s4gtr podName:ad569c0f-e4f1-42f2-aa2b-48f0a4f16a57 nodeName:}" failed. No retries permitted until 2026-09-11 03:40:18.091210312 +0000 UTC m=+6.249696633 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-s4gtr" (UniqueName: "kubernetes.io/projected/ad569c0f-e4f1-42f2-aa2b-48f0a4f16a57-kube-api-access-s4gtr") pod "network-check-target-f7hng" (UID: "ad569c0f-e4f1-42f2-aa2b-48f0a4f16a57") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 03:40:16.302477 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:16.302433    2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-10 03:35:14 +0000 UTC" deadline="2028-03-06 08:05:05.908531676 +0000 UTC"
Sep 11 03:40:16.302477 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:16.302475    2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13012h24m49.60606097s"
Sep 11 03:40:16.460862 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:16.460830    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qzhz6"
Sep 11 03:40:16.461018 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:40:16.460959    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qzhz6" podUID="eb00790b-1a90-4786-8b81-cbfc706f9c46"
Sep 11 03:40:16.503509 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:16.502642    2580 generic.go:356] "Generic (PLEG): container finished" podID="99e85cb9783bfb509f2d3b998351c555" containerID="9b8fe054bb40d6efef4bfcbedfffc6af0a70dfbb402eb61b6c6ff2e2d1f1f1b9" exitCode=0
Sep 11 03:40:16.503509 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:16.503335    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-121.ec2.internal" event={"ID":"99e85cb9783bfb509f2d3b998351c555","Type":"ContainerDied","Data":"9b8fe054bb40d6efef4bfcbedfffc6af0a70dfbb402eb61b6c6ff2e2d1f1f1b9"}
Sep 11 03:40:16.519751 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:16.519318    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-121.ec2.internal" podStartSLOduration=3.519298036 podStartE2EDuration="3.519298036s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 03:40:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:40:15.483163455 +0000 UTC m=+3.641649778" watchObservedRunningTime="2026-09-11 03:40:16.519298036 +0000 UTC m=+4.677784360"
Sep 11 03:40:17.457633 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:17.457593    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8knjw"
Sep 11 03:40:17.458075 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:40:17.457734    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8knjw" podUID="f6fa8b71-4ef9-4c91-b7ee-7ef235532ea0"
Sep 11 03:40:17.458149 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:17.457610    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f7hng"
Sep 11 03:40:17.458237 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:40:17.458212    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f7hng" podUID="ad569c0f-e4f1-42f2-aa2b-48f0a4f16a57"
Sep 11 03:40:17.526450 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:17.525771    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-121.ec2.internal" event={"ID":"99e85cb9783bfb509f2d3b998351c555","Type":"ContainerStarted","Data":"8c93b7908e04a997f2354c3156e6af503988458e6ccf48efb75a85c2f3307b7e"}
Sep 11 03:40:17.542195 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:17.542140    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-121.ec2.internal" podStartSLOduration=4.542119586 podStartE2EDuration="4.542119586s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 03:40:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:40:17.542099747 +0000 UTC m=+5.700586071" watchObservedRunningTime="2026-09-11 03:40:17.542119586 +0000 UTC m=+5.700605892"
Sep 11 03:40:17.907950 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:17.907912    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f6fa8b71-4ef9-4c91-b7ee-7ef235532ea0-original-pull-secret\") pod \"global-pull-secret-syncer-8knjw\" (UID: \"f6fa8b71-4ef9-4c91-b7ee-7ef235532ea0\") " pod="kube-system/global-pull-secret-syncer-8knjw"
Sep 11 03:40:17.908209 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:17.908016    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb00790b-1a90-4786-8b81-cbfc706f9c46-metrics-certs\") pod \"network-metrics-daemon-qzhz6\" (UID: \"eb00790b-1a90-4786-8b81-cbfc706f9c46\") " pod="openshift-multus/network-metrics-daemon-qzhz6"
Sep 11 03:40:17.908209 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:40:17.908151    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 03:40:17.908339 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:40:17.908213    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb00790b-1a90-4786-8b81-cbfc706f9c46-metrics-certs podName:eb00790b-1a90-4786-8b81-cbfc706f9c46 nodeName:}" failed. No retries permitted until 2026-09-11 03:40:21.90819415 +0000 UTC m=+10.066680454 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/eb00790b-1a90-4786-8b81-cbfc706f9c46-metrics-certs") pod "network-metrics-daemon-qzhz6" (UID: "eb00790b-1a90-4786-8b81-cbfc706f9c46") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 03:40:17.908766 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:40:17.908600    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 03:40:17.908766 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:40:17.908653    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f6fa8b71-4ef9-4c91-b7ee-7ef235532ea0-original-pull-secret podName:f6fa8b71-4ef9-4c91-b7ee-7ef235532ea0 nodeName:}" failed. No retries permitted until 2026-09-11 03:40:21.908639178 +0000 UTC m=+10.067125482 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f6fa8b71-4ef9-4c91-b7ee-7ef235532ea0-original-pull-secret") pod "global-pull-secret-syncer-8knjw" (UID: "f6fa8b71-4ef9-4c91-b7ee-7ef235532ea0") : object "kube-system"/"original-pull-secret" not registered
Sep 11 03:40:18.109236 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:18.109195    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4gtr\" (UniqueName: \"kubernetes.io/projected/ad569c0f-e4f1-42f2-aa2b-48f0a4f16a57-kube-api-access-s4gtr\") pod \"network-check-target-f7hng\" (UID: \"ad569c0f-e4f1-42f2-aa2b-48f0a4f16a57\") " pod="openshift-network-diagnostics/network-check-target-f7hng"
Sep 11 03:40:18.109400 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:40:18.109379    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 03:40:18.109477 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:40:18.109402    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 03:40:18.109477 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:40:18.109415    2580 projected.go:196] Error preparing data for projected volume kube-api-access-s4gtr for pod openshift-network-diagnostics/network-check-target-f7hng: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 03:40:18.109477 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:40:18.109470    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ad569c0f-e4f1-42f2-aa2b-48f0a4f16a57-kube-api-access-s4gtr podName:ad569c0f-e4f1-42f2-aa2b-48f0a4f16a57 nodeName:}" failed. No retries permitted until 2026-09-11 03:40:22.109452525 +0000 UTC m=+10.267938830 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-s4gtr" (UniqueName: "kubernetes.io/projected/ad569c0f-e4f1-42f2-aa2b-48f0a4f16a57-kube-api-access-s4gtr") pod "network-check-target-f7hng" (UID: "ad569c0f-e4f1-42f2-aa2b-48f0a4f16a57") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 03:40:18.458925 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:18.458892    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qzhz6"
Sep 11 03:40:18.459364 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:40:18.459038    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qzhz6" podUID="eb00790b-1a90-4786-8b81-cbfc706f9c46"
Sep 11 03:40:19.458394 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:19.458300    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f7hng"
Sep 11 03:40:19.458571 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:40:19.458434    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f7hng" podUID="ad569c0f-e4f1-42f2-aa2b-48f0a4f16a57"
Sep 11 03:40:19.458571 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:19.458523    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8knjw"
Sep 11 03:40:19.458825 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:40:19.458638    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8knjw" podUID="f6fa8b71-4ef9-4c91-b7ee-7ef235532ea0"
Sep 11 03:40:20.459299 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:20.459265    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qzhz6"
Sep 11 03:40:20.459763 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:40:20.459421    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qzhz6" podUID="eb00790b-1a90-4786-8b81-cbfc706f9c46"
Sep 11 03:40:21.458492 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:21.458462    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f7hng"
Sep 11 03:40:21.458690 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:21.458499    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8knjw"
Sep 11 03:40:21.458690 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:40:21.458589    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f7hng" podUID="ad569c0f-e4f1-42f2-aa2b-48f0a4f16a57"
Sep 11 03:40:21.458877 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:40:21.458728    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8knjw" podUID="f6fa8b71-4ef9-4c91-b7ee-7ef235532ea0"
Sep 11 03:40:21.942492 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:21.942451    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb00790b-1a90-4786-8b81-cbfc706f9c46-metrics-certs\") pod \"network-metrics-daemon-qzhz6\" (UID: \"eb00790b-1a90-4786-8b81-cbfc706f9c46\") " pod="openshift-multus/network-metrics-daemon-qzhz6"
Sep 11 03:40:21.942958 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:21.942508    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f6fa8b71-4ef9-4c91-b7ee-7ef235532ea0-original-pull-secret\") pod \"global-pull-secret-syncer-8knjw\" (UID: \"f6fa8b71-4ef9-4c91-b7ee-7ef235532ea0\") " pod="kube-system/global-pull-secret-syncer-8knjw"
Sep 11 03:40:21.942958 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:40:21.942640    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 03:40:21.942958 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:40:21.942709    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f6fa8b71-4ef9-4c91-b7ee-7ef235532ea0-original-pull-secret podName:f6fa8b71-4ef9-4c91-b7ee-7ef235532ea0 nodeName:}" failed. No retries permitted until 2026-09-11 03:40:29.942690925 +0000 UTC m=+18.101177228 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f6fa8b71-4ef9-4c91-b7ee-7ef235532ea0-original-pull-secret") pod "global-pull-secret-syncer-8knjw" (UID: "f6fa8b71-4ef9-4c91-b7ee-7ef235532ea0") : object "kube-system"/"original-pull-secret" not registered
Sep 11 03:40:21.943132 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:40:21.943076    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 03:40:21.943183 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:40:21.943136    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb00790b-1a90-4786-8b81-cbfc706f9c46-metrics-certs podName:eb00790b-1a90-4786-8b81-cbfc706f9c46 nodeName:}" failed. No retries permitted until 2026-09-11 03:40:29.943118269 +0000 UTC m=+18.101604574 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/eb00790b-1a90-4786-8b81-cbfc706f9c46-metrics-certs") pod "network-metrics-daemon-qzhz6" (UID: "eb00790b-1a90-4786-8b81-cbfc706f9c46") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 03:40:22.145001 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:22.144960    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4gtr\" (UniqueName: \"kubernetes.io/projected/ad569c0f-e4f1-42f2-aa2b-48f0a4f16a57-kube-api-access-s4gtr\") pod \"network-check-target-f7hng\" (UID: \"ad569c0f-e4f1-42f2-aa2b-48f0a4f16a57\") " pod="openshift-network-diagnostics/network-check-target-f7hng"
Sep 11 03:40:22.145184 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:40:22.145120    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 03:40:22.145184 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:40:22.145142    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 03:40:22.145184 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:40:22.145160    2580 projected.go:196] Error preparing data for projected volume kube-api-access-s4gtr for pod openshift-network-diagnostics/network-check-target-f7hng: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 03:40:22.145339 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:40:22.145220    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ad569c0f-e4f1-42f2-aa2b-48f0a4f16a57-kube-api-access-s4gtr podName:ad569c0f-e4f1-42f2-aa2b-48f0a4f16a57 nodeName:}" failed. No retries permitted until 2026-09-11 03:40:30.145199858 +0000 UTC m=+18.303686182 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-s4gtr" (UniqueName: "kubernetes.io/projected/ad569c0f-e4f1-42f2-aa2b-48f0a4f16a57-kube-api-access-s4gtr") pod "network-check-target-f7hng" (UID: "ad569c0f-e4f1-42f2-aa2b-48f0a4f16a57") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 03:40:22.461382 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:22.461337    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qzhz6"
Sep 11 03:40:22.461542 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:40:22.461511    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qzhz6" podUID="eb00790b-1a90-4786-8b81-cbfc706f9c46"
Sep 11 03:40:23.458236 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:23.458123    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8knjw"
Sep 11 03:40:23.458709 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:40:23.458248    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8knjw" podUID="f6fa8b71-4ef9-4c91-b7ee-7ef235532ea0"
Sep 11 03:40:23.458709 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:23.458283    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f7hng"
Sep 11 03:40:23.458709 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:40:23.458387    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f7hng" podUID="ad569c0f-e4f1-42f2-aa2b-48f0a4f16a57"
Sep 11 03:40:24.458521 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:24.458487    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qzhz6"
Sep 11 03:40:24.458890 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:40:24.458599    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qzhz6" podUID="eb00790b-1a90-4786-8b81-cbfc706f9c46"
Sep 11 03:40:25.458195 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:25.458165    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8knjw"
Sep 11 03:40:25.458365 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:25.458169    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f7hng"
Sep 11 03:40:25.458365 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:40:25.458264    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8knjw" podUID="f6fa8b71-4ef9-4c91-b7ee-7ef235532ea0"
Sep 11 03:40:25.458439 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:40:25.458373    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f7hng" podUID="ad569c0f-e4f1-42f2-aa2b-48f0a4f16a57"
Sep 11 03:40:26.460854 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:26.460824    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qzhz6"
Sep 11 03:40:26.461309 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:40:26.460954    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qzhz6" podUID="eb00790b-1a90-4786-8b81-cbfc706f9c46"
Sep 11 03:40:27.458341 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:27.458297    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8knjw"
Sep 11 03:40:27.458512 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:27.458318    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f7hng"
Sep 11 03:40:27.458512 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:40:27.458420    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8knjw" podUID="f6fa8b71-4ef9-4c91-b7ee-7ef235532ea0"
Sep 11 03:40:27.458512 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:40:27.458480    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f7hng" podUID="ad569c0f-e4f1-42f2-aa2b-48f0a4f16a57"
Sep 11 03:40:28.457701 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:28.457670    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qzhz6"
Sep 11 03:40:28.458119 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:40:28.457790    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qzhz6" podUID="eb00790b-1a90-4786-8b81-cbfc706f9c46"
Sep 11 03:40:29.457615 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:29.457576    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f7hng"
Sep 11 03:40:29.457822 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:29.457575    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8knjw"
Sep 11 03:40:29.457822 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:40:29.457723    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f7hng" podUID="ad569c0f-e4f1-42f2-aa2b-48f0a4f16a57"
Sep 11 03:40:29.457822 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:40:29.457749    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8knjw" podUID="f6fa8b71-4ef9-4c91-b7ee-7ef235532ea0"
Sep 11 03:40:30.005786 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:30.005745    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb00790b-1a90-4786-8b81-cbfc706f9c46-metrics-certs\") pod \"network-metrics-daemon-qzhz6\" (UID: \"eb00790b-1a90-4786-8b81-cbfc706f9c46\") " pod="openshift-multus/network-metrics-daemon-qzhz6"
Sep 11 03:40:30.005973 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:30.005810    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f6fa8b71-4ef9-4c91-b7ee-7ef235532ea0-original-pull-secret\") pod \"global-pull-secret-syncer-8knjw\" (UID: \"f6fa8b71-4ef9-4c91-b7ee-7ef235532ea0\") " pod="kube-system/global-pull-secret-syncer-8knjw"
Sep 11 03:40:30.005973 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:40:30.005862    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 03:40:30.005973 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:40:30.005900    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 03:40:30.005973 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:40:30.005935    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb00790b-1a90-4786-8b81-cbfc706f9c46-metrics-certs podName:eb00790b-1a90-4786-8b81-cbfc706f9c46 nodeName:}" failed. No retries permitted until 2026-09-11 03:40:46.005914278 +0000 UTC m=+34.164400594 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/eb00790b-1a90-4786-8b81-cbfc706f9c46-metrics-certs") pod "network-metrics-daemon-qzhz6" (UID: "eb00790b-1a90-4786-8b81-cbfc706f9c46") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 03:40:30.005973 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:40:30.005953    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f6fa8b71-4ef9-4c91-b7ee-7ef235532ea0-original-pull-secret podName:f6fa8b71-4ef9-4c91-b7ee-7ef235532ea0 nodeName:}" failed. No retries permitted until 2026-09-11 03:40:46.005944169 +0000 UTC m=+34.164430470 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f6fa8b71-4ef9-4c91-b7ee-7ef235532ea0-original-pull-secret") pod "global-pull-secret-syncer-8knjw" (UID: "f6fa8b71-4ef9-4c91-b7ee-7ef235532ea0") : object "kube-system"/"original-pull-secret" not registered
Sep 11 03:40:30.208226 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:30.208186    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4gtr\" (UniqueName: \"kubernetes.io/projected/ad569c0f-e4f1-42f2-aa2b-48f0a4f16a57-kube-api-access-s4gtr\") pod \"network-check-target-f7hng\" (UID: \"ad569c0f-e4f1-42f2-aa2b-48f0a4f16a57\") " pod="openshift-network-diagnostics/network-check-target-f7hng"
Sep 11 03:40:30.208412 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:40:30.208367    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 03:40:30.208412 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:40:30.208395    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 03:40:30.208412 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:40:30.208406    2580 projected.go:196] Error preparing data for projected volume kube-api-access-s4gtr for pod openshift-network-diagnostics/network-check-target-f7hng: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 03:40:30.208555 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:40:30.208471    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ad569c0f-e4f1-42f2-aa2b-48f0a4f16a57-kube-api-access-s4gtr podName:ad569c0f-e4f1-42f2-aa2b-48f0a4f16a57 nodeName:}" failed. No retries permitted until 2026-09-11 03:40:46.208452717 +0000 UTC m=+34.366939035 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-s4gtr" (UniqueName: "kubernetes.io/projected/ad569c0f-e4f1-42f2-aa2b-48f0a4f16a57-kube-api-access-s4gtr") pod "network-check-target-f7hng" (UID: "ad569c0f-e4f1-42f2-aa2b-48f0a4f16a57") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 03:40:30.457942 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:30.457907    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qzhz6"
Sep 11 03:40:30.458386 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:40:30.458050    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qzhz6" podUID="eb00790b-1a90-4786-8b81-cbfc706f9c46"
Sep 11 03:40:31.458516 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:31.458479    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f7hng"
Sep 11 03:40:31.458975 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:40:31.458612    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f7hng" podUID="ad569c0f-e4f1-42f2-aa2b-48f0a4f16a57"
Sep 11 03:40:31.458975 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:31.458685    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8knjw"
Sep 11 03:40:31.458975 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:40:31.458811    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8knjw" podUID="f6fa8b71-4ef9-4c91-b7ee-7ef235532ea0"
Sep 11 03:40:32.458365 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:32.458332    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qzhz6"
Sep 11 03:40:32.458545 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:40:32.458446    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qzhz6" podUID="eb00790b-1a90-4786-8b81-cbfc706f9c46"
Sep 11 03:40:33.457860 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:33.457696    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8knjw"
Sep 11 03:40:33.457964 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:33.457732    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f7hng"
Sep 11 03:40:33.457964 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:40:33.457946    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8knjw" podUID="f6fa8b71-4ef9-4c91-b7ee-7ef235532ea0"
Sep 11 03:40:33.458025 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:40:33.458005    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f7hng" podUID="ad569c0f-e4f1-42f2-aa2b-48f0a4f16a57"
Sep 11 03:40:33.559384 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:33.559358    2580 generic.go:356] "Generic (PLEG): container finished" podID="a91c03cc-9ab6-45df-a99a-3f197542c920" containerID="a94aeac558f9cda8d388f02abac55f73c4e88f3268d543f1de4d8fc9f9bddd32" exitCode=0
Sep 11 03:40:33.560216 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:33.559402    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tr4l2" event={"ID":"a91c03cc-9ab6-45df-a99a-3f197542c920","Type":"ContainerDied","Data":"a94aeac558f9cda8d388f02abac55f73c4e88f3268d543f1de4d8fc9f9bddd32"}
Sep 11 03:40:33.562614 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:33.562596    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j4pgw_28b30dae-a2d3-4251-99e8-04bf5153015b/ovn-acl-logging/0.log"
Sep 11 03:40:33.563025 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:33.563000    2580 generic.go:356] "Generic (PLEG): container finished" podID="28b30dae-a2d3-4251-99e8-04bf5153015b" containerID="efa87e42d4caceb23eb287e2a86eee0e6af86b66ab5ab58a5407b73e4d4d2b21" exitCode=1
Sep 11 03:40:33.563096 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:33.563063    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j4pgw" event={"ID":"28b30dae-a2d3-4251-99e8-04bf5153015b","Type":"ContainerStarted","Data":"1ed715af0de795859a1bbf4d4c9965d13feee4a674712cd72ae20de2f625947a"}
Sep 11 03:40:33.563191 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:33.563136    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j4pgw" event={"ID":"28b30dae-a2d3-4251-99e8-04bf5153015b","Type":"ContainerStarted","Data":"3a270644f6679c16a9cf21210dfa4e3f3d21049421ef9ab4d0f7f1fd3dad0c8e"}
Sep 11 03:40:33.563224 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:33.563190    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j4pgw" event={"ID":"28b30dae-a2d3-4251-99e8-04bf5153015b","Type":"ContainerStarted","Data":"528119abd6db28640b3efc08cd1f0e3a58c555dc4816b3e8f32489f8b1502ec1"}
Sep 11 03:40:33.563255 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:33.563241    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j4pgw" event={"ID":"28b30dae-a2d3-4251-99e8-04bf5153015b","Type":"ContainerDied","Data":"efa87e42d4caceb23eb287e2a86eee0e6af86b66ab5ab58a5407b73e4d4d2b21"}
Sep 11 03:40:33.563306 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:33.563292    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j4pgw" event={"ID":"28b30dae-a2d3-4251-99e8-04bf5153015b","Type":"ContainerStarted","Data":"22a44ab092cb646769002f082ff5fd3cfbf307415206f9ad1c12aedafb3dad74"}
Sep 11 03:40:33.564438 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:33.564418    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-wkqmr" event={"ID":"896af27b-fde3-43f0-bdbb-5bdf0815873a","Type":"ContainerStarted","Data":"a8abf1d1f15daaf4ac9057029a0512f5a9289c644a0358e329bdf2173f9e35be"}
Sep 11 03:40:33.565740 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:33.565720    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-tm9tp" event={"ID":"a7954230-378f-4cf1-b574-3c3b9f0ec4fb","Type":"ContainerStarted","Data":"171d391c94e8f35e5e7be341824621635b70407033df7f0e3009ee0063f1296d"}
Sep 11 03:40:33.567184 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:33.567162    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-5gs25" event={"ID":"4e99df15-f68d-4668-a7bd-7f1da9aa9792","Type":"ContainerStarted","Data":"5664e9796ec0e4c2923672bc8521ecaa827ec482cd3ce5dd7034097ef0f5ee25"}
Sep 11 03:40:33.568553 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:33.568533    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l42ph" event={"ID":"aae093a3-d3d4-4291-947e-3a59a9b4c84d","Type":"ContainerStarted","Data":"8ed6716603c7e079c9d79f951538650ef84fe2defd714ad9d5723dad5b74bae0"}
Sep 11 03:40:33.569829 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:33.569810    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-d9t58" event={"ID":"bc0076f9-9364-41f4-aadc-7e4cef25221a","Type":"ContainerStarted","Data":"7a4b8e10f6d7e21e98505e52ba95d56a56b4bb0b11a63776d024fd39866e2835"}
Sep 11 03:40:33.571027 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:33.571011    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-mlcbg" event={"ID":"84aade61-b056-43d4-bbcd-fd68eba2cb48","Type":"ContainerStarted","Data":"f1f300b43c02ef84be198a5dd5c4c25a845a140ddb988f9126e4ff992c40791f"}
Sep 11 03:40:33.593901 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:33.593853    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-tm9tp" podStartSLOduration=7.623756504 podStartE2EDuration="21.593835508s" totalImagesPullingTime="13.970079004s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 03:40:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:40:33.593609962 +0000 UTC m=+21.752096287" watchObservedRunningTime="2026-09-11 03:40:33.593835508 +0000 UTC m=+21.752321833"
Sep 11 03:40:33.607408 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:33.607368    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-mlcbg" podStartSLOduration=3.840584197 podStartE2EDuration="21.607357433s" totalImagesPullingTime="17.766773236s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 03:40:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:40:33.606972063 +0000 UTC m=+21.765458387" watchObservedRunningTime="2026-09-11 03:40:33.607357433 +0000 UTC m=+21.765843757"
Sep 11 03:40:33.625676 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:33.625619    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-d9t58" podStartSLOduration=3.7261931710000002 podStartE2EDuration="21.625605133s" totalImagesPullingTime="17.899411962s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 03:40:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:40:33.625084964 +0000 UTC m=+21.783571288" watchObservedRunningTime="2026-09-11 03:40:33.625605133 +0000 UTC m=+21.784091456"
Sep 11 03:40:33.640206 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:33.639993    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-5gs25" podStartSLOduration=3.882244599 podStartE2EDuration="21.639976305s" totalImagesPullingTime="17.757731706s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 03:40:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:40:33.639586181 +0000 UTC m=+21.798072505" watchObservedRunningTime="2026-09-11 03:40:33.639976305 +0000 UTC m=+21.798462628"
Sep 11 03:40:33.652790 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:33.652745    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-wkqmr" podStartSLOduration=3.893440957 podStartE2EDuration="21.652723988s" totalImagesPullingTime="17.759283031s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 03:40:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:40:33.652023111 +0000 UTC m=+21.810509446" watchObservedRunningTime="2026-09-11 03:40:33.652723988 +0000 UTC m=+21.811210313"
Sep 11 03:40:34.049879 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:34.049849    2580 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 11 03:40:34.332992 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:34.332858    2580 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-11T03:40:34.04987451Z","UUID":"83598c58-0627-4897-b6e5-4db09c6f3f00","Handler":null,"Name":"","Endpoint":""}
Sep 11 03:40:34.334525 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:34.334502    2580 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 11 03:40:34.334655 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:34.334536    2580 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 11 03:40:34.461971 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:34.461948    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qzhz6"
Sep 11 03:40:34.462084 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:40:34.462058    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qzhz6" podUID="eb00790b-1a90-4786-8b81-cbfc706f9c46"
Sep 11 03:40:34.576744 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:34.576721    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j4pgw_28b30dae-a2d3-4251-99e8-04bf5153015b/ovn-acl-logging/0.log"
Sep 11 03:40:34.577200 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:34.577168    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j4pgw" event={"ID":"28b30dae-a2d3-4251-99e8-04bf5153015b","Type":"ContainerStarted","Data":"c0f3d4c040a51508a439e8309ff12ebde3f83dc7e46ef0d8b5d22f0a5ba49598"}
Sep 11 03:40:34.578889 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:34.578696    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-gqqk2" event={"ID":"cc4dd8ce-127a-4e14-9949-0b7075d1df37","Type":"ContainerStarted","Data":"3c842975ca46b12e72cc11fece93e7ac4295c58c9e9029862cd9911da002afbb"}
Sep 11 03:40:34.580797 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:34.580765    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l42ph" event={"ID":"aae093a3-d3d4-4291-947e-3a59a9b4c84d","Type":"ContainerStarted","Data":"dd1f8fa080a415ae114833e297c9dae7caeb8278bb9d90191ca406276aa36854"}
Sep 11 03:40:34.592743 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:34.592654    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-gqqk2" podStartSLOduration=4.820256601 podStartE2EDuration="22.592640197s" totalImagesPullingTime="17.772383596s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 03:40:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:40:34.592553012 +0000 UTC m=+22.751039335" watchObservedRunningTime="2026-09-11 03:40:34.592640197 +0000 UTC m=+22.751126526"
Sep 11 03:40:35.457875 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:35.457847    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8knjw"
Sep 11 03:40:35.457875 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:35.457880    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f7hng"
Sep 11 03:40:35.458122 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:40:35.457955    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8knjw" podUID="f6fa8b71-4ef9-4c91-b7ee-7ef235532ea0"
Sep 11 03:40:35.458122 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:40:35.458043    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f7hng" podUID="ad569c0f-e4f1-42f2-aa2b-48f0a4f16a57"
Sep 11 03:40:35.585762 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:35.585723    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l42ph" event={"ID":"aae093a3-d3d4-4291-947e-3a59a9b4c84d","Type":"ContainerStarted","Data":"55e0965dc202c8789788a64f5d56716bc96af13ee8815592dc4b8eece43903e1"}
Sep 11 03:40:35.603403 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:35.603356    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l42ph" podStartSLOduration=4.077200235 podStartE2EDuration="23.603336694s" totalImagesPullingTime="19.526136459s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 03:40:12 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:40:35.602221725 +0000 UTC m=+23.760708048" watchObservedRunningTime="2026-09-11 03:40:35.603336694 +0000 UTC m=+23.761823016"
Sep 11 03:40:35.959978 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:35.959949    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-tm9tp"
Sep 11 03:40:35.960552 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:35.960535    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-tm9tp"
Sep 11 03:40:36.457887 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:36.457852    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qzhz6"
Sep 11 03:40:36.458051 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:40:36.458000    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qzhz6" podUID="eb00790b-1a90-4786-8b81-cbfc706f9c46"
Sep 11 03:40:36.591642 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:36.591610    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j4pgw_28b30dae-a2d3-4251-99e8-04bf5153015b/ovn-acl-logging/0.log"
Sep 11 03:40:36.592089 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:36.592070    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j4pgw" event={"ID":"28b30dae-a2d3-4251-99e8-04bf5153015b","Type":"ContainerStarted","Data":"24b858c99f29b5feb9cc7d2ba598fde57778bd597d275d7055c5651518c9d43b"}
Sep 11 03:40:36.592352 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:36.592330    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-tm9tp"
Sep 11 03:40:36.592928 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:36.592909    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-tm9tp"
Sep 11 03:40:37.458183 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:37.458148    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f7hng"
Sep 11 03:40:37.458350 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:37.458159    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8knjw"
Sep 11 03:40:37.458350 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:40:37.458246    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f7hng" podUID="ad569c0f-e4f1-42f2-aa2b-48f0a4f16a57"
Sep 11 03:40:37.458350 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:40:37.458323    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8knjw" podUID="f6fa8b71-4ef9-4c91-b7ee-7ef235532ea0"
Sep 11 03:40:38.458545 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:38.458369    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qzhz6"
Sep 11 03:40:38.459106 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:40:38.458628    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qzhz6" podUID="eb00790b-1a90-4786-8b81-cbfc706f9c46"
Sep 11 03:40:38.598142 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:38.598115    2580 generic.go:356] "Generic (PLEG): container finished" podID="a91c03cc-9ab6-45df-a99a-3f197542c920" containerID="496a3e2ae4ad2f1480e901ed77f3f6a368545ba1123ca020069e8234d14e07f5" exitCode=0
Sep 11 03:40:38.598320 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:38.598183    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tr4l2" event={"ID":"a91c03cc-9ab6-45df-a99a-3f197542c920","Type":"ContainerDied","Data":"496a3e2ae4ad2f1480e901ed77f3f6a368545ba1123ca020069e8234d14e07f5"}
Sep 11 03:40:38.601975 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:38.601929    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j4pgw_28b30dae-a2d3-4251-99e8-04bf5153015b/ovn-acl-logging/0.log"
Sep 11 03:40:38.605682 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:38.605636    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j4pgw" event={"ID":"28b30dae-a2d3-4251-99e8-04bf5153015b","Type":"ContainerStarted","Data":"4775717c4e92a2774fdde8fe9a9bc35558156894d262714d1e84bbb91d2a0d1c"}
Sep 11 03:40:38.606016 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:38.605997    2580 scope.go:117] "RemoveContainer" containerID="efa87e42d4caceb23eb287e2a86eee0e6af86b66ab5ab58a5407b73e4d4d2b21"
Sep 11 03:40:39.458530 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:39.458316    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8knjw"
Sep 11 03:40:39.458651 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:39.458352    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f7hng"
Sep 11 03:40:39.458651 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:40:39.458635    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8knjw" podUID="f6fa8b71-4ef9-4c91-b7ee-7ef235532ea0"
Sep 11 03:40:39.459177 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:40:39.458686    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f7hng" podUID="ad569c0f-e4f1-42f2-aa2b-48f0a4f16a57"
Sep 11 03:40:39.609748 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:39.609629    2580 generic.go:356] "Generic (PLEG): container finished" podID="a91c03cc-9ab6-45df-a99a-3f197542c920" containerID="91f0449f253b1e715feadc76d22667c3a17339966be6e36a6402f09be9dcf5a1" exitCode=0
Sep 11 03:40:39.609748 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:39.609716    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tr4l2" event={"ID":"a91c03cc-9ab6-45df-a99a-3f197542c920","Type":"ContainerDied","Data":"91f0449f253b1e715feadc76d22667c3a17339966be6e36a6402f09be9dcf5a1"}
Sep 11 03:40:39.613626 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:39.613609    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j4pgw_28b30dae-a2d3-4251-99e8-04bf5153015b/ovn-acl-logging/0.log"
Sep 11 03:40:39.613992 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:39.613972    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j4pgw" event={"ID":"28b30dae-a2d3-4251-99e8-04bf5153015b","Type":"ContainerStarted","Data":"e549d63af1ab7dbead8c4e2041eb1628b7f472024ce50ebd4d32882fb763a7e9"}
Sep 11 03:40:39.614304 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:39.614288    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-j4pgw"
Sep 11 03:40:39.614403 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:39.614352    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-j4pgw"
Sep 11 03:40:39.614403 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:39.614392    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-j4pgw"
Sep 11 03:40:39.629273 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:39.629252    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-j4pgw"
Sep 11 03:40:39.629375 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:39.629348    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-j4pgw"
Sep 11 03:40:39.664376 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:39.664329    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-j4pgw" podStartSLOduration=9.859012364 podStartE2EDuration="27.664312889s" totalImagesPullingTime="17.805300525s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 03:40:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:40:39.663613105 +0000 UTC m=+27.822099427" watchObservedRunningTime="2026-09-11 03:40:39.664312889 +0000 UTC m=+27.822799212"
Sep 11 03:40:39.703381 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:39.703352    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-8knjw"]
Sep 11 03:40:39.703502 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:39.703467    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8knjw"
Sep 11 03:40:39.703599 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:40:39.703578    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8knjw" podUID="f6fa8b71-4ef9-4c91-b7ee-7ef235532ea0"
Sep 11 03:40:39.706570 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:39.706548    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-f7hng"]
Sep 11 03:40:39.706687 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:39.706634    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f7hng"
Sep 11 03:40:39.706743 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:40:39.706726    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f7hng" podUID="ad569c0f-e4f1-42f2-aa2b-48f0a4f16a57"
Sep 11 03:40:39.707101 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:39.707080    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-qzhz6"]
Sep 11 03:40:39.707205 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:39.707175    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qzhz6"
Sep 11 03:40:39.707276 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:40:39.707257    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qzhz6" podUID="eb00790b-1a90-4786-8b81-cbfc706f9c46"
Sep 11 03:40:40.618710 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:40.618601    2580 generic.go:356] "Generic (PLEG): container finished" podID="a91c03cc-9ab6-45df-a99a-3f197542c920" containerID="6c75f5f895d012a9384cead7d33d0ec2b61d921a19e1aef2f75e63038abd7624" exitCode=0
Sep 11 03:40:40.619133 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:40.618720    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tr4l2" event={"ID":"a91c03cc-9ab6-45df-a99a-3f197542c920","Type":"ContainerDied","Data":"6c75f5f895d012a9384cead7d33d0ec2b61d921a19e1aef2f75e63038abd7624"}
Sep 11 03:40:41.458171 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:41.458140    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8knjw"
Sep 11 03:40:41.458359 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:41.458262    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f7hng"
Sep 11 03:40:41.458359 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:40:41.458268    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8knjw" podUID="f6fa8b71-4ef9-4c91-b7ee-7ef235532ea0"
Sep 11 03:40:41.458473 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:40:41.458372    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f7hng" podUID="ad569c0f-e4f1-42f2-aa2b-48f0a4f16a57"
Sep 11 03:40:41.458473 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:41.458272    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qzhz6"
Sep 11 03:40:41.458567 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:40:41.458513    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qzhz6" podUID="eb00790b-1a90-4786-8b81-cbfc706f9c46"
Sep 11 03:40:43.458068 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:43.458030    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f7hng"
Sep 11 03:40:43.458489 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:43.458161    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qzhz6"
Sep 11 03:40:43.458489 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:43.458176    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8knjw"
Sep 11 03:40:43.458489 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:40:43.458164    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f7hng" podUID="ad569c0f-e4f1-42f2-aa2b-48f0a4f16a57"
Sep 11 03:40:43.458489 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:40:43.458293    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qzhz6" podUID="eb00790b-1a90-4786-8b81-cbfc706f9c46"
Sep 11 03:40:43.458489 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:40:43.458365    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8knjw" podUID="f6fa8b71-4ef9-4c91-b7ee-7ef235532ea0"
Sep 11 03:40:45.458030 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:45.457847    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f7hng"
Sep 11 03:40:45.458417 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:45.457903    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8knjw"
Sep 11 03:40:45.458417 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:40:45.458099    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f7hng" podUID="ad569c0f-e4f1-42f2-aa2b-48f0a4f16a57"
Sep 11 03:40:45.458417 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:45.457906    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qzhz6"
Sep 11 03:40:45.458417 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:40:45.458182    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8knjw" podUID="f6fa8b71-4ef9-4c91-b7ee-7ef235532ea0"
Sep 11 03:40:45.458417 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:40:45.458268    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qzhz6" podUID="eb00790b-1a90-4786-8b81-cbfc706f9c46"
Sep 11 03:40:45.616252 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:45.616232    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-121.ec2.internal" event="NodeReady"
Sep 11 03:40:45.616361 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:45.616351    2580 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 11 03:40:45.633685 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:45.633642    2580 generic.go:356] "Generic (PLEG): container finished" podID="a91c03cc-9ab6-45df-a99a-3f197542c920" containerID="e5e995bb3cfce9bd52028cca7938404608bddc722a2a19bbef12268ea05f5280" exitCode=0
Sep 11 03:40:45.633835 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:45.633697    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tr4l2" event={"ID":"a91c03cc-9ab6-45df-a99a-3f197542c920","Type":"ContainerDied","Data":"e5e995bb3cfce9bd52028cca7938404608bddc722a2a19bbef12268ea05f5280"}
Sep 11 03:40:45.662224 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:45.662198    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-f5d7k"]
Sep 11 03:40:45.679391 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:45.679339    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-pwk5f"]
Sep 11 03:40:45.680110 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:45.679651    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-f5d7k"
Sep 11 03:40:45.682320 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:45.682295    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 11 03:40:45.682585 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:45.682295    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 11 03:40:45.682652 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:45.682614    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-qd2rt\""
Sep 11 03:40:45.682825 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:45.682803    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 11 03:40:45.690912 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:45.690889    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-f5d7k"]
Sep 11 03:40:45.691024 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:45.690922    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-pwk5f"
Sep 11 03:40:45.691024 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:45.690953    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-pwk5f"]
Sep 11 03:40:45.693175 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:45.693090    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-qlqn4\""
Sep 11 03:40:45.693270 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:45.693090    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 11 03:40:45.693270 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:45.693215    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 11 03:40:45.813979 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:45.813948    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/46e75c33-310a-420f-b9c9-f92bbe261dd7-cert\") pod \"ingress-canary-f5d7k\" (UID: \"46e75c33-310a-420f-b9c9-f92bbe261dd7\") " pod="openshift-ingress-canary/ingress-canary-f5d7k"
Sep 11 03:40:45.814089 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:45.813996    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/61f124d5-c24b-4150-b19a-ae500bce32fe-metrics-tls\") pod \"dns-default-pwk5f\" (UID: \"61f124d5-c24b-4150-b19a-ae500bce32fe\") " pod="openshift-dns/dns-default-pwk5f"
Sep 11 03:40:45.814089 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:45.814024    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nx5cm\" (UniqueName: \"kubernetes.io/projected/61f124d5-c24b-4150-b19a-ae500bce32fe-kube-api-access-nx5cm\") pod \"dns-default-pwk5f\" (UID: \"61f124d5-c24b-4150-b19a-ae500bce32fe\") " pod="openshift-dns/dns-default-pwk5f"
Sep 11 03:40:45.814089 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:45.814067    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6t7r7\" (UniqueName: \"kubernetes.io/projected/46e75c33-310a-420f-b9c9-f92bbe261dd7-kube-api-access-6t7r7\") pod \"ingress-canary-f5d7k\" (UID: \"46e75c33-310a-420f-b9c9-f92bbe261dd7\") " pod="openshift-ingress-canary/ingress-canary-f5d7k"
Sep 11 03:40:45.814089 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:45.814081    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/61f124d5-c24b-4150-b19a-ae500bce32fe-config-volume\") pod \"dns-default-pwk5f\" (UID: \"61f124d5-c24b-4150-b19a-ae500bce32fe\") " pod="openshift-dns/dns-default-pwk5f"
Sep 11 03:40:45.814208 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:45.814094    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61f124d5-c24b-4150-b19a-ae500bce32fe-tmp-dir\") pod \"dns-default-pwk5f\" (UID: \"61f124d5-c24b-4150-b19a-ae500bce32fe\") " pod="openshift-dns/dns-default-pwk5f"
Sep 11 03:40:45.915274 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:45.915242    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/46e75c33-310a-420f-b9c9-f92bbe261dd7-cert\") pod \"ingress-canary-f5d7k\" (UID: \"46e75c33-310a-420f-b9c9-f92bbe261dd7\") " pod="openshift-ingress-canary/ingress-canary-f5d7k"
Sep 11 03:40:45.915403 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:45.915293    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/61f124d5-c24b-4150-b19a-ae500bce32fe-metrics-tls\") pod \"dns-default-pwk5f\" (UID: \"61f124d5-c24b-4150-b19a-ae500bce32fe\") " pod="openshift-dns/dns-default-pwk5f"
Sep 11 03:40:45.915403 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:40:45.915383    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 03:40:45.915468 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:45.915414    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nx5cm\" (UniqueName: \"kubernetes.io/projected/61f124d5-c24b-4150-b19a-ae500bce32fe-kube-api-access-nx5cm\") pod \"dns-default-pwk5f\" (UID: \"61f124d5-c24b-4150-b19a-ae500bce32fe\") " pod="openshift-dns/dns-default-pwk5f"
Sep 11 03:40:45.915468 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:40:45.915438    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61f124d5-c24b-4150-b19a-ae500bce32fe-metrics-tls podName:61f124d5-c24b-4150-b19a-ae500bce32fe nodeName:}" failed. No retries permitted until 2026-09-11 03:40:46.415420074 +0000 UTC m=+34.573906393 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/61f124d5-c24b-4150-b19a-ae500bce32fe-metrics-tls") pod "dns-default-pwk5f" (UID: "61f124d5-c24b-4150-b19a-ae500bce32fe") : secret "dns-default-metrics-tls" not found
Sep 11 03:40:45.915544 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:45.915475    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6t7r7\" (UniqueName: \"kubernetes.io/projected/46e75c33-310a-420f-b9c9-f92bbe261dd7-kube-api-access-6t7r7\") pod \"ingress-canary-f5d7k\" (UID: \"46e75c33-310a-420f-b9c9-f92bbe261dd7\") " pod="openshift-ingress-canary/ingress-canary-f5d7k"
Sep 11 03:40:45.915544 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:45.915491    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/61f124d5-c24b-4150-b19a-ae500bce32fe-config-volume\") pod \"dns-default-pwk5f\" (UID: \"61f124d5-c24b-4150-b19a-ae500bce32fe\") " pod="openshift-dns/dns-default-pwk5f"
Sep 11 03:40:45.915544 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:45.915515    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61f124d5-c24b-4150-b19a-ae500bce32fe-tmp-dir\") pod \"dns-default-pwk5f\" (UID: \"61f124d5-c24b-4150-b19a-ae500bce32fe\") " pod="openshift-dns/dns-default-pwk5f"
Sep 11 03:40:45.915839 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:45.915821    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61f124d5-c24b-4150-b19a-ae500bce32fe-tmp-dir\") pod \"dns-default-pwk5f\" (UID: \"61f124d5-c24b-4150-b19a-ae500bce32fe\") " pod="openshift-dns/dns-default-pwk5f"
Sep 11 03:40:45.916087 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:45.916057    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/61f124d5-c24b-4150-b19a-ae500bce32fe-config-volume\") pod \"dns-default-pwk5f\" (UID: \"61f124d5-c24b-4150-b19a-ae500bce32fe\") " pod="openshift-dns/dns-default-pwk5f"
Sep 11 03:40:45.919478 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:45.919459    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/46e75c33-310a-420f-b9c9-f92bbe261dd7-cert\") pod \"ingress-canary-f5d7k\" (UID: \"46e75c33-310a-420f-b9c9-f92bbe261dd7\") " pod="openshift-ingress-canary/ingress-canary-f5d7k"
Sep 11 03:40:45.923263 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:45.923243    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nx5cm\" (UniqueName: \"kubernetes.io/projected/61f124d5-c24b-4150-b19a-ae500bce32fe-kube-api-access-nx5cm\") pod \"dns-default-pwk5f\" (UID: \"61f124d5-c24b-4150-b19a-ae500bce32fe\") " pod="openshift-dns/dns-default-pwk5f"
Sep 11 03:40:45.923498 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:45.923481    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6t7r7\" (UniqueName: \"kubernetes.io/projected/46e75c33-310a-420f-b9c9-f92bbe261dd7-kube-api-access-6t7r7\") pod \"ingress-canary-f5d7k\" (UID: \"46e75c33-310a-420f-b9c9-f92bbe261dd7\") " pod="openshift-ingress-canary/ingress-canary-f5d7k"
Sep 11 03:40:45.991650 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:45.991580    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-f5d7k"
Sep 11 03:40:46.015792 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:46.015771    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb00790b-1a90-4786-8b81-cbfc706f9c46-metrics-certs\") pod \"network-metrics-daemon-qzhz6\" (UID: \"eb00790b-1a90-4786-8b81-cbfc706f9c46\") " pod="openshift-multus/network-metrics-daemon-qzhz6"
Sep 11 03:40:46.015877 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:46.015818    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f6fa8b71-4ef9-4c91-b7ee-7ef235532ea0-original-pull-secret\") pod \"global-pull-secret-syncer-8knjw\" (UID: \"f6fa8b71-4ef9-4c91-b7ee-7ef235532ea0\") " pod="kube-system/global-pull-secret-syncer-8knjw"
Sep 11 03:40:46.015931 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:40:46.015907    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 03:40:46.015986 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:40:46.015947    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 03:40:46.015986 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:40:46.015968    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb00790b-1a90-4786-8b81-cbfc706f9c46-metrics-certs podName:eb00790b-1a90-4786-8b81-cbfc706f9c46 nodeName:}" failed. No retries permitted until 2026-09-11 03:41:18.015952826 +0000 UTC m=+66.174439126 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/eb00790b-1a90-4786-8b81-cbfc706f9c46-metrics-certs") pod "network-metrics-daemon-qzhz6" (UID: "eb00790b-1a90-4786-8b81-cbfc706f9c46") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 03:40:46.016062 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:40:46.015998    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f6fa8b71-4ef9-4c91-b7ee-7ef235532ea0-original-pull-secret podName:f6fa8b71-4ef9-4c91-b7ee-7ef235532ea0 nodeName:}" failed. No retries permitted until 2026-09-11 03:41:18.015982654 +0000 UTC m=+66.174468970 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f6fa8b71-4ef9-4c91-b7ee-7ef235532ea0-original-pull-secret") pod "global-pull-secret-syncer-8knjw" (UID: "f6fa8b71-4ef9-4c91-b7ee-7ef235532ea0") : object "kube-system"/"original-pull-secret" not registered
Sep 11 03:40:46.217627 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:46.217592    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4gtr\" (UniqueName: \"kubernetes.io/projected/ad569c0f-e4f1-42f2-aa2b-48f0a4f16a57-kube-api-access-s4gtr\") pod \"network-check-target-f7hng\" (UID: \"ad569c0f-e4f1-42f2-aa2b-48f0a4f16a57\") " pod="openshift-network-diagnostics/network-check-target-f7hng"
Sep 11 03:40:46.217804 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:40:46.217732    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 03:40:46.217804 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:40:46.217746    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 03:40:46.217804 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:40:46.217755    2580 projected.go:196] Error preparing data for projected volume kube-api-access-s4gtr for pod openshift-network-diagnostics/network-check-target-f7hng: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 03:40:46.217913 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:40:46.217806    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ad569c0f-e4f1-42f2-aa2b-48f0a4f16a57-kube-api-access-s4gtr podName:ad569c0f-e4f1-42f2-aa2b-48f0a4f16a57 nodeName:}" failed. No retries permitted until 2026-09-11 03:41:18.217792063 +0000 UTC m=+66.376278363 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-s4gtr" (UniqueName: "kubernetes.io/projected/ad569c0f-e4f1-42f2-aa2b-48f0a4f16a57-kube-api-access-s4gtr") pod "network-check-target-f7hng" (UID: "ad569c0f-e4f1-42f2-aa2b-48f0a4f16a57") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 03:40:46.295019 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:46.294948    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-f5d7k"]
Sep 11 03:40:46.298931 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:40:46.298905    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46e75c33_310a_420f_b9c9_f92bbe261dd7.slice/crio-520e3f3d400c3095e15fde75f760726409942e2bb654e362b9f6bf3495361681 WatchSource:0}: Error finding container 520e3f3d400c3095e15fde75f760726409942e2bb654e362b9f6bf3495361681: Status 404 returned error can't find the container with id 520e3f3d400c3095e15fde75f760726409942e2bb654e362b9f6bf3495361681
Sep 11 03:40:46.419289 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:46.419247    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/61f124d5-c24b-4150-b19a-ae500bce32fe-metrics-tls\") pod \"dns-default-pwk5f\" (UID: \"61f124d5-c24b-4150-b19a-ae500bce32fe\") " pod="openshift-dns/dns-default-pwk5f"
Sep 11 03:40:46.419468 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:40:46.419394    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 03:40:46.419468 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:40:46.419453    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61f124d5-c24b-4150-b19a-ae500bce32fe-metrics-tls podName:61f124d5-c24b-4150-b19a-ae500bce32fe nodeName:}" failed. No retries permitted until 2026-09-11 03:40:47.419439358 +0000 UTC m=+35.577925664 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/61f124d5-c24b-4150-b19a-ae500bce32fe-metrics-tls") pod "dns-default-pwk5f" (UID: "61f124d5-c24b-4150-b19a-ae500bce32fe") : secret "dns-default-metrics-tls" not found
Sep 11 03:40:46.637587 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:46.637498    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-f5d7k" event={"ID":"46e75c33-310a-420f-b9c9-f92bbe261dd7","Type":"ContainerStarted","Data":"520e3f3d400c3095e15fde75f760726409942e2bb654e362b9f6bf3495361681"}
Sep 11 03:40:46.640475 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:46.640451    2580 generic.go:356] "Generic (PLEG): container finished" podID="a91c03cc-9ab6-45df-a99a-3f197542c920" containerID="10e5cf6137b4362f1576f38941563a27cc76f15bfddb51aec8e865cdad83b624" exitCode=0
Sep 11 03:40:46.640602 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:46.640514    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tr4l2" event={"ID":"a91c03cc-9ab6-45df-a99a-3f197542c920","Type":"ContainerDied","Data":"10e5cf6137b4362f1576f38941563a27cc76f15bfddb51aec8e865cdad83b624"}
Sep 11 03:40:47.427259 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:47.427069    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/61f124d5-c24b-4150-b19a-ae500bce32fe-metrics-tls\") pod \"dns-default-pwk5f\" (UID: \"61f124d5-c24b-4150-b19a-ae500bce32fe\") " pod="openshift-dns/dns-default-pwk5f"
Sep 11 03:40:47.427536 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:40:47.427218    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 03:40:47.427536 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:40:47.427390    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61f124d5-c24b-4150-b19a-ae500bce32fe-metrics-tls podName:61f124d5-c24b-4150-b19a-ae500bce32fe nodeName:}" failed. No retries permitted until 2026-09-11 03:40:49.427373651 +0000 UTC m=+37.585859956 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/61f124d5-c24b-4150-b19a-ae500bce32fe-metrics-tls") pod "dns-default-pwk5f" (UID: "61f124d5-c24b-4150-b19a-ae500bce32fe") : secret "dns-default-metrics-tls" not found
Sep 11 03:40:47.458400 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:47.458365    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qzhz6"
Sep 11 03:40:47.458558 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:47.458371    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8knjw"
Sep 11 03:40:47.458558 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:47.458500    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f7hng"
Sep 11 03:40:47.461307 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:47.461281    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 11 03:40:47.462072 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:47.462050    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-hqw27\""
Sep 11 03:40:47.462186 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:47.462074    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 11 03:40:47.462186 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:47.462086    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-sfchv\""
Sep 11 03:40:47.462186 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:47.462153    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 11 03:40:47.462186 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:47.462060    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 11 03:40:47.646540 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:47.646507    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tr4l2" event={"ID":"a91c03cc-9ab6-45df-a99a-3f197542c920","Type":"ContainerStarted","Data":"a78df6f4f8a1774a2afff586ee15823caa66a9f344718f2fd918f9fa2339a7f9"}
Sep 11 03:40:47.665286 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:47.665248    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-tr4l2" podStartSLOduration=7.946204028 podStartE2EDuration="35.665234608s" totalImagesPullingTime="27.71903058s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 03:40:12 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:40:47.66492675 +0000 UTC m=+35.823413110" watchObservedRunningTime="2026-09-11 03:40:47.665234608 +0000 UTC m=+35.823720930"
Sep 11 03:40:49.442752 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:49.442715    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/61f124d5-c24b-4150-b19a-ae500bce32fe-metrics-tls\") pod \"dns-default-pwk5f\" (UID: \"61f124d5-c24b-4150-b19a-ae500bce32fe\") " pod="openshift-dns/dns-default-pwk5f"
Sep 11 03:40:49.443108 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:40:49.442833    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 03:40:49.443108 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:40:49.442891    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61f124d5-c24b-4150-b19a-ae500bce32fe-metrics-tls podName:61f124d5-c24b-4150-b19a-ae500bce32fe nodeName:}" failed. No retries permitted until 2026-09-11 03:40:53.44287454 +0000 UTC m=+41.601360852 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/61f124d5-c24b-4150-b19a-ae500bce32fe-metrics-tls") pod "dns-default-pwk5f" (UID: "61f124d5-c24b-4150-b19a-ae500bce32fe") : secret "dns-default-metrics-tls" not found
Sep 11 03:40:49.652293 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:49.652206    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-f5d7k" event={"ID":"46e75c33-310a-420f-b9c9-f92bbe261dd7","Type":"ContainerStarted","Data":"4b524a253ae9b02cfa3c545815c0624b3450e9dce19995507663807e18ce5fad"}
Sep 11 03:40:49.666770 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:49.666722    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-f5d7k" podStartSLOduration=2.180914565 podStartE2EDuration="4.666708117s" totalImagesPullingTime="2.485793552s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 03:40:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:40:49.666559731 +0000 UTC m=+37.825046055" watchObservedRunningTime="2026-09-11 03:40:49.666708117 +0000 UTC m=+37.825194440"
Sep 11 03:40:53.465066 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:40:53.465026    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/61f124d5-c24b-4150-b19a-ae500bce32fe-metrics-tls\") pod \"dns-default-pwk5f\" (UID: \"61f124d5-c24b-4150-b19a-ae500bce32fe\") " pod="openshift-dns/dns-default-pwk5f"
Sep 11 03:40:53.465407 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:40:53.465136    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 03:40:53.465407 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:40:53.465196    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61f124d5-c24b-4150-b19a-ae500bce32fe-metrics-tls podName:61f124d5-c24b-4150-b19a-ae500bce32fe nodeName:}" failed. No retries permitted until 2026-09-11 03:41:01.465183326 +0000 UTC m=+49.623669626 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/61f124d5-c24b-4150-b19a-ae500bce32fe-metrics-tls") pod "dns-default-pwk5f" (UID: "61f124d5-c24b-4150-b19a-ae500bce32fe") : secret "dns-default-metrics-tls" not found
Sep 11 03:41:01.521607 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:01.521567    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/61f124d5-c24b-4150-b19a-ae500bce32fe-metrics-tls\") pod \"dns-default-pwk5f\" (UID: \"61f124d5-c24b-4150-b19a-ae500bce32fe\") " pod="openshift-dns/dns-default-pwk5f"
Sep 11 03:41:01.522341 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:41:01.522240    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 03:41:01.522474 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:41:01.522373    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61f124d5-c24b-4150-b19a-ae500bce32fe-metrics-tls podName:61f124d5-c24b-4150-b19a-ae500bce32fe nodeName:}" failed. No retries permitted until 2026-09-11 03:41:17.522342346 +0000 UTC m=+65.680828661 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/61f124d5-c24b-4150-b19a-ae500bce32fe-metrics-tls") pod "dns-default-pwk5f" (UID: "61f124d5-c24b-4150-b19a-ae500bce32fe") : secret "dns-default-metrics-tls" not found
Sep 11 03:41:04.572527 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:04.572494    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-ks6ld"]
Sep 11 03:41:04.575308 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:04.575290    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-ks6ld"
Sep 11 03:41:04.577916 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:04.577892    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Sep 11 03:41:04.578245 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:04.578226    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-ndxzq\""
Sep 11 03:41:04.579002 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:04.578982    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Sep 11 03:41:04.579002 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:04.578993    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Sep 11 03:41:04.586183 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:04.586164    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-ks6ld"]
Sep 11 03:41:04.692713 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:04.692672    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-76b857c477-qw9fr"]
Sep 11 03:41:04.695698 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:04.695677    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-76b857c477-qw9fr"
Sep 11 03:41:04.697531 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:04.697507    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5844b856db-l7n7s"]
Sep 11 03:41:04.698974 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:04.698955    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 11 03:41:04.699107 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:04.699001    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-2llnq\""
Sep 11 03:41:04.699415 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:04.699402    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 11 03:41:04.700218 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:04.700201    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5844b856db-l7n7s"
Sep 11 03:41:04.701350 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:04.701317    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 11 03:41:04.705682 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:04.705543    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-p8x24\""
Sep 11 03:41:04.705904 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:04.705881    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Sep 11 03:41:04.705997 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:04.705904    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 11 03:41:04.706167 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:04.706142    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Sep 11 03:41:04.706414 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:04.706398    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 11 03:41:04.710231 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:04.710202    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 11 03:41:04.711832 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:04.711803    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Sep 11 03:41:04.715738 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:04.715717    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-76b857c477-qw9fr"]
Sep 11 03:41:04.718981 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:04.718959    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5844b856db-l7n7s"]
Sep 11 03:41:04.744613 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:04.744587    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7d68f71b-9559-41dd-ad14-cf3a556262f8-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-ks6ld\" (UID: \"7d68f71b-9559-41dd-ad14-cf3a556262f8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-ks6ld"
Sep 11 03:41:04.744773 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:04.744624    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xgzj\" (UniqueName: \"kubernetes.io/projected/7d68f71b-9559-41dd-ad14-cf3a556262f8-kube-api-access-9xgzj\") pod \"cluster-samples-operator-86499c89c4-ks6ld\" (UID: \"7d68f71b-9559-41dd-ad14-cf3a556262f8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-ks6ld"
Sep 11 03:41:04.777963 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:04.777929    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-m45tt"]
Sep 11 03:41:04.781011 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:04.780987    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-m45tt"
Sep 11 03:41:04.783374 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:04.783350    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Sep 11 03:41:04.783521 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:04.783503    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Sep 11 03:41:04.783690 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:04.783509    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Sep 11 03:41:04.783690 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:04.783585    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-jpmxc\""
Sep 11 03:41:04.783690 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:04.783615    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Sep 11 03:41:04.795045 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:04.795022    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-m45tt"]
Sep 11 03:41:04.845220 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:04.845139    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2a8f992e-1768-401c-afd6-b87e5a289d61-serving-cert\") pod \"insights-operator-5844b856db-l7n7s\" (UID: \"2a8f992e-1768-401c-afd6-b87e5a289d61\") " pod="openshift-insights/insights-operator-5844b856db-l7n7s"
Sep 11 03:41:04.845220 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:04.845171    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a0327861-5035-4a54-8d03-79c55efdfaa1-installation-pull-secrets\") pod \"image-registry-76b857c477-qw9fr\" (UID: \"a0327861-5035-4a54-8d03-79c55efdfaa1\") " pod="openshift-image-registry/image-registry-76b857c477-qw9fr"
Sep 11 03:41:04.845220 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:04.845194    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2a8f992e-1768-401c-afd6-b87e5a289d61-service-ca-bundle\") pod \"insights-operator-5844b856db-l7n7s\" (UID: \"2a8f992e-1768-401c-afd6-b87e5a289d61\") " pod="openshift-insights/insights-operator-5844b856db-l7n7s"
Sep 11 03:41:04.845469 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:04.845226    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28b4b\" (UniqueName: \"kubernetes.io/projected/2a8f992e-1768-401c-afd6-b87e5a289d61-kube-api-access-28b4b\") pod \"insights-operator-5844b856db-l7n7s\" (UID: \"2a8f992e-1768-401c-afd6-b87e5a289d61\") " pod="openshift-insights/insights-operator-5844b856db-l7n7s"
Sep 11 03:41:04.845469 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:04.845244    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a0327861-5035-4a54-8d03-79c55efdfaa1-trusted-ca\") pod \"image-registry-76b857c477-qw9fr\" (UID: \"a0327861-5035-4a54-8d03-79c55efdfaa1\") " pod="openshift-image-registry/image-registry-76b857c477-qw9fr"
Sep 11 03:41:04.845469 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:04.845265    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7d68f71b-9559-41dd-ad14-cf3a556262f8-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-ks6ld\" (UID: \"7d68f71b-9559-41dd-ad14-cf3a556262f8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-ks6ld"
Sep 11 03:41:04.845469 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:04.845286    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a0327861-5035-4a54-8d03-79c55efdfaa1-image-registry-private-configuration\") pod \"image-registry-76b857c477-qw9fr\" (UID: \"a0327861-5035-4a54-8d03-79c55efdfaa1\") " pod="openshift-image-registry/image-registry-76b857c477-qw9fr"
Sep 11 03:41:04.845469 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:04.845368    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xgzj\" (UniqueName: \"kubernetes.io/projected/7d68f71b-9559-41dd-ad14-cf3a556262f8-kube-api-access-9xgzj\") pod \"cluster-samples-operator-86499c89c4-ks6ld\" (UID: \"7d68f71b-9559-41dd-ad14-cf3a556262f8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-ks6ld"
Sep 11 03:41:04.845469 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:04.845400    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2a8f992e-1768-401c-afd6-b87e5a289d61-tmp\") pod \"insights-operator-5844b856db-l7n7s\" (UID: \"2a8f992e-1768-401c-afd6-b87e5a289d61\") " pod="openshift-insights/insights-operator-5844b856db-l7n7s"
Sep 11 03:41:04.845469 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:41:04.845412    2580 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 11 03:41:04.845469 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:04.845416    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a0327861-5035-4a54-8d03-79c55efdfaa1-ca-trust-extracted\") pod \"image-registry-76b857c477-qw9fr\" (UID: \"a0327861-5035-4a54-8d03-79c55efdfaa1\") " pod="openshift-image-registry/image-registry-76b857c477-qw9fr"
Sep 11 03:41:04.845835 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:41:04.845489    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d68f71b-9559-41dd-ad14-cf3a556262f8-samples-operator-tls podName:7d68f71b-9559-41dd-ad14-cf3a556262f8 nodeName:}" failed. No retries permitted until 2026-09-11 03:41:05.345468247 +0000 UTC m=+53.503954554 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/7d68f71b-9559-41dd-ad14-cf3a556262f8-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-ks6ld" (UID: "7d68f71b-9559-41dd-ad14-cf3a556262f8") : secret "samples-operator-tls" not found
Sep 11 03:41:04.845835 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:04.845514    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a0327861-5035-4a54-8d03-79c55efdfaa1-bound-sa-token\") pod \"image-registry-76b857c477-qw9fr\" (UID: \"a0327861-5035-4a54-8d03-79c55efdfaa1\") " pod="openshift-image-registry/image-registry-76b857c477-qw9fr"
Sep 11 03:41:04.845835 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:04.845592    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/2a8f992e-1768-401c-afd6-b87e5a289d61-snapshots\") pod \"insights-operator-5844b856db-l7n7s\" (UID: \"2a8f992e-1768-401c-afd6-b87e5a289d61\") " pod="openshift-insights/insights-operator-5844b856db-l7n7s"
Sep 11 03:41:04.845835 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:04.845618    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2a8f992e-1768-401c-afd6-b87e5a289d61-trusted-ca-bundle\") pod \"insights-operator-5844b856db-l7n7s\" (UID: \"2a8f992e-1768-401c-afd6-b87e5a289d61\") " pod="openshift-insights/insights-operator-5844b856db-l7n7s"
Sep 11 03:41:04.845835 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:04.845656    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a0327861-5035-4a54-8d03-79c55efdfaa1-registry-tls\") pod \"image-registry-76b857c477-qw9fr\" (UID: \"a0327861-5035-4a54-8d03-79c55efdfaa1\") " pod="openshift-image-registry/image-registry-76b857c477-qw9fr"
Sep 11 03:41:04.845835 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:04.845727    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a0327861-5035-4a54-8d03-79c55efdfaa1-registry-certificates\") pod \"image-registry-76b857c477-qw9fr\" (UID: \"a0327861-5035-4a54-8d03-79c55efdfaa1\") " pod="openshift-image-registry/image-registry-76b857c477-qw9fr"
Sep 11 03:41:04.845835 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:04.845763    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gb8w4\" (UniqueName: \"kubernetes.io/projected/a0327861-5035-4a54-8d03-79c55efdfaa1-kube-api-access-gb8w4\") pod \"image-registry-76b857c477-qw9fr\" (UID: \"a0327861-5035-4a54-8d03-79c55efdfaa1\") " pod="openshift-image-registry/image-registry-76b857c477-qw9fr"
Sep 11 03:41:04.860156 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:04.860125    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xgzj\" (UniqueName: \"kubernetes.io/projected/7d68f71b-9559-41dd-ad14-cf3a556262f8-kube-api-access-9xgzj\") pod \"cluster-samples-operator-86499c89c4-ks6ld\" (UID: \"7d68f71b-9559-41dd-ad14-cf3a556262f8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-ks6ld"
Sep 11 03:41:04.946746 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:04.946707    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a0327861-5035-4a54-8d03-79c55efdfaa1-image-registry-private-configuration\") pod \"image-registry-76b857c477-qw9fr\" (UID: \"a0327861-5035-4a54-8d03-79c55efdfaa1\") " pod="openshift-image-registry/image-registry-76b857c477-qw9fr"
Sep 11 03:41:04.946924 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:04.946754    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2a8f992e-1768-401c-afd6-b87e5a289d61-tmp\") pod \"insights-operator-5844b856db-l7n7s\" (UID: \"2a8f992e-1768-401c-afd6-b87e5a289d61\") " pod="openshift-insights/insights-operator-5844b856db-l7n7s"
Sep 11 03:41:04.946924 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:04.946779    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a0327861-5035-4a54-8d03-79c55efdfaa1-ca-trust-extracted\") pod \"image-registry-76b857c477-qw9fr\" (UID: \"a0327861-5035-4a54-8d03-79c55efdfaa1\") " pod="openshift-image-registry/image-registry-76b857c477-qw9fr"
Sep 11 03:41:04.946924 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:04.946802    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a0327861-5035-4a54-8d03-79c55efdfaa1-bound-sa-token\") pod \"image-registry-76b857c477-qw9fr\" (UID: \"a0327861-5035-4a54-8d03-79c55efdfaa1\") " pod="openshift-image-registry/image-registry-76b857c477-qw9fr"
Sep 11 03:41:04.946924 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:04.946851    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/2a8f992e-1768-401c-afd6-b87e5a289d61-snapshots\") pod \"insights-operator-5844b856db-l7n7s\" (UID: \"2a8f992e-1768-401c-afd6-b87e5a289d61\") " pod="openshift-insights/insights-operator-5844b856db-l7n7s"
Sep 11 03:41:04.946924 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:04.946874    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2a8f992e-1768-401c-afd6-b87e5a289d61-trusted-ca-bundle\") pod \"insights-operator-5844b856db-l7n7s\" (UID: \"2a8f992e-1768-401c-afd6-b87e5a289d61\") " pod="openshift-insights/insights-operator-5844b856db-l7n7s"
Sep 11 03:41:04.947187 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:04.947029    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a0327861-5035-4a54-8d03-79c55efdfaa1-registry-tls\") pod \"image-registry-76b857c477-qw9fr\" (UID: \"a0327861-5035-4a54-8d03-79c55efdfaa1\") " pod="openshift-image-registry/image-registry-76b857c477-qw9fr"
Sep 11 03:41:04.947187 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:04.947074    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a0327861-5035-4a54-8d03-79c55efdfaa1-registry-certificates\") pod \"image-registry-76b857c477-qw9fr\" (UID: \"a0327861-5035-4a54-8d03-79c55efdfaa1\") " pod="openshift-image-registry/image-registry-76b857c477-qw9fr"
Sep 11 03:41:04.947187 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:04.947103    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gb8w4\" (UniqueName: \"kubernetes.io/projected/a0327861-5035-4a54-8d03-79c55efdfaa1-kube-api-access-gb8w4\") pod \"image-registry-76b857c477-qw9fr\" (UID: \"a0327861-5035-4a54-8d03-79c55efdfaa1\") " pod="openshift-image-registry/image-registry-76b857c477-qw9fr"
Sep 11 03:41:04.947187 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:04.947147    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2a8f992e-1768-401c-afd6-b87e5a289d61-serving-cert\") pod \"insights-operator-5844b856db-l7n7s\" (UID: \"2a8f992e-1768-401c-afd6-b87e5a289d61\") " pod="openshift-insights/insights-operator-5844b856db-l7n7s"
Sep 11 03:41:04.947187 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:04.947161    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2a8f992e-1768-401c-afd6-b87e5a289d61-tmp\") pod \"insights-operator-5844b856db-l7n7s\" (UID: \"2a8f992e-1768-401c-afd6-b87e5a289d61\") " pod="openshift-insights/insights-operator-5844b856db-l7n7s"
Sep 11 03:41:04.947187 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:04.947173    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a0327861-5035-4a54-8d03-79c55efdfaa1-installation-pull-secrets\") pod \"image-registry-76b857c477-qw9fr\" (UID: \"a0327861-5035-4a54-8d03-79c55efdfaa1\") " pod="openshift-image-registry/image-registry-76b857c477-qw9fr"
Sep 11 03:41:04.947518 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:04.947229    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2a8f992e-1768-401c-afd6-b87e5a289d61-service-ca-bundle\") pod \"insights-operator-5844b856db-l7n7s\" (UID: \"2a8f992e-1768-401c-afd6-b87e5a289d61\") " pod="openshift-insights/insights-operator-5844b856db-l7n7s"
Sep 11 03:41:04.947518 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:04.947248    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a0327861-5035-4a54-8d03-79c55efdfaa1-ca-trust-extracted\") pod \"image-registry-76b857c477-qw9fr\" (UID: \"a0327861-5035-4a54-8d03-79c55efdfaa1\") " pod="openshift-image-registry/image-registry-76b857c477-qw9fr"
Sep 11 03:41:04.947518 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:04.947264    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f273ceed-c140-4434-b114-220a14a5f02b-config\") pod \"service-ca-operator-cf7c776b7-m45tt\" (UID: \"f273ceed-c140-4434-b114-220a14a5f02b\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-m45tt"
Sep 11 03:41:04.947518 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:41:04.947356    2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 11 03:41:04.947518 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:41:04.947371    2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-76b857c477-qw9fr: secret "image-registry-tls" not found
Sep 11 03:41:04.947518 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:41:04.947436    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a0327861-5035-4a54-8d03-79c55efdfaa1-registry-tls podName:a0327861-5035-4a54-8d03-79c55efdfaa1 nodeName:}" failed. No retries permitted until 2026-09-11 03:41:05.447417515 +0000 UTC m=+53.605903816 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a0327861-5035-4a54-8d03-79c55efdfaa1-registry-tls") pod "image-registry-76b857c477-qw9fr" (UID: "a0327861-5035-4a54-8d03-79c55efdfaa1") : secret "image-registry-tls" not found
Sep 11 03:41:04.947518 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:04.947497    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/2a8f992e-1768-401c-afd6-b87e5a289d61-snapshots\") pod \"insights-operator-5844b856db-l7n7s\" (UID: \"2a8f992e-1768-401c-afd6-b87e5a289d61\") " pod="openshift-insights/insights-operator-5844b856db-l7n7s"
Sep 11 03:41:04.947900 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:04.947650    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28b4b\" (UniqueName: \"kubernetes.io/projected/2a8f992e-1768-401c-afd6-b87e5a289d61-kube-api-access-28b4b\") pod \"insights-operator-5844b856db-l7n7s\" (UID: \"2a8f992e-1768-401c-afd6-b87e5a289d61\") " pod="openshift-insights/insights-operator-5844b856db-l7n7s"
Sep 11 03:41:04.947900 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:04.947704    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a0327861-5035-4a54-8d03-79c55efdfaa1-trusted-ca\") pod \"image-registry-76b857c477-qw9fr\" (UID: \"a0327861-5035-4a54-8d03-79c55efdfaa1\") " pod="openshift-image-registry/image-registry-76b857c477-qw9fr"
Sep 11 03:41:04.947900 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:04.947734    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f273ceed-c140-4434-b114-220a14a5f02b-serving-cert\") pod \"service-ca-operator-cf7c776b7-m45tt\" (UID: \"f273ceed-c140-4434-b114-220a14a5f02b\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-m45tt"
Sep 11 03:41:04.947900 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:04.947762    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2f5zw\" (UniqueName: \"kubernetes.io/projected/f273ceed-c140-4434-b114-220a14a5f02b-kube-api-access-2f5zw\") pod \"service-ca-operator-cf7c776b7-m45tt\" (UID: \"f273ceed-c140-4434-b114-220a14a5f02b\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-m45tt"
Sep 11 03:41:04.947900 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:04.947773    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a0327861-5035-4a54-8d03-79c55efdfaa1-registry-certificates\") pod \"image-registry-76b857c477-qw9fr\" (UID: \"a0327861-5035-4a54-8d03-79c55efdfaa1\") " pod="openshift-image-registry/image-registry-76b857c477-qw9fr"
Sep 11 03:41:04.948101 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:04.947937    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2a8f992e-1768-401c-afd6-b87e5a289d61-trusted-ca-bundle\") pod \"insights-operator-5844b856db-l7n7s\" (UID: \"2a8f992e-1768-401c-afd6-b87e5a289d61\") " pod="openshift-insights/insights-operator-5844b856db-l7n7s"
Sep 11 03:41:04.948241 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:04.948221    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2a8f992e-1768-401c-afd6-b87e5a289d61-service-ca-bundle\") pod \"insights-operator-5844b856db-l7n7s\" (UID: \"2a8f992e-1768-401c-afd6-b87e5a289d61\") " pod="openshift-insights/insights-operator-5844b856db-l7n7s"
Sep 11 03:41:04.948807 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:04.948785    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a0327861-5035-4a54-8d03-79c55efdfaa1-trusted-ca\") pod \"image-registry-76b857c477-qw9fr\" (UID: \"a0327861-5035-4a54-8d03-79c55efdfaa1\") " pod="openshift-image-registry/image-registry-76b857c477-qw9fr"
Sep 11 03:41:04.949518 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:04.949491    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a0327861-5035-4a54-8d03-79c55efdfaa1-image-registry-private-configuration\") pod \"image-registry-76b857c477-qw9fr\" (UID: \"a0327861-5035-4a54-8d03-79c55efdfaa1\") " pod="openshift-image-registry/image-registry-76b857c477-qw9fr"
Sep 11 03:41:04.949621 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:04.949535    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a0327861-5035-4a54-8d03-79c55efdfaa1-installation-pull-secrets\") pod \"image-registry-76b857c477-qw9fr\" (UID: \"a0327861-5035-4a54-8d03-79c55efdfaa1\") " pod="openshift-image-registry/image-registry-76b857c477-qw9fr"
Sep 11 03:41:04.949761 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:04.949743    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2a8f992e-1768-401c-afd6-b87e5a289d61-serving-cert\") pod \"insights-operator-5844b856db-l7n7s\" (UID: \"2a8f992e-1768-401c-afd6-b87e5a289d61\") " pod="openshift-insights/insights-operator-5844b856db-l7n7s"
Sep 11 03:41:04.955453 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:04.955430    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gb8w4\" (UniqueName: \"kubernetes.io/projected/a0327861-5035-4a54-8d03-79c55efdfaa1-kube-api-access-gb8w4\") pod \"image-registry-76b857c477-qw9fr\" (UID: \"a0327861-5035-4a54-8d03-79c55efdfaa1\") " pod="openshift-image-registry/image-registry-76b857c477-qw9fr"
Sep 11 03:41:04.955913 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:04.955894    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a0327861-5035-4a54-8d03-79c55efdfaa1-bound-sa-token\") pod \"image-registry-76b857c477-qw9fr\" (UID: \"a0327861-5035-4a54-8d03-79c55efdfaa1\") " pod="openshift-image-registry/image-registry-76b857c477-qw9fr"
Sep 11 03:41:04.956402 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:04.956385    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-28b4b\" (UniqueName: \"kubernetes.io/projected/2a8f992e-1768-401c-afd6-b87e5a289d61-kube-api-access-28b4b\") pod \"insights-operator-5844b856db-l7n7s\" (UID: \"2a8f992e-1768-401c-afd6-b87e5a289d61\") " pod="openshift-insights/insights-operator-5844b856db-l7n7s"
Sep 11 03:41:05.014574 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:05.014526    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5844b856db-l7n7s"
Sep 11 03:41:05.049187 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:05.049140    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f273ceed-c140-4434-b114-220a14a5f02b-config\") pod \"service-ca-operator-cf7c776b7-m45tt\" (UID: \"f273ceed-c140-4434-b114-220a14a5f02b\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-m45tt"
Sep 11 03:41:05.049342 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:05.049206    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f273ceed-c140-4434-b114-220a14a5f02b-serving-cert\") pod \"service-ca-operator-cf7c776b7-m45tt\" (UID: \"f273ceed-c140-4434-b114-220a14a5f02b\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-m45tt"
Sep 11 03:41:05.049342 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:05.049238    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2f5zw\" (UniqueName: \"kubernetes.io/projected/f273ceed-c140-4434-b114-220a14a5f02b-kube-api-access-2f5zw\") pod \"service-ca-operator-cf7c776b7-m45tt\" (UID: \"f273ceed-c140-4434-b114-220a14a5f02b\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-m45tt"
Sep 11 03:41:05.049707 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:05.049686    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f273ceed-c140-4434-b114-220a14a5f02b-config\") pod \"service-ca-operator-cf7c776b7-m45tt\" (UID: \"f273ceed-c140-4434-b114-220a14a5f02b\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-m45tt"
Sep 11 03:41:05.051354 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:05.051332    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f273ceed-c140-4434-b114-220a14a5f02b-serving-cert\") pod \"service-ca-operator-cf7c776b7-m45tt\" (UID: \"f273ceed-c140-4434-b114-220a14a5f02b\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-m45tt"
Sep 11 03:41:05.057751 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:05.057730    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2f5zw\" (UniqueName: \"kubernetes.io/projected/f273ceed-c140-4434-b114-220a14a5f02b-kube-api-access-2f5zw\") pod \"service-ca-operator-cf7c776b7-m45tt\" (UID: \"f273ceed-c140-4434-b114-220a14a5f02b\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-m45tt"
Sep 11 03:41:05.089092 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:05.089060    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-m45tt"
Sep 11 03:41:05.286234 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:05.286205    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5844b856db-l7n7s"]
Sep 11 03:41:05.289528 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:41:05.289505    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a8f992e_1768_401c_afd6_b87e5a289d61.slice/crio-a3c313742f043484941db025867ca88c0a01b6fe3354cce7b905cf0c288c54d3 WatchSource:0}: Error finding container a3c313742f043484941db025867ca88c0a01b6fe3354cce7b905cf0c288c54d3: Status 404 returned error can't find the container with id a3c313742f043484941db025867ca88c0a01b6fe3354cce7b905cf0c288c54d3
Sep 11 03:41:05.352065 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:05.351991    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7d68f71b-9559-41dd-ad14-cf3a556262f8-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-ks6ld\" (UID: \"7d68f71b-9559-41dd-ad14-cf3a556262f8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-ks6ld"
Sep 11 03:41:05.352200 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:41:05.352126    2580 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 11 03:41:05.352242 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:41:05.352220    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d68f71b-9559-41dd-ad14-cf3a556262f8-samples-operator-tls podName:7d68f71b-9559-41dd-ad14-cf3a556262f8 nodeName:}" failed. No retries permitted until 2026-09-11 03:41:06.352201104 +0000 UTC m=+54.510687405 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/7d68f71b-9559-41dd-ad14-cf3a556262f8-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-ks6ld" (UID: "7d68f71b-9559-41dd-ad14-cf3a556262f8") : secret "samples-operator-tls" not found
Sep 11 03:41:05.357871 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:05.357846    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-m45tt"]
Sep 11 03:41:05.361063 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:41:05.361040    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf273ceed_c140_4434_b114_220a14a5f02b.slice/crio-bbcd01f6455547670327fa2df9d720a90dc2e2421f2ed2f86fd75eb2e235a3fc WatchSource:0}: Error finding container bbcd01f6455547670327fa2df9d720a90dc2e2421f2ed2f86fd75eb2e235a3fc: Status 404 returned error can't find the container with id bbcd01f6455547670327fa2df9d720a90dc2e2421f2ed2f86fd75eb2e235a3fc
Sep 11 03:41:05.453043 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:05.453003    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a0327861-5035-4a54-8d03-79c55efdfaa1-registry-tls\") pod \"image-registry-76b857c477-qw9fr\" (UID: \"a0327861-5035-4a54-8d03-79c55efdfaa1\") " pod="openshift-image-registry/image-registry-76b857c477-qw9fr"
Sep 11 03:41:05.453217 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:41:05.453113    2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 11 03:41:05.453217 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:41:05.453130    2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-76b857c477-qw9fr: secret "image-registry-tls" not found
Sep 11 03:41:05.453217 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:41:05.453179    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a0327861-5035-4a54-8d03-79c55efdfaa1-registry-tls podName:a0327861-5035-4a54-8d03-79c55efdfaa1 nodeName:}" failed. No retries permitted until 2026-09-11 03:41:06.453165849 +0000 UTC m=+54.611652151 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a0327861-5035-4a54-8d03-79c55efdfaa1-registry-tls") pod "image-registry-76b857c477-qw9fr" (UID: "a0327861-5035-4a54-8d03-79c55efdfaa1") : secret "image-registry-tls" not found
Sep 11 03:41:05.689010 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:05.688977    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-m45tt" event={"ID":"f273ceed-c140-4434-b114-220a14a5f02b","Type":"ContainerStarted","Data":"bbcd01f6455547670327fa2df9d720a90dc2e2421f2ed2f86fd75eb2e235a3fc"}
Sep 11 03:41:05.690050 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:05.690030    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-l7n7s" event={"ID":"2a8f992e-1768-401c-afd6-b87e5a289d61","Type":"ContainerStarted","Data":"a3c313742f043484941db025867ca88c0a01b6fe3354cce7b905cf0c288c54d3"}
Sep 11 03:41:06.360979 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:06.360936    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7d68f71b-9559-41dd-ad14-cf3a556262f8-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-ks6ld\" (UID: \"7d68f71b-9559-41dd-ad14-cf3a556262f8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-ks6ld"
Sep 11 03:41:06.361310 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:41:06.361160    2580 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 11 03:41:06.361310 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:41:06.361256    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d68f71b-9559-41dd-ad14-cf3a556262f8-samples-operator-tls podName:7d68f71b-9559-41dd-ad14-cf3a556262f8 nodeName:}" failed. No retries permitted until 2026-09-11 03:41:08.361236384 +0000 UTC m=+56.519722699 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/7d68f71b-9559-41dd-ad14-cf3a556262f8-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-ks6ld" (UID: "7d68f71b-9559-41dd-ad14-cf3a556262f8") : secret "samples-operator-tls" not found
Sep 11 03:41:06.461520 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:06.461485    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a0327861-5035-4a54-8d03-79c55efdfaa1-registry-tls\") pod \"image-registry-76b857c477-qw9fr\" (UID: \"a0327861-5035-4a54-8d03-79c55efdfaa1\") " pod="openshift-image-registry/image-registry-76b857c477-qw9fr"
Sep 11 03:41:06.461711 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:41:06.461604    2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 11 03:41:06.461711 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:41:06.461618    2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-76b857c477-qw9fr: secret "image-registry-tls" not found
Sep 11 03:41:06.461711 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:41:06.461699    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a0327861-5035-4a54-8d03-79c55efdfaa1-registry-tls podName:a0327861-5035-4a54-8d03-79c55efdfaa1 nodeName:}" failed. No retries permitted until 2026-09-11 03:41:08.461678731 +0000 UTC m=+56.620165049 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a0327861-5035-4a54-8d03-79c55efdfaa1-registry-tls") pod "image-registry-76b857c477-qw9fr" (UID: "a0327861-5035-4a54-8d03-79c55efdfaa1") : secret "image-registry-tls" not found
Sep 11 03:41:08.377162 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:08.377121    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7d68f71b-9559-41dd-ad14-cf3a556262f8-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-ks6ld\" (UID: \"7d68f71b-9559-41dd-ad14-cf3a556262f8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-ks6ld"
Sep 11 03:41:08.377576 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:41:08.377311    2580 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 11 03:41:08.377576 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:41:08.377390    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d68f71b-9559-41dd-ad14-cf3a556262f8-samples-operator-tls podName:7d68f71b-9559-41dd-ad14-cf3a556262f8 nodeName:}" failed. No retries permitted until 2026-09-11 03:41:12.377369071 +0000 UTC m=+60.535855387 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/7d68f71b-9559-41dd-ad14-cf3a556262f8-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-ks6ld" (UID: "7d68f71b-9559-41dd-ad14-cf3a556262f8") : secret "samples-operator-tls" not found
Sep 11 03:41:08.477876 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:08.477841    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a0327861-5035-4a54-8d03-79c55efdfaa1-registry-tls\") pod \"image-registry-76b857c477-qw9fr\" (UID: \"a0327861-5035-4a54-8d03-79c55efdfaa1\") " pod="openshift-image-registry/image-registry-76b857c477-qw9fr"
Sep 11 03:41:08.478027 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:41:08.477974    2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 11 03:41:08.478027 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:41:08.477991    2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-76b857c477-qw9fr: secret "image-registry-tls" not found
Sep 11 03:41:08.478094 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:41:08.478042    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a0327861-5035-4a54-8d03-79c55efdfaa1-registry-tls podName:a0327861-5035-4a54-8d03-79c55efdfaa1 nodeName:}" failed. No retries permitted until 2026-09-11 03:41:12.478026277 +0000 UTC m=+60.636512584 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a0327861-5035-4a54-8d03-79c55efdfaa1-registry-tls") pod "image-registry-76b857c477-qw9fr" (UID: "a0327861-5035-4a54-8d03-79c55efdfaa1") : secret "image-registry-tls" not found
Sep 11 03:41:08.699937 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:08.699900    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-m45tt" event={"ID":"f273ceed-c140-4434-b114-220a14a5f02b","Type":"ContainerStarted","Data":"8edc3636ad5519bac29b3c967c6a0f60053155032a5020c2f640c76bd3a62a0f"}
Sep 11 03:41:08.701439 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:08.701414    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-l7n7s" event={"ID":"2a8f992e-1768-401c-afd6-b87e5a289d61","Type":"ContainerStarted","Data":"4fee7a1e9d37160f5e0cd2e2a7e46f0191a1f8864bfbd2df0eda9924daa4b42a"}
Sep 11 03:41:08.714577 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:08.714528    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-m45tt" podStartSLOduration=2.300254342 podStartE2EDuration="4.714507729s" totalImagesPullingTime="2.414253387s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 03:41:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:41:08.713635046 +0000 UTC m=+56.872121371" watchObservedRunningTime="2026-09-11 03:41:08.714507729 +0000 UTC m=+56.872994057"
Sep 11 03:41:08.728222 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:08.728174    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5844b856db-l7n7s" podStartSLOduration=2.231946163 podStartE2EDuration="4.728144186s" totalImagesPullingTime="2.496198023s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 03:41:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:41:08.727345828 +0000 UTC m=+56.885832155" watchObservedRunningTime="2026-09-11 03:41:08.728144186 +0000 UTC m=+56.886630508"
Sep 11 03:41:08.877842 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:08.877804    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-qgz5l"]
Sep 11 03:41:08.909008 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:08.908973    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-qgz5l"]
Sep 11 03:41:08.909277 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:08.909110    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-qgz5l"
Sep 11 03:41:08.911494 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:08.911470    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 11 03:41:08.911612 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:08.911472    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 11 03:41:08.911612 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:08.911538    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-8h9gh\""
Sep 11 03:41:08.985639 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:08.985564    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/40db11ed-2328-4a59-9db8-6efeaf55b54b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qgz5l\" (UID: \"40db11ed-2328-4a59-9db8-6efeaf55b54b\") " pod="openshift-insights/insights-runtime-extractor-qgz5l"
Sep 11 03:41:08.985639 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:08.985597    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/40db11ed-2328-4a59-9db8-6efeaf55b54b-crio-socket\") pod \"insights-runtime-extractor-qgz5l\" (UID: \"40db11ed-2328-4a59-9db8-6efeaf55b54b\") " pod="openshift-insights/insights-runtime-extractor-qgz5l"
Sep 11 03:41:08.985639 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:08.985623    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/40db11ed-2328-4a59-9db8-6efeaf55b54b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qgz5l\" (UID: \"40db11ed-2328-4a59-9db8-6efeaf55b54b\") " pod="openshift-insights/insights-runtime-extractor-qgz5l"
Sep 11 03:41:08.985865 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:08.985648    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvqpr\" (UniqueName: \"kubernetes.io/projected/40db11ed-2328-4a59-9db8-6efeaf55b54b-kube-api-access-bvqpr\") pod \"insights-runtime-extractor-qgz5l\" (UID: \"40db11ed-2328-4a59-9db8-6efeaf55b54b\") " pod="openshift-insights/insights-runtime-extractor-qgz5l"
Sep 11 03:41:08.985865 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:08.985709    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/40db11ed-2328-4a59-9db8-6efeaf55b54b-data-volume\") pod \"insights-runtime-extractor-qgz5l\" (UID: \"40db11ed-2328-4a59-9db8-6efeaf55b54b\") " pod="openshift-insights/insights-runtime-extractor-qgz5l"
Sep 11 03:41:09.086869 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:09.086836    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/40db11ed-2328-4a59-9db8-6efeaf55b54b-data-volume\") pod \"insights-runtime-extractor-qgz5l\" (UID: \"40db11ed-2328-4a59-9db8-6efeaf55b54b\") " pod="openshift-insights/insights-runtime-extractor-qgz5l"
Sep 11 03:41:09.087029 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:09.086918    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/40db11ed-2328-4a59-9db8-6efeaf55b54b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qgz5l\" (UID: \"40db11ed-2328-4a59-9db8-6efeaf55b54b\") " pod="openshift-insights/insights-runtime-extractor-qgz5l"
Sep 11 03:41:09.087029 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:09.086937    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/40db11ed-2328-4a59-9db8-6efeaf55b54b-crio-socket\") pod \"insights-runtime-extractor-qgz5l\" (UID: \"40db11ed-2328-4a59-9db8-6efeaf55b54b\") " pod="openshift-insights/insights-runtime-extractor-qgz5l"
Sep 11 03:41:09.087029 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:09.086958    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/40db11ed-2328-4a59-9db8-6efeaf55b54b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qgz5l\" (UID: \"40db11ed-2328-4a59-9db8-6efeaf55b54b\") " pod="openshift-insights/insights-runtime-extractor-qgz5l"
Sep 11 03:41:09.087029 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:09.086975    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvqpr\" (UniqueName: \"kubernetes.io/projected/40db11ed-2328-4a59-9db8-6efeaf55b54b-kube-api-access-bvqpr\") pod \"insights-runtime-extractor-qgz5l\" (UID: \"40db11ed-2328-4a59-9db8-6efeaf55b54b\") " pod="openshift-insights/insights-runtime-extractor-qgz5l"
Sep 11 03:41:09.087222 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:41:09.087052    2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 11 03:41:09.087222 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:41:09.087103    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 03:41:09.087222 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:41:09.087128    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/40db11ed-2328-4a59-9db8-6efeaf55b54b-kube-rbac-proxy-cm podName:40db11ed-2328-4a59-9db8-6efeaf55b54b nodeName:}" failed. No retries permitted until 2026-09-11 03:41:09.587110632 +0000 UTC m=+57.745596949 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/40db11ed-2328-4a59-9db8-6efeaf55b54b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-qgz5l" (UID: "40db11ed-2328-4a59-9db8-6efeaf55b54b") : configmap "kube-rbac-proxy" not found
Sep 11 03:41:09.087222 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:09.087126    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/40db11ed-2328-4a59-9db8-6efeaf55b54b-crio-socket\") pod \"insights-runtime-extractor-qgz5l\" (UID: \"40db11ed-2328-4a59-9db8-6efeaf55b54b\") " pod="openshift-insights/insights-runtime-extractor-qgz5l"
Sep 11 03:41:09.087222 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:41:09.087165    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40db11ed-2328-4a59-9db8-6efeaf55b54b-insights-runtime-extractor-tls podName:40db11ed-2328-4a59-9db8-6efeaf55b54b nodeName:}" failed. No retries permitted until 2026-09-11 03:41:09.587147293 +0000 UTC m=+57.745633616 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/40db11ed-2328-4a59-9db8-6efeaf55b54b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qgz5l" (UID: "40db11ed-2328-4a59-9db8-6efeaf55b54b") : secret "insights-runtime-extractor-tls" not found
Sep 11 03:41:09.087222 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:09.087163    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/40db11ed-2328-4a59-9db8-6efeaf55b54b-data-volume\") pod \"insights-runtime-extractor-qgz5l\" (UID: \"40db11ed-2328-4a59-9db8-6efeaf55b54b\") " pod="openshift-insights/insights-runtime-extractor-qgz5l"
Sep 11 03:41:09.094926 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:09.094903    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvqpr\" (UniqueName: \"kubernetes.io/projected/40db11ed-2328-4a59-9db8-6efeaf55b54b-kube-api-access-bvqpr\") pod \"insights-runtime-extractor-qgz5l\" (UID: \"40db11ed-2328-4a59-9db8-6efeaf55b54b\") " pod="openshift-insights/insights-runtime-extractor-qgz5l"
Sep 11 03:41:09.591622 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:09.591584    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/40db11ed-2328-4a59-9db8-6efeaf55b54b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qgz5l\" (UID: \"40db11ed-2328-4a59-9db8-6efeaf55b54b\") " pod="openshift-insights/insights-runtime-extractor-qgz5l"
Sep 11 03:41:09.592079 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:41:09.591753    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 03:41:09.592079 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:09.591762    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/40db11ed-2328-4a59-9db8-6efeaf55b54b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qgz5l\" (UID: \"40db11ed-2328-4a59-9db8-6efeaf55b54b\") " pod="openshift-insights/insights-runtime-extractor-qgz5l"
Sep 11 03:41:09.592079 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:41:09.591817    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40db11ed-2328-4a59-9db8-6efeaf55b54b-insights-runtime-extractor-tls podName:40db11ed-2328-4a59-9db8-6efeaf55b54b nodeName:}" failed. No retries permitted until 2026-09-11 03:41:10.591802143 +0000 UTC m=+58.750288448 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/40db11ed-2328-4a59-9db8-6efeaf55b54b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qgz5l" (UID: "40db11ed-2328-4a59-9db8-6efeaf55b54b") : secret "insights-runtime-extractor-tls" not found
Sep 11 03:41:09.592079 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:41:09.591865    2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 11 03:41:09.592079 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:41:09.591925    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/40db11ed-2328-4a59-9db8-6efeaf55b54b-kube-rbac-proxy-cm podName:40db11ed-2328-4a59-9db8-6efeaf55b54b nodeName:}" failed. No retries permitted until 2026-09-11 03:41:10.591908731 +0000 UTC m=+58.750395046 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/40db11ed-2328-4a59-9db8-6efeaf55b54b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-qgz5l" (UID: "40db11ed-2328-4a59-9db8-6efeaf55b54b") : configmap "kube-rbac-proxy" not found
Sep 11 03:41:10.601866 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:10.601816    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/40db11ed-2328-4a59-9db8-6efeaf55b54b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qgz5l\" (UID: \"40db11ed-2328-4a59-9db8-6efeaf55b54b\") " pod="openshift-insights/insights-runtime-extractor-qgz5l"
Sep 11 03:41:10.602371 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:10.601881    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/40db11ed-2328-4a59-9db8-6efeaf55b54b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qgz5l\" (UID: \"40db11ed-2328-4a59-9db8-6efeaf55b54b\") " pod="openshift-insights/insights-runtime-extractor-qgz5l"
Sep 11 03:41:10.602371 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:41:10.601936    2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 11 03:41:10.602371 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:41:10.602003    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/40db11ed-2328-4a59-9db8-6efeaf55b54b-kube-rbac-proxy-cm podName:40db11ed-2328-4a59-9db8-6efeaf55b54b nodeName:}" failed. No retries permitted until 2026-09-11 03:41:12.601986852 +0000 UTC m=+60.760473157 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/40db11ed-2328-4a59-9db8-6efeaf55b54b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-qgz5l" (UID: "40db11ed-2328-4a59-9db8-6efeaf55b54b") : configmap "kube-rbac-proxy" not found
Sep 11 03:41:10.602371 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:41:10.602015    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 03:41:10.602371 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:41:10.602069    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40db11ed-2328-4a59-9db8-6efeaf55b54b-insights-runtime-extractor-tls podName:40db11ed-2328-4a59-9db8-6efeaf55b54b nodeName:}" failed. No retries permitted until 2026-09-11 03:41:12.602054456 +0000 UTC m=+60.760540770 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/40db11ed-2328-4a59-9db8-6efeaf55b54b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qgz5l" (UID: "40db11ed-2328-4a59-9db8-6efeaf55b54b") : secret "insights-runtime-extractor-tls" not found
Sep 11 03:41:11.091244 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:11.091213    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-wkqmr_896af27b-fde3-43f0-bdbb-5bdf0815873a/dns-node-resolver/0.log"
Sep 11 03:41:11.631581 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:11.631546    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-j4pgw"
Sep 11 03:41:11.890065 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:11.889990    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-mlcbg_84aade61-b056-43d4-bbcd-fd68eba2cb48/node-ca/0.log"
Sep 11 03:41:12.415406 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:12.415376    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7d68f71b-9559-41dd-ad14-cf3a556262f8-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-ks6ld\" (UID: \"7d68f71b-9559-41dd-ad14-cf3a556262f8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-ks6ld"
Sep 11 03:41:12.415595 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:41:12.415491    2580 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 11 03:41:12.415595 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:41:12.415561    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d68f71b-9559-41dd-ad14-cf3a556262f8-samples-operator-tls podName:7d68f71b-9559-41dd-ad14-cf3a556262f8 nodeName:}" failed. No retries permitted until 2026-09-11 03:41:20.415543422 +0000 UTC m=+68.574029736 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/7d68f71b-9559-41dd-ad14-cf3a556262f8-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-ks6ld" (UID: "7d68f71b-9559-41dd-ad14-cf3a556262f8") : secret "samples-operator-tls" not found
Sep 11 03:41:12.489903 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:12.489872    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-f5d7k_46e75c33-310a-420f-b9c9-f92bbe261dd7/serve-healthcheck-canary/0.log"
Sep 11 03:41:12.515835 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:12.515803    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a0327861-5035-4a54-8d03-79c55efdfaa1-registry-tls\") pod \"image-registry-76b857c477-qw9fr\" (UID: \"a0327861-5035-4a54-8d03-79c55efdfaa1\") " pod="openshift-image-registry/image-registry-76b857c477-qw9fr"
Sep 11 03:41:12.515985 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:41:12.515956    2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 11 03:41:12.515985 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:41:12.515971    2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-76b857c477-qw9fr: secret "image-registry-tls" not found
Sep 11 03:41:12.516053 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:41:12.516032    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a0327861-5035-4a54-8d03-79c55efdfaa1-registry-tls podName:a0327861-5035-4a54-8d03-79c55efdfaa1 nodeName:}" failed. No retries permitted until 2026-09-11 03:41:20.516017635 +0000 UTC m=+68.674503946 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a0327861-5035-4a54-8d03-79c55efdfaa1-registry-tls") pod "image-registry-76b857c477-qw9fr" (UID: "a0327861-5035-4a54-8d03-79c55efdfaa1") : secret "image-registry-tls" not found
Sep 11 03:41:12.616771 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:12.616736    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/40db11ed-2328-4a59-9db8-6efeaf55b54b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qgz5l\" (UID: \"40db11ed-2328-4a59-9db8-6efeaf55b54b\") " pod="openshift-insights/insights-runtime-extractor-qgz5l"
Sep 11 03:41:12.616931 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:12.616777    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/40db11ed-2328-4a59-9db8-6efeaf55b54b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qgz5l\" (UID: \"40db11ed-2328-4a59-9db8-6efeaf55b54b\") " pod="openshift-insights/insights-runtime-extractor-qgz5l"
Sep 11 03:41:12.616931 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:41:12.616863    2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 11 03:41:12.616931 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:41:12.616889    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 03:41:12.617026 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:41:12.616936    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40db11ed-2328-4a59-9db8-6efeaf55b54b-insights-runtime-extractor-tls podName:40db11ed-2328-4a59-9db8-6efeaf55b54b nodeName:}" failed. No retries permitted until 2026-09-11 03:41:16.6169234 +0000 UTC m=+64.775409700 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/40db11ed-2328-4a59-9db8-6efeaf55b54b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qgz5l" (UID: "40db11ed-2328-4a59-9db8-6efeaf55b54b") : secret "insights-runtime-extractor-tls" not found
Sep 11 03:41:12.617026 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:41:12.616951    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/40db11ed-2328-4a59-9db8-6efeaf55b54b-kube-rbac-proxy-cm podName:40db11ed-2328-4a59-9db8-6efeaf55b54b nodeName:}" failed. No retries permitted until 2026-09-11 03:41:16.616945031 +0000 UTC m=+64.775431332 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/40db11ed-2328-4a59-9db8-6efeaf55b54b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-qgz5l" (UID: "40db11ed-2328-4a59-9db8-6efeaf55b54b") : configmap "kube-rbac-proxy" not found
Sep 11 03:41:16.648392 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:16.648354    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/40db11ed-2328-4a59-9db8-6efeaf55b54b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qgz5l\" (UID: \"40db11ed-2328-4a59-9db8-6efeaf55b54b\") " pod="openshift-insights/insights-runtime-extractor-qgz5l"
Sep 11 03:41:16.648392 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:16.648396    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/40db11ed-2328-4a59-9db8-6efeaf55b54b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qgz5l\" (UID: \"40db11ed-2328-4a59-9db8-6efeaf55b54b\") " pod="openshift-insights/insights-runtime-extractor-qgz5l"
Sep 11 03:41:16.648892 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:41:16.648476    2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 11 03:41:16.648892 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:41:16.648492    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 03:41:16.648892 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:41:16.648534    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/40db11ed-2328-4a59-9db8-6efeaf55b54b-kube-rbac-proxy-cm podName:40db11ed-2328-4a59-9db8-6efeaf55b54b nodeName:}" failed. No retries permitted until 2026-09-11 03:41:24.648519203 +0000 UTC m=+72.807005504 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/40db11ed-2328-4a59-9db8-6efeaf55b54b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-qgz5l" (UID: "40db11ed-2328-4a59-9db8-6efeaf55b54b") : configmap "kube-rbac-proxy" not found
Sep 11 03:41:16.648892 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:41:16.648612    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40db11ed-2328-4a59-9db8-6efeaf55b54b-insights-runtime-extractor-tls podName:40db11ed-2328-4a59-9db8-6efeaf55b54b nodeName:}" failed. No retries permitted until 2026-09-11 03:41:24.648590648 +0000 UTC m=+72.807076957 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/40db11ed-2328-4a59-9db8-6efeaf55b54b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qgz5l" (UID: "40db11ed-2328-4a59-9db8-6efeaf55b54b") : secret "insights-runtime-extractor-tls" not found
Sep 11 03:41:17.556756 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:17.556714    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/61f124d5-c24b-4150-b19a-ae500bce32fe-metrics-tls\") pod \"dns-default-pwk5f\" (UID: \"61f124d5-c24b-4150-b19a-ae500bce32fe\") " pod="openshift-dns/dns-default-pwk5f"
Sep 11 03:41:17.556927 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:41:17.556838    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 03:41:17.556927 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:41:17.556888    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61f124d5-c24b-4150-b19a-ae500bce32fe-metrics-tls podName:61f124d5-c24b-4150-b19a-ae500bce32fe nodeName:}" failed. No retries permitted until 2026-09-11 03:41:49.556875279 +0000 UTC m=+97.715361580 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/61f124d5-c24b-4150-b19a-ae500bce32fe-metrics-tls") pod "dns-default-pwk5f" (UID: "61f124d5-c24b-4150-b19a-ae500bce32fe") : secret "dns-default-metrics-tls" not found
Sep 11 03:41:18.061001 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:18.060955    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb00790b-1a90-4786-8b81-cbfc706f9c46-metrics-certs\") pod \"network-metrics-daemon-qzhz6\" (UID: \"eb00790b-1a90-4786-8b81-cbfc706f9c46\") " pod="openshift-multus/network-metrics-daemon-qzhz6"
Sep 11 03:41:18.061001 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:18.061008    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f6fa8b71-4ef9-4c91-b7ee-7ef235532ea0-original-pull-secret\") pod \"global-pull-secret-syncer-8knjw\" (UID: \"f6fa8b71-4ef9-4c91-b7ee-7ef235532ea0\") " pod="kube-system/global-pull-secret-syncer-8knjw"
Sep 11 03:41:18.064105 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:18.064085    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 11 03:41:18.064105 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:18.064100    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 11 03:41:18.071283 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:41:18.071259    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 11 03:41:18.071396 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:41:18.071354    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb00790b-1a90-4786-8b81-cbfc706f9c46-metrics-certs podName:eb00790b-1a90-4786-8b81-cbfc706f9c46 nodeName:}" failed. No retries permitted until 2026-09-11 03:42:22.071331874 +0000 UTC m=+130.229818198 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/eb00790b-1a90-4786-8b81-cbfc706f9c46-metrics-certs") pod "network-metrics-daemon-qzhz6" (UID: "eb00790b-1a90-4786-8b81-cbfc706f9c46") : secret "metrics-daemon-secret" not found
Sep 11 03:41:18.073567 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:18.073540    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f6fa8b71-4ef9-4c91-b7ee-7ef235532ea0-original-pull-secret\") pod \"global-pull-secret-syncer-8knjw\" (UID: \"f6fa8b71-4ef9-4c91-b7ee-7ef235532ea0\") " pod="kube-system/global-pull-secret-syncer-8knjw"
Sep 11 03:41:18.075472 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:18.075455    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8knjw"
Sep 11 03:41:18.263716 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:18.263658    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4gtr\" (UniqueName: \"kubernetes.io/projected/ad569c0f-e4f1-42f2-aa2b-48f0a4f16a57-kube-api-access-s4gtr\") pod \"network-check-target-f7hng\" (UID: \"ad569c0f-e4f1-42f2-aa2b-48f0a4f16a57\") " pod="openshift-network-diagnostics/network-check-target-f7hng"
Sep 11 03:41:18.266106 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:18.266087    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 11 03:41:18.276352 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:18.276333    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 11 03:41:18.286656 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:18.286629    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4gtr\" (UniqueName: \"kubernetes.io/projected/ad569c0f-e4f1-42f2-aa2b-48f0a4f16a57-kube-api-access-s4gtr\") pod \"network-check-target-f7hng\" (UID: \"ad569c0f-e4f1-42f2-aa2b-48f0a4f16a57\") " pod="openshift-network-diagnostics/network-check-target-f7hng"
Sep 11 03:41:18.341518 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:18.341440    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-8knjw"]
Sep 11 03:41:18.382609 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:18.382582    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-hqw27\""
Sep 11 03:41:18.391362 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:18.391340    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f7hng"
Sep 11 03:41:18.661979 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:18.661948    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-f7hng"]
Sep 11 03:41:18.665063 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:41:18.665036    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad569c0f_e4f1_42f2_aa2b_48f0a4f16a57.slice/crio-75dec0c0e62e92c030d8c5c70c22c664de6e7cd4a2eb223e6da148eb35f0132f WatchSource:0}: Error finding container 75dec0c0e62e92c030d8c5c70c22c664de6e7cd4a2eb223e6da148eb35f0132f: Status 404 returned error can't find the container with id 75dec0c0e62e92c030d8c5c70c22c664de6e7cd4a2eb223e6da148eb35f0132f
Sep 11 03:41:18.728771 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:18.728735    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-f7hng" event={"ID":"ad569c0f-e4f1-42f2-aa2b-48f0a4f16a57","Type":"ContainerStarted","Data":"75dec0c0e62e92c030d8c5c70c22c664de6e7cd4a2eb223e6da148eb35f0132f"}
Sep 11 03:41:18.729772 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:18.729738    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-8knjw" event={"ID":"f6fa8b71-4ef9-4c91-b7ee-7ef235532ea0","Type":"ContainerStarted","Data":"d0792cbabc68c0a646ab403cdcd419712c9cffbc75cbd54d2556be13b8bfa354"}
Sep 11 03:41:20.481815 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:20.481782    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7d68f71b-9559-41dd-ad14-cf3a556262f8-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-ks6ld\" (UID: \"7d68f71b-9559-41dd-ad14-cf3a556262f8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-ks6ld"
Sep 11 03:41:20.484942 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:20.484898    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7d68f71b-9559-41dd-ad14-cf3a556262f8-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-ks6ld\" (UID: \"7d68f71b-9559-41dd-ad14-cf3a556262f8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-ks6ld"
Sep 11 03:41:20.582830 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:20.582796    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a0327861-5035-4a54-8d03-79c55efdfaa1-registry-tls\") pod \"image-registry-76b857c477-qw9fr\" (UID: \"a0327861-5035-4a54-8d03-79c55efdfaa1\") " pod="openshift-image-registry/image-registry-76b857c477-qw9fr"
Sep 11 03:41:20.585802 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:20.585770    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a0327861-5035-4a54-8d03-79c55efdfaa1-registry-tls\") pod \"image-registry-76b857c477-qw9fr\" (UID: \"a0327861-5035-4a54-8d03-79c55efdfaa1\") " pod="openshift-image-registry/image-registry-76b857c477-qw9fr"
Sep 11 03:41:20.611827 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:20.611792    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-2llnq\""
Sep 11 03:41:20.620109 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:20.620076    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-76b857c477-qw9fr"
Sep 11 03:41:20.786266 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:20.786188    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-ndxzq\""
Sep 11 03:41:20.794128 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:20.794100    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-ks6ld"
Sep 11 03:41:20.902269 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:20.902212    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-76b857c477-qw9fr"]
Sep 11 03:41:22.048561 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:41:22.048529    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0327861_5035_4a54_8d03_79c55efdfaa1.slice/crio-c81573126548418291d9c1d4b11113f72076655cee992f1c8d7fdb6e30c3caab WatchSource:0}: Error finding container c81573126548418291d9c1d4b11113f72076655cee992f1c8d7fdb6e30c3caab: Status 404 returned error can't find the container with id c81573126548418291d9c1d4b11113f72076655cee992f1c8d7fdb6e30c3caab
Sep 11 03:41:22.741881 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:22.741834    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-76b857c477-qw9fr" event={"ID":"a0327861-5035-4a54-8d03-79c55efdfaa1","Type":"ContainerStarted","Data":"c81573126548418291d9c1d4b11113f72076655cee992f1c8d7fdb6e30c3caab"}
Sep 11 03:41:23.340614 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:23.340541    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-ks6ld"]
Sep 11 03:41:23.745930 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:23.745892    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-76b857c477-qw9fr" event={"ID":"a0327861-5035-4a54-8d03-79c55efdfaa1","Type":"ContainerStarted","Data":"5835cba5c871c03d3a8f029bbde030b633c83b55db336581b7ea3efaca31de06"}
Sep 11 03:41:23.746124 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:23.746020    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-76b857c477-qw9fr"
Sep 11 03:41:23.747138 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:23.747112    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-ks6ld" event={"ID":"7d68f71b-9559-41dd-ad14-cf3a556262f8","Type":"ContainerStarted","Data":"86a5ba1b3c8c32a91a43050561777703c53b02c328eba04a24d0aa129a3ae80c"}
Sep 11 03:41:23.748504 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:23.748485    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-f7hng" event={"ID":"ad569c0f-e4f1-42f2-aa2b-48f0a4f16a57","Type":"ContainerStarted","Data":"1747420515d01ae7cbca848802b680ae3b1c134b5aa704b376b8298ff605ba18"}
Sep 11 03:41:23.748574 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:23.748532    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-f7hng"
Sep 11 03:41:23.750005 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:23.749982    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-8knjw" event={"ID":"f6fa8b71-4ef9-4c91-b7ee-7ef235532ea0","Type":"ContainerStarted","Data":"32cfb6b565154a470fe40fcc4b3d1545f1e208270b46b5f0222a672863fccc88"}
Sep 11 03:41:23.769146 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:23.769097    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-76b857c477-qw9fr" podStartSLOduration=19.769083103 podStartE2EDuration="19.769083103s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 03:41:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:41:23.768396547 +0000 UTC m=+71.926882869" watchObservedRunningTime="2026-09-11 03:41:23.769083103 +0000 UTC m=+71.927569423"
Sep 11 03:41:23.783986 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:23.783944    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-8knjw" podStartSLOduration=67.046315154 podStartE2EDuration="1m11.783930005s" totalImagesPullingTime="4.737614851s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 03:40:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:41:23.782935631 +0000 UTC m=+71.941421954" watchObservedRunningTime="2026-09-11 03:41:23.783930005 +0000 UTC m=+71.942416328"
Sep 11 03:41:23.799441 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:23.799392    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-f7hng" podStartSLOduration=67.3823952 podStartE2EDuration="1m11.799377074s" totalImagesPullingTime="4.416981874s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 03:40:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:41:23.799100715 +0000 UTC m=+71.957587039" watchObservedRunningTime="2026-09-11 03:41:23.799377074 +0000 UTC m=+71.957863396"
Sep 11 03:41:24.718272 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:24.718231    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/40db11ed-2328-4a59-9db8-6efeaf55b54b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qgz5l\" (UID: \"40db11ed-2328-4a59-9db8-6efeaf55b54b\") " pod="openshift-insights/insights-runtime-extractor-qgz5l"
Sep 11 03:41:24.718703 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:24.718284    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/40db11ed-2328-4a59-9db8-6efeaf55b54b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qgz5l\" (UID: \"40db11ed-2328-4a59-9db8-6efeaf55b54b\") " pod="openshift-insights/insights-runtime-extractor-qgz5l"
Sep 11 03:41:24.718703 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:41:24.718450    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 03:41:24.718703 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:41:24.718516    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40db11ed-2328-4a59-9db8-6efeaf55b54b-insights-runtime-extractor-tls podName:40db11ed-2328-4a59-9db8-6efeaf55b54b nodeName:}" failed. No retries permitted until 2026-09-11 03:41:40.718499578 +0000 UTC m=+88.876985902 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/40db11ed-2328-4a59-9db8-6efeaf55b54b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qgz5l" (UID: "40db11ed-2328-4a59-9db8-6efeaf55b54b") : secret "insights-runtime-extractor-tls" not found
Sep 11 03:41:24.718834 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:24.718777    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/40db11ed-2328-4a59-9db8-6efeaf55b54b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qgz5l\" (UID: \"40db11ed-2328-4a59-9db8-6efeaf55b54b\") " pod="openshift-insights/insights-runtime-extractor-qgz5l"
Sep 11 03:41:25.763694 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:25.763647    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-ks6ld" event={"ID":"7d68f71b-9559-41dd-ad14-cf3a556262f8","Type":"ContainerStarted","Data":"32e336a5a8c4ec3b1beae335d9676ca75dbb56e0314b5ea931438d54dd19fe2f"}
Sep 11 03:41:25.764185 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:25.763711    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-ks6ld" event={"ID":"7d68f71b-9559-41dd-ad14-cf3a556262f8","Type":"ContainerStarted","Data":"67accf0c649d2f58df9d2f38f98a1f6147888baed724eea54e54d9e1341ebfb9"}
Sep 11 03:41:25.782381 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:25.782332    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-ks6ld" podStartSLOduration=20.258214011 podStartE2EDuration="21.782316292s" totalImagesPullingTime="1.524102281s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 03:41:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:41:25.781895166 +0000 UTC m=+73.940381489" watchObservedRunningTime="2026-09-11 03:41:25.782316292 +0000 UTC m=+73.940802615"
Sep 11 03:41:32.111245 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:32.111212    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-78fccb448b-r5mt8"]
Sep 11 03:41:32.115699 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:32.115656    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-78fccb448b-r5mt8"
Sep 11 03:41:32.118957 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:32.118927    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 11 03:41:32.119109 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:32.118928    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 11 03:41:32.119109 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:32.118928    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Sep 11 03:41:32.119109 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:32.118986    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 11 03:41:32.128515 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:32.128491    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-78fccb448b-r5mt8"]
Sep 11 03:41:32.152642 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:32.152605    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-76b857c477-qw9fr"]
Sep 11 03:41:32.172649 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:32.172607    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f383d425-0a6e-4f91-9791-39a0cb15598c-tmp\") pod \"klusterlet-addon-workmgr-78fccb448b-r5mt8\" (UID: \"f383d425-0a6e-4f91-9791-39a0cb15598c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-78fccb448b-r5mt8"
Sep 11 03:41:32.172816 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:32.172685    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxts2\" (UniqueName: \"kubernetes.io/projected/f383d425-0a6e-4f91-9791-39a0cb15598c-kube-api-access-rxts2\") pod \"klusterlet-addon-workmgr-78fccb448b-r5mt8\" (UID: \"f383d425-0a6e-4f91-9791-39a0cb15598c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-78fccb448b-r5mt8"
Sep 11 03:41:32.172816 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:32.172729    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/f383d425-0a6e-4f91-9791-39a0cb15598c-klusterlet-config\") pod \"klusterlet-addon-workmgr-78fccb448b-r5mt8\" (UID: \"f383d425-0a6e-4f91-9791-39a0cb15598c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-78fccb448b-r5mt8"
Sep 11 03:41:32.196557 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:32.196523    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6dc5b6c9b6-rt5ml"]
Sep 11 03:41:32.199709 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:32.199692    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6dc5b6c9b6-rt5ml"
Sep 11 03:41:32.216259 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:32.216226    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6dc5b6c9b6-rt5ml"]
Sep 11 03:41:32.273205 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:32.273166    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/dc23fd71-e4f6-4e97-b9c6-b5d200a69802-installation-pull-secrets\") pod \"image-registry-6dc5b6c9b6-rt5ml\" (UID: \"dc23fd71-e4f6-4e97-b9c6-b5d200a69802\") " pod="openshift-image-registry/image-registry-6dc5b6c9b6-rt5ml"
Sep 11 03:41:32.273380 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:32.273214    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxts2\" (UniqueName: \"kubernetes.io/projected/f383d425-0a6e-4f91-9791-39a0cb15598c-kube-api-access-rxts2\") pod \"klusterlet-addon-workmgr-78fccb448b-r5mt8\" (UID: \"f383d425-0a6e-4f91-9791-39a0cb15598c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-78fccb448b-r5mt8"
Sep 11 03:41:32.273380 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:32.273241    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/dc23fd71-e4f6-4e97-b9c6-b5d200a69802-image-registry-private-configuration\") pod \"image-registry-6dc5b6c9b6-rt5ml\" (UID: \"dc23fd71-e4f6-4e97-b9c6-b5d200a69802\") " pod="openshift-image-registry/image-registry-6dc5b6c9b6-rt5ml"
Sep 11 03:41:32.273380 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:32.273308    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/f383d425-0a6e-4f91-9791-39a0cb15598c-klusterlet-config\") pod \"klusterlet-addon-workmgr-78fccb448b-r5mt8\" (UID: \"f383d425-0a6e-4f91-9791-39a0cb15598c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-78fccb448b-r5mt8"
Sep 11 03:41:32.273380 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:32.273335    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/dc23fd71-e4f6-4e97-b9c6-b5d200a69802-registry-certificates\") pod \"image-registry-6dc5b6c9b6-rt5ml\" (UID: \"dc23fd71-e4f6-4e97-b9c6-b5d200a69802\") " pod="openshift-image-registry/image-registry-6dc5b6c9b6-rt5ml"
Sep 11 03:41:32.273380 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:32.273359    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58prv\" (UniqueName: \"kubernetes.io/projected/dc23fd71-e4f6-4e97-b9c6-b5d200a69802-kube-api-access-58prv\") pod \"image-registry-6dc5b6c9b6-rt5ml\" (UID: \"dc23fd71-e4f6-4e97-b9c6-b5d200a69802\") " pod="openshift-image-registry/image-registry-6dc5b6c9b6-rt5ml"
Sep 11 03:41:32.273574 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:32.273502    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/dc23fd71-e4f6-4e97-b9c6-b5d200a69802-ca-trust-extracted\") pod \"image-registry-6dc5b6c9b6-rt5ml\" (UID: \"dc23fd71-e4f6-4e97-b9c6-b5d200a69802\") " pod="openshift-image-registry/image-registry-6dc5b6c9b6-rt5ml"
Sep 11 03:41:32.273574 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:32.273544    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/dc23fd71-e4f6-4e97-b9c6-b5d200a69802-trusted-ca\") pod \"image-registry-6dc5b6c9b6-rt5ml\" (UID: \"dc23fd71-e4f6-4e97-b9c6-b5d200a69802\") " pod="openshift-image-registry/image-registry-6dc5b6c9b6-rt5ml"
Sep 11 03:41:32.273647 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:32.273583    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f383d425-0a6e-4f91-9791-39a0cb15598c-tmp\") pod \"klusterlet-addon-workmgr-78fccb448b-r5mt8\" (UID: \"f383d425-0a6e-4f91-9791-39a0cb15598c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-78fccb448b-r5mt8"
Sep 11 03:41:32.273647 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:32.273604    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/dc23fd71-e4f6-4e97-b9c6-b5d200a69802-registry-tls\") pod \"image-registry-6dc5b6c9b6-rt5ml\" (UID: \"dc23fd71-e4f6-4e97-b9c6-b5d200a69802\") " pod="openshift-image-registry/image-registry-6dc5b6c9b6-rt5ml"
Sep 11 03:41:32.273647 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:32.273631    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/dc23fd71-e4f6-4e97-b9c6-b5d200a69802-bound-sa-token\") pod \"image-registry-6dc5b6c9b6-rt5ml\" (UID: \"dc23fd71-e4f6-4e97-b9c6-b5d200a69802\") " pod="openshift-image-registry/image-registry-6dc5b6c9b6-rt5ml"
Sep 11 03:41:32.273915 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:32.273895    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f383d425-0a6e-4f91-9791-39a0cb15598c-tmp\") pod \"klusterlet-addon-workmgr-78fccb448b-r5mt8\" (UID: \"f383d425-0a6e-4f91-9791-39a0cb15598c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-78fccb448b-r5mt8"
Sep 11 03:41:32.275854 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:32.275838    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/f383d425-0a6e-4f91-9791-39a0cb15598c-klusterlet-config\") pod \"klusterlet-addon-workmgr-78fccb448b-r5mt8\" (UID: \"f383d425-0a6e-4f91-9791-39a0cb15598c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-78fccb448b-r5mt8"
Sep 11 03:41:32.285518 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:32.285486    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxts2\" (UniqueName: \"kubernetes.io/projected/f383d425-0a6e-4f91-9791-39a0cb15598c-kube-api-access-rxts2\") pod \"klusterlet-addon-workmgr-78fccb448b-r5mt8\" (UID: \"f383d425-0a6e-4f91-9791-39a0cb15598c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-78fccb448b-r5mt8"
Sep 11 03:41:32.374512 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:32.374416    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/dc23fd71-e4f6-4e97-b9c6-b5d200a69802-registry-certificates\") pod \"image-registry-6dc5b6c9b6-rt5ml\" (UID: \"dc23fd71-e4f6-4e97-b9c6-b5d200a69802\") " pod="openshift-image-registry/image-registry-6dc5b6c9b6-rt5ml"
Sep 11 03:41:32.374512 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:32.374456    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58prv\" (UniqueName: \"kubernetes.io/projected/dc23fd71-e4f6-4e97-b9c6-b5d200a69802-kube-api-access-58prv\") pod \"image-registry-6dc5b6c9b6-rt5ml\" (UID: \"dc23fd71-e4f6-4e97-b9c6-b5d200a69802\") " pod="openshift-image-registry/image-registry-6dc5b6c9b6-rt5ml"
Sep 11 03:41:32.374512 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:32.374479    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/dc23fd71-e4f6-4e97-b9c6-b5d200a69802-ca-trust-extracted\") pod \"image-registry-6dc5b6c9b6-rt5ml\" (UID: \"dc23fd71-e4f6-4e97-b9c6-b5d200a69802\") " pod="openshift-image-registry/image-registry-6dc5b6c9b6-rt5ml"
Sep 11 03:41:32.374767 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:32.374608    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/dc23fd71-e4f6-4e97-b9c6-b5d200a69802-trusted-ca\") pod \"image-registry-6dc5b6c9b6-rt5ml\" (UID: \"dc23fd71-e4f6-4e97-b9c6-b5d200a69802\") " pod="openshift-image-registry/image-registry-6dc5b6c9b6-rt5ml"
Sep 11 03:41:32.374767 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:32.374691    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/dc23fd71-e4f6-4e97-b9c6-b5d200a69802-registry-tls\") pod \"image-registry-6dc5b6c9b6-rt5ml\" (UID: \"dc23fd71-e4f6-4e97-b9c6-b5d200a69802\") " pod="openshift-image-registry/image-registry-6dc5b6c9b6-rt5ml"
Sep 11 03:41:32.374767 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:32.374733    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/dc23fd71-e4f6-4e97-b9c6-b5d200a69802-bound-sa-token\") pod \"image-registry-6dc5b6c9b6-rt5ml\" (UID: \"dc23fd71-e4f6-4e97-b9c6-b5d200a69802\") " pod="openshift-image-registry/image-registry-6dc5b6c9b6-rt5ml"
Sep 11 03:41:32.374951 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:32.374778    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/dc23fd71-e4f6-4e97-b9c6-b5d200a69802-installation-pull-secrets\") pod \"image-registry-6dc5b6c9b6-rt5ml\" (UID: \"dc23fd71-e4f6-4e97-b9c6-b5d200a69802\") " pod="openshift-image-registry/image-registry-6dc5b6c9b6-rt5ml"
Sep 11 03:41:32.374951 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:32.374869    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/dc23fd71-e4f6-4e97-b9c6-b5d200a69802-ca-trust-extracted\") pod \"image-registry-6dc5b6c9b6-rt5ml\" (UID: \"dc23fd71-e4f6-4e97-b9c6-b5d200a69802\") " pod="openshift-image-registry/image-registry-6dc5b6c9b6-rt5ml"
Sep 11 03:41:32.374951 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:32.374907    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/dc23fd71-e4f6-4e97-b9c6-b5d200a69802-image-registry-private-configuration\") pod \"image-registry-6dc5b6c9b6-rt5ml\" (UID: \"dc23fd71-e4f6-4e97-b9c6-b5d200a69802\") " pod="openshift-image-registry/image-registry-6dc5b6c9b6-rt5ml"
Sep 11 03:41:32.375376 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:32.375349    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/dc23fd71-e4f6-4e97-b9c6-b5d200a69802-registry-certificates\") pod \"image-registry-6dc5b6c9b6-rt5ml\" (UID: \"dc23fd71-e4f6-4e97-b9c6-b5d200a69802\") " pod="openshift-image-registry/image-registry-6dc5b6c9b6-rt5ml"
Sep 11 03:41:32.375679 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:32.375644    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/dc23fd71-e4f6-4e97-b9c6-b5d200a69802-trusted-ca\") pod \"image-registry-6dc5b6c9b6-rt5ml\" (UID: \"dc23fd71-e4f6-4e97-b9c6-b5d200a69802\") " pod="openshift-image-registry/image-registry-6dc5b6c9b6-rt5ml"
Sep 11 03:41:32.377190 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:32.377154    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/dc23fd71-e4f6-4e97-b9c6-b5d200a69802-installation-pull-secrets\") pod \"image-registry-6dc5b6c9b6-rt5ml\" (UID: \"dc23fd71-e4f6-4e97-b9c6-b5d200a69802\") " pod="openshift-image-registry/image-registry-6dc5b6c9b6-rt5ml"
Sep 11 03:41:32.377292 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:32.377170    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/dc23fd71-e4f6-4e97-b9c6-b5d200a69802-registry-tls\") pod \"image-registry-6dc5b6c9b6-rt5ml\" (UID: \"dc23fd71-e4f6-4e97-b9c6-b5d200a69802\") " pod="openshift-image-registry/image-registry-6dc5b6c9b6-rt5ml"
Sep 11 03:41:32.377292 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:32.377264    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/dc23fd71-e4f6-4e97-b9c6-b5d200a69802-image-registry-private-configuration\") pod \"image-registry-6dc5b6c9b6-rt5ml\" (UID: \"dc23fd71-e4f6-4e97-b9c6-b5d200a69802\") " pod="openshift-image-registry/image-registry-6dc5b6c9b6-rt5ml"
Sep 11 03:41:32.385515 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:32.385493    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/dc23fd71-e4f6-4e97-b9c6-b5d200a69802-bound-sa-token\") pod \"image-registry-6dc5b6c9b6-rt5ml\" (UID: \"dc23fd71-e4f6-4e97-b9c6-b5d200a69802\") " pod="openshift-image-registry/image-registry-6dc5b6c9b6-rt5ml"
Sep 11 03:41:32.385944 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:32.385928    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-58prv\" (UniqueName: \"kubernetes.io/projected/dc23fd71-e4f6-4e97-b9c6-b5d200a69802-kube-api-access-58prv\") pod \"image-registry-6dc5b6c9b6-rt5ml\" (UID: \"dc23fd71-e4f6-4e97-b9c6-b5d200a69802\") " pod="openshift-image-registry/image-registry-6dc5b6c9b6-rt5ml"
Sep 11 03:41:32.424939 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:32.424903    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-78fccb448b-r5mt8"
Sep 11 03:41:32.508920 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:32.508884    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6dc5b6c9b6-rt5ml"
Sep 11 03:41:32.702181 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:32.702145    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-78fccb448b-r5mt8"]
Sep 11 03:41:32.705234 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:41:32.705203    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf383d425_0a6e_4f91_9791_39a0cb15598c.slice/crio-4ed0fee8834570e0252a82defbb94748770cba695b6ffddcd8b70249b633a5f1 WatchSource:0}: Error finding container 4ed0fee8834570e0252a82defbb94748770cba695b6ffddcd8b70249b633a5f1: Status 404 returned error can't find the container with id 4ed0fee8834570e0252a82defbb94748770cba695b6ffddcd8b70249b633a5f1
Sep 11 03:41:32.778788 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:32.778759    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6dc5b6c9b6-rt5ml"]
Sep 11 03:41:32.781742 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:41:32.781716    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc23fd71_e4f6_4e97_b9c6_b5d200a69802.slice/crio-d59f37d45c1e80617617fda9c85f7464a17ab1f1f9f14b4ca17e42905bc70331 WatchSource:0}: Error finding container d59f37d45c1e80617617fda9c85f7464a17ab1f1f9f14b4ca17e42905bc70331: Status 404 returned error can't find the container with id d59f37d45c1e80617617fda9c85f7464a17ab1f1f9f14b4ca17e42905bc70331
Sep 11 03:41:32.784140 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:32.784105    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-78fccb448b-r5mt8" event={"ID":"f383d425-0a6e-4f91-9791-39a0cb15598c","Type":"ContainerStarted","Data":"4ed0fee8834570e0252a82defbb94748770cba695b6ffddcd8b70249b633a5f1"}
Sep 11 03:41:33.788489 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:33.788452    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6dc5b6c9b6-rt5ml" event={"ID":"dc23fd71-e4f6-4e97-b9c6-b5d200a69802","Type":"ContainerStarted","Data":"59b14c3c9996130089ff40a3c5fcd2d9fcf0b49265b84a26291f7488e7f92565"}
Sep 11 03:41:33.788876 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:33.788505    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6dc5b6c9b6-rt5ml" event={"ID":"dc23fd71-e4f6-4e97-b9c6-b5d200a69802","Type":"ContainerStarted","Data":"d59f37d45c1e80617617fda9c85f7464a17ab1f1f9f14b4ca17e42905bc70331"}
Sep 11 03:41:33.788876 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:33.788580    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6dc5b6c9b6-rt5ml"
Sep 11 03:41:33.808809 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:33.808751    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-6dc5b6c9b6-rt5ml" podStartSLOduration=1.80873114 podStartE2EDuration="1.80873114s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 03:41:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:41:33.807298313 +0000 UTC m=+81.965784637" watchObservedRunningTime="2026-09-11 03:41:33.80873114 +0000 UTC m=+81.967217463"
Sep 11 03:41:40.742314 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:40.742182    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/40db11ed-2328-4a59-9db8-6efeaf55b54b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qgz5l\" (UID: \"40db11ed-2328-4a59-9db8-6efeaf55b54b\") " pod="openshift-insights/insights-runtime-extractor-qgz5l"
Sep 11 03:41:40.744854 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:40.744827    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/40db11ed-2328-4a59-9db8-6efeaf55b54b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qgz5l\" (UID: \"40db11ed-2328-4a59-9db8-6efeaf55b54b\") " pod="openshift-insights/insights-runtime-extractor-qgz5l"
Sep 11 03:41:40.967151 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:40.967120    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-xd7gn"]
Sep 11 03:41:40.970355 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:40.970329    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-xd7gn"
Sep 11 03:41:40.972717 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:40.972687    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Sep 11 03:41:40.972852 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:40.972749    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-dlm8b\""
Sep 11 03:41:40.978238 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:40.978211    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-xd7gn"]
Sep 11 03:41:41.020740 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:41.020649    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-8h9gh\""
Sep 11 03:41:41.028389 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:41.028367    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-qgz5l"
Sep 11 03:41:41.044500 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:41.044470    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/f30e3e6e-ad79-489c-bffa-99b81296ac25-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-xd7gn\" (UID: \"f30e3e6e-ad79-489c-bffa-99b81296ac25\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-xd7gn"
Sep 11 03:41:41.145861 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:41.145821    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/f30e3e6e-ad79-489c-bffa-99b81296ac25-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-xd7gn\" (UID: \"f30e3e6e-ad79-489c-bffa-99b81296ac25\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-xd7gn"
Sep 11 03:41:41.146035 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:41:41.146001    2580 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found
Sep 11 03:41:41.146087 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:41:41.146081    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f30e3e6e-ad79-489c-bffa-99b81296ac25-tls-certificates podName:f30e3e6e-ad79-489c-bffa-99b81296ac25 nodeName:}" failed. No retries permitted until 2026-09-11 03:41:41.64605743 +0000 UTC m=+89.804543734 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/f30e3e6e-ad79-489c-bffa-99b81296ac25-tls-certificates") pod "prometheus-operator-admission-webhook-5864cd99d7-xd7gn" (UID: "f30e3e6e-ad79-489c-bffa-99b81296ac25") : secret "prometheus-operator-admission-webhook-tls" not found
Sep 11 03:41:41.648958 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:41.648917    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/f30e3e6e-ad79-489c-bffa-99b81296ac25-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-xd7gn\" (UID: \"f30e3e6e-ad79-489c-bffa-99b81296ac25\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-xd7gn"
Sep 11 03:41:41.651732 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:41.651702    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/f30e3e6e-ad79-489c-bffa-99b81296ac25-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-xd7gn\" (UID: \"f30e3e6e-ad79-489c-bffa-99b81296ac25\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-xd7gn"
Sep 11 03:41:41.882139 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:41.882096    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-xd7gn"
Sep 11 03:41:42.157797 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:42.157761    2580 patch_prober.go:28] interesting pod/image-registry-76b857c477-qw9fr container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 11 03:41:42.157986 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:42.157822    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-76b857c477-qw9fr" podUID="a0327861-5035-4a54-8d03-79c55efdfaa1" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 11 03:41:42.177599 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:42.177234    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-xd7gn"]
Sep 11 03:41:42.180045 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:41:42.180004    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf30e3e6e_ad79_489c_bffa_99b81296ac25.slice/crio-f21c19998d6b81ccf18ee067a82594e5e09d0494bf47e80f775bc657d665e8cf WatchSource:0}: Error finding container f21c19998d6b81ccf18ee067a82594e5e09d0494bf47e80f775bc657d665e8cf: Status 404 returned error can't find the container with id f21c19998d6b81ccf18ee067a82594e5e09d0494bf47e80f775bc657d665e8cf
Sep 11 03:41:42.180842 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:42.180810    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-qgz5l"]
Sep 11 03:41:42.183976 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:41:42.183949    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40db11ed_2328_4a59_9db8_6efeaf55b54b.slice/crio-42ecd5c483ee1f996839ad7922c7c823ca313c2b467df40816d785e1299899c4 WatchSource:0}: Error finding container 42ecd5c483ee1f996839ad7922c7c823ca313c2b467df40816d785e1299899c4: Status 404 returned error can't find the container with id 42ecd5c483ee1f996839ad7922c7c823ca313c2b467df40816d785e1299899c4
Sep 11 03:41:42.820874 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:42.820836    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-qgz5l" event={"ID":"40db11ed-2328-4a59-9db8-6efeaf55b54b","Type":"ContainerStarted","Data":"35b65a14442cd7064c3168dfcff283bf83df6e92904e6a11785ec27132d1d2ae"}
Sep 11 03:41:42.821058 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:42.820895    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-qgz5l" event={"ID":"40db11ed-2328-4a59-9db8-6efeaf55b54b","Type":"ContainerStarted","Data":"42ecd5c483ee1f996839ad7922c7c823ca313c2b467df40816d785e1299899c4"}
Sep 11 03:41:42.822308 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:42.822284    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-78fccb448b-r5mt8" event={"ID":"f383d425-0a6e-4f91-9791-39a0cb15598c","Type":"ContainerStarted","Data":"cdf6fabd7d4e7a5fb23facab32492159214062ec5a9751df07d0b1937b22c647"}
Sep 11 03:41:42.822487 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:42.822462    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-78fccb448b-r5mt8"
Sep 11 03:41:42.823522 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:42.823502    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-xd7gn" event={"ID":"f30e3e6e-ad79-489c-bffa-99b81296ac25","Type":"ContainerStarted","Data":"f21c19998d6b81ccf18ee067a82594e5e09d0494bf47e80f775bc657d665e8cf"}
Sep 11 03:41:42.824364 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:42.824346    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-78fccb448b-r5mt8"
Sep 11 03:41:42.843077 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:42.843037    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-78fccb448b-r5mt8" podStartSLOduration=1.5698082549999999 podStartE2EDuration="10.843022223s" totalImagesPullingTime="9.273213968s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 03:41:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:41:42.842531612 +0000 UTC m=+91.001017936" watchObservedRunningTime="2026-09-11 03:41:42.843022223 +0000 UTC m=+91.001508567"
Sep 11 03:41:43.830265 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:43.830231    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-qgz5l" event={"ID":"40db11ed-2328-4a59-9db8-6efeaf55b54b","Type":"ContainerStarted","Data":"89fb095eb50dea16fd911abf573f12d937f2b338d1e0a466c10c9a612dbca05a"}
Sep 11 03:41:43.832004 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:43.831974    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-xd7gn" event={"ID":"f30e3e6e-ad79-489c-bffa-99b81296ac25","Type":"ContainerStarted","Data":"c4f12e56888f0a2eb1a833ba66ba709e713ff33e75d7a17c7a793048633ed6e0"}
Sep 11 03:41:43.849258 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:43.849204    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-xd7gn" podStartSLOduration=2.423839679 podStartE2EDuration="3.849176319s" totalImagesPullingTime="1.42533664s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 03:41:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:41:43.84843176 +0000 UTC m=+92.006918083" watchObservedRunningTime="2026-09-11 03:41:43.849176319 +0000 UTC m=+92.007662642"
Sep 11 03:41:44.838374 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:44.838340    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-qgz5l" event={"ID":"40db11ed-2328-4a59-9db8-6efeaf55b54b","Type":"ContainerStarted","Data":"793322ac884027ef023b383708a7247d977acdbe6f5cb99b44d87ae5ba70448a"}
Sep 11 03:41:44.838794 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:44.838645    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-xd7gn"
Sep 11 03:41:44.842956 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:44.842933    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-xd7gn"
Sep 11 03:41:44.855576 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:44.855535    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-qgz5l" podStartSLOduration=34.955096077 podStartE2EDuration="36.855519699s" totalImagesPullingTime="1.900423622s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 03:41:08 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:41:44.855114845 +0000 UTC m=+93.013601169" watchObservedRunningTime="2026-09-11 03:41:44.855519699 +0000 UTC m=+93.014006024"
Sep 11 03:41:45.089695 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:45.089571    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-c6zjw"]
Sep 11 03:41:45.092733 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:45.092711    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-68bcd7f987-c6zjw"
Sep 11 03:41:45.095053 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:45.095032    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 11 03:41:45.095166 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:45.095067    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 11 03:41:45.095166 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:45.095086    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 11 03:41:45.095166 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:45.095071    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-m4ckq\""
Sep 11 03:41:45.095166 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:45.095130    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Sep 11 03:41:45.096015 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:45.095993    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Sep 11 03:41:45.101487 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:45.101466    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-c6zjw"]
Sep 11 03:41:45.180320 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:45.180290    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/a5f45b3b-f5ae-427c-b253-399d0dcd896d-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-c6zjw\" (UID: \"a5f45b3b-f5ae-427c-b253-399d0dcd896d\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-c6zjw"
Sep 11 03:41:45.180476 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:45.180373    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a5f45b3b-f5ae-427c-b253-399d0dcd896d-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-c6zjw\" (UID: \"a5f45b3b-f5ae-427c-b253-399d0dcd896d\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-c6zjw"
Sep 11 03:41:45.180476 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:45.180404    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8g7x\" (UniqueName: \"kubernetes.io/projected/a5f45b3b-f5ae-427c-b253-399d0dcd896d-kube-api-access-p8g7x\") pod \"prometheus-operator-68bcd7f987-c6zjw\" (UID: \"a5f45b3b-f5ae-427c-b253-399d0dcd896d\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-c6zjw"
Sep 11 03:41:45.180476 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:45.180464    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a5f45b3b-f5ae-427c-b253-399d0dcd896d-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-c6zjw\" (UID: \"a5f45b3b-f5ae-427c-b253-399d0dcd896d\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-c6zjw"
Sep 11 03:41:45.281302 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:45.281262    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8g7x\" (UniqueName: \"kubernetes.io/projected/a5f45b3b-f5ae-427c-b253-399d0dcd896d-kube-api-access-p8g7x\") pod \"prometheus-operator-68bcd7f987-c6zjw\" (UID: \"a5f45b3b-f5ae-427c-b253-399d0dcd896d\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-c6zjw"
Sep 11 03:41:45.281458 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:45.281325    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a5f45b3b-f5ae-427c-b253-399d0dcd896d-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-c6zjw\" (UID: \"a5f45b3b-f5ae-427c-b253-399d0dcd896d\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-c6zjw"
Sep 11 03:41:45.281458 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:45.281357    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/a5f45b3b-f5ae-427c-b253-399d0dcd896d-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-c6zjw\" (UID: \"a5f45b3b-f5ae-427c-b253-399d0dcd896d\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-c6zjw"
Sep 11 03:41:45.281458 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:45.281414    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a5f45b3b-f5ae-427c-b253-399d0dcd896d-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-c6zjw\" (UID: \"a5f45b3b-f5ae-427c-b253-399d0dcd896d\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-c6zjw"
Sep 11 03:41:45.282137 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:45.282103    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a5f45b3b-f5ae-427c-b253-399d0dcd896d-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-c6zjw\" (UID: \"a5f45b3b-f5ae-427c-b253-399d0dcd896d\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-c6zjw"
Sep 11 03:41:45.283791 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:45.283771    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a5f45b3b-f5ae-427c-b253-399d0dcd896d-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-c6zjw\" (UID: \"a5f45b3b-f5ae-427c-b253-399d0dcd896d\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-c6zjw"
Sep 11 03:41:45.283933 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:45.283916    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/a5f45b3b-f5ae-427c-b253-399d0dcd896d-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-c6zjw\" (UID: \"a5f45b3b-f5ae-427c-b253-399d0dcd896d\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-c6zjw"
Sep 11 03:41:45.289137 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:45.289118    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8g7x\" (UniqueName: \"kubernetes.io/projected/a5f45b3b-f5ae-427c-b253-399d0dcd896d-kube-api-access-p8g7x\") pod \"prometheus-operator-68bcd7f987-c6zjw\" (UID: \"a5f45b3b-f5ae-427c-b253-399d0dcd896d\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-c6zjw"
Sep 11 03:41:45.401792 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:45.401712    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-68bcd7f987-c6zjw"
Sep 11 03:41:45.670549 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:45.670519    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-c6zjw"]
Sep 11 03:41:45.673627 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:41:45.673598    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5f45b3b_f5ae_427c_b253_399d0dcd896d.slice/crio-085cad4d1c81f3dcf55987500ecbce77fe54535b1b99b0038171f70a70aebd91 WatchSource:0}: Error finding container 085cad4d1c81f3dcf55987500ecbce77fe54535b1b99b0038171f70a70aebd91: Status 404 returned error can't find the container with id 085cad4d1c81f3dcf55987500ecbce77fe54535b1b99b0038171f70a70aebd91
Sep 11 03:41:45.842782 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:45.842750    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-c6zjw" event={"ID":"a5f45b3b-f5ae-427c-b253-399d0dcd896d","Type":"ContainerStarted","Data":"085cad4d1c81f3dcf55987500ecbce77fe54535b1b99b0038171f70a70aebd91"}
Sep 11 03:41:47.851016 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:47.850985    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-c6zjw" event={"ID":"a5f45b3b-f5ae-427c-b253-399d0dcd896d","Type":"ContainerStarted","Data":"8102f9faa5f417d1fb1640fd6f52854cb64213f66ed0f2e02c89b908ea24a555"}
Sep 11 03:41:47.851386 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:47.851037    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-c6zjw" event={"ID":"a5f45b3b-f5ae-427c-b253-399d0dcd896d","Type":"ContainerStarted","Data":"572a90620ef0ddbbd08943b104639676aec9d1f67244711e7ee83fd0bf41572e"}
Sep 11 03:41:47.869303 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:47.869258    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-68bcd7f987-c6zjw" podStartSLOduration=1.416993237 podStartE2EDuration="2.869241548s" totalImagesPullingTime="1.452248311s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 03:41:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:41:47.868344621 +0000 UTC m=+96.026830944" watchObservedRunningTime="2026-09-11 03:41:47.869241548 +0000 UTC m=+96.027727870"
Sep 11 03:41:49.486428 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:49.486378    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-lq5nl"]
Sep 11 03:41:49.490953 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:49.490896    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-lq5nl"
Sep 11 03:41:49.493320 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:49.493295    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Sep 11 03:41:49.493436 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:49.493387    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-dtwwn\""
Sep 11 03:41:49.493631 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:49.493615    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Sep 11 03:41:49.495472 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:49.495454    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-gppcl"]
Sep 11 03:41:49.498969 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:49.498945    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-gppcl"
Sep 11 03:41:49.499069 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:49.499030    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-lq5nl"]
Sep 11 03:41:49.501060 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:49.501040    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-cv25j\""
Sep 11 03:41:49.501328 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:49.501305    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 11 03:41:49.501426 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:49.501405    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 11 03:41:49.502259 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:49.502240    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 11 03:41:49.615467 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:49.615438    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c8257ffa-bddf-478e-8b16-e246ad50e29a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-gppcl\" (UID: \"c8257ffa-bddf-478e-8b16-e246ad50e29a\") " pod="openshift-monitoring/node-exporter-gppcl"
Sep 11 03:41:49.615657 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:49.615479    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c8257ffa-bddf-478e-8b16-e246ad50e29a-sys\") pod \"node-exporter-gppcl\" (UID: \"c8257ffa-bddf-478e-8b16-e246ad50e29a\") " pod="openshift-monitoring/node-exporter-gppcl"
Sep 11 03:41:49.615657 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:49.615497    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c8257ffa-bddf-478e-8b16-e246ad50e29a-node-exporter-wtmp\") pod \"node-exporter-gppcl\" (UID: \"c8257ffa-bddf-478e-8b16-e246ad50e29a\") " pod="openshift-monitoring/node-exporter-gppcl"
Sep 11 03:41:49.615657 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:49.615518    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pnsm8\" (UniqueName: \"kubernetes.io/projected/c8257ffa-bddf-478e-8b16-e246ad50e29a-kube-api-access-pnsm8\") pod \"node-exporter-gppcl\" (UID: \"c8257ffa-bddf-478e-8b16-e246ad50e29a\") " pod="openshift-monitoring/node-exporter-gppcl"
Sep 11 03:41:49.615657 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:49.615537    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hw5wf\" (UniqueName: \"kubernetes.io/projected/4859b737-e190-44e9-ae1b-0812bbd1c927-kube-api-access-hw5wf\") pod \"openshift-state-metrics-67dcf8c979-lq5nl\" (UID: \"4859b737-e190-44e9-ae1b-0812bbd1c927\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-lq5nl"
Sep 11 03:41:49.615657 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:49.615558    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c8257ffa-bddf-478e-8b16-e246ad50e29a-node-exporter-textfile\") pod \"node-exporter-gppcl\" (UID: \"c8257ffa-bddf-478e-8b16-e246ad50e29a\") " pod="openshift-monitoring/node-exporter-gppcl"
Sep 11 03:41:49.615657 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:49.615575    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c8257ffa-bddf-478e-8b16-e246ad50e29a-metrics-client-ca\") pod \"node-exporter-gppcl\" (UID: \"c8257ffa-bddf-478e-8b16-e246ad50e29a\") " pod="openshift-monitoring/node-exporter-gppcl"
Sep 11 03:41:49.615657 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:49.615608    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/61f124d5-c24b-4150-b19a-ae500bce32fe-metrics-tls\") pod \"dns-default-pwk5f\" (UID: \"61f124d5-c24b-4150-b19a-ae500bce32fe\") " pod="openshift-dns/dns-default-pwk5f"
Sep 11 03:41:49.616016 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:49.615683    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4859b737-e190-44e9-ae1b-0812bbd1c927-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-lq5nl\" (UID: \"4859b737-e190-44e9-ae1b-0812bbd1c927\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-lq5nl"
Sep 11 03:41:49.616016 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:49.615732    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c8257ffa-bddf-478e-8b16-e246ad50e29a-root\") pod \"node-exporter-gppcl\" (UID: \"c8257ffa-bddf-478e-8b16-e246ad50e29a\") " pod="openshift-monitoring/node-exporter-gppcl"
Sep 11 03:41:49.616016 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:49.615790    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/4859b737-e190-44e9-ae1b-0812bbd1c927-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-lq5nl\" (UID: \"4859b737-e190-44e9-ae1b-0812bbd1c927\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-lq5nl"
Sep 11 03:41:49.616016 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:49.615824    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c8257ffa-bddf-478e-8b16-e246ad50e29a-node-exporter-accelerators-collector-config\") pod \"node-exporter-gppcl\" (UID: \"c8257ffa-bddf-478e-8b16-e246ad50e29a\") " pod="openshift-monitoring/node-exporter-gppcl"
Sep 11 03:41:49.616016 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:49.615859    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c8257ffa-bddf-478e-8b16-e246ad50e29a-node-exporter-tls\") pod \"node-exporter-gppcl\" (UID: \"c8257ffa-bddf-478e-8b16-e246ad50e29a\") " pod="openshift-monitoring/node-exporter-gppcl"
Sep 11 03:41:49.616016 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:49.615894    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4859b737-e190-44e9-ae1b-0812bbd1c927-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-lq5nl\" (UID: \"4859b737-e190-44e9-ae1b-0812bbd1c927\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-lq5nl"
Sep 11 03:41:49.617851 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:49.617818    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/61f124d5-c24b-4150-b19a-ae500bce32fe-metrics-tls\") pod \"dns-default-pwk5f\" (UID: \"61f124d5-c24b-4150-b19a-ae500bce32fe\") " pod="openshift-dns/dns-default-pwk5f"
Sep 11 03:41:49.716882 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:49.716841    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4859b737-e190-44e9-ae1b-0812bbd1c927-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-lq5nl\" (UID: \"4859b737-e190-44e9-ae1b-0812bbd1c927\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-lq5nl"
Sep 11 03:41:49.717074 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:49.716894    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c8257ffa-bddf-478e-8b16-e246ad50e29a-root\") pod \"node-exporter-gppcl\" (UID: \"c8257ffa-bddf-478e-8b16-e246ad50e29a\") " pod="openshift-monitoring/node-exporter-gppcl"
Sep 11 03:41:49.717074 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:49.716931    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/4859b737-e190-44e9-ae1b-0812bbd1c927-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-lq5nl\" (UID: \"4859b737-e190-44e9-ae1b-0812bbd1c927\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-lq5nl"
Sep 11 03:41:49.717074 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:49.716956    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c8257ffa-bddf-478e-8b16-e246ad50e29a-node-exporter-accelerators-collector-config\") pod \"node-exporter-gppcl\" (UID: \"c8257ffa-bddf-478e-8b16-e246ad50e29a\") " pod="openshift-monitoring/node-exporter-gppcl"
Sep 11 03:41:49.717074 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:49.716981    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c8257ffa-bddf-478e-8b16-e246ad50e29a-node-exporter-tls\") pod \"node-exporter-gppcl\" (UID: \"c8257ffa-bddf-478e-8b16-e246ad50e29a\") " pod="openshift-monitoring/node-exporter-gppcl"
Sep 11 03:41:49.717074 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:49.717002    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4859b737-e190-44e9-ae1b-0812bbd1c927-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-lq5nl\" (UID: \"4859b737-e190-44e9-ae1b-0812bbd1c927\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-lq5nl"
Sep 11 03:41:49.717335 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:49.717070    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c8257ffa-bddf-478e-8b16-e246ad50e29a-root\") pod \"node-exporter-gppcl\" (UID: \"c8257ffa-bddf-478e-8b16-e246ad50e29a\") " pod="openshift-monitoring/node-exporter-gppcl"
Sep 11 03:41:49.717335 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:41:49.717079    2580 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found
Sep 11 03:41:49.717335 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:49.717109    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c8257ffa-bddf-478e-8b16-e246ad50e29a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-gppcl\" (UID: \"c8257ffa-bddf-478e-8b16-e246ad50e29a\") " pod="openshift-monitoring/node-exporter-gppcl"
Sep 11 03:41:49.717335 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:41:49.717168    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4859b737-e190-44e9-ae1b-0812bbd1c927-openshift-state-metrics-tls podName:4859b737-e190-44e9-ae1b-0812bbd1c927 nodeName:}" failed. No retries permitted until 2026-09-11 03:41:50.217145502 +0000 UTC m=+98.375631818 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/4859b737-e190-44e9-ae1b-0812bbd1c927-openshift-state-metrics-tls") pod "openshift-state-metrics-67dcf8c979-lq5nl" (UID: "4859b737-e190-44e9-ae1b-0812bbd1c927") : secret "openshift-state-metrics-tls" not found
Sep 11 03:41:49.717335 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:49.717189    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c8257ffa-bddf-478e-8b16-e246ad50e29a-sys\") pod \"node-exporter-gppcl\" (UID: \"c8257ffa-bddf-478e-8b16-e246ad50e29a\") " pod="openshift-monitoring/node-exporter-gppcl"
Sep 11 03:41:49.717335 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:49.717265    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c8257ffa-bddf-478e-8b16-e246ad50e29a-node-exporter-wtmp\") pod \"node-exporter-gppcl\" (UID: \"c8257ffa-bddf-478e-8b16-e246ad50e29a\") " pod="openshift-monitoring/node-exporter-gppcl"
Sep 11 03:41:49.717335 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:49.717300    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnsm8\" (UniqueName: \"kubernetes.io/projected/c8257ffa-bddf-478e-8b16-e246ad50e29a-kube-api-access-pnsm8\") pod \"node-exporter-gppcl\" (UID: \"c8257ffa-bddf-478e-8b16-e246ad50e29a\") " pod="openshift-monitoring/node-exporter-gppcl"
Sep 11 03:41:49.717335 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:49.717331    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hw5wf\" (UniqueName: \"kubernetes.io/projected/4859b737-e190-44e9-ae1b-0812bbd1c927-kube-api-access-hw5wf\") pod \"openshift-state-metrics-67dcf8c979-lq5nl\" (UID: \"4859b737-e190-44e9-ae1b-0812bbd1c927\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-lq5nl"
Sep 11 03:41:49.717764 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:49.717358    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c8257ffa-bddf-478e-8b16-e246ad50e29a-node-exporter-textfile\") pod \"node-exporter-gppcl\" (UID: \"c8257ffa-bddf-478e-8b16-e246ad50e29a\") " pod="openshift-monitoring/node-exporter-gppcl"
Sep 11 03:41:49.717764 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:49.717384    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c8257ffa-bddf-478e-8b16-e246ad50e29a-metrics-client-ca\") pod \"node-exporter-gppcl\" (UID: \"c8257ffa-bddf-478e-8b16-e246ad50e29a\") " pod="openshift-monitoring/node-exporter-gppcl"
Sep 11 03:41:49.717764 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:49.717736    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c8257ffa-bddf-478e-8b16-e246ad50e29a-node-exporter-accelerators-collector-config\") pod \"node-exporter-gppcl\" (UID: \"c8257ffa-bddf-478e-8b16-e246ad50e29a\") " pod="openshift-monitoring/node-exporter-gppcl"
Sep 11 03:41:49.717764 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:49.717757    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c8257ffa-bddf-478e-8b16-e246ad50e29a-sys\") pod \"node-exporter-gppcl\" (UID: \"c8257ffa-bddf-478e-8b16-e246ad50e29a\") " pod="openshift-monitoring/node-exporter-gppcl"
Sep 11 03:41:49.717982 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:49.717881    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c8257ffa-bddf-478e-8b16-e246ad50e29a-node-exporter-wtmp\") pod \"node-exporter-gppcl\" (UID: \"c8257ffa-bddf-478e-8b16-e246ad50e29a\") " pod="openshift-monitoring/node-exporter-gppcl"
Sep 11 03:41:49.717982 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:49.717888    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c8257ffa-bddf-478e-8b16-e246ad50e29a-metrics-client-ca\") pod \"node-exporter-gppcl\" (UID: \"c8257ffa-bddf-478e-8b16-e246ad50e29a\") " pod="openshift-monitoring/node-exporter-gppcl"
Sep 11 03:41:49.717982 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:49.717976    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4859b737-e190-44e9-ae1b-0812bbd1c927-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-lq5nl\" (UID: \"4859b737-e190-44e9-ae1b-0812bbd1c927\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-lq5nl"
Sep 11 03:41:49.718156 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:49.718131    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c8257ffa-bddf-478e-8b16-e246ad50e29a-node-exporter-textfile\") pod \"node-exporter-gppcl\" (UID: \"c8257ffa-bddf-478e-8b16-e246ad50e29a\") " pod="openshift-monitoring/node-exporter-gppcl"
Sep 11 03:41:49.719646 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:49.719612    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c8257ffa-bddf-478e-8b16-e246ad50e29a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-gppcl\" (UID: \"c8257ffa-bddf-478e-8b16-e246ad50e29a\") " pod="openshift-monitoring/node-exporter-gppcl"
Sep 11 03:41:49.719840 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:49.719824    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c8257ffa-bddf-478e-8b16-e246ad50e29a-node-exporter-tls\") pod \"node-exporter-gppcl\" (UID: \"c8257ffa-bddf-478e-8b16-e246ad50e29a\") " pod="openshift-monitoring/node-exporter-gppcl"
Sep 11 03:41:49.719888 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:49.719821    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4859b737-e190-44e9-ae1b-0812bbd1c927-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-lq5nl\" (UID: \"4859b737-e190-44e9-ae1b-0812bbd1c927\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-lq5nl"
Sep 11 03:41:49.725213 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:49.725179    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hw5wf\" (UniqueName: \"kubernetes.io/projected/4859b737-e190-44e9-ae1b-0812bbd1c927-kube-api-access-hw5wf\") pod \"openshift-state-metrics-67dcf8c979-lq5nl\" (UID: \"4859b737-e190-44e9-ae1b-0812bbd1c927\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-lq5nl"
Sep 11 03:41:49.725336 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:49.725226    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pnsm8\" (UniqueName: \"kubernetes.io/projected/c8257ffa-bddf-478e-8b16-e246ad50e29a-kube-api-access-pnsm8\") pod \"node-exporter-gppcl\" (UID: \"c8257ffa-bddf-478e-8b16-e246ad50e29a\") " pod="openshift-monitoring/node-exporter-gppcl"
Sep 11 03:41:49.807256 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:49.807162    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-gppcl"
Sep 11 03:41:49.818960 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:41:49.818931    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8257ffa_bddf_478e_8b16_e246ad50e29a.slice/crio-542867620a14ac66be2e78e52b4c37157ac4af4600fdbcb2c03d5ab78601acc8 WatchSource:0}: Error finding container 542867620a14ac66be2e78e52b4c37157ac4af4600fdbcb2c03d5ab78601acc8: Status 404 returned error can't find the container with id 542867620a14ac66be2e78e52b4c37157ac4af4600fdbcb2c03d5ab78601acc8
Sep 11 03:41:49.858519 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:49.858482    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gppcl" event={"ID":"c8257ffa-bddf-478e-8b16-e246ad50e29a","Type":"ContainerStarted","Data":"542867620a14ac66be2e78e52b4c37157ac4af4600fdbcb2c03d5ab78601acc8"}
Sep 11 03:41:49.905163 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:49.905130    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-qlqn4\""
Sep 11 03:41:49.913064 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:49.913042    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-pwk5f"
Sep 11 03:41:50.180071 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:50.180040    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-pwk5f"]
Sep 11 03:41:50.182593 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:41:50.182569    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61f124d5_c24b_4150_b19a_ae500bce32fe.slice/crio-91e88b507717fea10e1800489a33d0c09586a213d9d9469425f85f7e71b4831d WatchSource:0}: Error finding container 91e88b507717fea10e1800489a33d0c09586a213d9d9469425f85f7e71b4831d: Status 404 returned error can't find the container with id 91e88b507717fea10e1800489a33d0c09586a213d9d9469425f85f7e71b4831d
Sep 11 03:41:50.221069 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:50.221041    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/4859b737-e190-44e9-ae1b-0812bbd1c927-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-lq5nl\" (UID: \"4859b737-e190-44e9-ae1b-0812bbd1c927\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-lq5nl"
Sep 11 03:41:50.223485 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:50.223463    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/4859b737-e190-44e9-ae1b-0812bbd1c927-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-lq5nl\" (UID: \"4859b737-e190-44e9-ae1b-0812bbd1c927\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-lq5nl"
Sep 11 03:41:50.400864 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:50.400818    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-lq5nl"
Sep 11 03:41:50.684935 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:50.684882    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-lq5nl"]
Sep 11 03:41:50.864137 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:50.864050    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-pwk5f" event={"ID":"61f124d5-c24b-4150-b19a-ae500bce32fe","Type":"ContainerStarted","Data":"91e88b507717fea10e1800489a33d0c09586a213d9d9469425f85f7e71b4831d"}
Sep 11 03:41:50.887071 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:41:50.887026    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4859b737_e190_44e9_ae1b_0812bbd1c927.slice/crio-19c87ac11a83ba2717a59d0d899701bbc59130af8ccbb258ff12741dde508f44 WatchSource:0}: Error finding container 19c87ac11a83ba2717a59d0d899701bbc59130af8ccbb258ff12741dde508f44: Status 404 returned error can't find the container with id 19c87ac11a83ba2717a59d0d899701bbc59130af8ccbb258ff12741dde508f44
Sep 11 03:41:51.871372 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:51.871335    2580 generic.go:356] "Generic (PLEG): container finished" podID="c8257ffa-bddf-478e-8b16-e246ad50e29a" containerID="bc45c74f5bd265d3ba5c38b117362a69a0e9e5d3e5234e1c165144b2a2347524" exitCode=0
Sep 11 03:41:51.871773 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:51.871460    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gppcl" event={"ID":"c8257ffa-bddf-478e-8b16-e246ad50e29a","Type":"ContainerDied","Data":"bc45c74f5bd265d3ba5c38b117362a69a0e9e5d3e5234e1c165144b2a2347524"}
Sep 11 03:41:51.874657 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:51.874556    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-pwk5f" event={"ID":"61f124d5-c24b-4150-b19a-ae500bce32fe","Type":"ContainerStarted","Data":"5effcc8a4a87dcbe6660da7bb32f2c6dc133200a00bc2aabda79b32743a8e235"}
Sep 11 03:41:51.879395 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:51.879275    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-lq5nl" event={"ID":"4859b737-e190-44e9-ae1b-0812bbd1c927","Type":"ContainerStarted","Data":"d22b61c34caf58c29fc8f8279c2104386c84de5c064096f5230a87c924eb8a09"}
Sep 11 03:41:51.879395 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:51.879334    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-lq5nl" event={"ID":"4859b737-e190-44e9-ae1b-0812bbd1c927","Type":"ContainerStarted","Data":"31b5d6038095bc949a1facdd1c990de62a863a20f6e7fc7012df2d913c97bdcd"}
Sep 11 03:41:51.879395 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:51.879360    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-lq5nl" event={"ID":"4859b737-e190-44e9-ae1b-0812bbd1c927","Type":"ContainerStarted","Data":"19c87ac11a83ba2717a59d0d899701bbc59130af8ccbb258ff12741dde508f44"}
Sep 11 03:41:52.157931 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:52.157897    2580 patch_prober.go:28] interesting pod/image-registry-76b857c477-qw9fr container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 11 03:41:52.158073 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:52.157953    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-76b857c477-qw9fr" podUID="a0327861-5035-4a54-8d03-79c55efdfaa1" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 11 03:41:52.513448 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:52.513413    2580 patch_prober.go:28] interesting pod/image-registry-6dc5b6c9b6-rt5ml container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 11 03:41:52.513629 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:52.513502    2580 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-6dc5b6c9b6-rt5ml" podUID="dc23fd71-e4f6-4e97-b9c6-b5d200a69802" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 11 03:41:52.884853 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:52.884768    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gppcl" event={"ID":"c8257ffa-bddf-478e-8b16-e246ad50e29a","Type":"ContainerStarted","Data":"d368267db207e4f472c38616eb583bb53ba6320c516ff2f8567ff0bcd1810bb6"}
Sep 11 03:41:52.884853 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:52.884819    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gppcl" event={"ID":"c8257ffa-bddf-478e-8b16-e246ad50e29a","Type":"ContainerStarted","Data":"e3852aea4b195d02eebed70d3a90934cbd73ca4d411ce2b8658876e268ba2b22"}
Sep 11 03:41:52.886603 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:52.886576    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-pwk5f" event={"ID":"61f124d5-c24b-4150-b19a-ae500bce32fe","Type":"ContainerStarted","Data":"ebe7e10fd6b95f5b71d5e720b52e73b9aeb040d4b072a4f049267d4896c80749"}
Sep 11 03:41:52.886751 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:52.886636    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-pwk5f"
Sep 11 03:41:52.888638 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:52.888614    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-lq5nl" event={"ID":"4859b737-e190-44e9-ae1b-0812bbd1c927","Type":"ContainerStarted","Data":"6f058bac7d7a4a4fde2cbeaff6534c79cacd62d881e2a364de66b14c5cb1b104"}
Sep 11 03:41:52.901608 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:52.901557    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-gppcl" podStartSLOduration=2.704577111 podStartE2EDuration="3.901538337s" totalImagesPullingTime="1.122959182s" totalInitContainerRuntime="74.002044ms" isStatefulPod=true podCreationTimestamp="2026-09-11 03:41:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:41:52.901283348 +0000 UTC m=+101.059769673" watchObservedRunningTime="2026-09-11 03:41:52.901538337 +0000 UTC m=+101.060024663"
Sep 11 03:41:52.919410 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:52.919346    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-pwk5f" podStartSLOduration=66.342136836 podStartE2EDuration="1m7.919323597s" totalImagesPullingTime="1.577186761s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 03:40:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:41:52.91744262 +0000 UTC m=+101.075928944" watchObservedRunningTime="2026-09-11 03:41:52.919323597 +0000 UTC m=+101.077809922"
Sep 11 03:41:52.936183 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:52.936130    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-lq5nl" podStartSLOduration=2.590796932 podStartE2EDuration="3.936114481s" totalImagesPullingTime="1.345317549s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 03:41:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:41:52.934844247 +0000 UTC m=+101.093330570" watchObservedRunningTime="2026-09-11 03:41:52.936114481 +0000 UTC m=+101.094600803"
Sep 11 03:41:54.193027 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:54.192997    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-g4xkc"]
Sep 11 03:41:54.196271 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:54.196251    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-g4xkc"
Sep 11 03:41:54.198561 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:54.198537    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-9jfl2\""
Sep 11 03:41:54.198686 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:54.198544    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Sep 11 03:41:54.208239 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:54.207439    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-g4xkc"]
Sep 11 03:41:54.360794 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:54.360759    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/065b6bbb-a5f5-4602-9f4b-fc191f92a4bd-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-g4xkc\" (UID: \"065b6bbb-a5f5-4602-9f4b-fc191f92a4bd\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-g4xkc"
Sep 11 03:41:54.360952 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:54.360800    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pr2fz\" (UniqueName: \"kubernetes.io/projected/065b6bbb-a5f5-4602-9f4b-fc191f92a4bd-kube-api-access-pr2fz\") pod \"monitoring-plugin-5d85778b9d-g4xkc\" (UID: \"065b6bbb-a5f5-4602-9f4b-fc191f92a4bd\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-g4xkc"
Sep 11 03:41:54.461300 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:54.461214    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/065b6bbb-a5f5-4602-9f4b-fc191f92a4bd-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-g4xkc\" (UID: \"065b6bbb-a5f5-4602-9f4b-fc191f92a4bd\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-g4xkc"
Sep 11 03:41:54.461300 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:54.461259    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pr2fz\" (UniqueName: \"kubernetes.io/projected/065b6bbb-a5f5-4602-9f4b-fc191f92a4bd-kube-api-access-pr2fz\") pod \"monitoring-plugin-5d85778b9d-g4xkc\" (UID: \"065b6bbb-a5f5-4602-9f4b-fc191f92a4bd\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-g4xkc"
Sep 11 03:41:54.461475 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:41:54.461386    2580 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found
Sep 11 03:41:54.461475 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:41:54.461464    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/065b6bbb-a5f5-4602-9f4b-fc191f92a4bd-monitoring-plugin-cert podName:065b6bbb-a5f5-4602-9f4b-fc191f92a4bd nodeName:}" failed. No retries permitted until 2026-09-11 03:41:54.961441773 +0000 UTC m=+103.119928146 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/065b6bbb-a5f5-4602-9f4b-fc191f92a4bd-monitoring-plugin-cert") pod "monitoring-plugin-5d85778b9d-g4xkc" (UID: "065b6bbb-a5f5-4602-9f4b-fc191f92a4bd") : secret "monitoring-plugin-cert" not found
Sep 11 03:41:54.470078 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:54.470047    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pr2fz\" (UniqueName: \"kubernetes.io/projected/065b6bbb-a5f5-4602-9f4b-fc191f92a4bd-kube-api-access-pr2fz\") pod \"monitoring-plugin-5d85778b9d-g4xkc\" (UID: \"065b6bbb-a5f5-4602-9f4b-fc191f92a4bd\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-g4xkc"
Sep 11 03:41:54.670962 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:54.670924    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-67577cf4bb-9pb7v"]
Sep 11 03:41:54.674944 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:54.674916    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-67577cf4bb-9pb7v"
Sep 11 03:41:54.677450 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:54.677426    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Sep 11 03:41:54.677569 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:54.677496    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Sep 11 03:41:54.677569 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:54.677496    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Sep 11 03:41:54.677744 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:54.677730    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Sep 11 03:41:54.677858 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:54.677834    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Sep 11 03:41:54.677962 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:54.677877    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-tf496\""
Sep 11 03:41:54.682221 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:54.682197    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Sep 11 03:41:54.684758 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:54.684738    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-67577cf4bb-9pb7v"]
Sep 11 03:41:54.758425 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:54.758347    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-f7hng"
Sep 11 03:41:54.764765 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:54.764742    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/0f67bb82-1dc5-418b-8778-b43028862742-telemeter-client-tls\") pod \"telemeter-client-67577cf4bb-9pb7v\" (UID: \"0f67bb82-1dc5-418b-8778-b43028862742\") " pod="openshift-monitoring/telemeter-client-67577cf4bb-9pb7v"
Sep 11 03:41:54.764853 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:54.764773    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0f67bb82-1dc5-418b-8778-b43028862742-telemeter-trusted-ca-bundle\") pod \"telemeter-client-67577cf4bb-9pb7v\" (UID: \"0f67bb82-1dc5-418b-8778-b43028862742\") " pod="openshift-monitoring/telemeter-client-67577cf4bb-9pb7v"
Sep 11 03:41:54.764853 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:54.764805    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0f67bb82-1dc5-418b-8778-b43028862742-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-67577cf4bb-9pb7v\" (UID: \"0f67bb82-1dc5-418b-8778-b43028862742\") " pod="openshift-monitoring/telemeter-client-67577cf4bb-9pb7v"
Sep 11 03:41:54.764922 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:54.764865    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/0f67bb82-1dc5-418b-8778-b43028862742-federate-client-tls\") pod \"telemeter-client-67577cf4bb-9pb7v\" (UID: \"0f67bb82-1dc5-418b-8778-b43028862742\") " pod="openshift-monitoring/telemeter-client-67577cf4bb-9pb7v"
Sep 11 03:41:54.764960 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:54.764934    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0f67bb82-1dc5-418b-8778-b43028862742-serving-certs-ca-bundle\") pod \"telemeter-client-67577cf4bb-9pb7v\" (UID: \"0f67bb82-1dc5-418b-8778-b43028862742\") " pod="openshift-monitoring/telemeter-client-67577cf4bb-9pb7v"
Sep 11 03:41:54.764960 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:54.764951    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0f67bb82-1dc5-418b-8778-b43028862742-metrics-client-ca\") pod \"telemeter-client-67577cf4bb-9pb7v\" (UID: \"0f67bb82-1dc5-418b-8778-b43028862742\") " pod="openshift-monitoring/telemeter-client-67577cf4bb-9pb7v"
Sep 11 03:41:54.765032 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:54.764977    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nk65r\" (UniqueName: \"kubernetes.io/projected/0f67bb82-1dc5-418b-8778-b43028862742-kube-api-access-nk65r\") pod \"telemeter-client-67577cf4bb-9pb7v\" (UID: \"0f67bb82-1dc5-418b-8778-b43028862742\") " pod="openshift-monitoring/telemeter-client-67577cf4bb-9pb7v"
Sep 11 03:41:54.765032 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:54.765015    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/0f67bb82-1dc5-418b-8778-b43028862742-secret-telemeter-client\") pod \"telemeter-client-67577cf4bb-9pb7v\" (UID: \"0f67bb82-1dc5-418b-8778-b43028862742\") " pod="openshift-monitoring/telemeter-client-67577cf4bb-9pb7v"
Sep 11 03:41:54.796762 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:54.796729    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6dc5b6c9b6-rt5ml"
Sep 11 03:41:54.865612 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:54.865580    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0f67bb82-1dc5-418b-8778-b43028862742-serving-certs-ca-bundle\") pod \"telemeter-client-67577cf4bb-9pb7v\" (UID: \"0f67bb82-1dc5-418b-8778-b43028862742\") " pod="openshift-monitoring/telemeter-client-67577cf4bb-9pb7v"
Sep 11 03:41:54.865612 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:54.865613    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0f67bb82-1dc5-418b-8778-b43028862742-metrics-client-ca\") pod \"telemeter-client-67577cf4bb-9pb7v\" (UID: \"0f67bb82-1dc5-418b-8778-b43028862742\") " pod="openshift-monitoring/telemeter-client-67577cf4bb-9pb7v"
Sep 11 03:41:54.865834 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:54.865638    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nk65r\" (UniqueName: \"kubernetes.io/projected/0f67bb82-1dc5-418b-8778-b43028862742-kube-api-access-nk65r\") pod \"telemeter-client-67577cf4bb-9pb7v\" (UID: \"0f67bb82-1dc5-418b-8778-b43028862742\") " pod="openshift-monitoring/telemeter-client-67577cf4bb-9pb7v"
Sep 11 03:41:54.865834 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:54.865710    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/0f67bb82-1dc5-418b-8778-b43028862742-secret-telemeter-client\") pod \"telemeter-client-67577cf4bb-9pb7v\" (UID: \"0f67bb82-1dc5-418b-8778-b43028862742\") " pod="openshift-monitoring/telemeter-client-67577cf4bb-9pb7v"
Sep 11 03:41:54.865834 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:54.865749    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/0f67bb82-1dc5-418b-8778-b43028862742-telemeter-client-tls\") pod \"telemeter-client-67577cf4bb-9pb7v\" (UID: \"0f67bb82-1dc5-418b-8778-b43028862742\") " pod="openshift-monitoring/telemeter-client-67577cf4bb-9pb7v"
Sep 11 03:41:54.865834 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:54.865773    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0f67bb82-1dc5-418b-8778-b43028862742-telemeter-trusted-ca-bundle\") pod \"telemeter-client-67577cf4bb-9pb7v\" (UID: \"0f67bb82-1dc5-418b-8778-b43028862742\") " pod="openshift-monitoring/telemeter-client-67577cf4bb-9pb7v"
Sep 11 03:41:54.865834 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:54.865798    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0f67bb82-1dc5-418b-8778-b43028862742-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-67577cf4bb-9pb7v\" (UID: \"0f67bb82-1dc5-418b-8778-b43028862742\") " pod="openshift-monitoring/telemeter-client-67577cf4bb-9pb7v"
Sep 11 03:41:54.865834 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:54.865828    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/0f67bb82-1dc5-418b-8778-b43028862742-federate-client-tls\") pod \"telemeter-client-67577cf4bb-9pb7v\" (UID: \"0f67bb82-1dc5-418b-8778-b43028862742\") " pod="openshift-monitoring/telemeter-client-67577cf4bb-9pb7v"
Sep 11 03:41:54.866523 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:54.866444    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0f67bb82-1dc5-418b-8778-b43028862742-serving-certs-ca-bundle\") pod \"telemeter-client-67577cf4bb-9pb7v\" (UID: \"0f67bb82-1dc5-418b-8778-b43028862742\") " pod="openshift-monitoring/telemeter-client-67577cf4bb-9pb7v"
Sep 11 03:41:54.866523 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:54.866462    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0f67bb82-1dc5-418b-8778-b43028862742-metrics-client-ca\") pod \"telemeter-client-67577cf4bb-9pb7v\" (UID: \"0f67bb82-1dc5-418b-8778-b43028862742\") " pod="openshift-monitoring/telemeter-client-67577cf4bb-9pb7v"
Sep 11 03:41:54.866774 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:54.866753    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0f67bb82-1dc5-418b-8778-b43028862742-telemeter-trusted-ca-bundle\") pod \"telemeter-client-67577cf4bb-9pb7v\" (UID: \"0f67bb82-1dc5-418b-8778-b43028862742\") " pod="openshift-monitoring/telemeter-client-67577cf4bb-9pb7v"
Sep 11 03:41:54.868674 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:54.868644    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/0f67bb82-1dc5-418b-8778-b43028862742-secret-telemeter-client\") pod \"telemeter-client-67577cf4bb-9pb7v\" (UID: \"0f67bb82-1dc5-418b-8778-b43028862742\") " pod="openshift-monitoring/telemeter-client-67577cf4bb-9pb7v"
Sep 11 03:41:54.868857 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:54.868840    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/0f67bb82-1dc5-418b-8778-b43028862742-federate-client-tls\") pod \"telemeter-client-67577cf4bb-9pb7v\" (UID: \"0f67bb82-1dc5-418b-8778-b43028862742\") " pod="openshift-monitoring/telemeter-client-67577cf4bb-9pb7v"
Sep 11 03:41:54.868912 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:54.868896    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/0f67bb82-1dc5-418b-8778-b43028862742-telemeter-client-tls\") pod \"telemeter-client-67577cf4bb-9pb7v\" (UID: \"0f67bb82-1dc5-418b-8778-b43028862742\") " pod="openshift-monitoring/telemeter-client-67577cf4bb-9pb7v"
Sep 11 03:41:54.869058 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:54.869042    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0f67bb82-1dc5-418b-8778-b43028862742-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-67577cf4bb-9pb7v\" (UID: \"0f67bb82-1dc5-418b-8778-b43028862742\") " pod="openshift-monitoring/telemeter-client-67577cf4bb-9pb7v"
Sep 11 03:41:54.872980 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:54.872958    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nk65r\" (UniqueName: \"kubernetes.io/projected/0f67bb82-1dc5-418b-8778-b43028862742-kube-api-access-nk65r\") pod \"telemeter-client-67577cf4bb-9pb7v\" (UID: \"0f67bb82-1dc5-418b-8778-b43028862742\") " pod="openshift-monitoring/telemeter-client-67577cf4bb-9pb7v"
Sep 11 03:41:54.966724 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:54.966653    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/065b6bbb-a5f5-4602-9f4b-fc191f92a4bd-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-g4xkc\" (UID: \"065b6bbb-a5f5-4602-9f4b-fc191f92a4bd\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-g4xkc"
Sep 11 03:41:54.969158 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:54.969119    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/065b6bbb-a5f5-4602-9f4b-fc191f92a4bd-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-g4xkc\" (UID: \"065b6bbb-a5f5-4602-9f4b-fc191f92a4bd\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-g4xkc"
Sep 11 03:41:54.984543 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:54.984512    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-67577cf4bb-9pb7v"
Sep 11 03:41:55.111073 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:55.111037    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-g4xkc"
Sep 11 03:41:55.258949 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:55.258916    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-67577cf4bb-9pb7v"]
Sep 11 03:41:55.262713 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:41:55.262683    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f67bb82_1dc5_418b_8778_b43028862742.slice/crio-27aad6356784ef1be09317eeb03104909677dfd31458d9d715148b0b785210fb WatchSource:0}: Error finding container 27aad6356784ef1be09317eeb03104909677dfd31458d9d715148b0b785210fb: Status 404 returned error can't find the container with id 27aad6356784ef1be09317eeb03104909677dfd31458d9d715148b0b785210fb
Sep 11 03:41:55.382183 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:55.382060    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-g4xkc"]
Sep 11 03:41:55.386068 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:41:55.386034    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod065b6bbb_a5f5_4602_9f4b_fc191f92a4bd.slice/crio-06069ede54a4092905ff3d31be3557d9020f37c1875aa26b8f09f68010bb77fe WatchSource:0}: Error finding container 06069ede54a4092905ff3d31be3557d9020f37c1875aa26b8f09f68010bb77fe: Status 404 returned error can't find the container with id 06069ede54a4092905ff3d31be3557d9020f37c1875aa26b8f09f68010bb77fe
Sep 11 03:41:55.902151 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:55.902115    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-g4xkc" event={"ID":"065b6bbb-a5f5-4602-9f4b-fc191f92a4bd","Type":"ContainerStarted","Data":"06069ede54a4092905ff3d31be3557d9020f37c1875aa26b8f09f68010bb77fe"}
Sep 11 03:41:55.903446 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:55.903416    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-67577cf4bb-9pb7v" event={"ID":"0f67bb82-1dc5-418b-8778-b43028862742","Type":"ContainerStarted","Data":"27aad6356784ef1be09317eeb03104909677dfd31458d9d715148b0b785210fb"}
Sep 11 03:41:57.170982 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:57.170914    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-76b857c477-qw9fr" podUID="a0327861-5035-4a54-8d03-79c55efdfaa1" containerName="registry" containerID="cri-o://5835cba5c871c03d3a8f029bbde030b633c83b55db336581b7ea3efaca31de06" gracePeriod=30
Sep 11 03:41:57.889737 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:57.889713    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-76b857c477-qw9fr"
Sep 11 03:41:57.915047 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:57.914922    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-g4xkc" event={"ID":"065b6bbb-a5f5-4602-9f4b-fc191f92a4bd","Type":"ContainerStarted","Data":"cf7f5efaa7cb401879aaa62f3469ae0afee2d51bb1e0c9bb041ebe93cceadf0b"}
Sep 11 03:41:57.915191 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:57.915081    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-g4xkc"
Sep 11 03:41:57.917326 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:57.917253    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-67577cf4bb-9pb7v" event={"ID":"0f67bb82-1dc5-418b-8778-b43028862742","Type":"ContainerStarted","Data":"a5f95ade9c6b88d319a453dc9a0b00c2d522bc8550a1fc55ae78003179fbe471"}
Sep 11 03:41:57.918806 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:57.918781    2580 generic.go:356] "Generic (PLEG): container finished" podID="a0327861-5035-4a54-8d03-79c55efdfaa1" containerID="5835cba5c871c03d3a8f029bbde030b633c83b55db336581b7ea3efaca31de06" exitCode=0
Sep 11 03:41:57.918922 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:57.918824    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-76b857c477-qw9fr"
Sep 11 03:41:57.918922 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:57.918840    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-76b857c477-qw9fr" event={"ID":"a0327861-5035-4a54-8d03-79c55efdfaa1","Type":"ContainerDied","Data":"5835cba5c871c03d3a8f029bbde030b633c83b55db336581b7ea3efaca31de06"}
Sep 11 03:41:57.918922 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:57.918877    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-76b857c477-qw9fr" event={"ID":"a0327861-5035-4a54-8d03-79c55efdfaa1","Type":"ContainerDied","Data":"c81573126548418291d9c1d4b11113f72076655cee992f1c8d7fdb6e30c3caab"}
Sep 11 03:41:57.918922 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:57.918903    2580 scope.go:117] "RemoveContainer" containerID="5835cba5c871c03d3a8f029bbde030b633c83b55db336581b7ea3efaca31de06"
Sep 11 03:41:57.921099 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:57.921081    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-g4xkc"
Sep 11 03:41:57.929887 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:57.929849    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-g4xkc" podStartSLOduration=1.538500068 podStartE2EDuration="3.929835917s" totalImagesPullingTime="2.391335849s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 03:41:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:41:57.92873229 +0000 UTC m=+106.087218613" watchObservedRunningTime="2026-09-11 03:41:57.929835917 +0000 UTC m=+106.088322240"
Sep 11 03:41:57.936402 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:57.936386    2580 scope.go:117] "RemoveContainer" containerID="5835cba5c871c03d3a8f029bbde030b633c83b55db336581b7ea3efaca31de06"
Sep 11 03:41:57.936737 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:41:57.936707    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5835cba5c871c03d3a8f029bbde030b633c83b55db336581b7ea3efaca31de06\": container with ID starting with 5835cba5c871c03d3a8f029bbde030b633c83b55db336581b7ea3efaca31de06 not found: ID does not exist" containerID="5835cba5c871c03d3a8f029bbde030b633c83b55db336581b7ea3efaca31de06"
Sep 11 03:41:57.936808 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:57.936750    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5835cba5c871c03d3a8f029bbde030b633c83b55db336581b7ea3efaca31de06"} err="failed to get container status \"5835cba5c871c03d3a8f029bbde030b633c83b55db336581b7ea3efaca31de06\": rpc error: code = NotFound desc = could not find container \"5835cba5c871c03d3a8f029bbde030b633c83b55db336581b7ea3efaca31de06\": container with ID starting with 5835cba5c871c03d3a8f029bbde030b633c83b55db336581b7ea3efaca31de06 not found: ID does not exist"
Sep 11 03:41:57.994445 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:57.994399    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a0327861-5035-4a54-8d03-79c55efdfaa1-registry-certificates\") pod \"a0327861-5035-4a54-8d03-79c55efdfaa1\" (UID: \"a0327861-5035-4a54-8d03-79c55efdfaa1\") "
Sep 11 03:41:57.994445 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:57.994450    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a0327861-5035-4a54-8d03-79c55efdfaa1-installation-pull-secrets\") pod \"a0327861-5035-4a54-8d03-79c55efdfaa1\" (UID: \"a0327861-5035-4a54-8d03-79c55efdfaa1\") "
Sep 11 03:41:57.994712 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:57.994489    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gb8w4\" (UniqueName: \"kubernetes.io/projected/a0327861-5035-4a54-8d03-79c55efdfaa1-kube-api-access-gb8w4\") pod \"a0327861-5035-4a54-8d03-79c55efdfaa1\" (UID: \"a0327861-5035-4a54-8d03-79c55efdfaa1\") "
Sep 11 03:41:57.994712 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:57.994601    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a0327861-5035-4a54-8d03-79c55efdfaa1-image-registry-private-configuration\") pod \"a0327861-5035-4a54-8d03-79c55efdfaa1\" (UID: \"a0327861-5035-4a54-8d03-79c55efdfaa1\") "
Sep 11 03:41:57.994712 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:57.994638    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a0327861-5035-4a54-8d03-79c55efdfaa1-trusted-ca\") pod \"a0327861-5035-4a54-8d03-79c55efdfaa1\" (UID: \"a0327861-5035-4a54-8d03-79c55efdfaa1\") "
Sep 11 03:41:57.994712 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:57.994676    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a0327861-5035-4a54-8d03-79c55efdfaa1-ca-trust-extracted\") pod \"a0327861-5035-4a54-8d03-79c55efdfaa1\" (UID: \"a0327861-5035-4a54-8d03-79c55efdfaa1\") "
Sep 11 03:41:57.994916 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:57.994748    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a0327861-5035-4a54-8d03-79c55efdfaa1-registry-tls\") pod \"a0327861-5035-4a54-8d03-79c55efdfaa1\" (UID: \"a0327861-5035-4a54-8d03-79c55efdfaa1\") "
Sep 11 03:41:57.994916 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:57.994781    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a0327861-5035-4a54-8d03-79c55efdfaa1-bound-sa-token\") pod \"a0327861-5035-4a54-8d03-79c55efdfaa1\" (UID: \"a0327861-5035-4a54-8d03-79c55efdfaa1\") "
Sep 11 03:41:57.995015 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:57.994986    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0327861-5035-4a54-8d03-79c55efdfaa1-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "a0327861-5035-4a54-8d03-79c55efdfaa1" (UID: "a0327861-5035-4a54-8d03-79c55efdfaa1"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 03:41:57.995489 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:57.995099    2580 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a0327861-5035-4a54-8d03-79c55efdfaa1-registry-certificates\") on node \"ip-10-0-138-121.ec2.internal\" DevicePath \"\""
Sep 11 03:41:57.995489 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:57.995345    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0327861-5035-4a54-8d03-79c55efdfaa1-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "a0327861-5035-4a54-8d03-79c55efdfaa1" (UID: "a0327861-5035-4a54-8d03-79c55efdfaa1"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 03:41:57.997422 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:57.997390    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0327861-5035-4a54-8d03-79c55efdfaa1-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "a0327861-5035-4a54-8d03-79c55efdfaa1" (UID: "a0327861-5035-4a54-8d03-79c55efdfaa1"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 03:41:57.997527 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:57.997441    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0327861-5035-4a54-8d03-79c55efdfaa1-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "a0327861-5035-4a54-8d03-79c55efdfaa1" (UID: "a0327861-5035-4a54-8d03-79c55efdfaa1"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 03:41:57.997527 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:57.997453    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0327861-5035-4a54-8d03-79c55efdfaa1-kube-api-access-gb8w4" (OuterVolumeSpecName: "kube-api-access-gb8w4") pod "a0327861-5035-4a54-8d03-79c55efdfaa1" (UID: "a0327861-5035-4a54-8d03-79c55efdfaa1"). InnerVolumeSpecName "kube-api-access-gb8w4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 03:41:57.997679 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:57.997548    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0327861-5035-4a54-8d03-79c55efdfaa1-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "a0327861-5035-4a54-8d03-79c55efdfaa1" (UID: "a0327861-5035-4a54-8d03-79c55efdfaa1"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 03:41:57.997809 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:57.997787    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0327861-5035-4a54-8d03-79c55efdfaa1-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "a0327861-5035-4a54-8d03-79c55efdfaa1" (UID: "a0327861-5035-4a54-8d03-79c55efdfaa1"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 03:41:58.003855 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:58.003828    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0327861-5035-4a54-8d03-79c55efdfaa1-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "a0327861-5035-4a54-8d03-79c55efdfaa1" (UID: "a0327861-5035-4a54-8d03-79c55efdfaa1"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 03:41:58.096375 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:58.096334    2580 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a0327861-5035-4a54-8d03-79c55efdfaa1-installation-pull-secrets\") on node \"ip-10-0-138-121.ec2.internal\" DevicePath \"\""
Sep 11 03:41:58.096375 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:58.096371    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gb8w4\" (UniqueName: \"kubernetes.io/projected/a0327861-5035-4a54-8d03-79c55efdfaa1-kube-api-access-gb8w4\") on node \"ip-10-0-138-121.ec2.internal\" DevicePath \"\""
Sep 11 03:41:58.096375 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:58.096381    2580 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a0327861-5035-4a54-8d03-79c55efdfaa1-image-registry-private-configuration\") on node \"ip-10-0-138-121.ec2.internal\" DevicePath \"\""
Sep 11 03:41:58.096599 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:58.096391    2580 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a0327861-5035-4a54-8d03-79c55efdfaa1-trusted-ca\") on node \"ip-10-0-138-121.ec2.internal\" DevicePath \"\""
Sep 11 03:41:58.096599 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:58.096401    2580 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a0327861-5035-4a54-8d03-79c55efdfaa1-ca-trust-extracted\") on node \"ip-10-0-138-121.ec2.internal\" DevicePath \"\""
Sep 11 03:41:58.096599 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:58.096409    2580 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a0327861-5035-4a54-8d03-79c55efdfaa1-registry-tls\") on node \"ip-10-0-138-121.ec2.internal\" DevicePath \"\""
Sep 11 03:41:58.096599 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:58.096417    2580 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a0327861-5035-4a54-8d03-79c55efdfaa1-bound-sa-token\") on node \"ip-10-0-138-121.ec2.internal\" DevicePath \"\""
Sep 11 03:41:58.241628 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:58.241596    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-76b857c477-qw9fr"]
Sep 11 03:41:58.244896 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:58.244873    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-76b857c477-qw9fr"]
Sep 11 03:41:58.464585 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:58.464547    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a0327861-5035-4a54-8d03-79c55efdfaa1" path="/var/lib/kubelet/pods/a0327861-5035-4a54-8d03-79c55efdfaa1/volumes"
Sep 11 03:41:59.930417 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:59.930380    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-67577cf4bb-9pb7v" event={"ID":"0f67bb82-1dc5-418b-8778-b43028862742","Type":"ContainerStarted","Data":"95b08303d0b9570dcab738cdf4a0d1733656e8dbd5dbbaa509a7d06adac70214"}
Sep 11 03:41:59.930922 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:59.930464    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-67577cf4bb-9pb7v" event={"ID":"0f67bb82-1dc5-418b-8778-b43028862742","Type":"ContainerStarted","Data":"5691a2e75d09a0813a8d02f5aa1e210d5621ffb354e9a94e49608e4e712c81d5"}
Sep 11 03:41:59.954907 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:41:59.954855    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-67577cf4bb-9pb7v" podStartSLOduration=1.9348449410000002 podStartE2EDuration="5.9548363s" totalImagesPullingTime="4.019991359s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 03:41:54 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:41:59.954397359 +0000 UTC m=+108.112883686" watchObservedRunningTime="2026-09-11 03:41:59.9548363 +0000 UTC m=+108.113322622"
Sep 11 03:42:02.500422 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:42:02.500387    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-76f99584f4-xt54w"]
Sep 11 03:42:02.500836 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:42:02.500671    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a0327861-5035-4a54-8d03-79c55efdfaa1" containerName="registry"
Sep 11 03:42:02.500836 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:42:02.500683    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="a0327861-5035-4a54-8d03-79c55efdfaa1" containerName="registry"
Sep 11 03:42:02.500836 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:42:02.500753    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="a0327861-5035-4a54-8d03-79c55efdfaa1" containerName="registry"
Sep 11 03:42:02.503451 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:42:02.503435    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-76f99584f4-xt54w"
Sep 11 03:42:02.505670 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:42:02.505642    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 11 03:42:02.505814 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:42:02.505645    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 11 03:42:02.505814 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:42:02.505645    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-wsfc4\""
Sep 11 03:42:02.513272 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:42:02.513245    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-76f99584f4-xt54w"]
Sep 11 03:42:02.635421 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:42:02.635365    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxrcx\" (UniqueName: \"kubernetes.io/projected/15478e4f-2c48-4b48-b6cc-64da0510c1fa-kube-api-access-xxrcx\") pod \"downloads-76f99584f4-xt54w\" (UID: \"15478e4f-2c48-4b48-b6cc-64da0510c1fa\") " pod="openshift-console/downloads-76f99584f4-xt54w"
Sep 11 03:42:02.736209 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:42:02.736166    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxrcx\" (UniqueName: \"kubernetes.io/projected/15478e4f-2c48-4b48-b6cc-64da0510c1fa-kube-api-access-xxrcx\") pod \"downloads-76f99584f4-xt54w\" (UID: \"15478e4f-2c48-4b48-b6cc-64da0510c1fa\") " pod="openshift-console/downloads-76f99584f4-xt54w"
Sep 11 03:42:02.744141 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:42:02.744117    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxrcx\" (UniqueName: \"kubernetes.io/projected/15478e4f-2c48-4b48-b6cc-64da0510c1fa-kube-api-access-xxrcx\") pod \"downloads-76f99584f4-xt54w\" (UID: \"15478e4f-2c48-4b48-b6cc-64da0510c1fa\") " pod="openshift-console/downloads-76f99584f4-xt54w"
Sep 11 03:42:02.812731 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:42:02.812627    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-76f99584f4-xt54w"
Sep 11 03:42:02.894484 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:42:02.894454    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-pwk5f"
Sep 11 03:42:03.080352 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:42:03.080282    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-76f99584f4-xt54w"]
Sep 11 03:42:03.082997 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:42:03.082970    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15478e4f_2c48_4b48_b6cc_64da0510c1fa.slice/crio-9877fafc83f0d85f276a44aa9745c6dedaba2d765f766d892736ab94a25f101c WatchSource:0}: Error finding container 9877fafc83f0d85f276a44aa9745c6dedaba2d765f766d892736ab94a25f101c: Status 404 returned error can't find the container with id 9877fafc83f0d85f276a44aa9745c6dedaba2d765f766d892736ab94a25f101c
Sep 11 03:42:03.946944 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:42:03.946896    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-76f99584f4-xt54w" event={"ID":"15478e4f-2c48-4b48-b6cc-64da0510c1fa","Type":"ContainerStarted","Data":"9877fafc83f0d85f276a44aa9745c6dedaba2d765f766d892736ab94a25f101c"}
Sep 11 03:42:12.013249 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:42:12.013208    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-79dd8f66f8-fct9f"]
Sep 11 03:42:12.016825 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:42:12.016801    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-79dd8f66f8-fct9f"
Sep 11 03:42:12.019119 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:42:12.019028    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 11 03:42:12.019240 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:42:12.019132    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-x2hwp\""
Sep 11 03:42:12.020218 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:42:12.019975    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 11 03:42:12.020218 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:42:12.020063    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 11 03:42:12.020218 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:42:12.020077    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 11 03:42:12.020444 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:42:12.020264    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 11 03:42:12.027068 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:42:12.027042    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-79dd8f66f8-fct9f"]
Sep 11 03:42:12.118781 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:42:12.118746    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/938e0eac-244e-49b0-b616-c7c4418ac36a-service-ca\") pod \"console-79dd8f66f8-fct9f\" (UID: \"938e0eac-244e-49b0-b616-c7c4418ac36a\") " pod="openshift-console/console-79dd8f66f8-fct9f"
Sep 11 03:42:12.118961 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:42:12.118792    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8bmbf\" (UniqueName: \"kubernetes.io/projected/938e0eac-244e-49b0-b616-c7c4418ac36a-kube-api-access-8bmbf\") pod \"console-79dd8f66f8-fct9f\" (UID: \"938e0eac-244e-49b0-b616-c7c4418ac36a\") " pod="openshift-console/console-79dd8f66f8-fct9f"
Sep 11 03:42:12.118961 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:42:12.118872    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/938e0eac-244e-49b0-b616-c7c4418ac36a-console-oauth-config\") pod \"console-79dd8f66f8-fct9f\" (UID: \"938e0eac-244e-49b0-b616-c7c4418ac36a\") " pod="openshift-console/console-79dd8f66f8-fct9f"
Sep 11 03:42:12.118961 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:42:12.118896    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/938e0eac-244e-49b0-b616-c7c4418ac36a-oauth-serving-cert\") pod \"console-79dd8f66f8-fct9f\" (UID: \"938e0eac-244e-49b0-b616-c7c4418ac36a\") " pod="openshift-console/console-79dd8f66f8-fct9f"
Sep 11 03:42:12.118961 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:42:12.118922    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/938e0eac-244e-49b0-b616-c7c4418ac36a-console-serving-cert\") pod \"console-79dd8f66f8-fct9f\" (UID: \"938e0eac-244e-49b0-b616-c7c4418ac36a\") " pod="openshift-console/console-79dd8f66f8-fct9f"
Sep 11 03:42:12.118961 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:42:12.118949    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/938e0eac-244e-49b0-b616-c7c4418ac36a-console-config\") pod \"console-79dd8f66f8-fct9f\" (UID: \"938e0eac-244e-49b0-b616-c7c4418ac36a\") " pod="openshift-console/console-79dd8f66f8-fct9f"
Sep 11 03:42:12.219672 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:42:12.219618    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/938e0eac-244e-49b0-b616-c7c4418ac36a-service-ca\") pod \"console-79dd8f66f8-fct9f\" (UID: \"938e0eac-244e-49b0-b616-c7c4418ac36a\") " pod="openshift-console/console-79dd8f66f8-fct9f"
Sep 11 03:42:12.219672 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:42:12.219672    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bmbf\" (UniqueName: \"kubernetes.io/projected/938e0eac-244e-49b0-b616-c7c4418ac36a-kube-api-access-8bmbf\") pod \"console-79dd8f66f8-fct9f\" (UID: \"938e0eac-244e-49b0-b616-c7c4418ac36a\") " pod="openshift-console/console-79dd8f66f8-fct9f"
Sep 11 03:42:12.220033 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:42:12.219749    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/938e0eac-244e-49b0-b616-c7c4418ac36a-console-oauth-config\") pod \"console-79dd8f66f8-fct9f\" (UID: \"938e0eac-244e-49b0-b616-c7c4418ac36a\") " pod="openshift-console/console-79dd8f66f8-fct9f"
Sep 11 03:42:12.220033 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:42:12.219772    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/938e0eac-244e-49b0-b616-c7c4418ac36a-oauth-serving-cert\") pod \"console-79dd8f66f8-fct9f\" (UID: \"938e0eac-244e-49b0-b616-c7c4418ac36a\") " pod="openshift-console/console-79dd8f66f8-fct9f"
Sep 11 03:42:12.220033 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:42:12.219800    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/938e0eac-244e-49b0-b616-c7c4418ac36a-console-serving-cert\") pod \"console-79dd8f66f8-fct9f\" (UID: \"938e0eac-244e-49b0-b616-c7c4418ac36a\") " pod="openshift-console/console-79dd8f66f8-fct9f"
Sep 11 03:42:12.220033 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:42:12.219829    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/938e0eac-244e-49b0-b616-c7c4418ac36a-console-config\") pod \"console-79dd8f66f8-fct9f\" (UID: \"938e0eac-244e-49b0-b616-c7c4418ac36a\") " pod="openshift-console/console-79dd8f66f8-fct9f"
Sep 11 03:42:12.220494 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:42:12.220465    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/938e0eac-244e-49b0-b616-c7c4418ac36a-service-ca\") pod \"console-79dd8f66f8-fct9f\" (UID: \"938e0eac-244e-49b0-b616-c7c4418ac36a\") " pod="openshift-console/console-79dd8f66f8-fct9f"
Sep 11 03:42:12.220714 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:42:12.220691    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/938e0eac-244e-49b0-b616-c7c4418ac36a-oauth-serving-cert\") pod \"console-79dd8f66f8-fct9f\" (UID: \"938e0eac-244e-49b0-b616-c7c4418ac36a\") " pod="openshift-console/console-79dd8f66f8-fct9f"
Sep 11 03:42:12.220842 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:42:12.220825    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/938e0eac-244e-49b0-b616-c7c4418ac36a-console-config\") pod \"console-79dd8f66f8-fct9f\" (UID: \"938e0eac-244e-49b0-b616-c7c4418ac36a\") " pod="openshift-console/console-79dd8f66f8-fct9f"
Sep 11 03:42:12.222541 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:42:12.222464    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/938e0eac-244e-49b0-b616-c7c4418ac36a-console-serving-cert\") pod \"console-79dd8f66f8-fct9f\" (UID: \"938e0eac-244e-49b0-b616-c7c4418ac36a\") " pod="openshift-console/console-79dd8f66f8-fct9f"
Sep 11 03:42:12.222639 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:42:12.222615    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/938e0eac-244e-49b0-b616-c7c4418ac36a-console-oauth-config\") pod \"console-79dd8f66f8-fct9f\" (UID: \"938e0eac-244e-49b0-b616-c7c4418ac36a\") " pod="openshift-console/console-79dd8f66f8-fct9f"
Sep 11 03:42:12.227324 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:42:12.227301    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8bmbf\" (UniqueName: \"kubernetes.io/projected/938e0eac-244e-49b0-b616-c7c4418ac36a-kube-api-access-8bmbf\") pod \"console-79dd8f66f8-fct9f\" (UID: \"938e0eac-244e-49b0-b616-c7c4418ac36a\") " pod="openshift-console/console-79dd8f66f8-fct9f"
Sep 11 03:42:12.330369 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:42:12.330336    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-x2hwp\""
Sep 11 03:42:12.339258 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:42:12.339224    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-79dd8f66f8-fct9f"
Sep 11 03:42:19.012202 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:42:19.012166    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-76f99584f4-xt54w" event={"ID":"15478e4f-2c48-4b48-b6cc-64da0510c1fa","Type":"ContainerStarted","Data":"6e04197b322262e78d7ff99821c5cfa244f5ec4e0fdf027df5edbbcb39e7bb0c"}
Sep 11 03:42:19.012686 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:42:19.012375    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-76f99584f4-xt54w"
Sep 11 03:42:19.013891 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:42:19.013869    2580 patch_prober.go:28] interesting pod/downloads-76f99584f4-xt54w container/download-server namespace/openshift-console: Readiness probe status=failure output="Get \"http://10.133.0.20:8080/\": dial tcp 10.133.0.20:8080: connect: connection refused" start-of-body=
Sep 11 03:42:19.013996 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:42:19.013911    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console/downloads-76f99584f4-xt54w" podUID="15478e4f-2c48-4b48-b6cc-64da0510c1fa" containerName="download-server" probeResult="failure" output="Get \"http://10.133.0.20:8080/\": dial tcp 10.133.0.20:8080: connect: connection refused"
Sep 11 03:42:19.030672 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:42:19.030606    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-76f99584f4-xt54w" podStartSLOduration=1.221359777 podStartE2EDuration="17.030590619s" totalImagesPullingTime="15.809230842s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 03:42:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:42:19.028735186 +0000 UTC m=+127.187221510" watchObservedRunningTime="2026-09-11 03:42:19.030590619 +0000 UTC m=+127.189076960"
Sep 11 03:42:19.057649 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:42:19.057456    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-79dd8f66f8-fct9f"]
Sep 11 03:42:19.060571 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:42:19.060542    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod938e0eac_244e_49b0_b616_c7c4418ac36a.slice/crio-3a9b007b5ce2ae95ade606ed87e97f18a6d463f1951e2b12e2f660c604c8f761 WatchSource:0}: Error finding container 3a9b007b5ce2ae95ade606ed87e97f18a6d463f1951e2b12e2f660c604c8f761: Status 404 returned error can't find the container with id 3a9b007b5ce2ae95ade606ed87e97f18a6d463f1951e2b12e2f660c604c8f761
Sep 11 03:42:20.020171 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:42:20.020128    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-79dd8f66f8-fct9f" event={"ID":"938e0eac-244e-49b0-b616-c7c4418ac36a","Type":"ContainerStarted","Data":"3a9b007b5ce2ae95ade606ed87e97f18a6d463f1951e2b12e2f660c604c8f761"}
Sep 11 03:42:20.033328 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:42:20.033262    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-76f99584f4-xt54w"
Sep 11 03:42:22.112181 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:42:22.112142    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb00790b-1a90-4786-8b81-cbfc706f9c46-metrics-certs\") pod \"network-metrics-daemon-qzhz6\" (UID: \"eb00790b-1a90-4786-8b81-cbfc706f9c46\") " pod="openshift-multus/network-metrics-daemon-qzhz6"
Sep 11 03:42:22.115244 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:42:22.115217    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb00790b-1a90-4786-8b81-cbfc706f9c46-metrics-certs\") pod \"network-metrics-daemon-qzhz6\" (UID: \"eb00790b-1a90-4786-8b81-cbfc706f9c46\") " pod="openshift-multus/network-metrics-daemon-qzhz6"
Sep 11 03:42:22.271854 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:42:22.271814    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-sfchv\""
Sep 11 03:42:22.279822 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:42:22.279791    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qzhz6"
Sep 11 03:42:22.685701 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:42:22.685608    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-qzhz6"]
Sep 11 03:42:22.802389 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:42:22.802344    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb00790b_1a90_4786_8b81_cbfc706f9c46.slice/crio-70f08077838368aac63ab9775575a478fe7c03f90bda5b1eee390f2315a66400 WatchSource:0}: Error finding container 70f08077838368aac63ab9775575a478fe7c03f90bda5b1eee390f2315a66400: Status 404 returned error can't find the container with id 70f08077838368aac63ab9775575a478fe7c03f90bda5b1eee390f2315a66400
Sep 11 03:42:23.043332 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:42:23.043242    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-qzhz6" event={"ID":"eb00790b-1a90-4786-8b81-cbfc706f9c46","Type":"ContainerStarted","Data":"70f08077838368aac63ab9775575a478fe7c03f90bda5b1eee390f2315a66400"}
Sep 11 03:42:23.045390 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:42:23.045361    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-79dd8f66f8-fct9f" event={"ID":"938e0eac-244e-49b0-b616-c7c4418ac36a","Type":"ContainerStarted","Data":"68d356576998ca9fae1f1bf01f6a6b9943294999135e1a1344ae3259ce824c03"}
Sep 11 03:42:23.065186 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:42:23.065142    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-79dd8f66f8-fct9f" podStartSLOduration=8.301526769 podStartE2EDuration="12.065121939s" totalImagesPullingTime="3.76359517s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 03:42:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:42:23.064206669 +0000 UTC m=+131.222692994" watchObservedRunningTime="2026-09-11 03:42:23.065121939 +0000 UTC m=+131.223608264"
Sep 11 03:42:25.056407 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:42:25.056313    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-qzhz6" event={"ID":"eb00790b-1a90-4786-8b81-cbfc706f9c46","Type":"ContainerStarted","Data":"db01a18a2ab4586a3603c1610706d6ae9553334831d5b46ce0f8f9cc506d240d"}
Sep 11 03:42:25.056407 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:42:25.056386    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-qzhz6" event={"ID":"eb00790b-1a90-4786-8b81-cbfc706f9c46","Type":"ContainerStarted","Data":"7f668930eea6b6dc9775dcbaaa7622a5f115119c75e5e89fa462ae244f111235"}
Sep 11 03:42:25.072690 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:42:25.072614    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-qzhz6" podStartSLOduration=131.488592954 podStartE2EDuration="2m13.072595018s" totalImagesPullingTime="1.584002064s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 03:40:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:42:25.072065957 +0000 UTC m=+133.230552281" watchObservedRunningTime="2026-09-11 03:42:25.072595018 +0000 UTC m=+133.231081343"
Sep 11 03:42:31.472106 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:42:31.472070    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-79dd8f66f8-fct9f"]
Sep 11 03:42:32.340143 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:42:32.340101    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-79dd8f66f8-fct9f"
Sep 11 03:42:34.095969 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:42:34.095935    2580 generic.go:356] "Generic (PLEG): container finished" podID="f273ceed-c140-4434-b114-220a14a5f02b" containerID="8edc3636ad5519bac29b3c967c6a0f60053155032a5020c2f640c76bd3a62a0f" exitCode=0
Sep 11 03:42:34.096435 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:42:34.096014    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-m45tt" event={"ID":"f273ceed-c140-4434-b114-220a14a5f02b","Type":"ContainerDied","Data":"8edc3636ad5519bac29b3c967c6a0f60053155032a5020c2f640c76bd3a62a0f"}
Sep 11 03:42:34.096435 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:42:34.096389    2580 scope.go:117] "RemoveContainer" containerID="8edc3636ad5519bac29b3c967c6a0f60053155032a5020c2f640c76bd3a62a0f"
Sep 11 03:42:34.097677 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:42:34.097638    2580 generic.go:356] "Generic (PLEG): container finished" podID="2a8f992e-1768-401c-afd6-b87e5a289d61" containerID="4fee7a1e9d37160f5e0cd2e2a7e46f0191a1f8864bfbd2df0eda9924daa4b42a" exitCode=0
Sep 11 03:42:34.097766 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:42:34.097717    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-l7n7s" event={"ID":"2a8f992e-1768-401c-afd6-b87e5a289d61","Type":"ContainerDied","Data":"4fee7a1e9d37160f5e0cd2e2a7e46f0191a1f8864bfbd2df0eda9924daa4b42a"}
Sep 11 03:42:34.097998 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:42:34.097985    2580 scope.go:117] "RemoveContainer" containerID="4fee7a1e9d37160f5e0cd2e2a7e46f0191a1f8864bfbd2df0eda9924daa4b42a"
Sep 11 03:42:35.104132 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:42:35.104090    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-m45tt" event={"ID":"f273ceed-c140-4434-b114-220a14a5f02b","Type":"ContainerStarted","Data":"27d0f45174bd9639c8d858997568affd493d7ae5f090d10ca1dd20df32fbe5ae"}
Sep 11 03:42:35.106480 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:42:35.106455    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-l7n7s" event={"ID":"2a8f992e-1768-401c-afd6-b87e5a289d61","Type":"ContainerStarted","Data":"eaeac346caceac4c08d47e60c92a398393c7255341f9ec37c7c116cc72a7aa13"}
Sep 11 03:42:56.490736 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:42:56.490689    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-79dd8f66f8-fct9f" podUID="938e0eac-244e-49b0-b616-c7c4418ac36a" containerName="console" containerID="cri-o://68d356576998ca9fae1f1bf01f6a6b9943294999135e1a1344ae3259ce824c03" gracePeriod=15
Sep 11 03:42:56.767581 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:42:56.767519    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-79dd8f66f8-fct9f"
Sep 11 03:42:56.907834 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:42:56.907798    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/938e0eac-244e-49b0-b616-c7c4418ac36a-service-ca\") pod \"938e0eac-244e-49b0-b616-c7c4418ac36a\" (UID: \"938e0eac-244e-49b0-b616-c7c4418ac36a\") "
Sep 11 03:42:56.907834 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:42:56.907835    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8bmbf\" (UniqueName: \"kubernetes.io/projected/938e0eac-244e-49b0-b616-c7c4418ac36a-kube-api-access-8bmbf\") pod \"938e0eac-244e-49b0-b616-c7c4418ac36a\" (UID: \"938e0eac-244e-49b0-b616-c7c4418ac36a\") "
Sep 11 03:42:56.908023 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:42:56.907871    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/938e0eac-244e-49b0-b616-c7c4418ac36a-console-config\") pod \"938e0eac-244e-49b0-b616-c7c4418ac36a\" (UID: \"938e0eac-244e-49b0-b616-c7c4418ac36a\") "
Sep 11 03:42:56.908023 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:42:56.907970    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/938e0eac-244e-49b0-b616-c7c4418ac36a-oauth-serving-cert\") pod \"938e0eac-244e-49b0-b616-c7c4418ac36a\" (UID: \"938e0eac-244e-49b0-b616-c7c4418ac36a\") "
Sep 11 03:42:56.908101 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:42:56.908021    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/938e0eac-244e-49b0-b616-c7c4418ac36a-console-oauth-config\") pod \"938e0eac-244e-49b0-b616-c7c4418ac36a\" (UID: \"938e0eac-244e-49b0-b616-c7c4418ac36a\") "
Sep 11 03:42:56.908101 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:42:56.908048    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/938e0eac-244e-49b0-b616-c7c4418ac36a-console-serving-cert\") pod \"938e0eac-244e-49b0-b616-c7c4418ac36a\" (UID: \"938e0eac-244e-49b0-b616-c7c4418ac36a\") "
Sep 11 03:42:56.908255 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:42:56.908226    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/938e0eac-244e-49b0-b616-c7c4418ac36a-service-ca" (OuterVolumeSpecName: "service-ca") pod "938e0eac-244e-49b0-b616-c7c4418ac36a" (UID: "938e0eac-244e-49b0-b616-c7c4418ac36a"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 03:42:56.908322 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:42:56.908232    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/938e0eac-244e-49b0-b616-c7c4418ac36a-console-config" (OuterVolumeSpecName: "console-config") pod "938e0eac-244e-49b0-b616-c7c4418ac36a" (UID: "938e0eac-244e-49b0-b616-c7c4418ac36a"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 03:42:56.908377 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:42:56.908323    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/938e0eac-244e-49b0-b616-c7c4418ac36a-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "938e0eac-244e-49b0-b616-c7c4418ac36a" (UID: "938e0eac-244e-49b0-b616-c7c4418ac36a"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 03:42:56.910148 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:42:56.910117    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/938e0eac-244e-49b0-b616-c7c4418ac36a-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "938e0eac-244e-49b0-b616-c7c4418ac36a" (UID: "938e0eac-244e-49b0-b616-c7c4418ac36a"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 03:42:56.910245 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:42:56.910209    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/938e0eac-244e-49b0-b616-c7c4418ac36a-kube-api-access-8bmbf" (OuterVolumeSpecName: "kube-api-access-8bmbf") pod "938e0eac-244e-49b0-b616-c7c4418ac36a" (UID: "938e0eac-244e-49b0-b616-c7c4418ac36a"). InnerVolumeSpecName "kube-api-access-8bmbf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 03:42:56.910245 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:42:56.910190    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/938e0eac-244e-49b0-b616-c7c4418ac36a-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "938e0eac-244e-49b0-b616-c7c4418ac36a" (UID: "938e0eac-244e-49b0-b616-c7c4418ac36a"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 03:42:57.009353 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:42:57.009318    2580 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/938e0eac-244e-49b0-b616-c7c4418ac36a-service-ca\") on node \"ip-10-0-138-121.ec2.internal\" DevicePath \"\""
Sep 11 03:42:57.009353 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:42:57.009349    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8bmbf\" (UniqueName: \"kubernetes.io/projected/938e0eac-244e-49b0-b616-c7c4418ac36a-kube-api-access-8bmbf\") on node \"ip-10-0-138-121.ec2.internal\" DevicePath \"\""
Sep 11 03:42:57.009353 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:42:57.009359    2580 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/938e0eac-244e-49b0-b616-c7c4418ac36a-console-config\") on node \"ip-10-0-138-121.ec2.internal\" DevicePath \"\""
Sep 11 03:42:57.009570 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:42:57.009368    2580 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/938e0eac-244e-49b0-b616-c7c4418ac36a-oauth-serving-cert\") on node \"ip-10-0-138-121.ec2.internal\" DevicePath \"\""
Sep 11 03:42:57.009570 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:42:57.009376    2580 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/938e0eac-244e-49b0-b616-c7c4418ac36a-console-oauth-config\") on node \"ip-10-0-138-121.ec2.internal\" DevicePath \"\""
Sep 11 03:42:57.009570 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:42:57.009384    2580 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/938e0eac-244e-49b0-b616-c7c4418ac36a-console-serving-cert\") on node \"ip-10-0-138-121.ec2.internal\" DevicePath \"\""
Sep 11 03:42:57.191265 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:42:57.191231    2580 generic.go:356] "Generic (PLEG): container finished" podID="938e0eac-244e-49b0-b616-c7c4418ac36a" containerID="68d356576998ca9fae1f1bf01f6a6b9943294999135e1a1344ae3259ce824c03" exitCode=0
Sep 11 03:42:57.191431 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:42:57.191283    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-79dd8f66f8-fct9f" event={"ID":"938e0eac-244e-49b0-b616-c7c4418ac36a","Type":"ContainerDied","Data":"68d356576998ca9fae1f1bf01f6a6b9943294999135e1a1344ae3259ce824c03"}
Sep 11 03:42:57.191431 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:42:57.191317    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-79dd8f66f8-fct9f" event={"ID":"938e0eac-244e-49b0-b616-c7c4418ac36a","Type":"ContainerDied","Data":"3a9b007b5ce2ae95ade606ed87e97f18a6d463f1951e2b12e2f660c604c8f761"}
Sep 11 03:42:57.191431 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:42:57.191336    2580 scope.go:117] "RemoveContainer" containerID="68d356576998ca9fae1f1bf01f6a6b9943294999135e1a1344ae3259ce824c03"
Sep 11 03:42:57.191431 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:42:57.191283    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-79dd8f66f8-fct9f"
Sep 11 03:42:57.199905 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:42:57.199889    2580 scope.go:117] "RemoveContainer" containerID="68d356576998ca9fae1f1bf01f6a6b9943294999135e1a1344ae3259ce824c03"
Sep 11 03:42:57.200201 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:42:57.200181    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68d356576998ca9fae1f1bf01f6a6b9943294999135e1a1344ae3259ce824c03\": container with ID starting with 68d356576998ca9fae1f1bf01f6a6b9943294999135e1a1344ae3259ce824c03 not found: ID does not exist" containerID="68d356576998ca9fae1f1bf01f6a6b9943294999135e1a1344ae3259ce824c03"
Sep 11 03:42:57.200253 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:42:57.200209    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68d356576998ca9fae1f1bf01f6a6b9943294999135e1a1344ae3259ce824c03"} err="failed to get container status \"68d356576998ca9fae1f1bf01f6a6b9943294999135e1a1344ae3259ce824c03\": rpc error: code = NotFound desc = could not find container \"68d356576998ca9fae1f1bf01f6a6b9943294999135e1a1344ae3259ce824c03\": container with ID starting with 68d356576998ca9fae1f1bf01f6a6b9943294999135e1a1344ae3259ce824c03 not found: ID does not exist"
Sep 11 03:42:57.212579 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:42:57.212550    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-79dd8f66f8-fct9f"]
Sep 11 03:42:57.214771 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:42:57.214749    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-79dd8f66f8-fct9f"]
Sep 11 03:42:58.462417 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:42:58.462380    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="938e0eac-244e-49b0-b616-c7c4418ac36a" path="/var/lib/kubelet/pods/938e0eac-244e-49b0-b616-c7c4418ac36a/volumes"
Sep 11 03:43:11.804525 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:43:11.804445    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 11 03:43:11.805003 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:43:11.804883    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="938e0eac-244e-49b0-b616-c7c4418ac36a" containerName="console"
Sep 11 03:43:11.805003 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:43:11.804899    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="938e0eac-244e-49b0-b616-c7c4418ac36a" containerName="console"
Sep 11 03:43:11.805003 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:43:11.804953    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="938e0eac-244e-49b0-b616-c7c4418ac36a" containerName="console"
Sep 11 03:43:11.809112 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:43:11.809085    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 11 03:43:11.811301 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:43:11.811275    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 11 03:43:11.811492 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:43:11.811476    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 11 03:43:11.811492 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:43:11.811484    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 11 03:43:11.811624 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:43:11.811484    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-ffk29\""
Sep 11 03:43:11.811624 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:43:11.811483    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 11 03:43:11.811747 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:43:11.811730    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 11 03:43:11.811811 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:43:11.811736    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 11 03:43:11.811811 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:43:11.811765    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 11 03:43:11.811811 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:43:11.811778    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 11 03:43:11.817405 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:43:11.817383    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 11 03:43:11.820394 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:43:11.820365    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e5988370-558f-43e2-af52-5369936a43ae-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"e5988370-558f-43e2-af52-5369936a43ae\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 03:43:11.820507 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:43:11.820412    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/e5988370-558f-43e2-af52-5369936a43ae-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"e5988370-558f-43e2-af52-5369936a43ae\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 03:43:11.820507 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:43:11.820488    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e5988370-558f-43e2-af52-5369936a43ae-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"e5988370-558f-43e2-af52-5369936a43ae\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 03:43:11.820603 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:43:11.820529    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/e5988370-558f-43e2-af52-5369936a43ae-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"e5988370-558f-43e2-af52-5369936a43ae\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 03:43:11.820603 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:43:11.820583    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e5988370-558f-43e2-af52-5369936a43ae-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"e5988370-558f-43e2-af52-5369936a43ae\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 03:43:11.820719 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:43:11.820622    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/e5988370-558f-43e2-af52-5369936a43ae-config-volume\") pod \"alertmanager-main-0\" (UID: \"e5988370-558f-43e2-af52-5369936a43ae\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 03:43:11.820719 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:43:11.820646    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e5988370-558f-43e2-af52-5369936a43ae-web-config\") pod \"alertmanager-main-0\" (UID: \"e5988370-558f-43e2-af52-5369936a43ae\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 03:43:11.820719 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:43:11.820713    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/e5988370-558f-43e2-af52-5369936a43ae-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"e5988370-558f-43e2-af52-5369936a43ae\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 03:43:11.820832 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:43:11.820739    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e5988370-558f-43e2-af52-5369936a43ae-config-out\") pod \"alertmanager-main-0\" (UID: \"e5988370-558f-43e2-af52-5369936a43ae\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 03:43:11.820832 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:43:11.820780    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jnwp\" (UniqueName: \"kubernetes.io/projected/e5988370-558f-43e2-af52-5369936a43ae-kube-api-access-7jnwp\") pod \"alertmanager-main-0\" (UID: \"e5988370-558f-43e2-af52-5369936a43ae\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 03:43:11.820832 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:43:11.820797    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/e5988370-558f-43e2-af52-5369936a43ae-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"e5988370-558f-43e2-af52-5369936a43ae\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 03:43:11.820832 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:43:11.820815    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e5988370-558f-43e2-af52-5369936a43ae-tls-assets\") pod \"alertmanager-main-0\" (UID: \"e5988370-558f-43e2-af52-5369936a43ae\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 03:43:11.821041 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:43:11.820867    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e5988370-558f-43e2-af52-5369936a43ae-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"e5988370-558f-43e2-af52-5369936a43ae\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 03:43:11.821041 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:43:11.820912    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 11 03:43:11.921632 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:43:11.921593    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jnwp\" (UniqueName: \"kubernetes.io/projected/e5988370-558f-43e2-af52-5369936a43ae-kube-api-access-7jnwp\") pod \"alertmanager-main-0\" (UID: \"e5988370-558f-43e2-af52-5369936a43ae\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 03:43:11.921632 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:43:11.921632    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/e5988370-558f-43e2-af52-5369936a43ae-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"e5988370-558f-43e2-af52-5369936a43ae\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 03:43:11.921912 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:43:11.921654    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e5988370-558f-43e2-af52-5369936a43ae-tls-assets\") pod \"alertmanager-main-0\" (UID: \"e5988370-558f-43e2-af52-5369936a43ae\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 03:43:11.921912 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:43:11.921705    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e5988370-558f-43e2-af52-5369936a43ae-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"e5988370-558f-43e2-af52-5369936a43ae\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 03:43:11.921912 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:43:11.921727    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e5988370-558f-43e2-af52-5369936a43ae-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"e5988370-558f-43e2-af52-5369936a43ae\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 03:43:11.921912 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:43:11.921753    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/e5988370-558f-43e2-af52-5369936a43ae-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"e5988370-558f-43e2-af52-5369936a43ae\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 03:43:11.921912 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:43:11.921798    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e5988370-558f-43e2-af52-5369936a43ae-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"e5988370-558f-43e2-af52-5369936a43ae\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 03:43:11.921912 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:43:11.921834    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/e5988370-558f-43e2-af52-5369936a43ae-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"e5988370-558f-43e2-af52-5369936a43ae\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 03:43:11.921912 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:43:11.921871    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e5988370-558f-43e2-af52-5369936a43ae-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"e5988370-558f-43e2-af52-5369936a43ae\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 03:43:11.921912 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:43:11.921897    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/e5988370-558f-43e2-af52-5369936a43ae-config-volume\") pod \"alertmanager-main-0\" (UID: \"e5988370-558f-43e2-af52-5369936a43ae\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 03:43:11.922294 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:43:11.921918    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e5988370-558f-43e2-af52-5369936a43ae-web-config\") pod \"alertmanager-main-0\" (UID: \"e5988370-558f-43e2-af52-5369936a43ae\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 03:43:11.922294 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:43:11.921951    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/e5988370-558f-43e2-af52-5369936a43ae-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"e5988370-558f-43e2-af52-5369936a43ae\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 03:43:11.922294 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:43:11.921982    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e5988370-558f-43e2-af52-5369936a43ae-config-out\") pod \"alertmanager-main-0\" (UID: \"e5988370-558f-43e2-af52-5369936a43ae\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 03:43:11.923742 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:43:11.923711    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e5988370-558f-43e2-af52-5369936a43ae-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"e5988370-558f-43e2-af52-5369936a43ae\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 03:43:11.923957 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:43:11.923930    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e5988370-558f-43e2-af52-5369936a43ae-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"e5988370-558f-43e2-af52-5369936a43ae\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 03:43:11.924864 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:43:11.924394    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/e5988370-558f-43e2-af52-5369936a43ae-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"e5988370-558f-43e2-af52-5369936a43ae\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 03:43:11.924864 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:43:11.924715    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e5988370-558f-43e2-af52-5369936a43ae-config-out\") pod \"alertmanager-main-0\" (UID: \"e5988370-558f-43e2-af52-5369936a43ae\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 03:43:11.924864 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:43:11.924825    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/e5988370-558f-43e2-af52-5369936a43ae-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"e5988370-558f-43e2-af52-5369936a43ae\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 03:43:11.924864 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:43:11.924850    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/e5988370-558f-43e2-af52-5369936a43ae-config-volume\") pod \"alertmanager-main-0\" (UID: \"e5988370-558f-43e2-af52-5369936a43ae\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 03:43:11.925131 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:43:11.925011    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e5988370-558f-43e2-af52-5369936a43ae-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"e5988370-558f-43e2-af52-5369936a43ae\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 03:43:11.925131 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:43:11.925011    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/e5988370-558f-43e2-af52-5369936a43ae-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"e5988370-558f-43e2-af52-5369936a43ae\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 03:43:11.925411 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:43:11.925394    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e5988370-558f-43e2-af52-5369936a43ae-tls-assets\") pod \"alertmanager-main-0\" (UID: \"e5988370-558f-43e2-af52-5369936a43ae\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 03:43:11.926373 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:43:11.926349    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e5988370-558f-43e2-af52-5369936a43ae-web-config\") pod \"alertmanager-main-0\" (UID: \"e5988370-558f-43e2-af52-5369936a43ae\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 03:43:11.926628 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:43:11.926611    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/e5988370-558f-43e2-af52-5369936a43ae-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"e5988370-558f-43e2-af52-5369936a43ae\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 03:43:11.926731 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:43:11.926715    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e5988370-558f-43e2-af52-5369936a43ae-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"e5988370-558f-43e2-af52-5369936a43ae\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 03:43:11.928871 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:43:11.928851    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jnwp\" (UniqueName: \"kubernetes.io/projected/e5988370-558f-43e2-af52-5369936a43ae-kube-api-access-7jnwp\") pod \"alertmanager-main-0\" (UID: \"e5988370-558f-43e2-af52-5369936a43ae\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 03:43:12.119863 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:43:12.119764    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 11 03:43:12.396189 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:43:12.396168    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 11 03:43:12.397811 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:43:12.397785    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5988370_558f_43e2_af52_5369936a43ae.slice/crio-b4f994d37dc449a348eef5e2291a53d5c3f7da38d99f7eee149c930034ed459b WatchSource:0}: Error finding container b4f994d37dc449a348eef5e2291a53d5c3f7da38d99f7eee149c930034ed459b: Status 404 returned error can't find the container with id b4f994d37dc449a348eef5e2291a53d5c3f7da38d99f7eee149c930034ed459b
Sep 11 03:43:13.256040 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:43:13.256007    2580 generic.go:356] "Generic (PLEG): container finished" podID="e5988370-558f-43e2-af52-5369936a43ae" containerID="e5627acaa9074bcc1ac6abdee6e96ab22c70ebd16bb123ce37f9dc0c51f8f00d" exitCode=0
Sep 11 03:43:13.256393 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:43:13.256086    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e5988370-558f-43e2-af52-5369936a43ae","Type":"ContainerDied","Data":"e5627acaa9074bcc1ac6abdee6e96ab22c70ebd16bb123ce37f9dc0c51f8f00d"}
Sep 11 03:43:13.256393 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:43:13.256150    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e5988370-558f-43e2-af52-5369936a43ae","Type":"ContainerStarted","Data":"b4f994d37dc449a348eef5e2291a53d5c3f7da38d99f7eee149c930034ed459b"}
Sep 11 03:43:15.267054 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:43:15.266970    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e5988370-558f-43e2-af52-5369936a43ae","Type":"ContainerStarted","Data":"56d93bd45e9ab6f3ea06885bf94a1fdeb524d14fea875be2c951322fedf9a4fb"}
Sep 11 03:43:15.267054 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:43:15.267035    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e5988370-558f-43e2-af52-5369936a43ae","Type":"ContainerStarted","Data":"2a43ffb4fb734649249075d2aee29fadb52b7ac48d9572c79e573c199faa3ee7"}
Sep 11 03:43:15.267421 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:43:15.267061    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e5988370-558f-43e2-af52-5369936a43ae","Type":"ContainerStarted","Data":"42237b697ee6f46f979b932e4375e88ff56d806034918eaf775a8792f14fb1fe"}
Sep 11 03:43:15.267421 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:43:15.267085    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e5988370-558f-43e2-af52-5369936a43ae","Type":"ContainerStarted","Data":"a5aeb4dfab5168b365fa8edea811746eb77e92a7b55c40aa0a6ca319773f9e21"}
Sep 11 03:43:15.267421 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:43:15.267107    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e5988370-558f-43e2-af52-5369936a43ae","Type":"ContainerStarted","Data":"3f834eccea0f3f1925c16b647b4e57311ed129b3aa333afb4b91f6709e3d7684"}
Sep 11 03:43:16.274069 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:43:16.274034    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e5988370-558f-43e2-af52-5369936a43ae","Type":"ContainerStarted","Data":"abbdf26ef1bbf5fc35bf86c7e2f29b28d9339364dfaacfedbd53f4305625402f"}
Sep 11 03:43:16.299826 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:43:16.299765    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.163129564 podStartE2EDuration="5.299747942s" totalImagesPullingTime="2.121143657s" totalInitContainerRuntime="15.474721ms" isStatefulPod=false podCreationTimestamp="2026-09-11 03:43:11 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:43:16.297865939 +0000 UTC m=+184.456352262" watchObservedRunningTime="2026-09-11 03:43:16.299747942 +0000 UTC m=+184.458234265"
Sep 11 03:44:39.552466 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:44:39.552390    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-hnmlv"]
Sep 11 03:44:39.555467 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:44:39.555450    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-hnmlv"
Sep 11 03:44:39.558063 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:44:39.558043    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Sep 11 03:44:39.558823 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:44:39.558802    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Sep 11 03:44:39.558921 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:44:39.558810    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-k65gq\""
Sep 11 03:44:39.565982 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:44:39.565957    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-hnmlv"]
Sep 11 03:44:39.637595 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:44:39.637549    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tcqnw\" (UniqueName: \"kubernetes.io/projected/f617c80e-90ca-4535-b0d4-07ef872969a2-kube-api-access-tcqnw\") pod \"cert-manager-webhook-6486ffd997-hnmlv\" (UID: \"f617c80e-90ca-4535-b0d4-07ef872969a2\") " pod="cert-manager/cert-manager-webhook-6486ffd997-hnmlv"
Sep 11 03:44:39.637766 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:44:39.637636    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f617c80e-90ca-4535-b0d4-07ef872969a2-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-hnmlv\" (UID: \"f617c80e-90ca-4535-b0d4-07ef872969a2\") " pod="cert-manager/cert-manager-webhook-6486ffd997-hnmlv"
Sep 11 03:44:39.738213 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:44:39.738169    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tcqnw\" (UniqueName: \"kubernetes.io/projected/f617c80e-90ca-4535-b0d4-07ef872969a2-kube-api-access-tcqnw\") pod \"cert-manager-webhook-6486ffd997-hnmlv\" (UID: \"f617c80e-90ca-4535-b0d4-07ef872969a2\") " pod="cert-manager/cert-manager-webhook-6486ffd997-hnmlv"
Sep 11 03:44:39.738398 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:44:39.738225    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f617c80e-90ca-4535-b0d4-07ef872969a2-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-hnmlv\" (UID: \"f617c80e-90ca-4535-b0d4-07ef872969a2\") " pod="cert-manager/cert-manager-webhook-6486ffd997-hnmlv"
Sep 11 03:44:39.748198 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:44:39.748163    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f617c80e-90ca-4535-b0d4-07ef872969a2-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-hnmlv\" (UID: \"f617c80e-90ca-4535-b0d4-07ef872969a2\") " pod="cert-manager/cert-manager-webhook-6486ffd997-hnmlv"
Sep 11 03:44:39.748312 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:44:39.748245    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tcqnw\" (UniqueName: \"kubernetes.io/projected/f617c80e-90ca-4535-b0d4-07ef872969a2-kube-api-access-tcqnw\") pod \"cert-manager-webhook-6486ffd997-hnmlv\" (UID: \"f617c80e-90ca-4535-b0d4-07ef872969a2\") " pod="cert-manager/cert-manager-webhook-6486ffd997-hnmlv"
Sep 11 03:44:39.873118 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:44:39.873030    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-hnmlv"
Sep 11 03:44:40.145193 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:44:40.145167    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-hnmlv"]
Sep 11 03:44:40.147811 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:44:40.147779    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf617c80e_90ca_4535_b0d4_07ef872969a2.slice/crio-9967321ad9dd7a2a09e7aafaad4748d3cf61fb4a9d3f2c6fc76c5d8a7f841658 WatchSource:0}: Error finding container 9967321ad9dd7a2a09e7aafaad4748d3cf61fb4a9d3f2c6fc76c5d8a7f841658: Status 404 returned error can't find the container with id 9967321ad9dd7a2a09e7aafaad4748d3cf61fb4a9d3f2c6fc76c5d8a7f841658
Sep 11 03:44:40.589703 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:44:40.589645    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-hnmlv" event={"ID":"f617c80e-90ca-4535-b0d4-07ef872969a2","Type":"ContainerStarted","Data":"9967321ad9dd7a2a09e7aafaad4748d3cf61fb4a9d3f2c6fc76c5d8a7f841658"}
Sep 11 03:44:42.909613 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:44:42.909578    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-7kjj4"]
Sep 11 03:44:42.913158 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:44:42.913138    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-7kjj4"
Sep 11 03:44:42.915345 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:44:42.915321    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-s9554\""
Sep 11 03:44:42.919566 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:44:42.919531    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-7kjj4"]
Sep 11 03:44:43.068627 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:44:43.068592    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9b695990-79cf-47d4-ac85-35dc19cef19d-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-7kjj4\" (UID: \"9b695990-79cf-47d4-ac85-35dc19cef19d\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-7kjj4"
Sep 11 03:44:43.068835 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:44:43.068743    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6kn47\" (UniqueName: \"kubernetes.io/projected/9b695990-79cf-47d4-ac85-35dc19cef19d-kube-api-access-6kn47\") pod \"cert-manager-cainjector-559cdd8c87-7kjj4\" (UID: \"9b695990-79cf-47d4-ac85-35dc19cef19d\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-7kjj4"
Sep 11 03:44:43.169623 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:44:43.169538    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kn47\" (UniqueName: \"kubernetes.io/projected/9b695990-79cf-47d4-ac85-35dc19cef19d-kube-api-access-6kn47\") pod \"cert-manager-cainjector-559cdd8c87-7kjj4\" (UID: \"9b695990-79cf-47d4-ac85-35dc19cef19d\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-7kjj4"
Sep 11 03:44:43.169623 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:44:43.169595    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9b695990-79cf-47d4-ac85-35dc19cef19d-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-7kjj4\" (UID: \"9b695990-79cf-47d4-ac85-35dc19cef19d\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-7kjj4"
Sep 11 03:44:43.177610 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:44:43.177582    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9b695990-79cf-47d4-ac85-35dc19cef19d-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-7kjj4\" (UID: \"9b695990-79cf-47d4-ac85-35dc19cef19d\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-7kjj4"
Sep 11 03:44:43.177840 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:44:43.177813    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6kn47\" (UniqueName: \"kubernetes.io/projected/9b695990-79cf-47d4-ac85-35dc19cef19d-kube-api-access-6kn47\") pod \"cert-manager-cainjector-559cdd8c87-7kjj4\" (UID: \"9b695990-79cf-47d4-ac85-35dc19cef19d\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-7kjj4"
Sep 11 03:44:43.225789 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:44:43.225757    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-7kjj4"
Sep 11 03:44:43.755081 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:44:43.755021    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-7kjj4"]
Sep 11 03:44:43.757090 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:44:43.757061    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9b695990_79cf_47d4_ac85_35dc19cef19d.slice/crio-807718f2da279c8df1d63ac1e90b6e3f98b05df8379999cc7d9cc4a579ca5076 WatchSource:0}: Error finding container 807718f2da279c8df1d63ac1e90b6e3f98b05df8379999cc7d9cc4a579ca5076: Status 404 returned error can't find the container with id 807718f2da279c8df1d63ac1e90b6e3f98b05df8379999cc7d9cc4a579ca5076
Sep 11 03:44:44.608086 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:44:44.608043    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-7kjj4" event={"ID":"9b695990-79cf-47d4-ac85-35dc19cef19d","Type":"ContainerStarted","Data":"4d993763658d9633118f4066c65539e53bc9e7d963ba1898952ac7846b8e8b9d"}
Sep 11 03:44:44.608086 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:44:44.608092    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-7kjj4" event={"ID":"9b695990-79cf-47d4-ac85-35dc19cef19d","Type":"ContainerStarted","Data":"807718f2da279c8df1d63ac1e90b6e3f98b05df8379999cc7d9cc4a579ca5076"}
Sep 11 03:44:44.609537 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:44:44.609514    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-hnmlv" event={"ID":"f617c80e-90ca-4535-b0d4-07ef872969a2","Type":"ContainerStarted","Data":"1da7c3cccd5efbf8cb3bac5808122d1e356f466cd8ee81f05bde6a7ec3c02c12"}
Sep 11 03:44:44.609647 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:44:44.609634    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-hnmlv"
Sep 11 03:44:44.624464 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:44:44.624398    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-7kjj4" podStartSLOduration=2.6243830150000003 podStartE2EDuration="2.624383015s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 03:44:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:44:44.623454497 +0000 UTC m=+272.781940821" watchObservedRunningTime="2026-09-11 03:44:44.624383015 +0000 UTC m=+272.782869337"
Sep 11 03:44:44.638738 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:44:44.638683    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-hnmlv" podStartSLOduration=2.229686609 podStartE2EDuration="5.638647669s" totalImagesPullingTime="3.40896106s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 03:44:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:44:44.638294373 +0000 UTC m=+272.796780709" watchObservedRunningTime="2026-09-11 03:44:44.638647669 +0000 UTC m=+272.797133993"
Sep 11 03:44:50.614798 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:44:50.614764    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-hnmlv"
Sep 11 03:45:12.281218 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:45:12.281176    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j4pgw_28b30dae-a2d3-4251-99e8-04bf5153015b/ovn-acl-logging/0.log"
Sep 11 03:45:12.284421 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:45:12.284399    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j4pgw_28b30dae-a2d3-4251-99e8-04bf5153015b/ovn-acl-logging/0.log"
Sep 11 03:45:12.286365 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:45:12.286346    2580 kubelet.go:1648] "Image garbage collection succeeded"
Sep 11 03:45:14.019102 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:45:14.019063    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6cf4d46bbf-sbztx"]
Sep 11 03:45:14.022898 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:45:14.022877    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6cf4d46bbf-sbztx"
Sep 11 03:45:14.025251 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:45:14.025217    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 11 03:45:14.025365 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:45:14.025265    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 11 03:45:14.025595 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:45:14.025579    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 11 03:45:14.026060 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:45:14.026045    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 11 03:45:14.026219 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:45:14.026207    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-8t266\""
Sep 11 03:45:14.039233 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:45:14.039213    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6cf4d46bbf-sbztx"]
Sep 11 03:45:14.126085 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:45:14.126047    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/86bf55e1-90b9-436a-98a4-7997e90e0f84-tmp\") pod \"opendatahub-operator-controller-manager-6cf4d46bbf-sbztx\" (UID: \"86bf55e1-90b9-436a-98a4-7997e90e0f84\") " pod="opendatahub/opendatahub-operator-controller-manager-6cf4d46bbf-sbztx"
Sep 11 03:45:14.126265 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:45:14.126107    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/86bf55e1-90b9-436a-98a4-7997e90e0f84-webhook-cert\") pod \"opendatahub-operator-controller-manager-6cf4d46bbf-sbztx\" (UID: \"86bf55e1-90b9-436a-98a4-7997e90e0f84\") " pod="opendatahub/opendatahub-operator-controller-manager-6cf4d46bbf-sbztx"
Sep 11 03:45:14.126265 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:45:14.126156    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxrb5\" (UniqueName: \"kubernetes.io/projected/86bf55e1-90b9-436a-98a4-7997e90e0f84-kube-api-access-sxrb5\") pod \"opendatahub-operator-controller-manager-6cf4d46bbf-sbztx\" (UID: \"86bf55e1-90b9-436a-98a4-7997e90e0f84\") " pod="opendatahub/opendatahub-operator-controller-manager-6cf4d46bbf-sbztx"
Sep 11 03:45:14.126265 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:45:14.126240    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/86bf55e1-90b9-436a-98a4-7997e90e0f84-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6cf4d46bbf-sbztx\" (UID: \"86bf55e1-90b9-436a-98a4-7997e90e0f84\") " pod="opendatahub/opendatahub-operator-controller-manager-6cf4d46bbf-sbztx"
Sep 11 03:45:14.126382 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:45:14.126280    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/86bf55e1-90b9-436a-98a4-7997e90e0f84-manifests\") pod \"opendatahub-operator-controller-manager-6cf4d46bbf-sbztx\" (UID: \"86bf55e1-90b9-436a-98a4-7997e90e0f84\") " pod="opendatahub/opendatahub-operator-controller-manager-6cf4d46bbf-sbztx"
Sep 11 03:45:14.227056 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:45:14.227017    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/86bf55e1-90b9-436a-98a4-7997e90e0f84-tmp\") pod \"opendatahub-operator-controller-manager-6cf4d46bbf-sbztx\" (UID: \"86bf55e1-90b9-436a-98a4-7997e90e0f84\") " pod="opendatahub/opendatahub-operator-controller-manager-6cf4d46bbf-sbztx"
Sep 11 03:45:14.227223 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:45:14.227080    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/86bf55e1-90b9-436a-98a4-7997e90e0f84-webhook-cert\") pod \"opendatahub-operator-controller-manager-6cf4d46bbf-sbztx\" (UID: \"86bf55e1-90b9-436a-98a4-7997e90e0f84\") " pod="opendatahub/opendatahub-operator-controller-manager-6cf4d46bbf-sbztx"
Sep 11 03:45:14.227223 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:45:14.227098    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxrb5\" (UniqueName: \"kubernetes.io/projected/86bf55e1-90b9-436a-98a4-7997e90e0f84-kube-api-access-sxrb5\") pod \"opendatahub-operator-controller-manager-6cf4d46bbf-sbztx\" (UID: \"86bf55e1-90b9-436a-98a4-7997e90e0f84\") " pod="opendatahub/opendatahub-operator-controller-manager-6cf4d46bbf-sbztx"
Sep 11 03:45:14.227223 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:45:14.227128    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/86bf55e1-90b9-436a-98a4-7997e90e0f84-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6cf4d46bbf-sbztx\" (UID: \"86bf55e1-90b9-436a-98a4-7997e90e0f84\") " pod="opendatahub/opendatahub-operator-controller-manager-6cf4d46bbf-sbztx"
Sep 11 03:45:14.227223 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:45:14.227150    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/86bf55e1-90b9-436a-98a4-7997e90e0f84-manifests\") pod \"opendatahub-operator-controller-manager-6cf4d46bbf-sbztx\" (UID: \"86bf55e1-90b9-436a-98a4-7997e90e0f84\") " pod="opendatahub/opendatahub-operator-controller-manager-6cf4d46bbf-sbztx"
Sep 11 03:45:14.227603 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:45:14.227568    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/86bf55e1-90b9-436a-98a4-7997e90e0f84-manifests\") pod \"opendatahub-operator-controller-manager-6cf4d46bbf-sbztx\" (UID: \"86bf55e1-90b9-436a-98a4-7997e90e0f84\") " pod="opendatahub/opendatahub-operator-controller-manager-6cf4d46bbf-sbztx"
Sep 11 03:45:14.229373 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:45:14.229342    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/86bf55e1-90b9-436a-98a4-7997e90e0f84-tmp\") pod \"opendatahub-operator-controller-manager-6cf4d46bbf-sbztx\" (UID: \"86bf55e1-90b9-436a-98a4-7997e90e0f84\") " pod="opendatahub/opendatahub-operator-controller-manager-6cf4d46bbf-sbztx"
Sep 11 03:45:14.229581 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:45:14.229565    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/86bf55e1-90b9-436a-98a4-7997e90e0f84-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6cf4d46bbf-sbztx\" (UID: \"86bf55e1-90b9-436a-98a4-7997e90e0f84\") " pod="opendatahub/opendatahub-operator-controller-manager-6cf4d46bbf-sbztx"
Sep 11 03:45:14.229705 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:45:14.229643    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/86bf55e1-90b9-436a-98a4-7997e90e0f84-webhook-cert\") pod \"opendatahub-operator-controller-manager-6cf4d46bbf-sbztx\" (UID: \"86bf55e1-90b9-436a-98a4-7997e90e0f84\") " pod="opendatahub/opendatahub-operator-controller-manager-6cf4d46bbf-sbztx"
Sep 11 03:45:14.239720 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:45:14.239699    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxrb5\" (UniqueName: \"kubernetes.io/projected/86bf55e1-90b9-436a-98a4-7997e90e0f84-kube-api-access-sxrb5\") pod \"opendatahub-operator-controller-manager-6cf4d46bbf-sbztx\" (UID: \"86bf55e1-90b9-436a-98a4-7997e90e0f84\") " pod="opendatahub/opendatahub-operator-controller-manager-6cf4d46bbf-sbztx"
Sep 11 03:45:14.333110 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:45:14.333010    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6cf4d46bbf-sbztx"
Sep 11 03:45:14.608978 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:45:14.608947    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6cf4d46bbf-sbztx"]
Sep 11 03:45:14.609675 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:45:14.609637    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86bf55e1_90b9_436a_98a4_7997e90e0f84.slice/crio-3e00898102ea37e5a847f06ceef0a0de3139be6fe2765be6382c4579e3e23b28 WatchSource:0}: Error finding container 3e00898102ea37e5a847f06ceef0a0de3139be6fe2765be6382c4579e3e23b28: Status 404 returned error can't find the container with id 3e00898102ea37e5a847f06ceef0a0de3139be6fe2765be6382c4579e3e23b28
Sep 11 03:45:14.611405 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:45:14.611389    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 03:45:14.728556 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:45:14.728521    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6cf4d46bbf-sbztx" event={"ID":"86bf55e1-90b9-436a-98a4-7997e90e0f84","Type":"ContainerStarted","Data":"3e00898102ea37e5a847f06ceef0a0de3139be6fe2765be6382c4579e3e23b28"}
Sep 11 03:45:18.748352 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:45:18.748314    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6cf4d46bbf-sbztx" event={"ID":"86bf55e1-90b9-436a-98a4-7997e90e0f84","Type":"ContainerStarted","Data":"c93aac1c0a850cc2c0cc92194a50bd6ff35bef437d0f450c45bcef4ed7ea2724"}
Sep 11 03:45:19.753578 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:45:19.753539    2580 generic.go:356] "Generic (PLEG): container finished" podID="86bf55e1-90b9-436a-98a4-7997e90e0f84" containerID="c93aac1c0a850cc2c0cc92194a50bd6ff35bef437d0f450c45bcef4ed7ea2724" exitCode=0
Sep 11 03:45:19.753986 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:45:19.753587    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6cf4d46bbf-sbztx" event={"ID":"86bf55e1-90b9-436a-98a4-7997e90e0f84","Type":"ContainerDied","Data":"c93aac1c0a850cc2c0cc92194a50bd6ff35bef437d0f450c45bcef4ed7ea2724"}
Sep 11 03:45:20.759686 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:45:20.759634    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6cf4d46bbf-sbztx" event={"ID":"86bf55e1-90b9-436a-98a4-7997e90e0f84","Type":"ContainerStarted","Data":"ab58d64666fbf382be0fad7a7565d54d87e8d9dc3f9fec8d632a115137a34cb4"}
Sep 11 03:45:20.760065 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:45:20.759719    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-6cf4d46bbf-sbztx"
Sep 11 03:45:20.782882 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:45:20.782831    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-6cf4d46bbf-sbztx" podStartSLOduration=3.188690403 podStartE2EDuration="7.782815877s" totalImagesPullingTime="3.54631955s" totalInitContainerRuntime="1.047805924s" isStatefulPod=false podCreationTimestamp="2026-09-11 03:45:13 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:45:20.781325146 +0000 UTC m=+308.939811469" watchObservedRunningTime="2026-09-11 03:45:20.782815877 +0000 UTC m=+308.941302200"
Sep 11 03:45:31.765982 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:45:31.765945    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-6cf4d46bbf-sbztx"
Sep 11 03:45:33.789021 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:45:33.788988    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-67c8664749-8wbt4"]
Sep 11 03:45:33.797503 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:45:33.797483    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-67c8664749-8wbt4"
Sep 11 03:45:33.799798 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:45:33.799775    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 11 03:45:33.799911 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:45:33.799839    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 11 03:45:33.800876 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:45:33.800844    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 11 03:45:33.800876 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:45:33.800869    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 11 03:45:33.801030 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:45:33.800846    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-vfnc6\""
Sep 11 03:45:33.802770 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:45:33.802751    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 11 03:45:33.803367 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:45:33.803348    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-67c8664749-8wbt4"]
Sep 11 03:45:33.891582 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:45:33.891556    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/cd30e29b-6de1-44ea-9ba1-f67605725a2a-manager-config\") pod \"lws-controller-manager-67c8664749-8wbt4\" (UID: \"cd30e29b-6de1-44ea-9ba1-f67605725a2a\") " pod="openshift-lws-operator/lws-controller-manager-67c8664749-8wbt4"
Sep 11 03:45:33.891752 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:45:33.891591    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5gcr\" (UniqueName: \"kubernetes.io/projected/cd30e29b-6de1-44ea-9ba1-f67605725a2a-kube-api-access-z5gcr\") pod \"lws-controller-manager-67c8664749-8wbt4\" (UID: \"cd30e29b-6de1-44ea-9ba1-f67605725a2a\") " pod="openshift-lws-operator/lws-controller-manager-67c8664749-8wbt4"
Sep 11 03:45:33.891752 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:45:33.891623    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cd30e29b-6de1-44ea-9ba1-f67605725a2a-cert\") pod \"lws-controller-manager-67c8664749-8wbt4\" (UID: \"cd30e29b-6de1-44ea-9ba1-f67605725a2a\") " pod="openshift-lws-operator/lws-controller-manager-67c8664749-8wbt4"
Sep 11 03:45:33.891752 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:45:33.891708    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/cd30e29b-6de1-44ea-9ba1-f67605725a2a-metrics-cert\") pod \"lws-controller-manager-67c8664749-8wbt4\" (UID: \"cd30e29b-6de1-44ea-9ba1-f67605725a2a\") " pod="openshift-lws-operator/lws-controller-manager-67c8664749-8wbt4"
Sep 11 03:45:33.992419 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:45:33.992378    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/cd30e29b-6de1-44ea-9ba1-f67605725a2a-manager-config\") pod \"lws-controller-manager-67c8664749-8wbt4\" (UID: \"cd30e29b-6de1-44ea-9ba1-f67605725a2a\") " pod="openshift-lws-operator/lws-controller-manager-67c8664749-8wbt4"
Sep 11 03:45:33.992419 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:45:33.992424    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5gcr\" (UniqueName: \"kubernetes.io/projected/cd30e29b-6de1-44ea-9ba1-f67605725a2a-kube-api-access-z5gcr\") pod \"lws-controller-manager-67c8664749-8wbt4\" (UID: \"cd30e29b-6de1-44ea-9ba1-f67605725a2a\") " pod="openshift-lws-operator/lws-controller-manager-67c8664749-8wbt4"
Sep 11 03:45:33.992657 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:45:33.992460    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cd30e29b-6de1-44ea-9ba1-f67605725a2a-cert\") pod \"lws-controller-manager-67c8664749-8wbt4\" (UID: \"cd30e29b-6de1-44ea-9ba1-f67605725a2a\") " pod="openshift-lws-operator/lws-controller-manager-67c8664749-8wbt4"
Sep 11 03:45:33.992657 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:45:33.992498    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/cd30e29b-6de1-44ea-9ba1-f67605725a2a-metrics-cert\") pod \"lws-controller-manager-67c8664749-8wbt4\" (UID: \"cd30e29b-6de1-44ea-9ba1-f67605725a2a\") " pod="openshift-lws-operator/lws-controller-manager-67c8664749-8wbt4"
Sep 11 03:45:33.993060 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:45:33.993041    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/cd30e29b-6de1-44ea-9ba1-f67605725a2a-manager-config\") pod \"lws-controller-manager-67c8664749-8wbt4\" (UID: \"cd30e29b-6de1-44ea-9ba1-f67605725a2a\") " pod="openshift-lws-operator/lws-controller-manager-67c8664749-8wbt4"
Sep 11 03:45:33.994995 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:45:33.994968    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cd30e29b-6de1-44ea-9ba1-f67605725a2a-cert\") pod \"lws-controller-manager-67c8664749-8wbt4\" (UID: \"cd30e29b-6de1-44ea-9ba1-f67605725a2a\") " pod="openshift-lws-operator/lws-controller-manager-67c8664749-8wbt4"
Sep 11 03:45:33.995084 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:45:33.995066    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/cd30e29b-6de1-44ea-9ba1-f67605725a2a-metrics-cert\") pod \"lws-controller-manager-67c8664749-8wbt4\" (UID: \"cd30e29b-6de1-44ea-9ba1-f67605725a2a\") " pod="openshift-lws-operator/lws-controller-manager-67c8664749-8wbt4"
Sep 11 03:45:34.001427 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:45:34.001408    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5gcr\" (UniqueName: \"kubernetes.io/projected/cd30e29b-6de1-44ea-9ba1-f67605725a2a-kube-api-access-z5gcr\") pod \"lws-controller-manager-67c8664749-8wbt4\" (UID: \"cd30e29b-6de1-44ea-9ba1-f67605725a2a\") " pod="openshift-lws-operator/lws-controller-manager-67c8664749-8wbt4"
Sep 11 03:45:34.107083 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:45:34.106989    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-67c8664749-8wbt4"
Sep 11 03:45:34.376800 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:45:34.376721    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-67c8664749-8wbt4"]
Sep 11 03:45:34.379696 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:45:34.379649    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd30e29b_6de1_44ea_9ba1_f67605725a2a.slice/crio-753dfd681d42440fa35c2c6228a6750046ebdbd7fa1fc87cae1029531516dcfa WatchSource:0}: Error finding container 753dfd681d42440fa35c2c6228a6750046ebdbd7fa1fc87cae1029531516dcfa: Status 404 returned error can't find the container with id 753dfd681d42440fa35c2c6228a6750046ebdbd7fa1fc87cae1029531516dcfa
Sep 11 03:45:34.824491 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:45:34.824454    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-67c8664749-8wbt4" event={"ID":"cd30e29b-6de1-44ea-9ba1-f67605725a2a","Type":"ContainerStarted","Data":"753dfd681d42440fa35c2c6228a6750046ebdbd7fa1fc87cae1029531516dcfa"}
Sep 11 03:45:36.836647 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:45:36.836564    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-67c8664749-8wbt4" event={"ID":"cd30e29b-6de1-44ea-9ba1-f67605725a2a","Type":"ContainerStarted","Data":"8cd5b20292560975b4698f5352e46fb2fb202ad34fbbca5473435b016cbe1666"}
Sep 11 03:45:36.837000 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:45:36.836688    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-67c8664749-8wbt4"
Sep 11 03:45:36.852341 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:45:36.852298    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-67c8664749-8wbt4" podStartSLOduration=1.678467816 podStartE2EDuration="3.852284245s" totalImagesPullingTime="2.173816429s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 03:45:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:45:36.851170473 +0000 UTC m=+325.009656796" watchObservedRunningTime="2026-09-11 03:45:36.852284245 +0000 UTC m=+325.010770568"
Sep 11 03:45:47.843894 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:45:47.843858    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-67c8664749-8wbt4"
Sep 11 03:45:52.913063 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:45:52.913030    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-8kfvp"]
Sep 11 03:45:52.922202 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:45:52.922180    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-8kfvp"
Sep 11 03:45:52.925317 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:45:52.925290    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Sep 11 03:45:52.925493 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:45:52.925363    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 11 03:45:52.925818 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:45:52.925797    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Sep 11 03:45:52.925912 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:45:52.925862    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Sep 11 03:45:52.926099 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:45:52.926083    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 11 03:45:52.926198 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:45:52.926111    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 11 03:45:52.926358 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:45:52.926342    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-l9kcw\""
Sep 11 03:45:52.931947 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:45:52.931925    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-8kfvp"]
Sep 11 03:45:53.058115 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:45:53.058081    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/bf009cc0-6d19-4ecb-adbc-81b9751265d5-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-8kfvp\" (UID: \"bf009cc0-6d19-4ecb-adbc-81b9751265d5\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-8kfvp"
Sep 11 03:45:53.058287 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:45:53.058147    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/bf009cc0-6d19-4ecb-adbc-81b9751265d5-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-8kfvp\" (UID: \"bf009cc0-6d19-4ecb-adbc-81b9751265d5\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-8kfvp"
Sep 11 03:45:53.058287 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:45:53.058216    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ppbqz\" (UniqueName: \"kubernetes.io/projected/bf009cc0-6d19-4ecb-adbc-81b9751265d5-kube-api-access-ppbqz\") pod \"istiod-openshift-gateway-5677ccb49b-8kfvp\" (UID: \"bf009cc0-6d19-4ecb-adbc-81b9751265d5\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-8kfvp"
Sep 11 03:45:53.058287 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:45:53.058264    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/bf009cc0-6d19-4ecb-adbc-81b9751265d5-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-8kfvp\" (UID: \"bf009cc0-6d19-4ecb-adbc-81b9751265d5\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-8kfvp"
Sep 11 03:45:53.058415 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:45:53.058343    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf009cc0-6d19-4ecb-adbc-81b9751265d5-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-8kfvp\" (UID: \"bf009cc0-6d19-4ecb-adbc-81b9751265d5\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-8kfvp"
Sep 11 03:45:53.058415 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:45:53.058370    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf009cc0-6d19-4ecb-adbc-81b9751265d5-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-8kfvp\" (UID: \"bf009cc0-6d19-4ecb-adbc-81b9751265d5\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-8kfvp"
Sep 11 03:45:53.058415 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:45:53.058405    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/bf009cc0-6d19-4ecb-adbc-81b9751265d5-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-8kfvp\" (UID: \"bf009cc0-6d19-4ecb-adbc-81b9751265d5\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-8kfvp"
Sep 11 03:45:53.159613 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:45:53.159571    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/bf009cc0-6d19-4ecb-adbc-81b9751265d5-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-8kfvp\" (UID: \"bf009cc0-6d19-4ecb-adbc-81b9751265d5\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-8kfvp"
Sep 11 03:45:53.159613 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:45:53.159620    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppbqz\" (UniqueName: \"kubernetes.io/projected/bf009cc0-6d19-4ecb-adbc-81b9751265d5-kube-api-access-ppbqz\") pod \"istiod-openshift-gateway-5677ccb49b-8kfvp\" (UID: \"bf009cc0-6d19-4ecb-adbc-81b9751265d5\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-8kfvp"
Sep 11 03:45:53.159887 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:45:53.159646    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/bf009cc0-6d19-4ecb-adbc-81b9751265d5-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-8kfvp\" (UID: \"bf009cc0-6d19-4ecb-adbc-81b9751265d5\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-8kfvp"
Sep 11 03:45:53.159887 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:45:53.159717    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf009cc0-6d19-4ecb-adbc-81b9751265d5-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-8kfvp\" (UID: \"bf009cc0-6d19-4ecb-adbc-81b9751265d5\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-8kfvp"
Sep 11 03:45:53.159887 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:45:53.159733    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf009cc0-6d19-4ecb-adbc-81b9751265d5-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-8kfvp\" (UID: \"bf009cc0-6d19-4ecb-adbc-81b9751265d5\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-8kfvp"
Sep 11 03:45:53.159887 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:45:53.159764    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/bf009cc0-6d19-4ecb-adbc-81b9751265d5-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-8kfvp\" (UID: \"bf009cc0-6d19-4ecb-adbc-81b9751265d5\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-8kfvp"
Sep 11 03:45:53.159887 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:45:53.159796    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/bf009cc0-6d19-4ecb-adbc-81b9751265d5-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-8kfvp\" (UID: \"bf009cc0-6d19-4ecb-adbc-81b9751265d5\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-8kfvp"
Sep 11 03:45:53.160480 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:45:53.160441    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/bf009cc0-6d19-4ecb-adbc-81b9751265d5-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-8kfvp\" (UID: \"bf009cc0-6d19-4ecb-adbc-81b9751265d5\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-8kfvp"
Sep 11 03:45:53.162150 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:45:53.162118    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf009cc0-6d19-4ecb-adbc-81b9751265d5-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-8kfvp\" (UID: \"bf009cc0-6d19-4ecb-adbc-81b9751265d5\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-8kfvp"
Sep 11 03:45:53.162150 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:45:53.162133    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/bf009cc0-6d19-4ecb-adbc-81b9751265d5-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-8kfvp\" (UID: \"bf009cc0-6d19-4ecb-adbc-81b9751265d5\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-8kfvp"
Sep 11 03:45:53.162278 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:45:53.162118    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/bf009cc0-6d19-4ecb-adbc-81b9751265d5-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-8kfvp\" (UID: \"bf009cc0-6d19-4ecb-adbc-81b9751265d5\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-8kfvp"
Sep 11 03:45:53.162314 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:45:53.162284    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/bf009cc0-6d19-4ecb-adbc-81b9751265d5-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-8kfvp\" (UID: \"bf009cc0-6d19-4ecb-adbc-81b9751265d5\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-8kfvp"
Sep 11 03:45:53.177754 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:45:53.177695    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf009cc0-6d19-4ecb-adbc-81b9751265d5-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-8kfvp\" (UID: \"bf009cc0-6d19-4ecb-adbc-81b9751265d5\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-8kfvp"
Sep 11 03:45:53.180396 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:45:53.180372    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ppbqz\" (UniqueName: \"kubernetes.io/projected/bf009cc0-6d19-4ecb-adbc-81b9751265d5-kube-api-access-ppbqz\") pod \"istiod-openshift-gateway-5677ccb49b-8kfvp\" (UID: \"bf009cc0-6d19-4ecb-adbc-81b9751265d5\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-8kfvp"
Sep 11 03:45:53.232378 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:45:53.232347    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-8kfvp"
Sep 11 03:45:53.505637 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:45:53.505564    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-8kfvp"]
Sep 11 03:45:53.507835 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:45:53.507810    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf009cc0_6d19_4ecb_adbc_81b9751265d5.slice/crio-9333d50fb1decbb02a97a39ea53c17ca1bd8407d661bb5564b657c4e068731b4 WatchSource:0}: Error finding container 9333d50fb1decbb02a97a39ea53c17ca1bd8407d661bb5564b657c4e068731b4: Status 404 returned error can't find the container with id 9333d50fb1decbb02a97a39ea53c17ca1bd8407d661bb5564b657c4e068731b4
Sep 11 03:45:53.915155 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:45:53.915119    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-8kfvp" event={"ID":"bf009cc0-6d19-4ecb-adbc-81b9751265d5","Type":"ContainerStarted","Data":"9333d50fb1decbb02a97a39ea53c17ca1bd8407d661bb5564b657c4e068731b4"}
Sep 11 03:45:55.763858 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:45:55.763814    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 03:45:55.764240 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:45:55.763896    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 03:45:55.926537 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:45:55.926503    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-8kfvp" event={"ID":"bf009cc0-6d19-4ecb-adbc-81b9751265d5","Type":"ContainerStarted","Data":"fc8a76dd2361b7ba4f7a33691892bb7b0e619aaac9bcaab9b71b6cde4f33839a"}
Sep 11 03:45:55.926726 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:45:55.926593    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-8kfvp"
Sep 11 03:45:55.976293 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:45:55.976238    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-8kfvp" podStartSLOduration=1.722588718 podStartE2EDuration="3.976222106s" totalImagesPullingTime="2.253633388s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 03:45:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:45:55.974894379 +0000 UTC m=+344.133380702" watchObservedRunningTime="2026-09-11 03:45:55.976222106 +0000 UTC m=+344.134708429"
Sep 11 03:45:56.932866 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:45:56.932835    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-8kfvp"
Sep 11 03:47:14.382718 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:47:14.382678    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7886644f6d-qpjhb"]
Sep 11 03:47:14.386312 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:47:14.386287    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7886644f6d-qpjhb"
Sep 11 03:47:14.388722 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:47:14.388695    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 11 03:47:14.388852 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:47:14.388744    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 11 03:47:14.388852 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:47:14.388751    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 11 03:47:14.388852 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:47:14.388806    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 11 03:47:14.389820 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:47:14.389802    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-x2hwp\""
Sep 11 03:47:14.389943 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:47:14.389891    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 11 03:47:14.393442 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:47:14.393423    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 11 03:47:14.402192 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:47:14.402167    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7886644f6d-qpjhb"]
Sep 11 03:47:14.430537 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:47:14.430510    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9db50db7-59ba-4828-b3c4-2ef68b00d22c-console-config\") pod \"console-7886644f6d-qpjhb\" (UID: \"9db50db7-59ba-4828-b3c4-2ef68b00d22c\") " pod="openshift-console/console-7886644f6d-qpjhb"
Sep 11 03:47:14.430638 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:47:14.430550    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9db50db7-59ba-4828-b3c4-2ef68b00d22c-trusted-ca-bundle\") pod \"console-7886644f6d-qpjhb\" (UID: \"9db50db7-59ba-4828-b3c4-2ef68b00d22c\") " pod="openshift-console/console-7886644f6d-qpjhb"
Sep 11 03:47:14.430638 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:47:14.430579    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9db50db7-59ba-4828-b3c4-2ef68b00d22c-console-serving-cert\") pod \"console-7886644f6d-qpjhb\" (UID: \"9db50db7-59ba-4828-b3c4-2ef68b00d22c\") " pod="openshift-console/console-7886644f6d-qpjhb"
Sep 11 03:47:14.430638 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:47:14.430624    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9db50db7-59ba-4828-b3c4-2ef68b00d22c-console-oauth-config\") pod \"console-7886644f6d-qpjhb\" (UID: \"9db50db7-59ba-4828-b3c4-2ef68b00d22c\") " pod="openshift-console/console-7886644f6d-qpjhb"
Sep 11 03:47:14.430798 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:47:14.430655    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8dqf5\" (UniqueName: \"kubernetes.io/projected/9db50db7-59ba-4828-b3c4-2ef68b00d22c-kube-api-access-8dqf5\") pod \"console-7886644f6d-qpjhb\" (UID: \"9db50db7-59ba-4828-b3c4-2ef68b00d22c\") " pod="openshift-console/console-7886644f6d-qpjhb"
Sep 11 03:47:14.430798 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:47:14.430729    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9db50db7-59ba-4828-b3c4-2ef68b00d22c-service-ca\") pod \"console-7886644f6d-qpjhb\" (UID: \"9db50db7-59ba-4828-b3c4-2ef68b00d22c\") " pod="openshift-console/console-7886644f6d-qpjhb"
Sep 11 03:47:14.430798 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:47:14.430779    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9db50db7-59ba-4828-b3c4-2ef68b00d22c-oauth-serving-cert\") pod \"console-7886644f6d-qpjhb\" (UID: \"9db50db7-59ba-4828-b3c4-2ef68b00d22c\") " pod="openshift-console/console-7886644f6d-qpjhb"
Sep 11 03:47:14.531151 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:47:14.531120    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9db50db7-59ba-4828-b3c4-2ef68b00d22c-console-config\") pod \"console-7886644f6d-qpjhb\" (UID: \"9db50db7-59ba-4828-b3c4-2ef68b00d22c\") " pod="openshift-console/console-7886644f6d-qpjhb"
Sep 11 03:47:14.531303 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:47:14.531167    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9db50db7-59ba-4828-b3c4-2ef68b00d22c-trusted-ca-bundle\") pod \"console-7886644f6d-qpjhb\" (UID: \"9db50db7-59ba-4828-b3c4-2ef68b00d22c\") " pod="openshift-console/console-7886644f6d-qpjhb"
Sep 11 03:47:14.531303 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:47:14.531196    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9db50db7-59ba-4828-b3c4-2ef68b00d22c-console-serving-cert\") pod \"console-7886644f6d-qpjhb\" (UID: \"9db50db7-59ba-4828-b3c4-2ef68b00d22c\") " pod="openshift-console/console-7886644f6d-qpjhb"
Sep 11 03:47:14.531303 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:47:14.531216    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9db50db7-59ba-4828-b3c4-2ef68b00d22c-console-oauth-config\") pod \"console-7886644f6d-qpjhb\" (UID: \"9db50db7-59ba-4828-b3c4-2ef68b00d22c\") " pod="openshift-console/console-7886644f6d-qpjhb"
Sep 11 03:47:14.531303 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:47:14.531240    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8dqf5\" (UniqueName: \"kubernetes.io/projected/9db50db7-59ba-4828-b3c4-2ef68b00d22c-kube-api-access-8dqf5\") pod \"console-7886644f6d-qpjhb\" (UID: \"9db50db7-59ba-4828-b3c4-2ef68b00d22c\") " pod="openshift-console/console-7886644f6d-qpjhb"
Sep 11 03:47:14.531477 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:47:14.531411    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9db50db7-59ba-4828-b3c4-2ef68b00d22c-service-ca\") pod \"console-7886644f6d-qpjhb\" (UID: \"9db50db7-59ba-4828-b3c4-2ef68b00d22c\") " pod="openshift-console/console-7886644f6d-qpjhb"
Sep 11 03:47:14.531531 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:47:14.531473    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9db50db7-59ba-4828-b3c4-2ef68b00d22c-oauth-serving-cert\") pod \"console-7886644f6d-qpjhb\" (UID: \"9db50db7-59ba-4828-b3c4-2ef68b00d22c\") " pod="openshift-console/console-7886644f6d-qpjhb"
Sep 11 03:47:14.532016 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:47:14.531987    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9db50db7-59ba-4828-b3c4-2ef68b00d22c-console-config\") pod \"console-7886644f6d-qpjhb\" (UID: \"9db50db7-59ba-4828-b3c4-2ef68b00d22c\") " pod="openshift-console/console-7886644f6d-qpjhb"
Sep 11 03:47:14.532124 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:47:14.532071    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9db50db7-59ba-4828-b3c4-2ef68b00d22c-service-ca\") pod \"console-7886644f6d-qpjhb\" (UID: \"9db50db7-59ba-4828-b3c4-2ef68b00d22c\") " pod="openshift-console/console-7886644f6d-qpjhb"
Sep 11 03:47:14.532186 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:47:14.532160    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9db50db7-59ba-4828-b3c4-2ef68b00d22c-oauth-serving-cert\") pod \"console-7886644f6d-qpjhb\" (UID: \"9db50db7-59ba-4828-b3c4-2ef68b00d22c\") " pod="openshift-console/console-7886644f6d-qpjhb"
Sep 11 03:47:14.532229 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:47:14.532195    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9db50db7-59ba-4828-b3c4-2ef68b00d22c-trusted-ca-bundle\") pod \"console-7886644f6d-qpjhb\" (UID: \"9db50db7-59ba-4828-b3c4-2ef68b00d22c\") " pod="openshift-console/console-7886644f6d-qpjhb"
Sep 11 03:47:14.533687 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:47:14.533650    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9db50db7-59ba-4828-b3c4-2ef68b00d22c-console-oauth-config\") pod \"console-7886644f6d-qpjhb\" (UID: \"9db50db7-59ba-4828-b3c4-2ef68b00d22c\") " pod="openshift-console/console-7886644f6d-qpjhb"
Sep 11 03:47:14.533780 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:47:14.533757    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9db50db7-59ba-4828-b3c4-2ef68b00d22c-console-serving-cert\") pod \"console-7886644f6d-qpjhb\" (UID: \"9db50db7-59ba-4828-b3c4-2ef68b00d22c\") " pod="openshift-console/console-7886644f6d-qpjhb"
Sep 11 03:47:14.539423 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:47:14.539404    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8dqf5\" (UniqueName: \"kubernetes.io/projected/9db50db7-59ba-4828-b3c4-2ef68b00d22c-kube-api-access-8dqf5\") pod \"console-7886644f6d-qpjhb\" (UID: \"9db50db7-59ba-4828-b3c4-2ef68b00d22c\") " pod="openshift-console/console-7886644f6d-qpjhb"
Sep 11 03:47:14.696166 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:47:14.696125    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7886644f6d-qpjhb"
Sep 11 03:47:14.967233 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:47:14.967173    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7886644f6d-qpjhb"]
Sep 11 03:47:14.969585 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:47:14.969550    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9db50db7_59ba_4828_b3c4_2ef68b00d22c.slice/crio-eddc72ff9b5b566748daa2711e365786b928376efdd04c3760f72d815e20d008 WatchSource:0}: Error finding container eddc72ff9b5b566748daa2711e365786b928376efdd04c3760f72d815e20d008: Status 404 returned error can't find the container with id eddc72ff9b5b566748daa2711e365786b928376efdd04c3760f72d815e20d008
Sep 11 03:47:15.267616 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:47:15.267522    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7886644f6d-qpjhb" event={"ID":"9db50db7-59ba-4828-b3c4-2ef68b00d22c","Type":"ContainerStarted","Data":"4532f6ab4dcfe04c78877dcb7abfca2b300cc5ef5bf5e378833f8be7bdccabbd"}
Sep 11 03:47:15.267616 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:47:15.267574    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7886644f6d-qpjhb" event={"ID":"9db50db7-59ba-4828-b3c4-2ef68b00d22c","Type":"ContainerStarted","Data":"eddc72ff9b5b566748daa2711e365786b928376efdd04c3760f72d815e20d008"}
Sep 11 03:47:15.288578 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:47:15.288527    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7886644f6d-qpjhb" podStartSLOduration=1.288510267 podStartE2EDuration="1.288510267s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 03:47:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:47:15.286086307 +0000 UTC m=+423.444572630" watchObservedRunningTime="2026-09-11 03:47:15.288510267 +0000 UTC m=+423.446996629"
Sep 11 03:47:24.697129 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:47:24.697091    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7886644f6d-qpjhb"
Sep 11 03:47:24.697620 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:47:24.697159    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7886644f6d-qpjhb"
Sep 11 03:47:24.702032 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:47:24.702004    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7886644f6d-qpjhb"
Sep 11 03:47:25.316780 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:47:25.316751    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7886644f6d-qpjhb"
Sep 11 03:47:40.166716 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:47:40.166638    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-c6p8p"]
Sep 11 03:47:40.169897 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:47:40.169875    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-c6p8p"
Sep 11 03:47:40.172765 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:47:40.172742    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 11 03:47:40.173356 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:47:40.173339    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 11 03:47:40.173795 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:47:40.173778    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-f82vk\""
Sep 11 03:47:40.182272 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:47:40.182251    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-c6p8p"]
Sep 11 03:47:40.259658 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:47:40.259624    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zglv5\" (UniqueName: \"kubernetes.io/projected/1066c072-19b7-4e56-92dd-c9e46a945e68-kube-api-access-zglv5\") pod \"kuadrant-operator-controller-manager-5769f99d9f-c6p8p\" (UID: \"1066c072-19b7-4e56-92dd-c9e46a945e68\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-c6p8p"
Sep 11 03:47:40.259834 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:47:40.259777    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1066c072-19b7-4e56-92dd-c9e46a945e68-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-c6p8p\" (UID: \"1066c072-19b7-4e56-92dd-c9e46a945e68\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-c6p8p"
Sep 11 03:47:40.361051 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:47:40.361012    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zglv5\" (UniqueName: \"kubernetes.io/projected/1066c072-19b7-4e56-92dd-c9e46a945e68-kube-api-access-zglv5\") pod \"kuadrant-operator-controller-manager-5769f99d9f-c6p8p\" (UID: \"1066c072-19b7-4e56-92dd-c9e46a945e68\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-c6p8p"
Sep 11 03:47:40.361218 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:47:40.361173    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1066c072-19b7-4e56-92dd-c9e46a945e68-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-c6p8p\" (UID: \"1066c072-19b7-4e56-92dd-c9e46a945e68\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-c6p8p"
Sep 11 03:47:40.361557 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:47:40.361538    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1066c072-19b7-4e56-92dd-c9e46a945e68-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-c6p8p\" (UID: \"1066c072-19b7-4e56-92dd-c9e46a945e68\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-c6p8p"
Sep 11 03:47:40.376481 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:47:40.376446    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zglv5\" (UniqueName: \"kubernetes.io/projected/1066c072-19b7-4e56-92dd-c9e46a945e68-kube-api-access-zglv5\") pod \"kuadrant-operator-controller-manager-5769f99d9f-c6p8p\" (UID: \"1066c072-19b7-4e56-92dd-c9e46a945e68\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-c6p8p"
Sep 11 03:47:40.480395 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:47:40.480314    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-c6p8p"
Sep 11 03:47:40.760547 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:47:40.760460    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-c6p8p"]
Sep 11 03:47:40.764957 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:47:40.764932    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1066c072_19b7_4e56_92dd_c9e46a945e68.slice/crio-f70a84b6d7b3643eda5e753d8f46878e8daaffb5b2a9676f09f4b63f97aa7ebb WatchSource:0}: Error finding container f70a84b6d7b3643eda5e753d8f46878e8daaffb5b2a9676f09f4b63f97aa7ebb: Status 404 returned error can't find the container with id f70a84b6d7b3643eda5e753d8f46878e8daaffb5b2a9676f09f4b63f97aa7ebb
Sep 11 03:47:41.389918 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:47:41.389880    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-c6p8p" event={"ID":"1066c072-19b7-4e56-92dd-c9e46a945e68","Type":"ContainerStarted","Data":"f70a84b6d7b3643eda5e753d8f46878e8daaffb5b2a9676f09f4b63f97aa7ebb"}
Sep 11 03:47:44.407907 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:47:44.407872    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-c6p8p" event={"ID":"1066c072-19b7-4e56-92dd-c9e46a945e68","Type":"ContainerStarted","Data":"77c694e42788fed83af8bfd0466d8aee750dfdbc46621adbf2e380cd7f66c67e"}
Sep 11 03:47:44.408244 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:47:44.408000    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-c6p8p"
Sep 11 03:47:44.429813 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:47:44.429769    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-c6p8p" podStartSLOduration=0.883660636 podStartE2EDuration="4.429753895s" totalImagesPullingTime="3.546093259s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 03:47:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:47:44.428177892 +0000 UTC m=+452.586664211" watchObservedRunningTime="2026-09-11 03:47:44.429753895 +0000 UTC m=+452.588240217"
Sep 11 03:47:55.414645 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:47:55.414607    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-c6p8p"
Sep 11 03:49:54.599728 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:49:54.599685    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-2rvkj"]
Sep 11 03:49:54.603175 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:49:54.603150    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-2rvkj"
Sep 11 03:49:54.605504 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:49:54.605484    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 11 03:49:54.605615 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:49:54.605484    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 11 03:49:54.606453 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:49:54.606427    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Sep 11 03:49:54.606546 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:49:54.606457    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-d5247\""
Sep 11 03:49:54.612997 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:49:54.612974    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-2rvkj"]
Sep 11 03:49:54.748795 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:49:54.748759    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnstx\" (UniqueName: \"kubernetes.io/projected/3cd86dce-6008-4b5d-9455-069889bc79cd-kube-api-access-nnstx\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-2rvkj\" (UID: \"3cd86dce-6008-4b5d-9455-069889bc79cd\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-2rvkj"
Sep 11 03:49:54.748795 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:49:54.748801    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3cd86dce-6008-4b5d-9455-069889bc79cd-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-2rvkj\" (UID: \"3cd86dce-6008-4b5d-9455-069889bc79cd\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-2rvkj"
Sep 11 03:49:54.748997 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:49:54.748821    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3cd86dce-6008-4b5d-9455-069889bc79cd-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-2rvkj\" (UID: \"3cd86dce-6008-4b5d-9455-069889bc79cd\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-2rvkj"
Sep 11 03:49:54.748997 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:49:54.748906    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3cd86dce-6008-4b5d-9455-069889bc79cd-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-2rvkj\" (UID: \"3cd86dce-6008-4b5d-9455-069889bc79cd\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-2rvkj"
Sep 11 03:49:54.748997 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:49:54.748941    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3cd86dce-6008-4b5d-9455-069889bc79cd-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-2rvkj\" (UID: \"3cd86dce-6008-4b5d-9455-069889bc79cd\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-2rvkj"
Sep 11 03:49:54.748997 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:49:54.748974    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3cd86dce-6008-4b5d-9455-069889bc79cd-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-2rvkj\" (UID: \"3cd86dce-6008-4b5d-9455-069889bc79cd\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-2rvkj"
Sep 11 03:49:54.850265 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:49:54.850174    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnstx\" (UniqueName: \"kubernetes.io/projected/3cd86dce-6008-4b5d-9455-069889bc79cd-kube-api-access-nnstx\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-2rvkj\" (UID: \"3cd86dce-6008-4b5d-9455-069889bc79cd\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-2rvkj"
Sep 11 03:49:54.850265 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:49:54.850214    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3cd86dce-6008-4b5d-9455-069889bc79cd-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-2rvkj\" (UID: \"3cd86dce-6008-4b5d-9455-069889bc79cd\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-2rvkj"
Sep 11 03:49:54.850265 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:49:54.850231    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3cd86dce-6008-4b5d-9455-069889bc79cd-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-2rvkj\" (UID: \"3cd86dce-6008-4b5d-9455-069889bc79cd\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-2rvkj"
Sep 11 03:49:54.850265 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:49:54.850259    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3cd86dce-6008-4b5d-9455-069889bc79cd-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-2rvkj\" (UID: \"3cd86dce-6008-4b5d-9455-069889bc79cd\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-2rvkj"
Sep 11 03:49:54.850590 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:49:54.850275    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3cd86dce-6008-4b5d-9455-069889bc79cd-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-2rvkj\" (UID: \"3cd86dce-6008-4b5d-9455-069889bc79cd\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-2rvkj"
Sep 11 03:49:54.850590 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:49:54.850304    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3cd86dce-6008-4b5d-9455-069889bc79cd-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-2rvkj\" (UID: \"3cd86dce-6008-4b5d-9455-069889bc79cd\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-2rvkj"
Sep 11 03:49:54.850684 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:49:54.850645    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3cd86dce-6008-4b5d-9455-069889bc79cd-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-2rvkj\" (UID: \"3cd86dce-6008-4b5d-9455-069889bc79cd\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-2rvkj"
Sep 11 03:49:54.850732 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:49:54.850643    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3cd86dce-6008-4b5d-9455-069889bc79cd-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-2rvkj\" (UID: \"3cd86dce-6008-4b5d-9455-069889bc79cd\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-2rvkj"
Sep 11 03:49:54.850732 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:49:54.850716    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3cd86dce-6008-4b5d-9455-069889bc79cd-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-2rvkj\" (UID: \"3cd86dce-6008-4b5d-9455-069889bc79cd\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-2rvkj"
Sep 11 03:49:54.852604 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:49:54.852575    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3cd86dce-6008-4b5d-9455-069889bc79cd-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-2rvkj\" (UID: \"3cd86dce-6008-4b5d-9455-069889bc79cd\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-2rvkj"
Sep 11 03:49:54.852882 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:49:54.852863    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3cd86dce-6008-4b5d-9455-069889bc79cd-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-2rvkj\" (UID: \"3cd86dce-6008-4b5d-9455-069889bc79cd\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-2rvkj"
Sep 11 03:49:54.857811 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:49:54.857789    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnstx\" (UniqueName: \"kubernetes.io/projected/3cd86dce-6008-4b5d-9455-069889bc79cd-kube-api-access-nnstx\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-2rvkj\" (UID: \"3cd86dce-6008-4b5d-9455-069889bc79cd\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-2rvkj"
Sep 11 03:49:54.913557 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:49:54.913517    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-2rvkj"
Sep 11 03:49:55.186378 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:49:55.186332    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-2rvkj"]
Sep 11 03:49:55.190142 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:49:55.190117    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cd86dce_6008_4b5d_9455_069889bc79cd.slice/crio-419b1abc29732e22f4bac28a1bb9be80d83fb51652e456f14e729360ea77d570 WatchSource:0}: Error finding container 419b1abc29732e22f4bac28a1bb9be80d83fb51652e456f14e729360ea77d570: Status 404 returned error can't find the container with id 419b1abc29732e22f4bac28a1bb9be80d83fb51652e456f14e729360ea77d570
Sep 11 03:49:56.009264 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:49:56.009227    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-2rvkj" event={"ID":"3cd86dce-6008-4b5d-9455-069889bc79cd","Type":"ContainerStarted","Data":"419b1abc29732e22f4bac28a1bb9be80d83fb51652e456f14e729360ea77d570"}
Sep 11 03:49:57.015702 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:49:57.015650    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-2rvkj" event={"ID":"3cd86dce-6008-4b5d-9455-069889bc79cd","Type":"ContainerStarted","Data":"f42ae66367804f503a1e5c2bb2baeb5e40fc0c28f768554daa78e085cbecaa93"}
Sep 11 03:49:57.032594 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:49:57.032538    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-2rvkj" podStartSLOduration=2.057697268 podStartE2EDuration="3.032520452s" totalImagesPullingTime="974.823184ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 03:49:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:49:57.03146345 +0000 UTC m=+585.189949808" watchObservedRunningTime="2026-09-11 03:49:57.032520452 +0000 UTC m=+585.191006774"
Sep 11 03:50:04.914369 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:50:04.914323    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-2rvkj"
Sep 11 03:50:04.914898 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:50:04.914408    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-2rvkj"
Sep 11 03:50:04.926525 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:50:04.926501    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-2rvkj"
Sep 11 03:50:05.063640 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:50:05.063605    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-2rvkj"
Sep 11 03:50:12.313219 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:50:12.313190    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j4pgw_28b30dae-a2d3-4251-99e8-04bf5153015b/ovn-acl-logging/0.log"
Sep 11 03:50:12.313650 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:50:12.313445    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j4pgw_28b30dae-a2d3-4251-99e8-04bf5153015b/ovn-acl-logging/0.log"
Sep 11 03:50:12.974636 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:50:12.974598    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-nf79k"]
Sep 11 03:50:12.978172 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:50:12.978147    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-nf79k"
Sep 11 03:50:12.980378 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:50:12.980356    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\""
Sep 11 03:50:12.989649 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:50:12.989620    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-nf79k"]
Sep 11 03:50:13.013727 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:50:13.013692    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxdnv\" (UniqueName: \"kubernetes.io/projected/a2800068-b767-412b-ba7a-60a7f8ab4dc6-kube-api-access-dxdnv\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-nf79k\" (UID: \"a2800068-b767-412b-ba7a-60a7f8ab4dc6\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-nf79k"
Sep 11 03:50:13.013910 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:50:13.013753    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2800068-b767-412b-ba7a-60a7f8ab4dc6-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-nf79k\" (UID: \"a2800068-b767-412b-ba7a-60a7f8ab4dc6\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-nf79k"
Sep 11 03:50:13.013910 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:50:13.013798    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2800068-b767-412b-ba7a-60a7f8ab4dc6-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-nf79k\" (UID: \"a2800068-b767-412b-ba7a-60a7f8ab4dc6\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-nf79k"
Sep 11 03:50:13.013910 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:50:13.013835    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2800068-b767-412b-ba7a-60a7f8ab4dc6-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-nf79k\" (UID: \"a2800068-b767-412b-ba7a-60a7f8ab4dc6\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-nf79k"
Sep 11 03:50:13.014059 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:50:13.013915    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2800068-b767-412b-ba7a-60a7f8ab4dc6-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-nf79k\" (UID: \"a2800068-b767-412b-ba7a-60a7f8ab4dc6\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-nf79k"
Sep 11 03:50:13.014059 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:50:13.013959    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2800068-b767-412b-ba7a-60a7f8ab4dc6-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-nf79k\" (UID: \"a2800068-b767-412b-ba7a-60a7f8ab4dc6\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-nf79k"
Sep 11 03:50:13.114492 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:50:13.114450    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2800068-b767-412b-ba7a-60a7f8ab4dc6-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-nf79k\" (UID: \"a2800068-b767-412b-ba7a-60a7f8ab4dc6\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-nf79k"
Sep 11 03:50:13.114687 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:50:13.114505    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2800068-b767-412b-ba7a-60a7f8ab4dc6-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-nf79k\" (UID: \"a2800068-b767-412b-ba7a-60a7f8ab4dc6\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-nf79k"
Sep 11 03:50:13.114687 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:50:13.114544    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2800068-b767-412b-ba7a-60a7f8ab4dc6-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-nf79k\" (UID: \"a2800068-b767-412b-ba7a-60a7f8ab4dc6\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-nf79k"
Sep 11 03:50:13.114687 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:50:13.114581    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxdnv\" (UniqueName: \"kubernetes.io/projected/a2800068-b767-412b-ba7a-60a7f8ab4dc6-kube-api-access-dxdnv\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-nf79k\" (UID: \"a2800068-b767-412b-ba7a-60a7f8ab4dc6\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-nf79k"
Sep 11 03:50:13.114687 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:50:13.114637    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2800068-b767-412b-ba7a-60a7f8ab4dc6-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-nf79k\" (UID: \"a2800068-b767-412b-ba7a-60a7f8ab4dc6\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-nf79k"
Sep 11 03:50:13.114687 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:50:13.114682    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2800068-b767-412b-ba7a-60a7f8ab4dc6-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-nf79k\" (UID: \"a2800068-b767-412b-ba7a-60a7f8ab4dc6\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-nf79k"
Sep 11 03:50:13.114950 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:50:13.114904    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2800068-b767-412b-ba7a-60a7f8ab4dc6-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-nf79k\" (UID: \"a2800068-b767-412b-ba7a-60a7f8ab4dc6\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-nf79k"
Sep 11 03:50:13.115039 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:50:13.115017    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2800068-b767-412b-ba7a-60a7f8ab4dc6-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-nf79k\" (UID: \"a2800068-b767-412b-ba7a-60a7f8ab4dc6\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-nf79k"
Sep 11 03:50:13.115099 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:50:13.115052    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2800068-b767-412b-ba7a-60a7f8ab4dc6-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-nf79k\" (UID: \"a2800068-b767-412b-ba7a-60a7f8ab4dc6\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-nf79k"
Sep 11 03:50:13.116858 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:50:13.116838    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2800068-b767-412b-ba7a-60a7f8ab4dc6-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-nf79k\" (UID: \"a2800068-b767-412b-ba7a-60a7f8ab4dc6\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-nf79k"
Sep 11 03:50:13.117153 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:50:13.117131    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2800068-b767-412b-ba7a-60a7f8ab4dc6-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-nf79k\" (UID: \"a2800068-b767-412b-ba7a-60a7f8ab4dc6\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-nf79k"
Sep 11 03:50:13.122910 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:50:13.122888    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxdnv\" (UniqueName: \"kubernetes.io/projected/a2800068-b767-412b-ba7a-60a7f8ab4dc6-kube-api-access-dxdnv\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-nf79k\" (UID: \"a2800068-b767-412b-ba7a-60a7f8ab4dc6\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-nf79k"
Sep 11 03:50:13.291020 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:50:13.290927    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-nf79k"
Sep 11 03:50:13.565139 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:50:13.565106    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-nf79k"]
Sep 11 03:50:13.567434 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:50:13.567407    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2800068_b767_412b_ba7a_60a7f8ab4dc6.slice/crio-2757a37f63867e8de739cd46b0c9d4a318b7d212ecedfa876c5b60bceddf8547 WatchSource:0}: Error finding container 2757a37f63867e8de739cd46b0c9d4a318b7d212ecedfa876c5b60bceddf8547: Status 404 returned error can't find the container with id 2757a37f63867e8de739cd46b0c9d4a318b7d212ecedfa876c5b60bceddf8547
Sep 11 03:50:14.102436 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:50:14.102279    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-nf79k" event={"ID":"a2800068-b767-412b-ba7a-60a7f8ab4dc6","Type":"ContainerStarted","Data":"ca94b10c300dbfc0a5eafc4cd7eef3a109a70589e046e718441bde4ccd9904dd"}
Sep 11 03:50:14.102436 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:50:14.102336    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-nf79k" event={"ID":"a2800068-b767-412b-ba7a-60a7f8ab4dc6","Type":"ContainerStarted","Data":"2757a37f63867e8de739cd46b0c9d4a318b7d212ecedfa876c5b60bceddf8547"}
Sep 11 03:50:14.120625 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:50:14.120576    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-nf79k" podStartSLOduration=1.9209234880000001 podStartE2EDuration="2.120559741s" totalImagesPullingTime="199.636253ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 03:50:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:50:14.118565393 +0000 UTC m=+602.277051710" watchObservedRunningTime="2026-09-11 03:50:14.120559741 +0000 UTC m=+602.279046064"
Sep 11 03:50:23.291204 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:50:23.291166    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-nf79k"
Sep 11 03:50:23.291798 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:50:23.291238    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-nf79k"
Sep 11 03:50:23.304363 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:50:23.304335    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-nf79k"
Sep 11 03:50:24.161640 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:50:24.161606    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-nf79k"
Sep 11 03:52:54.525793 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:52:54.525756    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-4c62f1-8468fc7f57-x72s8"]
Sep 11 03:52:54.529155 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:52:54.529136    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-4c62f1-8468fc7f57-x72s8"
Sep 11 03:52:54.531408 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:52:54.531391    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-4c62f1\""
Sep 11 03:52:54.531473 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:52:54.531391    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-4c62f1-dockercfg-tkh6c\""
Sep 11 03:52:54.538800 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:52:54.538776    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-4c62f1-8468fc7f57-x72s8"]
Sep 11 03:52:54.563233 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:52:54.563204    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pcb4d\" (UniqueName: \"kubernetes.io/projected/cc8734e6-1130-4626-9e55-1d10cf9492b7-kube-api-access-pcb4d\") pod \"payload-processing-e2e-inf-a-4c62f1-8468fc7f57-x72s8\" (UID: \"cc8734e6-1130-4626-9e55-1d10cf9492b7\") " pod="openshift-ingress/payload-processing-e2e-inf-a-4c62f1-8468fc7f57-x72s8"
Sep 11 03:52:54.563400 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:52:54.563292    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc8734e6-1130-4626-9e55-1d10cf9492b7-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-4c62f1-8468fc7f57-x72s8\" (UID: \"cc8734e6-1130-4626-9e55-1d10cf9492b7\") " pod="openshift-ingress/payload-processing-e2e-inf-a-4c62f1-8468fc7f57-x72s8"
Sep 11 03:52:54.664095 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:52:54.664059    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pcb4d\" (UniqueName: \"kubernetes.io/projected/cc8734e6-1130-4626-9e55-1d10cf9492b7-kube-api-access-pcb4d\") pod \"payload-processing-e2e-inf-a-4c62f1-8468fc7f57-x72s8\" (UID: \"cc8734e6-1130-4626-9e55-1d10cf9492b7\") " pod="openshift-ingress/payload-processing-e2e-inf-a-4c62f1-8468fc7f57-x72s8"
Sep 11 03:52:54.664269 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:52:54.664128    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc8734e6-1130-4626-9e55-1d10cf9492b7-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-4c62f1-8468fc7f57-x72s8\" (UID: \"cc8734e6-1130-4626-9e55-1d10cf9492b7\") " pod="openshift-ingress/payload-processing-e2e-inf-a-4c62f1-8468fc7f57-x72s8"
Sep 11 03:52:54.664842 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:52:54.664822    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc8734e6-1130-4626-9e55-1d10cf9492b7-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-4c62f1-8468fc7f57-x72s8\" (UID: \"cc8734e6-1130-4626-9e55-1d10cf9492b7\") " pod="openshift-ingress/payload-processing-e2e-inf-a-4c62f1-8468fc7f57-x72s8"
Sep 11 03:52:54.671864 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:52:54.671839    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pcb4d\" (UniqueName: \"kubernetes.io/projected/cc8734e6-1130-4626-9e55-1d10cf9492b7-kube-api-access-pcb4d\") pod \"payload-processing-e2e-inf-a-4c62f1-8468fc7f57-x72s8\" (UID: \"cc8734e6-1130-4626-9e55-1d10cf9492b7\") " pod="openshift-ingress/payload-processing-e2e-inf-a-4c62f1-8468fc7f57-x72s8"
Sep 11 03:52:54.839792 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:52:54.839711    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-4c62f1-8468fc7f57-x72s8"
Sep 11 03:52:55.111579 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:52:55.111504    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-4c62f1-8468fc7f57-x72s8"]
Sep 11 03:52:55.114965 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:52:55.114938    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc8734e6_1130_4626_9e55_1d10cf9492b7.slice/crio-429627c4ed68117fce4ea85529bd50f5199dc6685457624c886201dc7cae3b24 WatchSource:0}: Error finding container 429627c4ed68117fce4ea85529bd50f5199dc6685457624c886201dc7cae3b24: Status 404 returned error can't find the container with id 429627c4ed68117fce4ea85529bd50f5199dc6685457624c886201dc7cae3b24
Sep 11 03:52:55.116871 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:52:55.116855    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 03:52:55.854645 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:52:55.854559    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-4c62f1-8468fc7f57-x72s8" event={"ID":"cc8734e6-1130-4626-9e55-1d10cf9492b7","Type":"ContainerStarted","Data":"429627c4ed68117fce4ea85529bd50f5199dc6685457624c886201dc7cae3b24"}
Sep 11 03:52:56.481815 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:52:56.481781    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-conflict-a-08754c30-kserve-6d9bddf55-p4kjr"]
Sep 11 03:52:56.485233 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:52:56.485206    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-08754c30-kserve-6d9bddf55-p4kjr"
Sep 11 03:52:56.487490 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:52:56.487464    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-conflict-a-08754c30-kserve-self-signed-certs\""
Sep 11 03:52:56.492983 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:52:56.492709    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-a-08754c30-kserve-6d9bddf55-p4kjr"]
Sep 11 03:52:56.579335 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:52:56.579297    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a778771-3dcb-4ca4-9a6a-ce05cd87d081-dshm\") pod \"e2e-conflict-a-08754c30-kserve-6d9bddf55-p4kjr\" (UID: \"8a778771-3dcb-4ca4-9a6a-ce05cd87d081\") " pod="llm/e2e-conflict-a-08754c30-kserve-6d9bddf55-p4kjr"
Sep 11 03:52:56.579335 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:52:56.579348    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a778771-3dcb-4ca4-9a6a-ce05cd87d081-tmp-dir\") pod \"e2e-conflict-a-08754c30-kserve-6d9bddf55-p4kjr\" (UID: \"8a778771-3dcb-4ca4-9a6a-ce05cd87d081\") " pod="llm/e2e-conflict-a-08754c30-kserve-6d9bddf55-p4kjr"
Sep 11 03:52:56.579603 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:52:56.579393    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a778771-3dcb-4ca4-9a6a-ce05cd87d081-model-cache\") pod \"e2e-conflict-a-08754c30-kserve-6d9bddf55-p4kjr\" (UID: \"8a778771-3dcb-4ca4-9a6a-ce05cd87d081\") " pod="llm/e2e-conflict-a-08754c30-kserve-6d9bddf55-p4kjr"
Sep 11 03:52:56.579603 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:52:56.579437    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a778771-3dcb-4ca4-9a6a-ce05cd87d081-tls-certs\") pod \"e2e-conflict-a-08754c30-kserve-6d9bddf55-p4kjr\" (UID: \"8a778771-3dcb-4ca4-9a6a-ce05cd87d081\") " pod="llm/e2e-conflict-a-08754c30-kserve-6d9bddf55-p4kjr"
Sep 11 03:52:56.579603 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:52:56.579511    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a778771-3dcb-4ca4-9a6a-ce05cd87d081-home\") pod \"e2e-conflict-a-08754c30-kserve-6d9bddf55-p4kjr\" (UID: \"8a778771-3dcb-4ca4-9a6a-ce05cd87d081\") " pod="llm/e2e-conflict-a-08754c30-kserve-6d9bddf55-p4kjr"
Sep 11 03:52:56.579798 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:52:56.579610    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zzdgw\" (UniqueName: \"kubernetes.io/projected/8a778771-3dcb-4ca4-9a6a-ce05cd87d081-kube-api-access-zzdgw\") pod \"e2e-conflict-a-08754c30-kserve-6d9bddf55-p4kjr\" (UID: \"8a778771-3dcb-4ca4-9a6a-ce05cd87d081\") " pod="llm/e2e-conflict-a-08754c30-kserve-6d9bddf55-p4kjr"
Sep 11 03:52:56.680930 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:52:56.680889    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a778771-3dcb-4ca4-9a6a-ce05cd87d081-model-cache\") pod \"e2e-conflict-a-08754c30-kserve-6d9bddf55-p4kjr\" (UID: \"8a778771-3dcb-4ca4-9a6a-ce05cd87d081\") " pod="llm/e2e-conflict-a-08754c30-kserve-6d9bddf55-p4kjr"
Sep 11 03:52:56.680930 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:52:56.680943    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a778771-3dcb-4ca4-9a6a-ce05cd87d081-tls-certs\") pod \"e2e-conflict-a-08754c30-kserve-6d9bddf55-p4kjr\" (UID: \"8a778771-3dcb-4ca4-9a6a-ce05cd87d081\") " pod="llm/e2e-conflict-a-08754c30-kserve-6d9bddf55-p4kjr"
Sep 11 03:52:56.681225 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:52:56.680990    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a778771-3dcb-4ca4-9a6a-ce05cd87d081-home\") pod \"e2e-conflict-a-08754c30-kserve-6d9bddf55-p4kjr\" (UID: \"8a778771-3dcb-4ca4-9a6a-ce05cd87d081\") " pod="llm/e2e-conflict-a-08754c30-kserve-6d9bddf55-p4kjr"
Sep 11 03:52:56.681225 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:52:56.681064    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzdgw\" (UniqueName: \"kubernetes.io/projected/8a778771-3dcb-4ca4-9a6a-ce05cd87d081-kube-api-access-zzdgw\") pod \"e2e-conflict-a-08754c30-kserve-6d9bddf55-p4kjr\" (UID: \"8a778771-3dcb-4ca4-9a6a-ce05cd87d081\") " pod="llm/e2e-conflict-a-08754c30-kserve-6d9bddf55-p4kjr"
Sep 11 03:52:56.681225 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:52:56.681130    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a778771-3dcb-4ca4-9a6a-ce05cd87d081-dshm\") pod \"e2e-conflict-a-08754c30-kserve-6d9bddf55-p4kjr\" (UID: \"8a778771-3dcb-4ca4-9a6a-ce05cd87d081\") " pod="llm/e2e-conflict-a-08754c30-kserve-6d9bddf55-p4kjr"
Sep 11 03:52:56.681225 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:52:56.681165    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a778771-3dcb-4ca4-9a6a-ce05cd87d081-tmp-dir\") pod \"e2e-conflict-a-08754c30-kserve-6d9bddf55-p4kjr\" (UID: \"8a778771-3dcb-4ca4-9a6a-ce05cd87d081\") " pod="llm/e2e-conflict-a-08754c30-kserve-6d9bddf55-p4kjr"
Sep 11 03:52:56.681531 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:52:56.681343    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a778771-3dcb-4ca4-9a6a-ce05cd87d081-model-cache\") pod \"e2e-conflict-a-08754c30-kserve-6d9bddf55-p4kjr\" (UID: \"8a778771-3dcb-4ca4-9a6a-ce05cd87d081\") " pod="llm/e2e-conflict-a-08754c30-kserve-6d9bddf55-p4kjr"
Sep 11 03:52:56.681694 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:52:56.681648    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a778771-3dcb-4ca4-9a6a-ce05cd87d081-tmp-dir\") pod \"e2e-conflict-a-08754c30-kserve-6d9bddf55-p4kjr\" (UID: \"8a778771-3dcb-4ca4-9a6a-ce05cd87d081\") " pod="llm/e2e-conflict-a-08754c30-kserve-6d9bddf55-p4kjr"
Sep 11 03:52:56.682060 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:52:56.682030    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a778771-3dcb-4ca4-9a6a-ce05cd87d081-home\") pod \"e2e-conflict-a-08754c30-kserve-6d9bddf55-p4kjr\" (UID: \"8a778771-3dcb-4ca4-9a6a-ce05cd87d081\") " pod="llm/e2e-conflict-a-08754c30-kserve-6d9bddf55-p4kjr"
Sep 11 03:52:56.684844 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:52:56.684348    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a778771-3dcb-4ca4-9a6a-ce05cd87d081-tls-certs\") pod \"e2e-conflict-a-08754c30-kserve-6d9bddf55-p4kjr\" (UID: \"8a778771-3dcb-4ca4-9a6a-ce05cd87d081\") " pod="llm/e2e-conflict-a-08754c30-kserve-6d9bddf55-p4kjr"
Sep 11 03:52:56.684844 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:52:56.684611    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a778771-3dcb-4ca4-9a6a-ce05cd87d081-dshm\") pod \"e2e-conflict-a-08754c30-kserve-6d9bddf55-p4kjr\" (UID: \"8a778771-3dcb-4ca4-9a6a-ce05cd87d081\") " pod="llm/e2e-conflict-a-08754c30-kserve-6d9bddf55-p4kjr"
Sep 11 03:52:56.688929 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:52:56.688898    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzdgw\" (UniqueName: \"kubernetes.io/projected/8a778771-3dcb-4ca4-9a6a-ce05cd87d081-kube-api-access-zzdgw\") pod \"e2e-conflict-a-08754c30-kserve-6d9bddf55-p4kjr\" (UID: \"8a778771-3dcb-4ca4-9a6a-ce05cd87d081\") " pod="llm/e2e-conflict-a-08754c30-kserve-6d9bddf55-p4kjr"
Sep 11 03:52:56.796733 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:52:56.796400    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-08754c30-kserve-6d9bddf55-p4kjr"
Sep 11 03:52:57.074171 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:52:57.074095    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-a-08754c30-kserve-6d9bddf55-p4kjr"]
Sep 11 03:52:57.076414 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:52:57.076387    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8a778771_3dcb_4ca4_9a6a_ce05cd87d081.slice/crio-ef50e66810d29ab678e556466570a1c3442281652fa2fa15d07f80514a4815ae WatchSource:0}: Error finding container ef50e66810d29ab678e556466570a1c3442281652fa2fa15d07f80514a4815ae: Status 404 returned error can't find the container with id ef50e66810d29ab678e556466570a1c3442281652fa2fa15d07f80514a4815ae
Sep 11 03:52:57.874129 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:52:57.874078    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-4c62f1-8468fc7f57-x72s8" event={"ID":"cc8734e6-1130-4626-9e55-1d10cf9492b7","Type":"ContainerStarted","Data":"747348d41b9cc0424056168c352b64909ff3946a7ae9c5b37a1829e3b4ff00e8"}
Sep 11 03:52:57.874339 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:52:57.874238    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-4c62f1-8468fc7f57-x72s8"
Sep 11 03:52:57.877018 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:52:57.876985    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-08754c30-kserve-6d9bddf55-p4kjr" event={"ID":"8a778771-3dcb-4ca4-9a6a-ce05cd87d081","Type":"ContainerStarted","Data":"1dd5811c88116fd8798d754366b311c67fc6a50df5fd6c8cbf164148086edf86"}
Sep 11 03:52:57.877159 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:52:57.877027    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-08754c30-kserve-6d9bddf55-p4kjr" event={"ID":"8a778771-3dcb-4ca4-9a6a-ce05cd87d081","Type":"ContainerStarted","Data":"ef50e66810d29ab678e556466570a1c3442281652fa2fa15d07f80514a4815ae"}
Sep 11 03:52:57.891588 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:52:57.891535    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-4c62f1-8468fc7f57-x72s8" podStartSLOduration=2.161840853 podStartE2EDuration="3.891521095s" totalImagesPullingTime="1.729680242s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 03:52:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:52:57.890958897 +0000 UTC m=+766.049445221" watchObservedRunningTime="2026-09-11 03:52:57.891521095 +0000 UTC m=+766.050007421"
Sep 11 03:52:57.908076 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:52:57.908026    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-conflict-a-08754c30-kserve-6d9bddf55-p4kjr" podStartSLOduration=1.908009833 podStartE2EDuration="1.908009833s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 03:52:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:52:57.906163455 +0000 UTC m=+766.064649779" watchObservedRunningTime="2026-09-11 03:52:57.908009833 +0000 UTC m=+766.066496155"
Sep 11 03:53:01.204857 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:01.204817    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1eeb81b2-7b85c4ff8-ngtcl"]
Sep 11 03:53:01.208986 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:01.208963    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1eeb81b2-7b85c4ff8-ngtcl"
Sep 11 03:53:01.211447 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:01.211421    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-fcpfs\""
Sep 11 03:53:01.211557 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:01.211508    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 11 03:53:01.212426 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:01.212397    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 03:53:01.212535 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:01.212477    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 11 03:53:01.212589 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:01.212530    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-1eeb81b2-serving-cert\""
Sep 11 03:53:01.216627 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:01.216593    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1eeb81b2-7b85c4ff8-ngtcl"]
Sep 11 03:53:01.327068 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:01.327030    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8gdz\" (UniqueName: \"kubernetes.io/projected/8e0286a3-712f-46d6-8927-2b036878bda7-kube-api-access-w8gdz\") pod \"maas-api-e2e-ait-1eeb81b2-7b85c4ff8-ngtcl\" (UID: \"8e0286a3-712f-46d6-8927-2b036878bda7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1eeb81b2-7b85c4ff8-ngtcl"
Sep 11 03:53:01.327258 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:01.327081    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8e0286a3-712f-46d6-8927-2b036878bda7-metrics-certs\") pod \"maas-api-e2e-ait-1eeb81b2-7b85c4ff8-ngtcl\" (UID: \"8e0286a3-712f-46d6-8927-2b036878bda7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1eeb81b2-7b85c4ff8-ngtcl"
Sep 11 03:53:01.327258 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:01.327177    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8e0286a3-712f-46d6-8927-2b036878bda7-maas-api-tls\") pod \"maas-api-e2e-ait-1eeb81b2-7b85c4ff8-ngtcl\" (UID: \"8e0286a3-712f-46d6-8927-2b036878bda7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1eeb81b2-7b85c4ff8-ngtcl"
Sep 11 03:53:01.327258 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:01.327213    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e0286a3-712f-46d6-8927-2b036878bda7-openshift-service-ca\") pod \"maas-api-e2e-ait-1eeb81b2-7b85c4ff8-ngtcl\" (UID: \"8e0286a3-712f-46d6-8927-2b036878bda7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1eeb81b2-7b85c4ff8-ngtcl"
Sep 11 03:53:01.428347 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:01.428306    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8gdz\" (UniqueName: \"kubernetes.io/projected/8e0286a3-712f-46d6-8927-2b036878bda7-kube-api-access-w8gdz\") pod \"maas-api-e2e-ait-1eeb81b2-7b85c4ff8-ngtcl\" (UID: \"8e0286a3-712f-46d6-8927-2b036878bda7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1eeb81b2-7b85c4ff8-ngtcl"
Sep 11 03:53:01.428546 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:01.428365    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8e0286a3-712f-46d6-8927-2b036878bda7-metrics-certs\") pod \"maas-api-e2e-ait-1eeb81b2-7b85c4ff8-ngtcl\" (UID: \"8e0286a3-712f-46d6-8927-2b036878bda7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1eeb81b2-7b85c4ff8-ngtcl"
Sep 11 03:53:01.428546 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:01.428410    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8e0286a3-712f-46d6-8927-2b036878bda7-maas-api-tls\") pod \"maas-api-e2e-ait-1eeb81b2-7b85c4ff8-ngtcl\" (UID: \"8e0286a3-712f-46d6-8927-2b036878bda7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1eeb81b2-7b85c4ff8-ngtcl"
Sep 11 03:53:01.428546 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:01.428435    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e0286a3-712f-46d6-8927-2b036878bda7-openshift-service-ca\") pod \"maas-api-e2e-ait-1eeb81b2-7b85c4ff8-ngtcl\" (UID: \"8e0286a3-712f-46d6-8927-2b036878bda7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1eeb81b2-7b85c4ff8-ngtcl"
Sep 11 03:53:01.428757 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:53:01.428549    2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-1eeb81b2-serving-cert: secret "maas-api-e2e-ait-1eeb81b2-serving-cert" not found
Sep 11 03:53:01.428757 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:53:01.428628    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8e0286a3-712f-46d6-8927-2b036878bda7-maas-api-tls podName:8e0286a3-712f-46d6-8927-2b036878bda7 nodeName:}" failed. No retries permitted until 2026-09-11 03:53:01.928610757 +0000 UTC m=+770.087097058 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8e0286a3-712f-46d6-8927-2b036878bda7-maas-api-tls") pod "maas-api-e2e-ait-1eeb81b2-7b85c4ff8-ngtcl" (UID: "8e0286a3-712f-46d6-8927-2b036878bda7") : secret "maas-api-e2e-ait-1eeb81b2-serving-cert" not found
Sep 11 03:53:01.429201 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:01.429178    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e0286a3-712f-46d6-8927-2b036878bda7-openshift-service-ca\") pod \"maas-api-e2e-ait-1eeb81b2-7b85c4ff8-ngtcl\" (UID: \"8e0286a3-712f-46d6-8927-2b036878bda7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1eeb81b2-7b85c4ff8-ngtcl"
Sep 11 03:53:01.430833 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:01.430815    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8e0286a3-712f-46d6-8927-2b036878bda7-metrics-certs\") pod \"maas-api-e2e-ait-1eeb81b2-7b85c4ff8-ngtcl\" (UID: \"8e0286a3-712f-46d6-8927-2b036878bda7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1eeb81b2-7b85c4ff8-ngtcl"
Sep 11 03:53:01.441704 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:01.441678    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8gdz\" (UniqueName: \"kubernetes.io/projected/8e0286a3-712f-46d6-8927-2b036878bda7-kube-api-access-w8gdz\") pod \"maas-api-e2e-ait-1eeb81b2-7b85c4ff8-ngtcl\" (UID: \"8e0286a3-712f-46d6-8927-2b036878bda7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1eeb81b2-7b85c4ff8-ngtcl"
Sep 11 03:53:01.934729 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:01.934690    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8e0286a3-712f-46d6-8927-2b036878bda7-maas-api-tls\") pod \"maas-api-e2e-ait-1eeb81b2-7b85c4ff8-ngtcl\" (UID: \"8e0286a3-712f-46d6-8927-2b036878bda7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1eeb81b2-7b85c4ff8-ngtcl"
Sep 11 03:53:01.937047 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:01.937017    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8e0286a3-712f-46d6-8927-2b036878bda7-maas-api-tls\") pod \"maas-api-e2e-ait-1eeb81b2-7b85c4ff8-ngtcl\" (UID: \"8e0286a3-712f-46d6-8927-2b036878bda7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1eeb81b2-7b85c4ff8-ngtcl"
Sep 11 03:53:02.123733 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:02.123695    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1eeb81b2-7b85c4ff8-ngtcl"
Sep 11 03:53:02.396733 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:02.396606    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1eeb81b2-7b85c4ff8-ngtcl"]
Sep 11 03:53:02.399029 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:53:02.399001    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e0286a3_712f_46d6_8927_2b036878bda7.slice/crio-b4c25b5bd9800752745fb7b635f30c86d2a86f508e6edcfaa0b07bc17617f4a1 WatchSource:0}: Error finding container b4c25b5bd9800752745fb7b635f30c86d2a86f508e6edcfaa0b07bc17617f4a1: Status 404 returned error can't find the container with id b4c25b5bd9800752745fb7b635f30c86d2a86f508e6edcfaa0b07bc17617f4a1
Sep 11 03:53:02.906309 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:02.906160    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1eeb81b2-7b85c4ff8-ngtcl" event={"ID":"8e0286a3-712f-46d6-8927-2b036878bda7","Type":"ContainerStarted","Data":"b4c25b5bd9800752745fb7b635f30c86d2a86f508e6edcfaa0b07bc17617f4a1"}
Sep 11 03:53:04.919284 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:04.919240    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1eeb81b2-7b85c4ff8-ngtcl" event={"ID":"8e0286a3-712f-46d6-8927-2b036878bda7","Type":"ContainerStarted","Data":"bafcabbb2c0dbc2462f9a654351e45492dd0286bf9221f631fc1be73b8ad68cf"}
Sep 11 03:53:04.919752 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:04.919346    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1eeb81b2-7b85c4ff8-ngtcl"
Sep 11 03:53:04.935166 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:04.935118    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1eeb81b2-7b85c4ff8-ngtcl" podStartSLOduration=1.7663552569999998 podStartE2EDuration="3.935102527s" totalImagesPullingTime="2.16874727s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 03:53:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:53:04.9336515 +0000 UTC m=+773.092137822" watchObservedRunningTime="2026-09-11 03:53:04.935102527 +0000 UTC m=+773.093588850"
Sep 11 03:53:06.797136 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:06.797089    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-conflict-a-08754c30-kserve-6d9bddf55-p4kjr"
Sep 11 03:53:06.797523 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:06.797152    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-conflict-a-08754c30-kserve-6d9bddf55-p4kjr"
Sep 11 03:53:06.810574 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:06.810488    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-conflict-a-08754c30-kserve-6d9bddf55-p4kjr"
Sep 11 03:53:06.942618 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:06.942589    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-conflict-a-08754c30-kserve-6d9bddf55-p4kjr"
Sep 11 03:53:07.923135 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:07.923104    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-8efac4-openshift-default-66557bb4f8-dl5z8"]
Sep 11 03:53:07.926793 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:07.926767    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-8efac4-openshift-default-66557bb4f8-dl5z8"
Sep 11 03:53:07.929109 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:07.929079    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-8efac4-openshift-default-dockercfg-2j5hp\""
Sep 11 03:53:07.941420 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:07.941398    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-8efac4-openshift-default-66557bb4f8-dl5z8"]
Sep 11 03:53:07.995354 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:07.995314    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bfe5ee73-4698-420d-9729-3153ab5656fa-istio-token\") pod \"e2e-inf-b-8efac4-openshift-default-66557bb4f8-dl5z8\" (UID: \"bfe5ee73-4698-420d-9729-3153ab5656fa\") " pod="openshift-ingress/e2e-inf-b-8efac4-openshift-default-66557bb4f8-dl5z8"
Sep 11 03:53:07.995354 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:07.995359    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfe5ee73-4698-420d-9729-3153ab5656fa-workload-certs\") pod \"e2e-inf-b-8efac4-openshift-default-66557bb4f8-dl5z8\" (UID: \"bfe5ee73-4698-420d-9729-3153ab5656fa\") " pod="openshift-ingress/e2e-inf-b-8efac4-openshift-default-66557bb4f8-dl5z8"
Sep 11 03:53:07.995579 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:07.995383    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfe5ee73-4698-420d-9729-3153ab5656fa-credential-socket\") pod \"e2e-inf-b-8efac4-openshift-default-66557bb4f8-dl5z8\" (UID: \"bfe5ee73-4698-420d-9729-3153ab5656fa\") " pod="openshift-ingress/e2e-inf-b-8efac4-openshift-default-66557bb4f8-dl5z8"
Sep 11 03:53:07.995579 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:07.995409    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfe5ee73-4698-420d-9729-3153ab5656fa-istio-data\") pod \"e2e-inf-b-8efac4-openshift-default-66557bb4f8-dl5z8\" (UID: \"bfe5ee73-4698-420d-9729-3153ab5656fa\") " pod="openshift-ingress/e2e-inf-b-8efac4-openshift-default-66557bb4f8-dl5z8"
Sep 11 03:53:07.995579 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:07.995482    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfe5ee73-4698-420d-9729-3153ab5656fa-workload-socket\") pod \"e2e-inf-b-8efac4-openshift-default-66557bb4f8-dl5z8\" (UID: \"bfe5ee73-4698-420d-9729-3153ab5656fa\") " pod="openshift-ingress/e2e-inf-b-8efac4-openshift-default-66557bb4f8-dl5z8"
Sep 11 03:53:07.995579 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:07.995513    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tp2fp\" (UniqueName: \"kubernetes.io/projected/bfe5ee73-4698-420d-9729-3153ab5656fa-kube-api-access-tp2fp\") pod \"e2e-inf-b-8efac4-openshift-default-66557bb4f8-dl5z8\" (UID: \"bfe5ee73-4698-420d-9729-3153ab5656fa\") " pod="openshift-ingress/e2e-inf-b-8efac4-openshift-default-66557bb4f8-dl5z8"
Sep 11 03:53:07.995779 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:07.995571    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfe5ee73-4698-420d-9729-3153ab5656fa-istio-envoy\") pod \"e2e-inf-b-8efac4-openshift-default-66557bb4f8-dl5z8\" (UID: \"bfe5ee73-4698-420d-9729-3153ab5656fa\") " pod="openshift-ingress/e2e-inf-b-8efac4-openshift-default-66557bb4f8-dl5z8"
Sep 11 03:53:07.995779 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:07.995687    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfe5ee73-4698-420d-9729-3153ab5656fa-istiod-ca-cert\") pod \"e2e-inf-b-8efac4-openshift-default-66557bb4f8-dl5z8\" (UID: \"bfe5ee73-4698-420d-9729-3153ab5656fa\") " pod="openshift-ingress/e2e-inf-b-8efac4-openshift-default-66557bb4f8-dl5z8"
Sep 11 03:53:07.995779 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:07.995712    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfe5ee73-4698-420d-9729-3153ab5656fa-istio-podinfo\") pod \"e2e-inf-b-8efac4-openshift-default-66557bb4f8-dl5z8\" (UID: \"bfe5ee73-4698-420d-9729-3153ab5656fa\") " pod="openshift-ingress/e2e-inf-b-8efac4-openshift-default-66557bb4f8-dl5z8"
Sep 11 03:53:08.097077 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:08.097038    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bfe5ee73-4698-420d-9729-3153ab5656fa-istio-token\") pod \"e2e-inf-b-8efac4-openshift-default-66557bb4f8-dl5z8\" (UID: \"bfe5ee73-4698-420d-9729-3153ab5656fa\") " pod="openshift-ingress/e2e-inf-b-8efac4-openshift-default-66557bb4f8-dl5z8"
Sep 11 03:53:08.097282 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:08.097084    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfe5ee73-4698-420d-9729-3153ab5656fa-workload-certs\") pod \"e2e-inf-b-8efac4-openshift-default-66557bb4f8-dl5z8\" (UID: \"bfe5ee73-4698-420d-9729-3153ab5656fa\") " pod="openshift-ingress/e2e-inf-b-8efac4-openshift-default-66557bb4f8-dl5z8"
Sep 11 03:53:08.097282 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:08.097133    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfe5ee73-4698-420d-9729-3153ab5656fa-credential-socket\") pod \"e2e-inf-b-8efac4-openshift-default-66557bb4f8-dl5z8\" (UID: \"bfe5ee73-4698-420d-9729-3153ab5656fa\") " pod="openshift-ingress/e2e-inf-b-8efac4-openshift-default-66557bb4f8-dl5z8"
Sep 11 03:53:08.097282 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:08.097161    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfe5ee73-4698-420d-9729-3153ab5656fa-istio-data\") pod \"e2e-inf-b-8efac4-openshift-default-66557bb4f8-dl5z8\" (UID: \"bfe5ee73-4698-420d-9729-3153ab5656fa\") " pod="openshift-ingress/e2e-inf-b-8efac4-openshift-default-66557bb4f8-dl5z8"
Sep 11 03:53:08.097282 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:08.097248    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfe5ee73-4698-420d-9729-3153ab5656fa-workload-socket\") pod \"e2e-inf-b-8efac4-openshift-default-66557bb4f8-dl5z8\" (UID: \"bfe5ee73-4698-420d-9729-3153ab5656fa\") " pod="openshift-ingress/e2e-inf-b-8efac4-openshift-default-66557bb4f8-dl5z8"
Sep 11 03:53:08.097282 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:08.097280    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tp2fp\" (UniqueName: \"kubernetes.io/projected/bfe5ee73-4698-420d-9729-3153ab5656fa-kube-api-access-tp2fp\") pod \"e2e-inf-b-8efac4-openshift-default-66557bb4f8-dl5z8\" (UID: \"bfe5ee73-4698-420d-9729-3153ab5656fa\") " pod="openshift-ingress/e2e-inf-b-8efac4-openshift-default-66557bb4f8-dl5z8"
Sep 11 03:53:08.097539 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:08.097318    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfe5ee73-4698-420d-9729-3153ab5656fa-istio-envoy\") pod \"e2e-inf-b-8efac4-openshift-default-66557bb4f8-dl5z8\" (UID: \"bfe5ee73-4698-420d-9729-3153ab5656fa\") " pod="openshift-ingress/e2e-inf-b-8efac4-openshift-default-66557bb4f8-dl5z8"
Sep 11 03:53:08.097539 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:08.097393    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfe5ee73-4698-420d-9729-3153ab5656fa-istiod-ca-cert\") pod \"e2e-inf-b-8efac4-openshift-default-66557bb4f8-dl5z8\" (UID: \"bfe5ee73-4698-420d-9729-3153ab5656fa\") " pod="openshift-ingress/e2e-inf-b-8efac4-openshift-default-66557bb4f8-dl5z8"
Sep 11 03:53:08.097539 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:08.097424    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfe5ee73-4698-420d-9729-3153ab5656fa-istio-podinfo\") pod \"e2e-inf-b-8efac4-openshift-default-66557bb4f8-dl5z8\" (UID: \"bfe5ee73-4698-420d-9729-3153ab5656fa\") " pod="openshift-ingress/e2e-inf-b-8efac4-openshift-default-66557bb4f8-dl5z8"
Sep 11 03:53:08.097709 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:08.097595    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfe5ee73-4698-420d-9729-3153ab5656fa-credential-socket\") pod \"e2e-inf-b-8efac4-openshift-default-66557bb4f8-dl5z8\" (UID: \"bfe5ee73-4698-420d-9729-3153ab5656fa\") " pod="openshift-ingress/e2e-inf-b-8efac4-openshift-default-66557bb4f8-dl5z8"
Sep 11 03:53:08.097709 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:08.097626    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfe5ee73-4698-420d-9729-3153ab5656fa-istio-data\") pod \"e2e-inf-b-8efac4-openshift-default-66557bb4f8-dl5z8\" (UID: \"bfe5ee73-4698-420d-9729-3153ab5656fa\") " pod="openshift-ingress/e2e-inf-b-8efac4-openshift-default-66557bb4f8-dl5z8"
Sep 11 03:53:08.097817 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:08.097793    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfe5ee73-4698-420d-9729-3153ab5656fa-workload-certs\") pod \"e2e-inf-b-8efac4-openshift-default-66557bb4f8-dl5z8\" (UID: \"bfe5ee73-4698-420d-9729-3153ab5656fa\") " pod="openshift-ingress/e2e-inf-b-8efac4-openshift-default-66557bb4f8-dl5z8"
Sep 11 03:53:08.097870 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:08.097826    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfe5ee73-4698-420d-9729-3153ab5656fa-workload-socket\") pod \"e2e-inf-b-8efac4-openshift-default-66557bb4f8-dl5z8\" (UID: \"bfe5ee73-4698-420d-9729-3153ab5656fa\") " pod="openshift-ingress/e2e-inf-b-8efac4-openshift-default-66557bb4f8-dl5z8"
Sep 11 03:53:08.098204 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:08.098185    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfe5ee73-4698-420d-9729-3153ab5656fa-istiod-ca-cert\") pod \"e2e-inf-b-8efac4-openshift-default-66557bb4f8-dl5z8\" (UID: \"bfe5ee73-4698-420d-9729-3153ab5656fa\") " pod="openshift-ingress/e2e-inf-b-8efac4-openshift-default-66557bb4f8-dl5z8"
Sep 11 03:53:08.099722 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:08.099697    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfe5ee73-4698-420d-9729-3153ab5656fa-istio-envoy\") pod \"e2e-inf-b-8efac4-openshift-default-66557bb4f8-dl5z8\" (UID: \"bfe5ee73-4698-420d-9729-3153ab5656fa\") " pod="openshift-ingress/e2e-inf-b-8efac4-openshift-default-66557bb4f8-dl5z8"
Sep 11 03:53:08.099971 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:08.099950    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfe5ee73-4698-420d-9729-3153ab5656fa-istio-podinfo\") pod \"e2e-inf-b-8efac4-openshift-default-66557bb4f8-dl5z8\" (UID: \"bfe5ee73-4698-420d-9729-3153ab5656fa\") " pod="openshift-ingress/e2e-inf-b-8efac4-openshift-default-66557bb4f8-dl5z8"
Sep 11 03:53:08.110576 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:08.110554    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bfe5ee73-4698-420d-9729-3153ab5656fa-istio-token\") pod \"e2e-inf-b-8efac4-openshift-default-66557bb4f8-dl5z8\" (UID: \"bfe5ee73-4698-420d-9729-3153ab5656fa\") " pod="openshift-ingress/e2e-inf-b-8efac4-openshift-default-66557bb4f8-dl5z8"
Sep 11 03:53:08.110817 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:08.110796    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tp2fp\" (UniqueName: \"kubernetes.io/projected/bfe5ee73-4698-420d-9729-3153ab5656fa-kube-api-access-tp2fp\") pod \"e2e-inf-b-8efac4-openshift-default-66557bb4f8-dl5z8\" (UID: \"bfe5ee73-4698-420d-9729-3153ab5656fa\") " pod="openshift-ingress/e2e-inf-b-8efac4-openshift-default-66557bb4f8-dl5z8"
Sep 11 03:53:08.237996 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:08.237909    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-8efac4-openshift-default-66557bb4f8-dl5z8"
Sep 11 03:53:08.513733 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:08.513639    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-8efac4-openshift-default-66557bb4f8-dl5z8"]
Sep 11 03:53:08.517067 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:53:08.517034    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfe5ee73_4698_420d_9729_3153ab5656fa.slice/crio-129d106a6c6da6bbf23ea239901dc07537b9307f0e78fb292b44f7a47a98b2df WatchSource:0}: Error finding container 129d106a6c6da6bbf23ea239901dc07537b9307f0e78fb292b44f7a47a98b2df: Status 404 returned error can't find the container with id 129d106a6c6da6bbf23ea239901dc07537b9307f0e78fb292b44f7a47a98b2df
Sep 11 03:53:08.884776 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:08.884694    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-4c62f1-8468fc7f57-x72s8"
Sep 11 03:53:08.952331 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:08.952295    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-8efac4-openshift-default-66557bb4f8-dl5z8" event={"ID":"bfe5ee73-4698-420d-9729-3153ab5656fa","Type":"ContainerStarted","Data":"129d106a6c6da6bbf23ea239901dc07537b9307f0e78fb292b44f7a47a98b2df"}
Sep 11 03:53:10.933395 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:10.933365    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1eeb81b2-7b85c4ff8-ngtcl"
Sep 11 03:53:11.099681 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:11.099631    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 03:53:11.099742 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:11.099729    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 03:53:11.099781 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:11.099760    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 03:53:11.972539 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:11.972494    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-8efac4-openshift-default-66557bb4f8-dl5z8" event={"ID":"bfe5ee73-4698-420d-9729-3153ab5656fa","Type":"ContainerStarted","Data":"1856f2ca275da8b5a2f861e2ee2827b84fc738a8867abf2c136fb088d94e9b0e"}
Sep 11 03:53:11.990592 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:11.990535    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-8efac4-openshift-default-66557bb4f8-dl5z8" podStartSLOduration=2.410364811 podStartE2EDuration="4.990516744s" totalImagesPullingTime="2.580151933s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 03:53:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:53:11.988922508 +0000 UTC m=+780.147408831" watchObservedRunningTime="2026-09-11 03:53:11.990516744 +0000 UTC m=+780.149003067"
Sep 11 03:53:12.238212 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:12.238132    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-8efac4-openshift-default-66557bb4f8-dl5z8"
Sep 11 03:53:12.243447 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:12.243420    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-8efac4-openshift-default-66557bb4f8-dl5z8"
Sep 11 03:53:12.977778 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:12.977741    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-8efac4-openshift-default-66557bb4f8-dl5z8"
Sep 11 03:53:12.978656 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:12.978637    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-8efac4-openshift-default-66557bb4f8-dl5z8"
Sep 11 03:53:17.790736 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:17.790701    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-8efac4-5df6f86d78-8xm9m"]
Sep 11 03:53:17.794679 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:17.794642    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8efac4-5df6f86d78-8xm9m"
Sep 11 03:53:17.796982 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:17.796958    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-8efac4\""
Sep 11 03:53:17.797117 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:17.796986    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-8efac4-dockercfg-2jvlf\""
Sep 11 03:53:17.802631 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:17.802605    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-8efac4-5df6f86d78-8xm9m"]
Sep 11 03:53:17.893002 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:17.892965    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/094495b3-e2a6-413d-b2d3-6251ad5667bd-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-8efac4-5df6f86d78-8xm9m\" (UID: \"094495b3-e2a6-413d-b2d3-6251ad5667bd\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8efac4-5df6f86d78-8xm9m"
Sep 11 03:53:17.893198 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:17.893025    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbzjv\" (UniqueName: \"kubernetes.io/projected/094495b3-e2a6-413d-b2d3-6251ad5667bd-kube-api-access-rbzjv\") pod \"payload-pre-processing-e2e-inf-b-8efac4-5df6f86d78-8xm9m\" (UID: \"094495b3-e2a6-413d-b2d3-6251ad5667bd\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8efac4-5df6f86d78-8xm9m"
Sep 11 03:53:17.993908 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:17.993871    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/094495b3-e2a6-413d-b2d3-6251ad5667bd-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-8efac4-5df6f86d78-8xm9m\" (UID: \"094495b3-e2a6-413d-b2d3-6251ad5667bd\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8efac4-5df6f86d78-8xm9m"
Sep 11 03:53:17.994088 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:17.993916    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbzjv\" (UniqueName: \"kubernetes.io/projected/094495b3-e2a6-413d-b2d3-6251ad5667bd-kube-api-access-rbzjv\") pod \"payload-pre-processing-e2e-inf-b-8efac4-5df6f86d78-8xm9m\" (UID: \"094495b3-e2a6-413d-b2d3-6251ad5667bd\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8efac4-5df6f86d78-8xm9m"
Sep 11 03:53:17.994553 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:17.994529    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/094495b3-e2a6-413d-b2d3-6251ad5667bd-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-8efac4-5df6f86d78-8xm9m\" (UID: \"094495b3-e2a6-413d-b2d3-6251ad5667bd\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8efac4-5df6f86d78-8xm9m"
Sep 11 03:53:18.001546 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:18.001513    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbzjv\" (UniqueName: \"kubernetes.io/projected/094495b3-e2a6-413d-b2d3-6251ad5667bd-kube-api-access-rbzjv\") pod \"payload-pre-processing-e2e-inf-b-8efac4-5df6f86d78-8xm9m\" (UID: \"094495b3-e2a6-413d-b2d3-6251ad5667bd\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8efac4-5df6f86d78-8xm9m"
Sep 11 03:53:18.106152 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:18.106062    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8efac4-5df6f86d78-8xm9m"
Sep 11 03:53:18.378991 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:18.378962    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-8efac4-5df6f86d78-8xm9m"]
Sep 11 03:53:18.380828 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:53:18.380796    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod094495b3_e2a6_413d_b2d3_6251ad5667bd.slice/crio-73411ca29cc2713afb22d2ba1ec08a280bcce4537c009f3d7c2533b5ab4dbf62 WatchSource:0}: Error finding container 73411ca29cc2713afb22d2ba1ec08a280bcce4537c009f3d7c2533b5ab4dbf62: Status 404 returned error can't find the container with id 73411ca29cc2713afb22d2ba1ec08a280bcce4537c009f3d7c2533b5ab4dbf62
Sep 11 03:53:19.009248 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:19.009212    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8efac4-5df6f86d78-8xm9m" event={"ID":"094495b3-e2a6-413d-b2d3-6251ad5667bd","Type":"ContainerStarted","Data":"4b7b83ebb63a0850376438b3661871b033754dfee85f697b2062e5704c4695bd"}
Sep 11 03:53:19.009645 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:19.009265    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8efac4-5df6f86d78-8xm9m" event={"ID":"094495b3-e2a6-413d-b2d3-6251ad5667bd","Type":"ContainerStarted","Data":"73411ca29cc2713afb22d2ba1ec08a280bcce4537c009f3d7c2533b5ab4dbf62"}
Sep 11 03:53:19.009645 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:19.009340    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8efac4-5df6f86d78-8xm9m"
Sep 11 03:53:19.023570 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:19.023510    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8efac4-5df6f86d78-8xm9m" podStartSLOduration=2.023492486 podStartE2EDuration="2.023492486s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 03:53:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:53:19.023043315 +0000 UTC m=+787.181529639" watchObservedRunningTime="2026-09-11 03:53:19.023492486 +0000 UTC m=+787.181978810"
Sep 11 03:53:25.889143 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:25.889095    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1eeb81b2-7b85c4ff8-ngtcl"]
Sep 11 03:53:25.889795 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:25.889366    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1eeb81b2-7b85c4ff8-ngtcl" podUID="8e0286a3-712f-46d6-8927-2b036878bda7" containerName="maas-api" containerID="cri-o://bafcabbb2c0dbc2462f9a654351e45492dd0286bf9221f631fc1be73b8ad68cf" gracePeriod=30
Sep 11 03:53:25.927956 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:25.927913    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1eeb81b2-7b85c4ff8-ngtcl" podUID="8e0286a3-712f-46d6-8927-2b036878bda7" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.34:8443/health\": dial tcp 10.133.0.34:8443: connect: connection refused"
Sep 11 03:53:26.048303 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:26.048266    2580 generic.go:356] "Generic (PLEG): container finished" podID="8e0286a3-712f-46d6-8927-2b036878bda7" containerID="bafcabbb2c0dbc2462f9a654351e45492dd0286bf9221f631fc1be73b8ad68cf" exitCode=0
Sep 11 03:53:26.048467 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:26.048336    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1eeb81b2-7b85c4ff8-ngtcl" event={"ID":"8e0286a3-712f-46d6-8927-2b036878bda7","Type":"ContainerDied","Data":"bafcabbb2c0dbc2462f9a654351e45492dd0286bf9221f631fc1be73b8ad68cf"}
Sep 11 03:53:26.187498 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:26.187220    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1eeb81b2-7b85c4ff8-ngtcl"
Sep 11 03:53:26.271594 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:26.271555    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8e0286a3-712f-46d6-8927-2b036878bda7-maas-api-tls\") pod \"8e0286a3-712f-46d6-8927-2b036878bda7\" (UID: \"8e0286a3-712f-46d6-8927-2b036878bda7\") "
Sep 11 03:53:26.271814 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:26.271610    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8e0286a3-712f-46d6-8927-2b036878bda7-metrics-certs\") pod \"8e0286a3-712f-46d6-8927-2b036878bda7\" (UID: \"8e0286a3-712f-46d6-8927-2b036878bda7\") "
Sep 11 03:53:26.271814 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:26.271712    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e0286a3-712f-46d6-8927-2b036878bda7-openshift-service-ca\") pod \"8e0286a3-712f-46d6-8927-2b036878bda7\" (UID: \"8e0286a3-712f-46d6-8927-2b036878bda7\") "
Sep 11 03:53:26.271814 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:26.271777    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w8gdz\" (UniqueName: \"kubernetes.io/projected/8e0286a3-712f-46d6-8927-2b036878bda7-kube-api-access-w8gdz\") pod \"8e0286a3-712f-46d6-8927-2b036878bda7\" (UID: \"8e0286a3-712f-46d6-8927-2b036878bda7\") "
Sep 11 03:53:26.272140 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:26.272105    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e0286a3-712f-46d6-8927-2b036878bda7-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "8e0286a3-712f-46d6-8927-2b036878bda7" (UID: "8e0286a3-712f-46d6-8927-2b036878bda7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 03:53:26.274094 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:26.274071    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e0286a3-712f-46d6-8927-2b036878bda7-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "8e0286a3-712f-46d6-8927-2b036878bda7" (UID: "8e0286a3-712f-46d6-8927-2b036878bda7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 03:53:26.274278 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:26.274251    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e0286a3-712f-46d6-8927-2b036878bda7-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "8e0286a3-712f-46d6-8927-2b036878bda7" (UID: "8e0286a3-712f-46d6-8927-2b036878bda7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 03:53:26.274377 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:26.274267    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e0286a3-712f-46d6-8927-2b036878bda7-kube-api-access-w8gdz" (OuterVolumeSpecName: "kube-api-access-w8gdz") pod "8e0286a3-712f-46d6-8927-2b036878bda7" (UID: "8e0286a3-712f-46d6-8927-2b036878bda7"). InnerVolumeSpecName "kube-api-access-w8gdz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 03:53:26.372561 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:26.372526    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w8gdz\" (UniqueName: \"kubernetes.io/projected/8e0286a3-712f-46d6-8927-2b036878bda7-kube-api-access-w8gdz\") on node \"ip-10-0-138-121.ec2.internal\" DevicePath \"\""
Sep 11 03:53:26.372561 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:26.372562    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8e0286a3-712f-46d6-8927-2b036878bda7-maas-api-tls\") on node \"ip-10-0-138-121.ec2.internal\" DevicePath \"\""
Sep 11 03:53:26.372782 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:26.372575    2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8e0286a3-712f-46d6-8927-2b036878bda7-metrics-certs\") on node \"ip-10-0-138-121.ec2.internal\" DevicePath \"\""
Sep 11 03:53:26.372782 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:26.372587    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e0286a3-712f-46d6-8927-2b036878bda7-openshift-service-ca\") on node \"ip-10-0-138-121.ec2.internal\" DevicePath \"\""
Sep 11 03:53:27.057030 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:27.056994    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1eeb81b2-7b85c4ff8-ngtcl"
Sep 11 03:53:27.057030 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:27.057015    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1eeb81b2-7b85c4ff8-ngtcl" event={"ID":"8e0286a3-712f-46d6-8927-2b036878bda7","Type":"ContainerDied","Data":"b4c25b5bd9800752745fb7b635f30c86d2a86f508e6edcfaa0b07bc17617f4a1"}
Sep 11 03:53:27.057622 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:27.057078    2580 scope.go:117] "RemoveContainer" containerID="bafcabbb2c0dbc2462f9a654351e45492dd0286bf9221f631fc1be73b8ad68cf"
Sep 11 03:53:27.074746 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:27.074712    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1eeb81b2-7b85c4ff8-ngtcl"]
Sep 11 03:53:27.078835 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:27.078806    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1eeb81b2-7b85c4ff8-ngtcl"]
Sep 11 03:53:27.244350 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:27.244312    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-2c6bf060-gw-openshift-default-684bf7746d-b5czv"]
Sep 11 03:53:27.244956 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:27.244935    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e0286a3-712f-46d6-8927-2b036878bda7" containerName="maas-api"
Sep 11 03:53:27.245005 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:27.244961    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e0286a3-712f-46d6-8927-2b036878bda7" containerName="maas-api"
Sep 11 03:53:27.245112 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:27.245088    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e0286a3-712f-46d6-8927-2b036878bda7" containerName="maas-api"
Sep 11 03:53:27.277566 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:27.277530    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-2c6bf060-gw-openshift-default-684bf7746d-b5czv"]
Sep 11 03:53:27.277756 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:27.277637    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-2c6bf060-gw-openshift-default-684bf7746d-b5czv"
Sep 11 03:53:27.279948 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:27.279919    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-2c6bf060-gw-openshift-default-dockercfg-pv72z\""
Sep 11 03:53:27.381298 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:27.381201    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b97d8c86-2972-4312-a5f0-43f69a3ff31a-istio-data\") pod \"e2e-migrate-2c6bf060-gw-openshift-default-684bf7746d-b5czv\" (UID: \"b97d8c86-2972-4312-a5f0-43f69a3ff31a\") " pod="openshift-ingress/e2e-migrate-2c6bf060-gw-openshift-default-684bf7746d-b5czv"
Sep 11 03:53:27.381298 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:27.381253    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b97d8c86-2972-4312-a5f0-43f69a3ff31a-istio-token\") pod \"e2e-migrate-2c6bf060-gw-openshift-default-684bf7746d-b5czv\" (UID: \"b97d8c86-2972-4312-a5f0-43f69a3ff31a\") " pod="openshift-ingress/e2e-migrate-2c6bf060-gw-openshift-default-684bf7746d-b5czv"
Sep 11 03:53:27.381481 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:27.381332    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b97d8c86-2972-4312-a5f0-43f69a3ff31a-istio-envoy\") pod \"e2e-migrate-2c6bf060-gw-openshift-default-684bf7746d-b5czv\" (UID: \"b97d8c86-2972-4312-a5f0-43f69a3ff31a\") " pod="openshift-ingress/e2e-migrate-2c6bf060-gw-openshift-default-684bf7746d-b5czv"
Sep 11 03:53:27.381481 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:27.381356    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b97d8c86-2972-4312-a5f0-43f69a3ff31a-credential-socket\") pod \"e2e-migrate-2c6bf060-gw-openshift-default-684bf7746d-b5czv\" (UID: \"b97d8c86-2972-4312-a5f0-43f69a3ff31a\") " pod="openshift-ingress/e2e-migrate-2c6bf060-gw-openshift-default-684bf7746d-b5czv"
Sep 11 03:53:27.381481 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:27.381372    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b97d8c86-2972-4312-a5f0-43f69a3ff31a-workload-certs\") pod \"e2e-migrate-2c6bf060-gw-openshift-default-684bf7746d-b5czv\" (UID: \"b97d8c86-2972-4312-a5f0-43f69a3ff31a\") " pod="openshift-ingress/e2e-migrate-2c6bf060-gw-openshift-default-684bf7746d-b5czv"
Sep 11 03:53:27.381481 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:27.381425    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b97d8c86-2972-4312-a5f0-43f69a3ff31a-istiod-ca-cert\") pod \"e2e-migrate-2c6bf060-gw-openshift-default-684bf7746d-b5czv\" (UID: \"b97d8c86-2972-4312-a5f0-43f69a3ff31a\") " pod="openshift-ingress/e2e-migrate-2c6bf060-gw-openshift-default-684bf7746d-b5czv"
Sep 11 03:53:27.381481 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:27.381454    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b97d8c86-2972-4312-a5f0-43f69a3ff31a-istio-podinfo\") pod \"e2e-migrate-2c6bf060-gw-openshift-default-684bf7746d-b5czv\" (UID: \"b97d8c86-2972-4312-a5f0-43f69a3ff31a\") " pod="openshift-ingress/e2e-migrate-2c6bf060-gw-openshift-default-684bf7746d-b5czv"
Sep 11 03:53:27.381481 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:27.381469    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9kmt\" (UniqueName: \"kubernetes.io/projected/b97d8c86-2972-4312-a5f0-43f69a3ff31a-kube-api-access-v9kmt\") pod \"e2e-migrate-2c6bf060-gw-openshift-default-684bf7746d-b5czv\" (UID: \"b97d8c86-2972-4312-a5f0-43f69a3ff31a\") " pod="openshift-ingress/e2e-migrate-2c6bf060-gw-openshift-default-684bf7746d-b5czv"
Sep 11 03:53:27.381675 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:27.381498    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b97d8c86-2972-4312-a5f0-43f69a3ff31a-workload-socket\") pod \"e2e-migrate-2c6bf060-gw-openshift-default-684bf7746d-b5czv\" (UID: \"b97d8c86-2972-4312-a5f0-43f69a3ff31a\") " pod="openshift-ingress/e2e-migrate-2c6bf060-gw-openshift-default-684bf7746d-b5czv"
Sep 11 03:53:27.482807 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:27.482765    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b97d8c86-2972-4312-a5f0-43f69a3ff31a-istio-podinfo\") pod \"e2e-migrate-2c6bf060-gw-openshift-default-684bf7746d-b5czv\" (UID: \"b97d8c86-2972-4312-a5f0-43f69a3ff31a\") " pod="openshift-ingress/e2e-migrate-2c6bf060-gw-openshift-default-684bf7746d-b5czv"
Sep 11 03:53:27.482807 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:27.482809    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9kmt\" (UniqueName: \"kubernetes.io/projected/b97d8c86-2972-4312-a5f0-43f69a3ff31a-kube-api-access-v9kmt\") pod \"e2e-migrate-2c6bf060-gw-openshift-default-684bf7746d-b5czv\" (UID: \"b97d8c86-2972-4312-a5f0-43f69a3ff31a\") " pod="openshift-ingress/e2e-migrate-2c6bf060-gw-openshift-default-684bf7746d-b5czv"
Sep 11 03:53:27.483053 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:27.482857    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b97d8c86-2972-4312-a5f0-43f69a3ff31a-workload-socket\") pod \"e2e-migrate-2c6bf060-gw-openshift-default-684bf7746d-b5czv\" (UID: \"b97d8c86-2972-4312-a5f0-43f69a3ff31a\") " pod="openshift-ingress/e2e-migrate-2c6bf060-gw-openshift-default-684bf7746d-b5czv"
Sep 11 03:53:27.483053 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:27.482895    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b97d8c86-2972-4312-a5f0-43f69a3ff31a-istio-data\") pod \"e2e-migrate-2c6bf060-gw-openshift-default-684bf7746d-b5czv\" (UID: \"b97d8c86-2972-4312-a5f0-43f69a3ff31a\") " pod="openshift-ingress/e2e-migrate-2c6bf060-gw-openshift-default-684bf7746d-b5czv"
Sep 11 03:53:27.483161 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:27.483094    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b97d8c86-2972-4312-a5f0-43f69a3ff31a-istio-token\") pod \"e2e-migrate-2c6bf060-gw-openshift-default-684bf7746d-b5czv\" (UID: \"b97d8c86-2972-4312-a5f0-43f69a3ff31a\") " pod="openshift-ingress/e2e-migrate-2c6bf060-gw-openshift-default-684bf7746d-b5czv"
Sep 11 03:53:27.483219 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:27.483191    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b97d8c86-2972-4312-a5f0-43f69a3ff31a-istio-envoy\") pod \"e2e-migrate-2c6bf060-gw-openshift-default-684bf7746d-b5czv\" (UID: \"b97d8c86-2972-4312-a5f0-43f69a3ff31a\") " pod="openshift-ingress/e2e-migrate-2c6bf060-gw-openshift-default-684bf7746d-b5czv"
Sep 11 03:53:27.483338 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:27.483318    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b97d8c86-2972-4312-a5f0-43f69a3ff31a-istio-data\") pod \"e2e-migrate-2c6bf060-gw-openshift-default-684bf7746d-b5czv\" (UID: \"b97d8c86-2972-4312-a5f0-43f69a3ff31a\") " pod="openshift-ingress/e2e-migrate-2c6bf060-gw-openshift-default-684bf7746d-b5czv"
Sep 11 03:53:27.483417 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:27.483353    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b97d8c86-2972-4312-a5f0-43f69a3ff31a-workload-socket\") pod \"e2e-migrate-2c6bf060-gw-openshift-default-684bf7746d-b5czv\" (UID: \"b97d8c86-2972-4312-a5f0-43f69a3ff31a\") " pod="openshift-ingress/e2e-migrate-2c6bf060-gw-openshift-default-684bf7746d-b5czv"
Sep 11 03:53:27.483417 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:27.483225    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b97d8c86-2972-4312-a5f0-43f69a3ff31a-credential-socket\") pod \"e2e-migrate-2c6bf060-gw-openshift-default-684bf7746d-b5czv\" (UID: \"b97d8c86-2972-4312-a5f0-43f69a3ff31a\") " pod="openshift-ingress/e2e-migrate-2c6bf060-gw-openshift-default-684bf7746d-b5czv"
Sep 11 03:53:27.483914 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:27.483439    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b97d8c86-2972-4312-a5f0-43f69a3ff31a-workload-certs\") pod \"e2e-migrate-2c6bf060-gw-openshift-default-684bf7746d-b5czv\" (UID: \"b97d8c86-2972-4312-a5f0-43f69a3ff31a\") " pod="openshift-ingress/e2e-migrate-2c6bf060-gw-openshift-default-684bf7746d-b5czv"
Sep 11 03:53:27.483914 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:27.483525    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b97d8c86-2972-4312-a5f0-43f69a3ff31a-istiod-ca-cert\") pod \"e2e-migrate-2c6bf060-gw-openshift-default-684bf7746d-b5czv\" (UID: \"b97d8c86-2972-4312-a5f0-43f69a3ff31a\") " pod="openshift-ingress/e2e-migrate-2c6bf060-gw-openshift-default-684bf7746d-b5czv"
Sep 11 03:53:27.483914 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:27.483561    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b97d8c86-2972-4312-a5f0-43f69a3ff31a-credential-socket\") pod \"e2e-migrate-2c6bf060-gw-openshift-default-684bf7746d-b5czv\" (UID: \"b97d8c86-2972-4312-a5f0-43f69a3ff31a\") " pod="openshift-ingress/e2e-migrate-2c6bf060-gw-openshift-default-684bf7746d-b5czv"
Sep 11 03:53:27.483914 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:27.483895    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b97d8c86-2972-4312-a5f0-43f69a3ff31a-workload-certs\") pod \"e2e-migrate-2c6bf060-gw-openshift-default-684bf7746d-b5czv\" (UID: \"b97d8c86-2972-4312-a5f0-43f69a3ff31a\") " pod="openshift-ingress/e2e-migrate-2c6bf060-gw-openshift-default-684bf7746d-b5czv"
Sep 11 03:53:27.484259 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:27.484238    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b97d8c86-2972-4312-a5f0-43f69a3ff31a-istiod-ca-cert\") pod \"e2e-migrate-2c6bf060-gw-openshift-default-684bf7746d-b5czv\" (UID: \"b97d8c86-2972-4312-a5f0-43f69a3ff31a\") " pod="openshift-ingress/e2e-migrate-2c6bf060-gw-openshift-default-684bf7746d-b5czv"
Sep 11 03:53:27.486008 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:27.485982    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b97d8c86-2972-4312-a5f0-43f69a3ff31a-istio-envoy\") pod \"e2e-migrate-2c6bf060-gw-openshift-default-684bf7746d-b5czv\" (UID: \"b97d8c86-2972-4312-a5f0-43f69a3ff31a\") " pod="openshift-ingress/e2e-migrate-2c6bf060-gw-openshift-default-684bf7746d-b5czv"
Sep 11 03:53:27.486370 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:27.486304    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b97d8c86-2972-4312-a5f0-43f69a3ff31a-istio-podinfo\") pod \"e2e-migrate-2c6bf060-gw-openshift-default-684bf7746d-b5czv\" (UID: \"b97d8c86-2972-4312-a5f0-43f69a3ff31a\") " pod="openshift-ingress/e2e-migrate-2c6bf060-gw-openshift-default-684bf7746d-b5czv"
Sep 11 03:53:27.492035 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:27.492010    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b97d8c86-2972-4312-a5f0-43f69a3ff31a-istio-token\") pod \"e2e-migrate-2c6bf060-gw-openshift-default-684bf7746d-b5czv\" (UID: \"b97d8c86-2972-4312-a5f0-43f69a3ff31a\") " pod="openshift-ingress/e2e-migrate-2c6bf060-gw-openshift-default-684bf7746d-b5czv"
Sep 11 03:53:27.492522 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:27.492505    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9kmt\" (UniqueName: \"kubernetes.io/projected/b97d8c86-2972-4312-a5f0-43f69a3ff31a-kube-api-access-v9kmt\") pod \"e2e-migrate-2c6bf060-gw-openshift-default-684bf7746d-b5czv\" (UID: \"b97d8c86-2972-4312-a5f0-43f69a3ff31a\") " pod="openshift-ingress/e2e-migrate-2c6bf060-gw-openshift-default-684bf7746d-b5czv"
Sep 11 03:53:27.589680 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:27.589627    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-2c6bf060-gw-openshift-default-684bf7746d-b5czv"
Sep 11 03:53:27.864429 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:27.864383    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-2c6bf060-gw-openshift-default-684bf7746d-b5czv"]
Sep 11 03:53:27.875054 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:27.873865    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 03:53:27.875054 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:27.873970    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 03:53:27.875054 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:27.874018    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 03:53:28.067275 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:28.067231    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-2c6bf060-gw-openshift-default-684bf7746d-b5czv" event={"ID":"b97d8c86-2972-4312-a5f0-43f69a3ff31a","Type":"ContainerStarted","Data":"8c003f8e21d57da83a9547ad4b48d0ac89755037dc2be85d75c3cc5e91606fc1"}
Sep 11 03:53:28.067816 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:28.067302    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-2c6bf060-gw-openshift-default-684bf7746d-b5czv" event={"ID":"b97d8c86-2972-4312-a5f0-43f69a3ff31a","Type":"ContainerStarted","Data":"93798222b247963b9955a26a8015b9950ec169bc1b8d5b77e566c159be595643"}
Sep 11 03:53:28.086291 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:28.086222    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-2c6bf060-gw-openshift-default-684bf7746d-b5czv" podStartSLOduration=1.086199464 podStartE2EDuration="1.086199464s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 03:53:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:53:28.085327286 +0000 UTC m=+796.243813609" watchObservedRunningTime="2026-09-11 03:53:28.086199464 +0000 UTC m=+796.244685785"
Sep 11 03:53:28.467453 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:28.467408    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8e0286a3-712f-46d6-8927-2b036878bda7" path="/var/lib/kubelet/pods/8e0286a3-712f-46d6-8927-2b036878bda7/volumes"
Sep 11 03:53:28.590806 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:28.590764    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-2c6bf060-gw-openshift-default-684bf7746d-b5czv"
Sep 11 03:53:28.592244 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:28.592216    2580 patch_prober.go:28] interesting pod/e2e-migrate-2c6bf060-gw-openshift-default-684bf7746d-b5czv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Sep 11 03:53:28.592364 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:28.592268    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-2c6bf060-gw-openshift-default-684bf7746d-b5czv" podUID="b97d8c86-2972-4312-a5f0-43f69a3ff31a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Sep 11 03:53:29.594980 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:29.594928    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-2c6bf060-gw-openshift-default-684bf7746d-b5czv"
Sep 11 03:53:30.017322 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:30.017290    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8efac4-5df6f86d78-8xm9m"
Sep 11 03:53:30.084983 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:30.084946    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-2c6bf060-gw-openshift-default-684bf7746d-b5czv"
Sep 11 03:53:30.086060 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:30.086036    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-2c6bf060-gw-openshift-default-684bf7746d-b5czv"
Sep 11 03:53:31.511295 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:31.511242    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-2c6bf060-5f7db68584-m6l4j"]
Sep 11 03:53:31.516148 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:31.516123    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-2c6bf060-5f7db68584-m6l4j"
Sep 11 03:53:31.519803 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:31.519696    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-fcpfs\""
Sep 11 03:53:31.519803 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:31.519706    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 11 03:53:31.519987 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:31.519823    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 11 03:53:31.519987 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:31.519887    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-2c6bf060-serving-cert\""
Sep 11 03:53:31.519987 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:31.519896    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 03:53:31.528230 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:31.528129    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-2c6bf060-5f7db68584-m6l4j"]
Sep 11 03:53:31.622998 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:31.622957    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/24f87fa9-0c0b-4d54-8a54-ba1ca6b81b90-maas-api-tls\") pod \"maas-api-e2e-migrate-2c6bf060-5f7db68584-m6l4j\" (UID: \"24f87fa9-0c0b-4d54-8a54-ba1ca6b81b90\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-2c6bf060-5f7db68584-m6l4j"
Sep 11 03:53:31.622998 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:31.623003    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/24f87fa9-0c0b-4d54-8a54-ba1ca6b81b90-metrics-certs\") pod \"maas-api-e2e-migrate-2c6bf060-5f7db68584-m6l4j\" (UID: \"24f87fa9-0c0b-4d54-8a54-ba1ca6b81b90\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-2c6bf060-5f7db68584-m6l4j"
Sep 11 03:53:31.623199 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:31.623073    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/24f87fa9-0c0b-4d54-8a54-ba1ca6b81b90-openshift-service-ca\") pod \"maas-api-e2e-migrate-2c6bf060-5f7db68584-m6l4j\" (UID: \"24f87fa9-0c0b-4d54-8a54-ba1ca6b81b90\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-2c6bf060-5f7db68584-m6l4j"
Sep 11 03:53:31.623199 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:31.623153    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xqs7\" (UniqueName: \"kubernetes.io/projected/24f87fa9-0c0b-4d54-8a54-ba1ca6b81b90-kube-api-access-9xqs7\") pod \"maas-api-e2e-migrate-2c6bf060-5f7db68584-m6l4j\" (UID: \"24f87fa9-0c0b-4d54-8a54-ba1ca6b81b90\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-2c6bf060-5f7db68584-m6l4j"
Sep 11 03:53:31.724110 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:31.724067    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/24f87fa9-0c0b-4d54-8a54-ba1ca6b81b90-maas-api-tls\") pod \"maas-api-e2e-migrate-2c6bf060-5f7db68584-m6l4j\" (UID: \"24f87fa9-0c0b-4d54-8a54-ba1ca6b81b90\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-2c6bf060-5f7db68584-m6l4j"
Sep 11 03:53:31.724110 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:31.724114    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/24f87fa9-0c0b-4d54-8a54-ba1ca6b81b90-metrics-certs\") pod \"maas-api-e2e-migrate-2c6bf060-5f7db68584-m6l4j\" (UID: \"24f87fa9-0c0b-4d54-8a54-ba1ca6b81b90\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-2c6bf060-5f7db68584-m6l4j"
Sep 11 03:53:31.724308 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:31.724147    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/24f87fa9-0c0b-4d54-8a54-ba1ca6b81b90-openshift-service-ca\") pod \"maas-api-e2e-migrate-2c6bf060-5f7db68584-m6l4j\" (UID: \"24f87fa9-0c0b-4d54-8a54-ba1ca6b81b90\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-2c6bf060-5f7db68584-m6l4j"
Sep 11 03:53:31.724308 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:31.724182    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xqs7\" (UniqueName: \"kubernetes.io/projected/24f87fa9-0c0b-4d54-8a54-ba1ca6b81b90-kube-api-access-9xqs7\") pod \"maas-api-e2e-migrate-2c6bf060-5f7db68584-m6l4j\" (UID: \"24f87fa9-0c0b-4d54-8a54-ba1ca6b81b90\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-2c6bf060-5f7db68584-m6l4j"
Sep 11 03:53:31.724940 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:31.724907    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/24f87fa9-0c0b-4d54-8a54-ba1ca6b81b90-openshift-service-ca\") pod \"maas-api-e2e-migrate-2c6bf060-5f7db68584-m6l4j\" (UID: \"24f87fa9-0c0b-4d54-8a54-ba1ca6b81b90\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-2c6bf060-5f7db68584-m6l4j"
Sep 11 03:53:31.726554 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:31.726526    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/24f87fa9-0c0b-4d54-8a54-ba1ca6b81b90-maas-api-tls\") pod \"maas-api-e2e-migrate-2c6bf060-5f7db68584-m6l4j\" (UID: \"24f87fa9-0c0b-4d54-8a54-ba1ca6b81b90\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-2c6bf060-5f7db68584-m6l4j"
Sep 11 03:53:31.726698 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:31.726560    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/24f87fa9-0c0b-4d54-8a54-ba1ca6b81b90-metrics-certs\") pod \"maas-api-e2e-migrate-2c6bf060-5f7db68584-m6l4j\" (UID: \"24f87fa9-0c0b-4d54-8a54-ba1ca6b81b90\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-2c6bf060-5f7db68584-m6l4j"
Sep 11 03:53:31.731330 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:31.731304    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xqs7\" (UniqueName: \"kubernetes.io/projected/24f87fa9-0c0b-4d54-8a54-ba1ca6b81b90-kube-api-access-9xqs7\") pod \"maas-api-e2e-migrate-2c6bf060-5f7db68584-m6l4j\" (UID: \"24f87fa9-0c0b-4d54-8a54-ba1ca6b81b90\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-2c6bf060-5f7db68584-m6l4j"
Sep 11 03:53:31.829796 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:31.829709    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-2c6bf060-5f7db68584-m6l4j"
Sep 11 03:53:32.098596 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:32.098512    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-2c6bf060-c5c9797d6-pctzf"]
Sep 11 03:53:32.102829 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:32.102805    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-2c6bf060-c5c9797d6-pctzf"
Sep 11 03:53:32.104941 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:32.104916    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-2c6bf060-dockercfg-mvfxb\""
Sep 11 03:53:32.105045 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:32.104988    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-2c6bf060\""
Sep 11 03:53:32.110504 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:32.110471    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-2c6bf060-c5c9797d6-pctzf"]
Sep 11 03:53:32.128820 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:32.128292    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/098240a8-8c72-4966-b7ff-3f30af8587ba-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-2c6bf060-c5c9797d6-pctzf\" (UID: \"098240a8-8c72-4966-b7ff-3f30af8587ba\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-2c6bf060-c5c9797d6-pctzf"
Sep 11 03:53:32.128820 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:32.128494    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-snvmg\" (UniqueName: \"kubernetes.io/projected/098240a8-8c72-4966-b7ff-3f30af8587ba-kube-api-access-snvmg\") pod \"payload-pre-processing-e2e-migrate-2c6bf060-c5c9797d6-pctzf\" (UID: \"098240a8-8c72-4966-b7ff-3f30af8587ba\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-2c6bf060-c5c9797d6-pctzf"
Sep 11 03:53:32.140443 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:32.140401    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-2c6bf060-5f7db68584-m6l4j"]
Sep 11 03:53:32.143704 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:53:32.143652    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24f87fa9_0c0b_4d54_8a54_ba1ca6b81b90.slice/crio-c8533c0a423ed751ebffb6a57833f4f2a02ab83186d6c0f58a6e198462f3d881 WatchSource:0}: Error finding container c8533c0a423ed751ebffb6a57833f4f2a02ab83186d6c0f58a6e198462f3d881: Status 404 returned error can't find the container with id c8533c0a423ed751ebffb6a57833f4f2a02ab83186d6c0f58a6e198462f3d881
Sep 11 03:53:32.229977 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:32.229937    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-snvmg\" (UniqueName: \"kubernetes.io/projected/098240a8-8c72-4966-b7ff-3f30af8587ba-kube-api-access-snvmg\") pod \"payload-pre-processing-e2e-migrate-2c6bf060-c5c9797d6-pctzf\" (UID: \"098240a8-8c72-4966-b7ff-3f30af8587ba\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-2c6bf060-c5c9797d6-pctzf"
Sep 11 03:53:32.230174 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:32.229990    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/098240a8-8c72-4966-b7ff-3f30af8587ba-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-2c6bf060-c5c9797d6-pctzf\" (UID: \"098240a8-8c72-4966-b7ff-3f30af8587ba\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-2c6bf060-c5c9797d6-pctzf"
Sep 11 03:53:32.230715 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:32.230692    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/098240a8-8c72-4966-b7ff-3f30af8587ba-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-2c6bf060-c5c9797d6-pctzf\" (UID: \"098240a8-8c72-4966-b7ff-3f30af8587ba\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-2c6bf060-c5c9797d6-pctzf"
Sep 11 03:53:32.236842 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:32.236816    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-snvmg\" (UniqueName: \"kubernetes.io/projected/098240a8-8c72-4966-b7ff-3f30af8587ba-kube-api-access-snvmg\") pod \"payload-pre-processing-e2e-migrate-2c6bf060-c5c9797d6-pctzf\" (UID: \"098240a8-8c72-4966-b7ff-3f30af8587ba\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-2c6bf060-c5c9797d6-pctzf"
Sep 11 03:53:32.417108 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:32.416983    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-2c6bf060-c5c9797d6-pctzf"
Sep 11 03:53:32.713674 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:32.713629    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-2c6bf060-c5c9797d6-pctzf"]
Sep 11 03:53:32.716377 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:53:32.716353    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod098240a8_8c72_4966_b7ff_3f30af8587ba.slice/crio-32d6d7fa8f1705d07affbe2231b24832a7c2b4b63ad290ddf44660b3fad645f8 WatchSource:0}: Error finding container 32d6d7fa8f1705d07affbe2231b24832a7c2b4b63ad290ddf44660b3fad645f8: Status 404 returned error can't find the container with id 32d6d7fa8f1705d07affbe2231b24832a7c2b4b63ad290ddf44660b3fad645f8
Sep 11 03:53:33.101627 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:33.101516    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-2c6bf060-c5c9797d6-pctzf" event={"ID":"098240a8-8c72-4966-b7ff-3f30af8587ba","Type":"ContainerStarted","Data":"3820e21fa141ea4534859f8293c25fa4cb6c192de769d47f11784347f260efd2"}
Sep 11 03:53:33.101627 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:33.101581    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-2c6bf060-c5c9797d6-pctzf" event={"ID":"098240a8-8c72-4966-b7ff-3f30af8587ba","Type":"ContainerStarted","Data":"32d6d7fa8f1705d07affbe2231b24832a7c2b4b63ad290ddf44660b3fad645f8"}
Sep 11 03:53:33.101627 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:33.101611    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-2c6bf060-c5c9797d6-pctzf"
Sep 11 03:53:33.103145 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:33.103121    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-2c6bf060-5f7db68584-m6l4j" event={"ID":"24f87fa9-0c0b-4d54-8a54-ba1ca6b81b90","Type":"ContainerStarted","Data":"a36f0aa53a735a4f0635a303f8d7654a2cdd4b53f70c2fe2e53c6174b94245bf"}
Sep 11 03:53:33.103238 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:33.103162    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-2c6bf060-5f7db68584-m6l4j" event={"ID":"24f87fa9-0c0b-4d54-8a54-ba1ca6b81b90","Type":"ContainerStarted","Data":"c8533c0a423ed751ebffb6a57833f4f2a02ab83186d6c0f58a6e198462f3d881"}
Sep 11 03:53:33.103238 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:33.103183    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-2c6bf060-5f7db68584-m6l4j"
Sep 11 03:53:33.117335 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:33.117283    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-2c6bf060-c5c9797d6-pctzf" podStartSLOduration=1.117270236 podStartE2EDuration="1.117270236s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 03:53:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:53:33.116151717 +0000 UTC m=+801.274638041" watchObservedRunningTime="2026-09-11 03:53:33.117270236 +0000 UTC m=+801.275756550"
Sep 11 03:53:33.131442 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:33.131373    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-2c6bf060-5f7db68584-m6l4j" podStartSLOduration=1.764267755 podStartE2EDuration="2.13135847s" totalImagesPullingTime="367.090715ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 03:53:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:53:33.130969063 +0000 UTC m=+801.289455387" watchObservedRunningTime="2026-09-11 03:53:33.13135847 +0000 UTC m=+801.289844793"
Sep 11 03:53:36.119953 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:36.119874    2580 generic.go:356] "Generic (PLEG): container finished" podID="bfe5ee73-4698-420d-9729-3153ab5656fa" containerID="1856f2ca275da8b5a2f861e2ee2827b84fc738a8867abf2c136fb088d94e9b0e" exitCode=0
Sep 11 03:53:36.120341 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:36.119948    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-8efac4-openshift-default-66557bb4f8-dl5z8" event={"ID":"bfe5ee73-4698-420d-9729-3153ab5656fa","Type":"ContainerDied","Data":"1856f2ca275da8b5a2f861e2ee2827b84fc738a8867abf2c136fb088d94e9b0e"}
Sep 11 03:53:36.120391 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:36.120376    2580 scope.go:117] "RemoveContainer" containerID="1856f2ca275da8b5a2f861e2ee2827b84fc738a8867abf2c136fb088d94e9b0e"
Sep 11 03:53:36.121478 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:36.121445    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 03:53:36.121595 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:36.121530    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 03:53:36.121595 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:36.121581    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 03:53:37.126556 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:37.126520    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-8efac4-openshift-default-66557bb4f8-dl5z8" event={"ID":"bfe5ee73-4698-420d-9729-3153ab5656fa","Type":"ContainerStarted","Data":"552b8541e0a1981a238d9d627a711fa51e4d4c069425d744101e8ebd7b7ad04e"}
Sep 11 03:53:37.238552 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:37.238516    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-8efac4-openshift-default-66557bb4f8-dl5z8"
Sep 11 03:53:37.238781 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:37.238760    2580 patch_prober.go:28] interesting pod/e2e-inf-b-8efac4-openshift-default-66557bb4f8-dl5z8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body=
Sep 11 03:53:37.238837 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:37.238809    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8efac4-openshift-default-66557bb4f8-dl5z8" podUID="bfe5ee73-4698-420d-9729-3153ab5656fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused"
Sep 11 03:53:38.238444 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:38.238398    2580 patch_prober.go:28] interesting pod/e2e-inf-b-8efac4-openshift-default-66557bb4f8-dl5z8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body=
Sep 11 03:53:38.238880 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:38.238468    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8efac4-openshift-default-66557bb4f8-dl5z8" podUID="bfe5ee73-4698-420d-9729-3153ab5656fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused"
Sep 11 03:53:39.114901 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:39.114869    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-2c6bf060-5f7db68584-m6l4j"
Sep 11 03:53:39.238826 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:39.238785    2580 patch_prober.go:28] interesting pod/e2e-inf-b-8efac4-openshift-default-66557bb4f8-dl5z8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body=
Sep 11 03:53:39.239193 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:39.238844    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8efac4-openshift-default-66557bb4f8-dl5z8" podUID="bfe5ee73-4698-420d-9729-3153ab5656fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused"
Sep 11 03:53:40.238947 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:40.238907    2580 patch_prober.go:28] interesting pod/e2e-inf-b-8efac4-openshift-default-66557bb4f8-dl5z8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body=
Sep 11 03:53:40.239330 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:40.238997    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8efac4-openshift-default-66557bb4f8-dl5z8" podUID="bfe5ee73-4698-420d-9729-3153ab5656fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused"
Sep 11 03:53:41.238839 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:41.238802    2580 patch_prober.go:28] interesting pod/e2e-inf-b-8efac4-openshift-default-66557bb4f8-dl5z8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body=
Sep 11 03:53:41.239012 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:41.238856    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8efac4-openshift-default-66557bb4f8-dl5z8" podUID="bfe5ee73-4698-420d-9729-3153ab5656fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused"
Sep 11 03:53:42.285886 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:42.285846    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-8efac4-openshift-default-66557bb4f8-dl5z8"
Sep 11 03:53:42.286438 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:42.286417    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-8efac4-openshift-default-66557bb4f8-dl5z8"
Sep 11 03:53:42.287601 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:42.287575    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-8efac4-openshift-default-66557bb4f8-dl5z8"
Sep 11 03:53:44.111547 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:44.111517    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-2c6bf060-c5c9797d6-pctzf"
Sep 11 03:53:49.205848 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:49.205814    2580 generic.go:356] "Generic (PLEG): container finished" podID="b97d8c86-2972-4312-a5f0-43f69a3ff31a" containerID="8c003f8e21d57da83a9547ad4b48d0ac89755037dc2be85d75c3cc5e91606fc1" exitCode=0
Sep 11 03:53:49.206366 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:49.205895    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-2c6bf060-gw-openshift-default-684bf7746d-b5czv" event={"ID":"b97d8c86-2972-4312-a5f0-43f69a3ff31a","Type":"ContainerDied","Data":"8c003f8e21d57da83a9547ad4b48d0ac89755037dc2be85d75c3cc5e91606fc1"}
Sep 11 03:53:49.206452 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:49.206434    2580 scope.go:117] "RemoveContainer" containerID="8c003f8e21d57da83a9547ad4b48d0ac89755037dc2be85d75c3cc5e91606fc1"
Sep 11 03:53:49.207555 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:49.207529    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 03:53:49.207612 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:49.207596    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 03:53:49.207678 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:49.207652    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 03:53:50.212772 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:50.212742    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-2c6bf060-gw-openshift-default-684bf7746d-b5czv" event={"ID":"b97d8c86-2972-4312-a5f0-43f69a3ff31a","Type":"ContainerStarted","Data":"4a40fc960ff02ed5fcf5d221af572204e196122005199a93566c24e6e2d5693d"}
Sep 11 03:53:50.590352 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:50.590260    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-2c6bf060-gw-openshift-default-684bf7746d-b5czv"
Sep 11 03:53:50.590550 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:50.590499    2580 patch_prober.go:28] interesting pod/e2e-migrate-2c6bf060-gw-openshift-default-684bf7746d-b5czv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Sep 11 03:53:50.590590 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:50.590553    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-2c6bf060-gw-openshift-default-684bf7746d-b5czv" podUID="b97d8c86-2972-4312-a5f0-43f69a3ff31a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Sep 11 03:53:51.591106 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:51.591069    2580 patch_prober.go:28] interesting pod/e2e-migrate-2c6bf060-gw-openshift-default-684bf7746d-b5czv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Sep 11 03:53:51.591521 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:51.591135    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-2c6bf060-gw-openshift-default-684bf7746d-b5czv" podUID="b97d8c86-2972-4312-a5f0-43f69a3ff31a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Sep 11 03:53:52.590792 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:52.590750    2580 patch_prober.go:28] interesting pod/e2e-migrate-2c6bf060-gw-openshift-default-684bf7746d-b5czv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Sep 11 03:53:52.590975 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:52.590818    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-2c6bf060-gw-openshift-default-684bf7746d-b5czv" podUID="b97d8c86-2972-4312-a5f0-43f69a3ff31a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Sep 11 03:53:53.590824 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:53.590788    2580 patch_prober.go:28] interesting pod/e2e-migrate-2c6bf060-gw-openshift-default-684bf7746d-b5czv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Sep 11 03:53:53.591186 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:53.590845    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-2c6bf060-gw-openshift-default-684bf7746d-b5czv" podUID="b97d8c86-2972-4312-a5f0-43f69a3ff31a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Sep 11 03:53:54.590102 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:54.590060    2580 patch_prober.go:28] interesting pod/e2e-migrate-2c6bf060-gw-openshift-default-684bf7746d-b5czv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Sep 11 03:53:54.590315 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:54.590118    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-2c6bf060-gw-openshift-default-684bf7746d-b5czv" podUID="b97d8c86-2972-4312-a5f0-43f69a3ff31a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Sep 11 03:53:55.590193 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:55.590154    2580 patch_prober.go:28] interesting pod/e2e-migrate-2c6bf060-gw-openshift-default-684bf7746d-b5czv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Sep 11 03:53:55.590572 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:55.590214    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-2c6bf060-gw-openshift-default-684bf7746d-b5czv" podUID="b97d8c86-2972-4312-a5f0-43f69a3ff31a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Sep 11 03:53:56.595255 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:56.595221    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-2c6bf060-gw-openshift-default-684bf7746d-b5czv"
Sep 11 03:53:56.595747 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:56.595476    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-2c6bf060-gw-openshift-default-684bf7746d-b5czv"
Sep 11 03:53:56.596172 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:53:56.596152    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-2c6bf060-gw-openshift-default-684bf7746d-b5czv"
Sep 11 03:54:01.615780 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:01.615743    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-2c6bf060-5f7db68584-m6l4j"]
Sep 11 03:54:01.616320 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:01.616034    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-2c6bf060-5f7db68584-m6l4j" podUID="24f87fa9-0c0b-4d54-8a54-ba1ca6b81b90" containerName="maas-api" containerID="cri-o://a36f0aa53a735a4f0635a303f8d7654a2cdd4b53f70c2fe2e53c6174b94245bf" gracePeriod=30
Sep 11 03:54:01.670586 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:01.670552    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-2c6bf060-c5c9797d6-pctzf"]
Sep 11 03:54:01.671005 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:01.670956    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-2c6bf060-c5c9797d6-pctzf" podUID="098240a8-8c72-4966-b7ff-3f30af8587ba" containerName="payload-pre-processing" containerID="cri-o://3820e21fa141ea4534859f8293c25fa4cb6c192de769d47f11784347f260efd2" gracePeriod=30
Sep 11 03:54:01.914845 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:01.914820    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-2c6bf060-5f7db68584-m6l4j"
Sep 11 03:54:01.944949 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:01.944922    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/24f87fa9-0c0b-4d54-8a54-ba1ca6b81b90-metrics-certs\") pod \"24f87fa9-0c0b-4d54-8a54-ba1ca6b81b90\" (UID: \"24f87fa9-0c0b-4d54-8a54-ba1ca6b81b90\") "
Sep 11 03:54:01.945075 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:01.944955    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9xqs7\" (UniqueName: \"kubernetes.io/projected/24f87fa9-0c0b-4d54-8a54-ba1ca6b81b90-kube-api-access-9xqs7\") pod \"24f87fa9-0c0b-4d54-8a54-ba1ca6b81b90\" (UID: \"24f87fa9-0c0b-4d54-8a54-ba1ca6b81b90\") "
Sep 11 03:54:01.945075 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:01.944999    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/24f87fa9-0c0b-4d54-8a54-ba1ca6b81b90-openshift-service-ca\") pod \"24f87fa9-0c0b-4d54-8a54-ba1ca6b81b90\" (UID: \"24f87fa9-0c0b-4d54-8a54-ba1ca6b81b90\") "
Sep 11 03:54:01.945075 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:01.945051    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/24f87fa9-0c0b-4d54-8a54-ba1ca6b81b90-maas-api-tls\") pod \"24f87fa9-0c0b-4d54-8a54-ba1ca6b81b90\" (UID: \"24f87fa9-0c0b-4d54-8a54-ba1ca6b81b90\") "
Sep 11 03:54:01.945399 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:01.945368    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24f87fa9-0c0b-4d54-8a54-ba1ca6b81b90-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "24f87fa9-0c0b-4d54-8a54-ba1ca6b81b90" (UID: "24f87fa9-0c0b-4d54-8a54-ba1ca6b81b90"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 03:54:01.947022 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:01.946998    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-2c6bf060-c5c9797d6-pctzf"
Sep 11 03:54:01.947168 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:01.947150    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24f87fa9-0c0b-4d54-8a54-ba1ca6b81b90-kube-api-access-9xqs7" (OuterVolumeSpecName: "kube-api-access-9xqs7") pod "24f87fa9-0c0b-4d54-8a54-ba1ca6b81b90" (UID: "24f87fa9-0c0b-4d54-8a54-ba1ca6b81b90"). InnerVolumeSpecName "kube-api-access-9xqs7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 03:54:01.947234 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:01.947177    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/24f87fa9-0c0b-4d54-8a54-ba1ca6b81b90-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "24f87fa9-0c0b-4d54-8a54-ba1ca6b81b90" (UID: "24f87fa9-0c0b-4d54-8a54-ba1ca6b81b90"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 03:54:01.947428 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:01.947410    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/24f87fa9-0c0b-4d54-8a54-ba1ca6b81b90-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "24f87fa9-0c0b-4d54-8a54-ba1ca6b81b90" (UID: "24f87fa9-0c0b-4d54-8a54-ba1ca6b81b90"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 03:54:02.005978 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:02.005941    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-2c6bf060-gw-openshift-default-684bf7746d-b5czv"]
Sep 11 03:54:02.006176 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:02.006154    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-2c6bf060-gw-openshift-default-684bf7746d-b5czv" podUID="b97d8c86-2972-4312-a5f0-43f69a3ff31a" containerName="istio-proxy" containerID="cri-o://4a40fc960ff02ed5fcf5d221af572204e196122005199a93566c24e6e2d5693d" gracePeriod=30
Sep 11 03:54:02.046023 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:02.045989    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/098240a8-8c72-4966-b7ff-3f30af8587ba-plugins-config-volume\") pod \"098240a8-8c72-4966-b7ff-3f30af8587ba\" (UID: \"098240a8-8c72-4966-b7ff-3f30af8587ba\") "
Sep 11 03:54:02.046213 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:02.046102    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-snvmg\" (UniqueName: \"kubernetes.io/projected/098240a8-8c72-4966-b7ff-3f30af8587ba-kube-api-access-snvmg\") pod \"098240a8-8c72-4966-b7ff-3f30af8587ba\" (UID: \"098240a8-8c72-4966-b7ff-3f30af8587ba\") "
Sep 11 03:54:02.046392 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:02.046373    2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/24f87fa9-0c0b-4d54-8a54-ba1ca6b81b90-metrics-certs\") on node \"ip-10-0-138-121.ec2.internal\" DevicePath \"\""
Sep 11 03:54:02.046452 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:02.046406    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9xqs7\" (UniqueName: \"kubernetes.io/projected/24f87fa9-0c0b-4d54-8a54-ba1ca6b81b90-kube-api-access-9xqs7\") on node \"ip-10-0-138-121.ec2.internal\" DevicePath \"\""
Sep 11 03:54:02.046452 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:02.046420    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/24f87fa9-0c0b-4d54-8a54-ba1ca6b81b90-openshift-service-ca\") on node \"ip-10-0-138-121.ec2.internal\" DevicePath \"\""
Sep 11 03:54:02.046452 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:02.046436    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/24f87fa9-0c0b-4d54-8a54-ba1ca6b81b90-maas-api-tls\") on node \"ip-10-0-138-121.ec2.internal\" DevicePath \"\""
Sep 11 03:54:02.046452 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:02.046369    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/098240a8-8c72-4966-b7ff-3f30af8587ba-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "098240a8-8c72-4966-b7ff-3f30af8587ba" (UID: "098240a8-8c72-4966-b7ff-3f30af8587ba"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 03:54:02.048219 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:02.048202    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/098240a8-8c72-4966-b7ff-3f30af8587ba-kube-api-access-snvmg" (OuterVolumeSpecName: "kube-api-access-snvmg") pod "098240a8-8c72-4966-b7ff-3f30af8587ba" (UID: "098240a8-8c72-4966-b7ff-3f30af8587ba"). InnerVolumeSpecName "kube-api-access-snvmg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 03:54:02.147709 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:02.147598    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-snvmg\" (UniqueName: \"kubernetes.io/projected/098240a8-8c72-4966-b7ff-3f30af8587ba-kube-api-access-snvmg\") on node \"ip-10-0-138-121.ec2.internal\" DevicePath \"\""
Sep 11 03:54:02.147709 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:02.147632    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/098240a8-8c72-4966-b7ff-3f30af8587ba-plugins-config-volume\") on node \"ip-10-0-138-121.ec2.internal\" DevicePath \"\""
Sep 11 03:54:02.279727 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:02.279695    2580 generic.go:356] "Generic (PLEG): container finished" podID="24f87fa9-0c0b-4d54-8a54-ba1ca6b81b90" containerID="a36f0aa53a735a4f0635a303f8d7654a2cdd4b53f70c2fe2e53c6174b94245bf" exitCode=0
Sep 11 03:54:02.279912 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:02.279797    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-2c6bf060-5f7db68584-m6l4j"
Sep 11 03:54:02.279912 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:02.279827    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-2c6bf060-5f7db68584-m6l4j" event={"ID":"24f87fa9-0c0b-4d54-8a54-ba1ca6b81b90","Type":"ContainerDied","Data":"a36f0aa53a735a4f0635a303f8d7654a2cdd4b53f70c2fe2e53c6174b94245bf"}
Sep 11 03:54:02.279912 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:02.279871    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-2c6bf060-5f7db68584-m6l4j" event={"ID":"24f87fa9-0c0b-4d54-8a54-ba1ca6b81b90","Type":"ContainerDied","Data":"c8533c0a423ed751ebffb6a57833f4f2a02ab83186d6c0f58a6e198462f3d881"}
Sep 11 03:54:02.279912 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:02.279893    2580 scope.go:117] "RemoveContainer" containerID="a36f0aa53a735a4f0635a303f8d7654a2cdd4b53f70c2fe2e53c6174b94245bf"
Sep 11 03:54:02.281199 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:02.281174    2580 generic.go:356] "Generic (PLEG): container finished" podID="098240a8-8c72-4966-b7ff-3f30af8587ba" containerID="3820e21fa141ea4534859f8293c25fa4cb6c192de769d47f11784347f260efd2" exitCode=0
Sep 11 03:54:02.281294 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:02.281222    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-2c6bf060-c5c9797d6-pctzf"
Sep 11 03:54:02.281294 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:02.281257    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-2c6bf060-c5c9797d6-pctzf" event={"ID":"098240a8-8c72-4966-b7ff-3f30af8587ba","Type":"ContainerDied","Data":"3820e21fa141ea4534859f8293c25fa4cb6c192de769d47f11784347f260efd2"}
Sep 11 03:54:02.281372 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:02.281295    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-2c6bf060-c5c9797d6-pctzf" event={"ID":"098240a8-8c72-4966-b7ff-3f30af8587ba","Type":"ContainerDied","Data":"32d6d7fa8f1705d07affbe2231b24832a7c2b4b63ad290ddf44660b3fad645f8"}
Sep 11 03:54:02.288697 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:02.288654    2580 scope.go:117] "RemoveContainer" containerID="a36f0aa53a735a4f0635a303f8d7654a2cdd4b53f70c2fe2e53c6174b94245bf"
Sep 11 03:54:02.288962 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:54:02.288938    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a36f0aa53a735a4f0635a303f8d7654a2cdd4b53f70c2fe2e53c6174b94245bf\": container with ID starting with a36f0aa53a735a4f0635a303f8d7654a2cdd4b53f70c2fe2e53c6174b94245bf not found: ID does not exist" containerID="a36f0aa53a735a4f0635a303f8d7654a2cdd4b53f70c2fe2e53c6174b94245bf"
Sep 11 03:54:02.289043 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:02.288975    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a36f0aa53a735a4f0635a303f8d7654a2cdd4b53f70c2fe2e53c6174b94245bf"} err="failed to get container status \"a36f0aa53a735a4f0635a303f8d7654a2cdd4b53f70c2fe2e53c6174b94245bf\": rpc error: code = NotFound desc = could not find container \"a36f0aa53a735a4f0635a303f8d7654a2cdd4b53f70c2fe2e53c6174b94245bf\": container with ID starting with a36f0aa53a735a4f0635a303f8d7654a2cdd4b53f70c2fe2e53c6174b94245bf not found: ID does not exist"
Sep 11 03:54:02.289043 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:02.288999    2580 scope.go:117] "RemoveContainer" containerID="3820e21fa141ea4534859f8293c25fa4cb6c192de769d47f11784347f260efd2"
Sep 11 03:54:02.297378 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:02.297360    2580 scope.go:117] "RemoveContainer" containerID="3820e21fa141ea4534859f8293c25fa4cb6c192de769d47f11784347f260efd2"
Sep 11 03:54:02.297637 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:54:02.297620    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3820e21fa141ea4534859f8293c25fa4cb6c192de769d47f11784347f260efd2\": container with ID starting with 3820e21fa141ea4534859f8293c25fa4cb6c192de769d47f11784347f260efd2 not found: ID does not exist" containerID="3820e21fa141ea4534859f8293c25fa4cb6c192de769d47f11784347f260efd2"
Sep 11 03:54:02.297720 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:02.297644    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3820e21fa141ea4534859f8293c25fa4cb6c192de769d47f11784347f260efd2"} err="failed to get container status \"3820e21fa141ea4534859f8293c25fa4cb6c192de769d47f11784347f260efd2\": rpc error: code = NotFound desc = could not find container \"3820e21fa141ea4534859f8293c25fa4cb6c192de769d47f11784347f260efd2\": container with ID starting with 3820e21fa141ea4534859f8293c25fa4cb6c192de769d47f11784347f260efd2 not found: ID does not exist"
Sep 11 03:54:02.308697 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:02.308648    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-2c6bf060-c5c9797d6-pctzf"]
Sep 11 03:54:02.312474 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:02.312451    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-2c6bf060-c5c9797d6-pctzf"]
Sep 11 03:54:02.325135 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:02.325109    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-2c6bf060-5f7db68584-m6l4j"]
Sep 11 03:54:02.328748 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:02.328725    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-2c6bf060-5f7db68584-m6l4j"]
Sep 11 03:54:02.465093 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:02.465047    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="098240a8-8c72-4966-b7ff-3f30af8587ba" path="/var/lib/kubelet/pods/098240a8-8c72-4966-b7ff-3f30af8587ba/volumes"
Sep 11 03:54:02.465560 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:02.465538    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="24f87fa9-0c0b-4d54-8a54-ba1ca6b81b90" path="/var/lib/kubelet/pods/24f87fa9-0c0b-4d54-8a54-ba1ca6b81b90/volumes"
Sep 11 03:54:02.761885 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:02.760998    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-b65989d0-openshift-default-7bcbd59bb6-78l9k"]
Sep 11 03:54:02.762280 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:02.761935    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="24f87fa9-0c0b-4d54-8a54-ba1ca6b81b90" containerName="maas-api"
Sep 11 03:54:02.762280 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:02.762140    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="24f87fa9-0c0b-4d54-8a54-ba1ca6b81b90" containerName="maas-api"
Sep 11 03:54:02.762384 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:02.762371    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="098240a8-8c72-4966-b7ff-3f30af8587ba" containerName="payload-pre-processing"
Sep 11 03:54:02.762435 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:02.762386    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="098240a8-8c72-4966-b7ff-3f30af8587ba" containerName="payload-pre-processing"
Sep 11 03:54:02.762570 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:02.762554    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="24f87fa9-0c0b-4d54-8a54-ba1ca6b81b90" containerName="maas-api"
Sep 11 03:54:02.762641 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:02.762581    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="098240a8-8c72-4966-b7ff-3f30af8587ba" containerName="payload-pre-processing"
Sep 11 03:54:02.767580 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:02.767552    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b65989d0-openshift-default-7bcbd59bb6-78l9k"
Sep 11 03:54:02.770474 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:02.770445    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-b65989d0-openshift-default-dockercfg-cs24r\""
Sep 11 03:54:02.772767 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:02.772690    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-b65989d0-openshift-default-7bcbd59bb6-78l9k"]
Sep 11 03:54:02.857109 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:02.857066    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f155593-9c15-4514-8c71-aed691e89177-workload-certs\") pod \"e2e-ait-b65989d0-openshift-default-7bcbd59bb6-78l9k\" (UID: \"7f155593-9c15-4514-8c71-aed691e89177\") " pod="openshift-ingress/e2e-ait-b65989d0-openshift-default-7bcbd59bb6-78l9k"
Sep 11 03:54:02.857109 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:02.857111    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f155593-9c15-4514-8c71-aed691e89177-istio-podinfo\") pod \"e2e-ait-b65989d0-openshift-default-7bcbd59bb6-78l9k\" (UID: \"7f155593-9c15-4514-8c71-aed691e89177\") " pod="openshift-ingress/e2e-ait-b65989d0-openshift-default-7bcbd59bb6-78l9k"
Sep 11 03:54:02.857363 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:02.857137    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f155593-9c15-4514-8c71-aed691e89177-istio-token\") pod \"e2e-ait-b65989d0-openshift-default-7bcbd59bb6-78l9k\" (UID: \"7f155593-9c15-4514-8c71-aed691e89177\") " pod="openshift-ingress/e2e-ait-b65989d0-openshift-default-7bcbd59bb6-78l9k"
Sep 11 03:54:02.857363 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:02.857207    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f155593-9c15-4514-8c71-aed691e89177-istio-envoy\") pod \"e2e-ait-b65989d0-openshift-default-7bcbd59bb6-78l9k\" (UID: \"7f155593-9c15-4514-8c71-aed691e89177\") " pod="openshift-ingress/e2e-ait-b65989d0-openshift-default-7bcbd59bb6-78l9k"
Sep 11 03:54:02.857363 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:02.857253    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f155593-9c15-4514-8c71-aed691e89177-credential-socket\") pod \"e2e-ait-b65989d0-openshift-default-7bcbd59bb6-78l9k\" (UID: \"7f155593-9c15-4514-8c71-aed691e89177\") " pod="openshift-ingress/e2e-ait-b65989d0-openshift-default-7bcbd59bb6-78l9k"
Sep 11 03:54:02.857363 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:02.857296    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f155593-9c15-4514-8c71-aed691e89177-istio-data\") pod \"e2e-ait-b65989d0-openshift-default-7bcbd59bb6-78l9k\" (UID: \"7f155593-9c15-4514-8c71-aed691e89177\") " pod="openshift-ingress/e2e-ait-b65989d0-openshift-default-7bcbd59bb6-78l9k"
Sep 11 03:54:02.857363 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:02.857334    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f155593-9c15-4514-8c71-aed691e89177-workload-socket\") pod \"e2e-ait-b65989d0-openshift-default-7bcbd59bb6-78l9k\" (UID: \"7f155593-9c15-4514-8c71-aed691e89177\") " pod="openshift-ingress/e2e-ait-b65989d0-openshift-default-7bcbd59bb6-78l9k"
Sep 11 03:54:02.857363 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:02.857355    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jg9nb\" (UniqueName: \"kubernetes.io/projected/7f155593-9c15-4514-8c71-aed691e89177-kube-api-access-jg9nb\") pod \"e2e-ait-b65989d0-openshift-default-7bcbd59bb6-78l9k\" (UID: \"7f155593-9c15-4514-8c71-aed691e89177\") " pod="openshift-ingress/e2e-ait-b65989d0-openshift-default-7bcbd59bb6-78l9k"
Sep 11 03:54:02.857654 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:02.857383    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f155593-9c15-4514-8c71-aed691e89177-istiod-ca-cert\") pod \"e2e-ait-b65989d0-openshift-default-7bcbd59bb6-78l9k\" (UID: \"7f155593-9c15-4514-8c71-aed691e89177\") " pod="openshift-ingress/e2e-ait-b65989d0-openshift-default-7bcbd59bb6-78l9k"
Sep 11 03:54:02.958905 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:02.958871    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f155593-9c15-4514-8c71-aed691e89177-istiod-ca-cert\") pod \"e2e-ait-b65989d0-openshift-default-7bcbd59bb6-78l9k\" (UID: \"7f155593-9c15-4514-8c71-aed691e89177\") " pod="openshift-ingress/e2e-ait-b65989d0-openshift-default-7bcbd59bb6-78l9k"
Sep 11 03:54:02.959094 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:02.958924    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f155593-9c15-4514-8c71-aed691e89177-workload-certs\") pod \"e2e-ait-b65989d0-openshift-default-7bcbd59bb6-78l9k\" (UID: \"7f155593-9c15-4514-8c71-aed691e89177\") " pod="openshift-ingress/e2e-ait-b65989d0-openshift-default-7bcbd59bb6-78l9k"
Sep 11 03:54:02.959094 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:02.958939    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f155593-9c15-4514-8c71-aed691e89177-istio-podinfo\") pod \"e2e-ait-b65989d0-openshift-default-7bcbd59bb6-78l9k\" (UID: \"7f155593-9c15-4514-8c71-aed691e89177\") " pod="openshift-ingress/e2e-ait-b65989d0-openshift-default-7bcbd59bb6-78l9k"
Sep 11 03:54:02.959094 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:02.959057    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f155593-9c15-4514-8c71-aed691e89177-istio-token\") pod \"e2e-ait-b65989d0-openshift-default-7bcbd59bb6-78l9k\" (UID: \"7f155593-9c15-4514-8c71-aed691e89177\") " pod="openshift-ingress/e2e-ait-b65989d0-openshift-default-7bcbd59bb6-78l9k"
Sep 11 03:54:02.959273 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:02.959126    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f155593-9c15-4514-8c71-aed691e89177-istio-envoy\") pod \"e2e-ait-b65989d0-openshift-default-7bcbd59bb6-78l9k\" (UID: \"7f155593-9c15-4514-8c71-aed691e89177\") " pod="openshift-ingress/e2e-ait-b65989d0-openshift-default-7bcbd59bb6-78l9k"
Sep 11 03:54:02.959273 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:02.959183    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f155593-9c15-4514-8c71-aed691e89177-credential-socket\") pod \"e2e-ait-b65989d0-openshift-default-7bcbd59bb6-78l9k\" (UID: \"7f155593-9c15-4514-8c71-aed691e89177\") " pod="openshift-ingress/e2e-ait-b65989d0-openshift-default-7bcbd59bb6-78l9k"
Sep 11 03:54:02.959376 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:02.959308    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f155593-9c15-4514-8c71-aed691e89177-istio-data\") pod \"e2e-ait-b65989d0-openshift-default-7bcbd59bb6-78l9k\" (UID: \"7f155593-9c15-4514-8c71-aed691e89177\") " pod="openshift-ingress/e2e-ait-b65989d0-openshift-default-7bcbd59bb6-78l9k"
Sep 11 03:54:02.959431 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:02.959382    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f155593-9c15-4514-8c71-aed691e89177-workload-socket\") pod \"e2e-ait-b65989d0-openshift-default-7bcbd59bb6-78l9k\" (UID: \"7f155593-9c15-4514-8c71-aed691e89177\") " pod="openshift-ingress/e2e-ait-b65989d0-openshift-default-7bcbd59bb6-78l9k"
Sep 11 03:54:02.959431 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:02.959422    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jg9nb\" (UniqueName: \"kubernetes.io/projected/7f155593-9c15-4514-8c71-aed691e89177-kube-api-access-jg9nb\") pod \"e2e-ait-b65989d0-openshift-default-7bcbd59bb6-78l9k\" (UID: \"7f155593-9c15-4514-8c71-aed691e89177\") " pod="openshift-ingress/e2e-ait-b65989d0-openshift-default-7bcbd59bb6-78l9k"
Sep 11 03:54:02.959527 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:02.959513    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f155593-9c15-4514-8c71-aed691e89177-credential-socket\") pod \"e2e-ait-b65989d0-openshift-default-7bcbd59bb6-78l9k\" (UID: \"7f155593-9c15-4514-8c71-aed691e89177\") " pod="openshift-ingress/e2e-ait-b65989d0-openshift-default-7bcbd59bb6-78l9k"
Sep 11 03:54:02.959656 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:02.959612    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f155593-9c15-4514-8c71-aed691e89177-istio-data\") pod \"e2e-ait-b65989d0-openshift-default-7bcbd59bb6-78l9k\" (UID: \"7f155593-9c15-4514-8c71-aed691e89177\") " pod="openshift-ingress/e2e-ait-b65989d0-openshift-default-7bcbd59bb6-78l9k"
Sep 11 03:54:02.959840 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:02.959732    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f155593-9c15-4514-8c71-aed691e89177-istiod-ca-cert\") pod \"e2e-ait-b65989d0-openshift-default-7bcbd59bb6-78l9k\" (UID: \"7f155593-9c15-4514-8c71-aed691e89177\") " pod="openshift-ingress/e2e-ait-b65989d0-openshift-default-7bcbd59bb6-78l9k"
Sep 11 03:54:02.959840 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:02.959755    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f155593-9c15-4514-8c71-aed691e89177-workload-certs\") pod \"e2e-ait-b65989d0-openshift-default-7bcbd59bb6-78l9k\" (UID: \"7f155593-9c15-4514-8c71-aed691e89177\") " pod="openshift-ingress/e2e-ait-b65989d0-openshift-default-7bcbd59bb6-78l9k"
Sep 11 03:54:02.959840 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:02.959773    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f155593-9c15-4514-8c71-aed691e89177-workload-socket\") pod \"e2e-ait-b65989d0-openshift-default-7bcbd59bb6-78l9k\" (UID: \"7f155593-9c15-4514-8c71-aed691e89177\") " pod="openshift-ingress/e2e-ait-b65989d0-openshift-default-7bcbd59bb6-78l9k"
Sep 11 03:54:02.961488 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:02.961468    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f155593-9c15-4514-8c71-aed691e89177-istio-envoy\") pod \"e2e-ait-b65989d0-openshift-default-7bcbd59bb6-78l9k\" (UID: \"7f155593-9c15-4514-8c71-aed691e89177\") " pod="openshift-ingress/e2e-ait-b65989d0-openshift-default-7bcbd59bb6-78l9k"
Sep 11 03:54:02.961890 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:02.961871    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f155593-9c15-4514-8c71-aed691e89177-istio-podinfo\") pod \"e2e-ait-b65989d0-openshift-default-7bcbd59bb6-78l9k\" (UID: \"7f155593-9c15-4514-8c71-aed691e89177\") " pod="openshift-ingress/e2e-ait-b65989d0-openshift-default-7bcbd59bb6-78l9k"
Sep 11 03:54:02.967166 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:02.967135    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f155593-9c15-4514-8c71-aed691e89177-istio-token\") pod \"e2e-ait-b65989d0-openshift-default-7bcbd59bb6-78l9k\" (UID: \"7f155593-9c15-4514-8c71-aed691e89177\") " pod="openshift-ingress/e2e-ait-b65989d0-openshift-default-7bcbd59bb6-78l9k"
Sep 11 03:54:02.967337 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:02.967317    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jg9nb\" (UniqueName: \"kubernetes.io/projected/7f155593-9c15-4514-8c71-aed691e89177-kube-api-access-jg9nb\") pod \"e2e-ait-b65989d0-openshift-default-7bcbd59bb6-78l9k\" (UID: \"7f155593-9c15-4514-8c71-aed691e89177\") " pod="openshift-ingress/e2e-ait-b65989d0-openshift-default-7bcbd59bb6-78l9k"
Sep 11 03:54:03.079693 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:03.079562    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b65989d0-openshift-default-7bcbd59bb6-78l9k"
Sep 11 03:54:03.359473 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:03.359445    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-b65989d0-openshift-default-7bcbd59bb6-78l9k"]
Sep 11 03:54:03.361526 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:54:03.361496    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f155593_9c15_4514_8c71_aed691e89177.slice/crio-187e5b11989aaae6d70ce077a73749c0c83755ce673022904e70adad47ad08ad WatchSource:0}: Error finding container 187e5b11989aaae6d70ce077a73749c0c83755ce673022904e70adad47ad08ad: Status 404 returned error can't find the container with id 187e5b11989aaae6d70ce077a73749c0c83755ce673022904e70adad47ad08ad
Sep 11 03:54:03.363780 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:03.363750    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 03:54:03.363857 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:03.363816    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 03:54:03.363857 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:03.363848    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 03:54:04.295007 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:04.294933    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b65989d0-openshift-default-7bcbd59bb6-78l9k" event={"ID":"7f155593-9c15-4514-8c71-aed691e89177","Type":"ContainerStarted","Data":"213aadda92dc57306d32c258c3336abf6dfadf9efde88cc2c68d9132a9f39e3d"}
Sep 11 03:54:04.295007 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:04.295008    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b65989d0-openshift-default-7bcbd59bb6-78l9k" event={"ID":"7f155593-9c15-4514-8c71-aed691e89177","Type":"ContainerStarted","Data":"187e5b11989aaae6d70ce077a73749c0c83755ce673022904e70adad47ad08ad"}
Sep 11 03:54:04.314929 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:04.314878    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-b65989d0-openshift-default-7bcbd59bb6-78l9k" podStartSLOduration=2.314860972 podStartE2EDuration="2.314860972s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 03:54:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:54:04.312924391 +0000 UTC m=+832.471410713" watchObservedRunningTime="2026-09-11 03:54:04.314860972 +0000 UTC m=+832.473347295"
Sep 11 03:54:05.080123 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:05.080087    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-b65989d0-openshift-default-7bcbd59bb6-78l9k"
Sep 11 03:54:05.085111 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:05.085073    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-b65989d0-openshift-default-7bcbd59bb6-78l9k"
Sep 11 03:54:05.299966 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:05.299929    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-b65989d0-openshift-default-7bcbd59bb6-78l9k"
Sep 11 03:54:05.300855 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:05.300832    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-b65989d0-openshift-default-7bcbd59bb6-78l9k"
Sep 11 03:54:07.269131 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:07.269056    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-2c6bf060-gw-openshift-default-684bf7746d-b5czv"
Sep 11 03:54:07.303140 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:07.303103    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b97d8c86-2972-4312-a5f0-43f69a3ff31a-credential-socket\") pod \"b97d8c86-2972-4312-a5f0-43f69a3ff31a\" (UID: \"b97d8c86-2972-4312-a5f0-43f69a3ff31a\") "
Sep 11 03:54:07.303330 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:07.303155    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b97d8c86-2972-4312-a5f0-43f69a3ff31a-istio-token\") pod \"b97d8c86-2972-4312-a5f0-43f69a3ff31a\" (UID: \"b97d8c86-2972-4312-a5f0-43f69a3ff31a\") "
Sep 11 03:54:07.303330 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:07.303210    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b97d8c86-2972-4312-a5f0-43f69a3ff31a-istio-envoy\") pod \"b97d8c86-2972-4312-a5f0-43f69a3ff31a\" (UID: \"b97d8c86-2972-4312-a5f0-43f69a3ff31a\") "
Sep 11 03:54:07.303330 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:07.303302    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b97d8c86-2972-4312-a5f0-43f69a3ff31a-istio-data\") pod \"b97d8c86-2972-4312-a5f0-43f69a3ff31a\" (UID: \"b97d8c86-2972-4312-a5f0-43f69a3ff31a\") "
Sep 11 03:54:07.303330 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:07.303326    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b97d8c86-2972-4312-a5f0-43f69a3ff31a-workload-certs\") pod \"b97d8c86-2972-4312-a5f0-43f69a3ff31a\" (UID: \"b97d8c86-2972-4312-a5f0-43f69a3ff31a\") "
Sep 11 03:54:07.303533 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:07.303350    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b97d8c86-2972-4312-a5f0-43f69a3ff31a-istio-podinfo\") pod \"b97d8c86-2972-4312-a5f0-43f69a3ff31a\" (UID: \"b97d8c86-2972-4312-a5f0-43f69a3ff31a\") "
Sep 11 03:54:07.303533 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:07.303390    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b97d8c86-2972-4312-a5f0-43f69a3ff31a-istiod-ca-cert\") pod \"b97d8c86-2972-4312-a5f0-43f69a3ff31a\" (UID: \"b97d8c86-2972-4312-a5f0-43f69a3ff31a\") "
Sep 11 03:54:07.303533 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:07.303418    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v9kmt\" (UniqueName: \"kubernetes.io/projected/b97d8c86-2972-4312-a5f0-43f69a3ff31a-kube-api-access-v9kmt\") pod \"b97d8c86-2972-4312-a5f0-43f69a3ff31a\" (UID: \"b97d8c86-2972-4312-a5f0-43f69a3ff31a\") "
Sep 11 03:54:07.303533 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:07.303462    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b97d8c86-2972-4312-a5f0-43f69a3ff31a-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "b97d8c86-2972-4312-a5f0-43f69a3ff31a" (UID: "b97d8c86-2972-4312-a5f0-43f69a3ff31a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 03:54:07.303744 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:07.303546    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b97d8c86-2972-4312-a5f0-43f69a3ff31a-workload-socket\") pod \"b97d8c86-2972-4312-a5f0-43f69a3ff31a\" (UID: \"b97d8c86-2972-4312-a5f0-43f69a3ff31a\") "
Sep 11 03:54:07.304039 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:07.303984    2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b97d8c86-2972-4312-a5f0-43f69a3ff31a-credential-socket\") on node \"ip-10-0-138-121.ec2.internal\" DevicePath \"\""
Sep 11 03:54:07.304143 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:07.304085    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b97d8c86-2972-4312-a5f0-43f69a3ff31a-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "b97d8c86-2972-4312-a5f0-43f69a3ff31a" (UID: "b97d8c86-2972-4312-a5f0-43f69a3ff31a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 03:54:07.304247 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:07.304225    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b97d8c86-2972-4312-a5f0-43f69a3ff31a-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "b97d8c86-2972-4312-a5f0-43f69a3ff31a" (UID: "b97d8c86-2972-4312-a5f0-43f69a3ff31a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 03:54:07.304507 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:07.304473    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b97d8c86-2972-4312-a5f0-43f69a3ff31a-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "b97d8c86-2972-4312-a5f0-43f69a3ff31a" (UID: "b97d8c86-2972-4312-a5f0-43f69a3ff31a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 03:54:07.304723 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:07.304691    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b97d8c86-2972-4312-a5f0-43f69a3ff31a-istio-data" (OuterVolumeSpecName: "istio-data") pod "b97d8c86-2972-4312-a5f0-43f69a3ff31a" (UID: "b97d8c86-2972-4312-a5f0-43f69a3ff31a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 03:54:07.306314 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:07.306129    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b97d8c86-2972-4312-a5f0-43f69a3ff31a-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "b97d8c86-2972-4312-a5f0-43f69a3ff31a" (UID: "b97d8c86-2972-4312-a5f0-43f69a3ff31a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 03:54:07.306419 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:07.306377    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b97d8c86-2972-4312-a5f0-43f69a3ff31a-istio-token" (OuterVolumeSpecName: "istio-token") pod "b97d8c86-2972-4312-a5f0-43f69a3ff31a" (UID: "b97d8c86-2972-4312-a5f0-43f69a3ff31a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 03:54:07.306521 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:07.306497    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b97d8c86-2972-4312-a5f0-43f69a3ff31a-kube-api-access-v9kmt" (OuterVolumeSpecName: "kube-api-access-v9kmt") pod "b97d8c86-2972-4312-a5f0-43f69a3ff31a" (UID: "b97d8c86-2972-4312-a5f0-43f69a3ff31a"). InnerVolumeSpecName "kube-api-access-v9kmt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 03:54:07.307853 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:07.307803    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b97d8c86-2972-4312-a5f0-43f69a3ff31a-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "b97d8c86-2972-4312-a5f0-43f69a3ff31a" (UID: "b97d8c86-2972-4312-a5f0-43f69a3ff31a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 03:54:07.314702 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:07.314647    2580 generic.go:356] "Generic (PLEG): container finished" podID="b97d8c86-2972-4312-a5f0-43f69a3ff31a" containerID="4a40fc960ff02ed5fcf5d221af572204e196122005199a93566c24e6e2d5693d" exitCode=0
Sep 11 03:54:07.314798 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:07.314705    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-2c6bf060-gw-openshift-default-684bf7746d-b5czv" event={"ID":"b97d8c86-2972-4312-a5f0-43f69a3ff31a","Type":"ContainerDied","Data":"4a40fc960ff02ed5fcf5d221af572204e196122005199a93566c24e6e2d5693d"}
Sep 11 03:54:07.314855 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:07.314722    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-2c6bf060-gw-openshift-default-684bf7746d-b5czv"
Sep 11 03:54:07.314934 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:07.314914    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-2c6bf060-gw-openshift-default-684bf7746d-b5czv" event={"ID":"b97d8c86-2972-4312-a5f0-43f69a3ff31a","Type":"ContainerDied","Data":"93798222b247963b9955a26a8015b9950ec169bc1b8d5b77e566c159be595643"}
Sep 11 03:54:07.315019 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:07.314961    2580 scope.go:117] "RemoveContainer" containerID="4a40fc960ff02ed5fcf5d221af572204e196122005199a93566c24e6e2d5693d"
Sep 11 03:54:07.330430 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:07.330411    2580 scope.go:117] "RemoveContainer" containerID="8c003f8e21d57da83a9547ad4b48d0ac89755037dc2be85d75c3cc5e91606fc1"
Sep 11 03:54:07.339780 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:07.339739    2580 scope.go:117] "RemoveContainer" containerID="4a40fc960ff02ed5fcf5d221af572204e196122005199a93566c24e6e2d5693d"
Sep 11 03:54:07.340108 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:54:07.340091    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a40fc960ff02ed5fcf5d221af572204e196122005199a93566c24e6e2d5693d\": container with ID starting with 4a40fc960ff02ed5fcf5d221af572204e196122005199a93566c24e6e2d5693d not found: ID does not exist" containerID="4a40fc960ff02ed5fcf5d221af572204e196122005199a93566c24e6e2d5693d"
Sep 11 03:54:07.340164 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:07.340118    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a40fc960ff02ed5fcf5d221af572204e196122005199a93566c24e6e2d5693d"} err="failed to get container status \"4a40fc960ff02ed5fcf5d221af572204e196122005199a93566c24e6e2d5693d\": rpc error: code = NotFound desc = could not find container \"4a40fc960ff02ed5fcf5d221af572204e196122005199a93566c24e6e2d5693d\": container with ID starting with 4a40fc960ff02ed5fcf5d221af572204e196122005199a93566c24e6e2d5693d not found: ID does not exist"
Sep 11 03:54:07.340164 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:07.340136    2580 scope.go:117] "RemoveContainer" containerID="8c003f8e21d57da83a9547ad4b48d0ac89755037dc2be85d75c3cc5e91606fc1"
Sep 11 03:54:07.340416 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:54:07.340402    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c003f8e21d57da83a9547ad4b48d0ac89755037dc2be85d75c3cc5e91606fc1\": container with ID starting with 8c003f8e21d57da83a9547ad4b48d0ac89755037dc2be85d75c3cc5e91606fc1 not found: ID does not exist" containerID="8c003f8e21d57da83a9547ad4b48d0ac89755037dc2be85d75c3cc5e91606fc1"
Sep 11 03:54:07.340473 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:07.340420    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c003f8e21d57da83a9547ad4b48d0ac89755037dc2be85d75c3cc5e91606fc1"} err="failed to get container status \"8c003f8e21d57da83a9547ad4b48d0ac89755037dc2be85d75c3cc5e91606fc1\": rpc error: code = NotFound desc = could not find container \"8c003f8e21d57da83a9547ad4b48d0ac89755037dc2be85d75c3cc5e91606fc1\": container with ID starting with 8c003f8e21d57da83a9547ad4b48d0ac89755037dc2be85d75c3cc5e91606fc1 not found: ID does not exist"
Sep 11 03:54:07.340564 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:07.340545    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-2c6bf060-gw-openshift-default-684bf7746d-b5czv"]
Sep 11 03:54:07.346245 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:07.346216    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-2c6bf060-gw-openshift-default-684bf7746d-b5czv"]
Sep 11 03:54:07.404760 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:07.404717    2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b97d8c86-2972-4312-a5f0-43f69a3ff31a-istio-data\") on node \"ip-10-0-138-121.ec2.internal\" DevicePath \"\""
Sep 11 03:54:07.404760 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:07.404754    2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b97d8c86-2972-4312-a5f0-43f69a3ff31a-workload-certs\") on node \"ip-10-0-138-121.ec2.internal\" DevicePath \"\""
Sep 11 03:54:07.404760 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:07.404766    2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b97d8c86-2972-4312-a5f0-43f69a3ff31a-istio-podinfo\") on node \"ip-10-0-138-121.ec2.internal\" DevicePath \"\""
Sep 11 03:54:07.405000 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:07.404777    2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b97d8c86-2972-4312-a5f0-43f69a3ff31a-istiod-ca-cert\") on node \"ip-10-0-138-121.ec2.internal\" DevicePath \"\""
Sep 11 03:54:07.405000 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:07.404791    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v9kmt\" (UniqueName: \"kubernetes.io/projected/b97d8c86-2972-4312-a5f0-43f69a3ff31a-kube-api-access-v9kmt\") on node \"ip-10-0-138-121.ec2.internal\" DevicePath \"\""
Sep 11 03:54:07.405000 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:07.404804    2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b97d8c86-2972-4312-a5f0-43f69a3ff31a-workload-socket\") on node \"ip-10-0-138-121.ec2.internal\" DevicePath \"\""
Sep 11 03:54:07.405000 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:07.404815    2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b97d8c86-2972-4312-a5f0-43f69a3ff31a-istio-token\") on node \"ip-10-0-138-121.ec2.internal\" DevicePath \"\""
Sep 11 03:54:07.405000 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:07.404826    2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b97d8c86-2972-4312-a5f0-43f69a3ff31a-istio-envoy\") on node \"ip-10-0-138-121.ec2.internal\" DevicePath \"\""
Sep 11 03:54:08.110524 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:08.110489    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b65989d0-6fd77dcc6c-brr2m"]
Sep 11 03:54:08.111006 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:08.110986    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b97d8c86-2972-4312-a5f0-43f69a3ff31a" containerName="istio-proxy"
Sep 11 03:54:08.111112 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:08.111010    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="b97d8c86-2972-4312-a5f0-43f69a3ff31a" containerName="istio-proxy"
Sep 11 03:54:08.111163 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:08.111109    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="b97d8c86-2972-4312-a5f0-43f69a3ff31a" containerName="istio-proxy"
Sep 11 03:54:08.111163 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:08.111121    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="b97d8c86-2972-4312-a5f0-43f69a3ff31a" containerName="istio-proxy"
Sep 11 03:54:08.115853 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:08.115831    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b65989d0-6fd77dcc6c-brr2m"
Sep 11 03:54:08.117967 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:08.117948    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-fcpfs\""
Sep 11 03:54:08.118844 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:08.118825    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-b65989d0-serving-cert\""
Sep 11 03:54:08.118944 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:08.118861    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 03:54:08.118944 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:08.118868    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 11 03:54:08.118944 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:08.118865    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 11 03:54:08.126133 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:08.125192    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b65989d0-6fd77dcc6c-brr2m"]
Sep 11 03:54:08.212684 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:08.212641    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a7915b92-7fbe-46dd-b22e-6edf899ca470-metrics-certs\") pod \"maas-api-e2e-ait-b65989d0-6fd77dcc6c-brr2m\" (UID: \"a7915b92-7fbe-46dd-b22e-6edf899ca470\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b65989d0-6fd77dcc6c-brr2m"
Sep 11 03:54:08.212843 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:08.212796    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7915b92-7fbe-46dd-b22e-6edf899ca470-openshift-service-ca\") pod \"maas-api-e2e-ait-b65989d0-6fd77dcc6c-brr2m\" (UID: \"a7915b92-7fbe-46dd-b22e-6edf899ca470\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b65989d0-6fd77dcc6c-brr2m"
Sep 11 03:54:08.212843 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:08.212839    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mw4f\" (UniqueName: \"kubernetes.io/projected/a7915b92-7fbe-46dd-b22e-6edf899ca470-kube-api-access-4mw4f\") pod \"maas-api-e2e-ait-b65989d0-6fd77dcc6c-brr2m\" (UID: \"a7915b92-7fbe-46dd-b22e-6edf899ca470\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b65989d0-6fd77dcc6c-brr2m"
Sep 11 03:54:08.212918 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:08.212861    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a7915b92-7fbe-46dd-b22e-6edf899ca470-maas-api-tls\") pod \"maas-api-e2e-ait-b65989d0-6fd77dcc6c-brr2m\" (UID: \"a7915b92-7fbe-46dd-b22e-6edf899ca470\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b65989d0-6fd77dcc6c-brr2m"
Sep 11 03:54:08.314204 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:08.314166    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7915b92-7fbe-46dd-b22e-6edf899ca470-openshift-service-ca\") pod \"maas-api-e2e-ait-b65989d0-6fd77dcc6c-brr2m\" (UID: \"a7915b92-7fbe-46dd-b22e-6edf899ca470\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b65989d0-6fd77dcc6c-brr2m"
Sep 11 03:54:08.314571 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:08.314207    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mw4f\" (UniqueName: \"kubernetes.io/projected/a7915b92-7fbe-46dd-b22e-6edf899ca470-kube-api-access-4mw4f\") pod \"maas-api-e2e-ait-b65989d0-6fd77dcc6c-brr2m\" (UID: \"a7915b92-7fbe-46dd-b22e-6edf899ca470\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b65989d0-6fd77dcc6c-brr2m"
Sep 11 03:54:08.314571 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:08.314231    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a7915b92-7fbe-46dd-b22e-6edf899ca470-maas-api-tls\") pod \"maas-api-e2e-ait-b65989d0-6fd77dcc6c-brr2m\" (UID: \"a7915b92-7fbe-46dd-b22e-6edf899ca470\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b65989d0-6fd77dcc6c-brr2m"
Sep 11 03:54:08.314571 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:08.314262    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a7915b92-7fbe-46dd-b22e-6edf899ca470-metrics-certs\") pod \"maas-api-e2e-ait-b65989d0-6fd77dcc6c-brr2m\" (UID: \"a7915b92-7fbe-46dd-b22e-6edf899ca470\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b65989d0-6fd77dcc6c-brr2m"
Sep 11 03:54:08.314571 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:54:08.314340    2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-b65989d0-serving-cert: secret "maas-api-e2e-ait-b65989d0-serving-cert" not found
Sep 11 03:54:08.314571 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:54:08.314414    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a7915b92-7fbe-46dd-b22e-6edf899ca470-maas-api-tls podName:a7915b92-7fbe-46dd-b22e-6edf899ca470 nodeName:}" failed. No retries permitted until 2026-09-11 03:54:08.8143942 +0000 UTC m=+836.972880515 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a7915b92-7fbe-46dd-b22e-6edf899ca470-maas-api-tls") pod "maas-api-e2e-ait-b65989d0-6fd77dcc6c-brr2m" (UID: "a7915b92-7fbe-46dd-b22e-6edf899ca470") : secret "maas-api-e2e-ait-b65989d0-serving-cert" not found
Sep 11 03:54:08.314955 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:08.314933    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7915b92-7fbe-46dd-b22e-6edf899ca470-openshift-service-ca\") pod \"maas-api-e2e-ait-b65989d0-6fd77dcc6c-brr2m\" (UID: \"a7915b92-7fbe-46dd-b22e-6edf899ca470\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b65989d0-6fd77dcc6c-brr2m"
Sep 11 03:54:08.316803 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:08.316780    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a7915b92-7fbe-46dd-b22e-6edf899ca470-metrics-certs\") pod \"maas-api-e2e-ait-b65989d0-6fd77dcc6c-brr2m\" (UID: \"a7915b92-7fbe-46dd-b22e-6edf899ca470\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b65989d0-6fd77dcc6c-brr2m"
Sep 11 03:54:08.324248 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:08.324221    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mw4f\" (UniqueName: \"kubernetes.io/projected/a7915b92-7fbe-46dd-b22e-6edf899ca470-kube-api-access-4mw4f\") pod \"maas-api-e2e-ait-b65989d0-6fd77dcc6c-brr2m\" (UID: \"a7915b92-7fbe-46dd-b22e-6edf899ca470\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b65989d0-6fd77dcc6c-brr2m"
Sep 11 03:54:08.466298 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:08.466262    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b97d8c86-2972-4312-a5f0-43f69a3ff31a" path="/var/lib/kubelet/pods/b97d8c86-2972-4312-a5f0-43f69a3ff31a/volumes"
Sep 11 03:54:08.816178 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:08.816097    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b65989d0-65d7f8f6f8-z7bgh"]
Sep 11 03:54:08.816530 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:08.816517    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b97d8c86-2972-4312-a5f0-43f69a3ff31a" containerName="istio-proxy"
Sep 11 03:54:08.816576 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:08.816532    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="b97d8c86-2972-4312-a5f0-43f69a3ff31a" containerName="istio-proxy"
Sep 11 03:54:08.819709 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:08.819690    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-b65989d0-65d7f8f6f8-z7bgh"
Sep 11 03:54:08.820132 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:08.820109    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a7915b92-7fbe-46dd-b22e-6edf899ca470-maas-api-tls\") pod \"maas-api-e2e-ait-b65989d0-6fd77dcc6c-brr2m\" (UID: \"a7915b92-7fbe-46dd-b22e-6edf899ca470\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b65989d0-6fd77dcc6c-brr2m"
Sep 11 03:54:08.821962 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:08.821939    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-b65989d0\""
Sep 11 03:54:08.822074 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:08.822024    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-b65989d0-dockercfg-2wt5d\""
Sep 11 03:54:08.822614 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:08.822594    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a7915b92-7fbe-46dd-b22e-6edf899ca470-maas-api-tls\") pod \"maas-api-e2e-ait-b65989d0-6fd77dcc6c-brr2m\" (UID: \"a7915b92-7fbe-46dd-b22e-6edf899ca470\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b65989d0-6fd77dcc6c-brr2m"
Sep 11 03:54:08.829299 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:08.829275    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b65989d0-65d7f8f6f8-z7bgh"]
Sep 11 03:54:08.842621 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:08.842594    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b65989d0-6f5f5b8565-wzfkq"]
Sep 11 03:54:08.846095 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:08.846078    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-b65989d0-6f5f5b8565-wzfkq"
Sep 11 03:54:08.855996 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:08.855970    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b65989d0-6f5f5b8565-wzfkq"]
Sep 11 03:54:08.921177 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:08.921139    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1fc8c1c7-3ddb-4d48-94fa-f8cc49ae79b8-plugins-config-volume\") pod \"payload-processing-e2e-ait-b65989d0-6f5f5b8565-wzfkq\" (UID: \"1fc8c1c7-3ddb-4d48-94fa-f8cc49ae79b8\") " pod="openshift-ingress/payload-processing-e2e-ait-b65989d0-6f5f5b8565-wzfkq"
Sep 11 03:54:08.921338 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:08.921255    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51a74db3-49db-4aa3-94e3-3a6e629cf554-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-b65989d0-65d7f8f6f8-z7bgh\" (UID: \"51a74db3-49db-4aa3-94e3-3a6e629cf554\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b65989d0-65d7f8f6f8-z7bgh"
Sep 11 03:54:08.921338 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:08.921313    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tgvvw\" (UniqueName: \"kubernetes.io/projected/51a74db3-49db-4aa3-94e3-3a6e629cf554-kube-api-access-tgvvw\") pod \"payload-pre-processing-e2e-ait-b65989d0-65d7f8f6f8-z7bgh\" (UID: \"51a74db3-49db-4aa3-94e3-3a6e629cf554\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b65989d0-65d7f8f6f8-z7bgh"
Sep 11 03:54:08.921338 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:08.921337    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5d78\" (UniqueName: \"kubernetes.io/projected/1fc8c1c7-3ddb-4d48-94fa-f8cc49ae79b8-kube-api-access-c5d78\") pod \"payload-processing-e2e-ait-b65989d0-6f5f5b8565-wzfkq\" (UID: \"1fc8c1c7-3ddb-4d48-94fa-f8cc49ae79b8\") " pod="openshift-ingress/payload-processing-e2e-ait-b65989d0-6f5f5b8565-wzfkq"
Sep 11 03:54:09.022614 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:09.022568    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1fc8c1c7-3ddb-4d48-94fa-f8cc49ae79b8-plugins-config-volume\") pod \"payload-processing-e2e-ait-b65989d0-6f5f5b8565-wzfkq\" (UID: \"1fc8c1c7-3ddb-4d48-94fa-f8cc49ae79b8\") " pod="openshift-ingress/payload-processing-e2e-ait-b65989d0-6f5f5b8565-wzfkq"
Sep 11 03:54:09.022801 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:09.022704    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51a74db3-49db-4aa3-94e3-3a6e629cf554-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-b65989d0-65d7f8f6f8-z7bgh\" (UID: \"51a74db3-49db-4aa3-94e3-3a6e629cf554\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b65989d0-65d7f8f6f8-z7bgh"
Sep 11 03:54:09.022801 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:09.022781    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgvvw\" (UniqueName: \"kubernetes.io/projected/51a74db3-49db-4aa3-94e3-3a6e629cf554-kube-api-access-tgvvw\") pod \"payload-pre-processing-e2e-ait-b65989d0-65d7f8f6f8-z7bgh\" (UID: \"51a74db3-49db-4aa3-94e3-3a6e629cf554\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b65989d0-65d7f8f6f8-z7bgh"
Sep 11 03:54:09.022926 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:09.022815    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5d78\" (UniqueName: \"kubernetes.io/projected/1fc8c1c7-3ddb-4d48-94fa-f8cc49ae79b8-kube-api-access-c5d78\") pod \"payload-processing-e2e-ait-b65989d0-6f5f5b8565-wzfkq\" (UID: \"1fc8c1c7-3ddb-4d48-94fa-f8cc49ae79b8\") " pod="openshift-ingress/payload-processing-e2e-ait-b65989d0-6f5f5b8565-wzfkq"
Sep 11 03:54:09.023409 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:09.023390    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51a74db3-49db-4aa3-94e3-3a6e629cf554-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-b65989d0-65d7f8f6f8-z7bgh\" (UID: \"51a74db3-49db-4aa3-94e3-3a6e629cf554\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b65989d0-65d7f8f6f8-z7bgh"
Sep 11 03:54:09.023462 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:09.023427    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1fc8c1c7-3ddb-4d48-94fa-f8cc49ae79b8-plugins-config-volume\") pod \"payload-processing-e2e-ait-b65989d0-6f5f5b8565-wzfkq\" (UID: \"1fc8c1c7-3ddb-4d48-94fa-f8cc49ae79b8\") " pod="openshift-ingress/payload-processing-e2e-ait-b65989d0-6f5f5b8565-wzfkq"
Sep 11 03:54:09.029581 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:09.029554    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b65989d0-6fd77dcc6c-brr2m"
Sep 11 03:54:09.030233 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:09.030211    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5d78\" (UniqueName: \"kubernetes.io/projected/1fc8c1c7-3ddb-4d48-94fa-f8cc49ae79b8-kube-api-access-c5d78\") pod \"payload-processing-e2e-ait-b65989d0-6f5f5b8565-wzfkq\" (UID: \"1fc8c1c7-3ddb-4d48-94fa-f8cc49ae79b8\") " pod="openshift-ingress/payload-processing-e2e-ait-b65989d0-6f5f5b8565-wzfkq"
Sep 11 03:54:09.030325 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:09.030261    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgvvw\" (UniqueName: \"kubernetes.io/projected/51a74db3-49db-4aa3-94e3-3a6e629cf554-kube-api-access-tgvvw\") pod \"payload-pre-processing-e2e-ait-b65989d0-65d7f8f6f8-z7bgh\" (UID: \"51a74db3-49db-4aa3-94e3-3a6e629cf554\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b65989d0-65d7f8f6f8-z7bgh"
Sep 11 03:54:09.141968 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:09.141917    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-b65989d0-65d7f8f6f8-z7bgh"
Sep 11 03:54:09.158075 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:09.158048    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-b65989d0-6f5f5b8565-wzfkq"
Sep 11 03:54:09.304804 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:09.304784    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b65989d0-6fd77dcc6c-brr2m"]
Sep 11 03:54:09.307168 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:54:09.307140    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7915b92_7fbe_46dd_b22e_6edf899ca470.slice/crio-4a386eea781c18c58b7ff5e2989c17bf19325a6616128e81e88ac2d9b8532b00 WatchSource:0}: Error finding container 4a386eea781c18c58b7ff5e2989c17bf19325a6616128e81e88ac2d9b8532b00: Status 404 returned error can't find the container with id 4a386eea781c18c58b7ff5e2989c17bf19325a6616128e81e88ac2d9b8532b00
Sep 11 03:54:09.327983 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:09.327930    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b65989d0-6fd77dcc6c-brr2m" event={"ID":"a7915b92-7fbe-46dd-b22e-6edf899ca470","Type":"ContainerStarted","Data":"4a386eea781c18c58b7ff5e2989c17bf19325a6616128e81e88ac2d9b8532b00"}
Sep 11 03:54:09.416924 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:09.416895    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b65989d0-65d7f8f6f8-z7bgh"]
Sep 11 03:54:09.418873 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:54:09.418847    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51a74db3_49db_4aa3_94e3_3a6e629cf554.slice/crio-65cd72232f78fdf2a4bf00df6745b993a019871439f8ee3cbc0474a79ac22322 WatchSource:0}: Error finding container 65cd72232f78fdf2a4bf00df6745b993a019871439f8ee3cbc0474a79ac22322: Status 404 returned error can't find the container with id 65cd72232f78fdf2a4bf00df6745b993a019871439f8ee3cbc0474a79ac22322
Sep 11 03:54:09.446183 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:09.446161    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b65989d0-6f5f5b8565-wzfkq"]
Sep 11 03:54:09.457014 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:54:09.456984    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fc8c1c7_3ddb_4d48_94fa_f8cc49ae79b8.slice/crio-d57bfe5c207dceffbe7f05d294edce73ce898e728d4c7c94a652a4887f79e214 WatchSource:0}: Error finding container d57bfe5c207dceffbe7f05d294edce73ce898e728d4c7c94a652a4887f79e214: Status 404 returned error can't find the container with id d57bfe5c207dceffbe7f05d294edce73ce898e728d4c7c94a652a4887f79e214
Sep 11 03:54:10.335348 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:10.335306    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-b65989d0-65d7f8f6f8-z7bgh" event={"ID":"51a74db3-49db-4aa3-94e3-3a6e629cf554","Type":"ContainerStarted","Data":"c17b27c11ebc0e3572416832f3f46cb458f0bdc48907c5e7526ec2c1b970b589"}
Sep 11 03:54:10.335786 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:10.335407    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-b65989d0-65d7f8f6f8-z7bgh" event={"ID":"51a74db3-49db-4aa3-94e3-3a6e629cf554","Type":"ContainerStarted","Data":"65cd72232f78fdf2a4bf00df6745b993a019871439f8ee3cbc0474a79ac22322"}
Sep 11 03:54:10.335786 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:10.335442    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-b65989d0-65d7f8f6f8-z7bgh"
Sep 11 03:54:10.337079 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:10.337051    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-b65989d0-6f5f5b8565-wzfkq" event={"ID":"1fc8c1c7-3ddb-4d48-94fa-f8cc49ae79b8","Type":"ContainerStarted","Data":"1fd25441abbb0c104fdc68e7cc1d6a1183925a7149f4908884be0820a9d2bf43"}
Sep 11 03:54:10.337212 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:10.337102    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-b65989d0-6f5f5b8565-wzfkq" event={"ID":"1fc8c1c7-3ddb-4d48-94fa-f8cc49ae79b8","Type":"ContainerStarted","Data":"d57bfe5c207dceffbe7f05d294edce73ce898e728d4c7c94a652a4887f79e214"}
Sep 11 03:54:10.337212 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:10.337154    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-b65989d0-6f5f5b8565-wzfkq"
Sep 11 03:54:10.338531 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:10.338506    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b65989d0-6fd77dcc6c-brr2m" event={"ID":"a7915b92-7fbe-46dd-b22e-6edf899ca470","Type":"ContainerStarted","Data":"385ba9cb5a9a5cb3ccac15667d40833675eb5416a025d8260d69a6736555944e"}
Sep 11 03:54:10.338671 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:10.338642    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b65989d0-6fd77dcc6c-brr2m"
Sep 11 03:54:10.352457 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:10.352404    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-b65989d0-65d7f8f6f8-z7bgh" podStartSLOduration=2.35238511 podStartE2EDuration="2.35238511s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 03:54:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:54:10.349336753 +0000 UTC m=+838.507823085" watchObservedRunningTime="2026-09-11 03:54:10.35238511 +0000 UTC m=+838.510871434"
Sep 11 03:54:10.379879 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:10.379818    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-b65989d0-6f5f5b8565-wzfkq" podStartSLOduration=2.379802495 podStartE2EDuration="2.379802495s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 03:54:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:54:10.363140082 +0000 UTC m=+838.521626406" watchObservedRunningTime="2026-09-11 03:54:10.379802495 +0000 UTC m=+838.538288875"
Sep 11 03:54:10.381191 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:10.381159    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b65989d0-6fd77dcc6c-brr2m" podStartSLOduration=1.9611093259999999 podStartE2EDuration="2.381147141s" totalImagesPullingTime="420.037815ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 03:54:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:54:10.378996269 +0000 UTC m=+838.537482593" watchObservedRunningTime="2026-09-11 03:54:10.381147141 +0000 UTC m=+838.539633464"
Sep 11 03:54:16.350785 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:16.350752    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b65989d0-6fd77dcc6c-brr2m"
Sep 11 03:54:21.347516 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:21.347480    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-b65989d0-6f5f5b8565-wzfkq"
Sep 11 03:54:21.347908 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:21.347569    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-b65989d0-65d7f8f6f8-z7bgh"
Sep 11 03:54:28.438001 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:28.437964    2580 generic.go:356] "Generic (PLEG): container finished" podID="7f155593-9c15-4514-8c71-aed691e89177" containerID="213aadda92dc57306d32c258c3336abf6dfadf9efde88cc2c68d9132a9f39e3d" exitCode=0
Sep 11 03:54:28.438508 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:28.438042    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b65989d0-openshift-default-7bcbd59bb6-78l9k" event={"ID":"7f155593-9c15-4514-8c71-aed691e89177","Type":"ContainerDied","Data":"213aadda92dc57306d32c258c3336abf6dfadf9efde88cc2c68d9132a9f39e3d"}
Sep 11 03:54:28.438601 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:28.438584    2580 scope.go:117] "RemoveContainer" containerID="213aadda92dc57306d32c258c3336abf6dfadf9efde88cc2c68d9132a9f39e3d"
Sep 11 03:54:28.439817 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:28.439789    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 03:54:28.439883 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:28.439874    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 03:54:28.439922 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:28.439908    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 03:54:29.445916 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:29.445882    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b65989d0-openshift-default-7bcbd59bb6-78l9k" event={"ID":"7f155593-9c15-4514-8c71-aed691e89177","Type":"ContainerStarted","Data":"a72f86ac39a5d547f711761df557d13d68d385b067ccd544101c711ee7354a7a"}
Sep 11 03:54:30.080674 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:30.080617    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-b65989d0-openshift-default-7bcbd59bb6-78l9k"
Sep 11 03:54:30.080924 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:30.080894    2580 patch_prober.go:28] interesting pod/e2e-ait-b65989d0-openshift-default-7bcbd59bb6-78l9k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body=
Sep 11 03:54:30.081001 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:30.080979    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-b65989d0-openshift-default-7bcbd59bb6-78l9k" podUID="7f155593-9c15-4514-8c71-aed691e89177" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused"
Sep 11 03:54:31.080866 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:31.080826    2580 patch_prober.go:28] interesting pod/e2e-ait-b65989d0-openshift-default-7bcbd59bb6-78l9k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body=
Sep 11 03:54:31.081265 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:31.080884    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-b65989d0-openshift-default-7bcbd59bb6-78l9k" podUID="7f155593-9c15-4514-8c71-aed691e89177" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused"
Sep 11 03:54:32.080405 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:32.080365    2580 patch_prober.go:28] interesting pod/e2e-ait-b65989d0-openshift-default-7bcbd59bb6-78l9k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body=
Sep 11 03:54:32.080583 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:32.080436    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-b65989d0-openshift-default-7bcbd59bb6-78l9k" podUID="7f155593-9c15-4514-8c71-aed691e89177" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused"
Sep 11 03:54:33.080948 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:33.080909    2580 patch_prober.go:28] interesting pod/e2e-ait-b65989d0-openshift-default-7bcbd59bb6-78l9k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body=
Sep 11 03:54:33.081526 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:33.081211    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-b65989d0-openshift-default-7bcbd59bb6-78l9k" podUID="7f155593-9c15-4514-8c71-aed691e89177" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused"
Sep 11 03:54:34.080571 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:34.080530    2580 patch_prober.go:28] interesting pod/e2e-ait-b65989d0-openshift-default-7bcbd59bb6-78l9k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body=
Sep 11 03:54:34.080786 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:34.080591    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-b65989d0-openshift-default-7bcbd59bb6-78l9k" podUID="7f155593-9c15-4514-8c71-aed691e89177" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused"
Sep 11 03:54:35.084532 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:35.084496    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-b65989d0-openshift-default-7bcbd59bb6-78l9k"
Sep 11 03:54:35.084924 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:35.084878    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-b65989d0-openshift-default-7bcbd59bb6-78l9k"
Sep 11 03:54:35.085764 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:35.085734    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-b65989d0-openshift-default-7bcbd59bb6-78l9k"
Sep 11 03:54:37.467975 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:37.467944    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b65989d0-6fd77dcc6c-brr2m"]
Sep 11 03:54:37.468474 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:37.468193    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b65989d0-6fd77dcc6c-brr2m" podUID="a7915b92-7fbe-46dd-b22e-6edf899ca470" containerName="maas-api" containerID="cri-o://385ba9cb5a9a5cb3ccac15667d40833675eb5416a025d8260d69a6736555944e" gracePeriod=30
Sep 11 03:54:37.530164 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:37.530129    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b65989d0-6f5f5b8565-wzfkq"]
Sep 11 03:54:37.530368 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:37.530340    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-b65989d0-6f5f5b8565-wzfkq" podUID="1fc8c1c7-3ddb-4d48-94fa-f8cc49ae79b8" containerName="payload-processing" containerID="cri-o://1fd25441abbb0c104fdc68e7cc1d6a1183925a7149f4908884be0820a9d2bf43" gracePeriod=30
Sep 11 03:54:37.537504 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:37.537282    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b65989d0-65d7f8f6f8-z7bgh"]
Sep 11 03:54:37.537601 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:37.537511    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-b65989d0-65d7f8f6f8-z7bgh" podUID="51a74db3-49db-4aa3-94e3-3a6e629cf554" containerName="payload-pre-processing" containerID="cri-o://c17b27c11ebc0e3572416832f3f46cb458f0bdc48907c5e7526ec2c1b970b589" gracePeriod=30
Sep 11 03:54:37.824383 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:37.824358    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-b65989d0-65d7f8f6f8-z7bgh"
Sep 11 03:54:37.827495 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:37.827468    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b65989d0-6fd77dcc6c-brr2m"
Sep 11 03:54:37.845950 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:37.845923    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-b65989d0-6f5f5b8565-wzfkq"
Sep 11 03:54:37.914349 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:37.914314    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7915b92-7fbe-46dd-b22e-6edf899ca470-openshift-service-ca\") pod \"a7915b92-7fbe-46dd-b22e-6edf899ca470\" (UID: \"a7915b92-7fbe-46dd-b22e-6edf899ca470\") "
Sep 11 03:54:37.914498 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:37.914367    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1fc8c1c7-3ddb-4d48-94fa-f8cc49ae79b8-plugins-config-volume\") pod \"1fc8c1c7-3ddb-4d48-94fa-f8cc49ae79b8\" (UID: \"1fc8c1c7-3ddb-4d48-94fa-f8cc49ae79b8\") "
Sep 11 03:54:37.914498 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:37.914398    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c5d78\" (UniqueName: \"kubernetes.io/projected/1fc8c1c7-3ddb-4d48-94fa-f8cc49ae79b8-kube-api-access-c5d78\") pod \"1fc8c1c7-3ddb-4d48-94fa-f8cc49ae79b8\" (UID: \"1fc8c1c7-3ddb-4d48-94fa-f8cc49ae79b8\") "
Sep 11 03:54:37.914498 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:37.914415    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a7915b92-7fbe-46dd-b22e-6edf899ca470-maas-api-tls\") pod \"a7915b92-7fbe-46dd-b22e-6edf899ca470\" (UID: \"a7915b92-7fbe-46dd-b22e-6edf899ca470\") "
Sep 11 03:54:37.914498 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:37.914468    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a7915b92-7fbe-46dd-b22e-6edf899ca470-metrics-certs\") pod \"a7915b92-7fbe-46dd-b22e-6edf899ca470\" (UID: \"a7915b92-7fbe-46dd-b22e-6edf899ca470\") "
Sep 11 03:54:37.914498 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:37.914494    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tgvvw\" (UniqueName: \"kubernetes.io/projected/51a74db3-49db-4aa3-94e3-3a6e629cf554-kube-api-access-tgvvw\") pod \"51a74db3-49db-4aa3-94e3-3a6e629cf554\" (UID: \"51a74db3-49db-4aa3-94e3-3a6e629cf554\") "
Sep 11 03:54:37.914773 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:37.914538    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4mw4f\" (UniqueName: \"kubernetes.io/projected/a7915b92-7fbe-46dd-b22e-6edf899ca470-kube-api-access-4mw4f\") pod \"a7915b92-7fbe-46dd-b22e-6edf899ca470\" (UID: \"a7915b92-7fbe-46dd-b22e-6edf899ca470\") "
Sep 11 03:54:37.914773 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:37.914581    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51a74db3-49db-4aa3-94e3-3a6e629cf554-plugins-config-volume\") pod \"51a74db3-49db-4aa3-94e3-3a6e629cf554\" (UID: \"51a74db3-49db-4aa3-94e3-3a6e629cf554\") "
Sep 11 03:54:37.914773 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:37.914742    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7915b92-7fbe-46dd-b22e-6edf899ca470-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a7915b92-7fbe-46dd-b22e-6edf899ca470" (UID: "a7915b92-7fbe-46dd-b22e-6edf899ca470"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 03:54:37.914936 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:37.914867    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1fc8c1c7-3ddb-4d48-94fa-f8cc49ae79b8-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1fc8c1c7-3ddb-4d48-94fa-f8cc49ae79b8" (UID: "1fc8c1c7-3ddb-4d48-94fa-f8cc49ae79b8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 03:54:37.915112 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:37.915085    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51a74db3-49db-4aa3-94e3-3a6e629cf554-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "51a74db3-49db-4aa3-94e3-3a6e629cf554" (UID: "51a74db3-49db-4aa3-94e3-3a6e629cf554"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 03:54:37.915179 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:37.915099    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7915b92-7fbe-46dd-b22e-6edf899ca470-openshift-service-ca\") on node \"ip-10-0-138-121.ec2.internal\" DevicePath \"\""
Sep 11 03:54:37.915179 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:37.915139    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1fc8c1c7-3ddb-4d48-94fa-f8cc49ae79b8-plugins-config-volume\") on node \"ip-10-0-138-121.ec2.internal\" DevicePath \"\""
Sep 11 03:54:37.916988 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:37.916962    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fc8c1c7-3ddb-4d48-94fa-f8cc49ae79b8-kube-api-access-c5d78" (OuterVolumeSpecName: "kube-api-access-c5d78") pod "1fc8c1c7-3ddb-4d48-94fa-f8cc49ae79b8" (UID: "1fc8c1c7-3ddb-4d48-94fa-f8cc49ae79b8"). InnerVolumeSpecName "kube-api-access-c5d78". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 03:54:37.917160 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:37.917143    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7915b92-7fbe-46dd-b22e-6edf899ca470-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a7915b92-7fbe-46dd-b22e-6edf899ca470" (UID: "a7915b92-7fbe-46dd-b22e-6edf899ca470"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 03:54:37.917248 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:37.917221    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7915b92-7fbe-46dd-b22e-6edf899ca470-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "a7915b92-7fbe-46dd-b22e-6edf899ca470" (UID: "a7915b92-7fbe-46dd-b22e-6edf899ca470"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 03:54:37.917307 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:37.917269    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51a74db3-49db-4aa3-94e3-3a6e629cf554-kube-api-access-tgvvw" (OuterVolumeSpecName: "kube-api-access-tgvvw") pod "51a74db3-49db-4aa3-94e3-3a6e629cf554" (UID: "51a74db3-49db-4aa3-94e3-3a6e629cf554"). InnerVolumeSpecName "kube-api-access-tgvvw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 03:54:37.917346 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:37.917297    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7915b92-7fbe-46dd-b22e-6edf899ca470-kube-api-access-4mw4f" (OuterVolumeSpecName: "kube-api-access-4mw4f") pod "a7915b92-7fbe-46dd-b22e-6edf899ca470" (UID: "a7915b92-7fbe-46dd-b22e-6edf899ca470"). InnerVolumeSpecName "kube-api-access-4mw4f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 03:54:38.016164 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:38.016080    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c5d78\" (UniqueName: \"kubernetes.io/projected/1fc8c1c7-3ddb-4d48-94fa-f8cc49ae79b8-kube-api-access-c5d78\") on node \"ip-10-0-138-121.ec2.internal\" DevicePath \"\""
Sep 11 03:54:38.016164 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:38.016112    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a7915b92-7fbe-46dd-b22e-6edf899ca470-maas-api-tls\") on node \"ip-10-0-138-121.ec2.internal\" DevicePath \"\""
Sep 11 03:54:38.016164 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:38.016121    2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a7915b92-7fbe-46dd-b22e-6edf899ca470-metrics-certs\") on node \"ip-10-0-138-121.ec2.internal\" DevicePath \"\""
Sep 11 03:54:38.016164 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:38.016129    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tgvvw\" (UniqueName: \"kubernetes.io/projected/51a74db3-49db-4aa3-94e3-3a6e629cf554-kube-api-access-tgvvw\") on node \"ip-10-0-138-121.ec2.internal\" DevicePath \"\""
Sep 11 03:54:38.016164 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:38.016142    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4mw4f\" (UniqueName: \"kubernetes.io/projected/a7915b92-7fbe-46dd-b22e-6edf899ca470-kube-api-access-4mw4f\") on node \"ip-10-0-138-121.ec2.internal\" DevicePath \"\""
Sep 11 03:54:38.016164 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:38.016153    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51a74db3-49db-4aa3-94e3-3a6e629cf554-plugins-config-volume\") on node \"ip-10-0-138-121.ec2.internal\" DevicePath \"\""
Sep 11 03:54:38.533187 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:38.533154    2580 generic.go:356] "Generic (PLEG): container finished" podID="a7915b92-7fbe-46dd-b22e-6edf899ca470" containerID="385ba9cb5a9a5cb3ccac15667d40833675eb5416a025d8260d69a6736555944e" exitCode=0
Sep 11 03:54:38.533851 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:38.533196    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b65989d0-6fd77dcc6c-brr2m"
Sep 11 03:54:38.533851 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:38.533234    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b65989d0-6fd77dcc6c-brr2m" event={"ID":"a7915b92-7fbe-46dd-b22e-6edf899ca470","Type":"ContainerDied","Data":"385ba9cb5a9a5cb3ccac15667d40833675eb5416a025d8260d69a6736555944e"}
Sep 11 03:54:38.533851 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:38.533293    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b65989d0-6fd77dcc6c-brr2m" event={"ID":"a7915b92-7fbe-46dd-b22e-6edf899ca470","Type":"ContainerDied","Data":"4a386eea781c18c58b7ff5e2989c17bf19325a6616128e81e88ac2d9b8532b00"}
Sep 11 03:54:38.533851 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:38.533328    2580 scope.go:117] "RemoveContainer" containerID="385ba9cb5a9a5cb3ccac15667d40833675eb5416a025d8260d69a6736555944e"
Sep 11 03:54:38.535505 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:38.534896    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-b65989d0-65d7f8f6f8-z7bgh"
Sep 11 03:54:38.535505 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:38.534897    2580 generic.go:356] "Generic (PLEG): container finished" podID="51a74db3-49db-4aa3-94e3-3a6e629cf554" containerID="c17b27c11ebc0e3572416832f3f46cb458f0bdc48907c5e7526ec2c1b970b589" exitCode=0
Sep 11 03:54:38.535505 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:38.535017    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-b65989d0-65d7f8f6f8-z7bgh" event={"ID":"51a74db3-49db-4aa3-94e3-3a6e629cf554","Type":"ContainerDied","Data":"c17b27c11ebc0e3572416832f3f46cb458f0bdc48907c5e7526ec2c1b970b589"}
Sep 11 03:54:38.535505 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:38.535069    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-b65989d0-65d7f8f6f8-z7bgh" event={"ID":"51a74db3-49db-4aa3-94e3-3a6e629cf554","Type":"ContainerDied","Data":"65cd72232f78fdf2a4bf00df6745b993a019871439f8ee3cbc0474a79ac22322"}
Sep 11 03:54:38.537219 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:38.537187    2580 generic.go:356] "Generic (PLEG): container finished" podID="1fc8c1c7-3ddb-4d48-94fa-f8cc49ae79b8" containerID="1fd25441abbb0c104fdc68e7cc1d6a1183925a7149f4908884be0820a9d2bf43" exitCode=0
Sep 11 03:54:38.537340 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:38.537229    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-b65989d0-6f5f5b8565-wzfkq" event={"ID":"1fc8c1c7-3ddb-4d48-94fa-f8cc49ae79b8","Type":"ContainerDied","Data":"1fd25441abbb0c104fdc68e7cc1d6a1183925a7149f4908884be0820a9d2bf43"}
Sep 11 03:54:38.537340 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:38.537238    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-b65989d0-6f5f5b8565-wzfkq"
Sep 11 03:54:38.537340 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:38.537261    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-b65989d0-6f5f5b8565-wzfkq" event={"ID":"1fc8c1c7-3ddb-4d48-94fa-f8cc49ae79b8","Type":"ContainerDied","Data":"d57bfe5c207dceffbe7f05d294edce73ce898e728d4c7c94a652a4887f79e214"}
Sep 11 03:54:38.543820 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:38.543801    2580 scope.go:117] "RemoveContainer" containerID="385ba9cb5a9a5cb3ccac15667d40833675eb5416a025d8260d69a6736555944e"
Sep 11 03:54:38.544113 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:54:38.544091    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"385ba9cb5a9a5cb3ccac15667d40833675eb5416a025d8260d69a6736555944e\": container with ID starting with 385ba9cb5a9a5cb3ccac15667d40833675eb5416a025d8260d69a6736555944e not found: ID does not exist" containerID="385ba9cb5a9a5cb3ccac15667d40833675eb5416a025d8260d69a6736555944e"
Sep 11 03:54:38.544194 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:38.544125    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"385ba9cb5a9a5cb3ccac15667d40833675eb5416a025d8260d69a6736555944e"} err="failed to get container status \"385ba9cb5a9a5cb3ccac15667d40833675eb5416a025d8260d69a6736555944e\": rpc error: code = NotFound desc = could not find container \"385ba9cb5a9a5cb3ccac15667d40833675eb5416a025d8260d69a6736555944e\": container with ID starting with 385ba9cb5a9a5cb3ccac15667d40833675eb5416a025d8260d69a6736555944e not found: ID does not exist"
Sep 11 03:54:38.544194 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:38.544147    2580 scope.go:117] "RemoveContainer" containerID="c17b27c11ebc0e3572416832f3f46cb458f0bdc48907c5e7526ec2c1b970b589"
Sep 11 03:54:38.552881 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:38.552855    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b65989d0-6fd77dcc6c-brr2m"]
Sep 11 03:54:38.553915 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:38.553885    2580 scope.go:117] "RemoveContainer" containerID="c17b27c11ebc0e3572416832f3f46cb458f0bdc48907c5e7526ec2c1b970b589"
Sep 11 03:54:38.554246 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:54:38.554223    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c17b27c11ebc0e3572416832f3f46cb458f0bdc48907c5e7526ec2c1b970b589\": container with ID starting with c17b27c11ebc0e3572416832f3f46cb458f0bdc48907c5e7526ec2c1b970b589 not found: ID does not exist" containerID="c17b27c11ebc0e3572416832f3f46cb458f0bdc48907c5e7526ec2c1b970b589"
Sep 11 03:54:38.554302 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:38.554255    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c17b27c11ebc0e3572416832f3f46cb458f0bdc48907c5e7526ec2c1b970b589"} err="failed to get container status \"c17b27c11ebc0e3572416832f3f46cb458f0bdc48907c5e7526ec2c1b970b589\": rpc error: code = NotFound desc = could not find container \"c17b27c11ebc0e3572416832f3f46cb458f0bdc48907c5e7526ec2c1b970b589\": container with ID starting with c17b27c11ebc0e3572416832f3f46cb458f0bdc48907c5e7526ec2c1b970b589 not found: ID does not exist"
Sep 11 03:54:38.554302 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:38.554277    2580 scope.go:117] "RemoveContainer" containerID="1fd25441abbb0c104fdc68e7cc1d6a1183925a7149f4908884be0820a9d2bf43"
Sep 11 03:54:38.562505 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:38.562479    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b65989d0-6fd77dcc6c-brr2m"]
Sep 11 03:54:38.565483 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:38.565465    2580 scope.go:117] "RemoveContainer" containerID="1fd25441abbb0c104fdc68e7cc1d6a1183925a7149f4908884be0820a9d2bf43"
Sep 11 03:54:38.565844 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:54:38.565823    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1fd25441abbb0c104fdc68e7cc1d6a1183925a7149f4908884be0820a9d2bf43\": container with ID starting with 1fd25441abbb0c104fdc68e7cc1d6a1183925a7149f4908884be0820a9d2bf43 not found: ID does not exist" containerID="1fd25441abbb0c104fdc68e7cc1d6a1183925a7149f4908884be0820a9d2bf43"
Sep 11 03:54:38.565915 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:38.565851    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1fd25441abbb0c104fdc68e7cc1d6a1183925a7149f4908884be0820a9d2bf43"} err="failed to get container status \"1fd25441abbb0c104fdc68e7cc1d6a1183925a7149f4908884be0820a9d2bf43\": rpc error: code = NotFound desc = could not find container \"1fd25441abbb0c104fdc68e7cc1d6a1183925a7149f4908884be0820a9d2bf43\": container with ID starting with 1fd25441abbb0c104fdc68e7cc1d6a1183925a7149f4908884be0820a9d2bf43 not found: ID does not exist"
Sep 11 03:54:38.575922 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:38.575891    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b65989d0-65d7f8f6f8-z7bgh"]
Sep 11 03:54:38.580025 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:38.580000    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b65989d0-65d7f8f6f8-z7bgh"]
Sep 11 03:54:38.589599 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:38.589539    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b65989d0-6f5f5b8565-wzfkq"]
Sep 11 03:54:38.591493 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:38.591472    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b65989d0-6f5f5b8565-wzfkq"]
Sep 11 03:54:38.894533 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:38.894442    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-b65989d0-openshift-default-7bcbd59bb6-78l9k"]
Sep 11 03:54:38.894707 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:38.894686    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-b65989d0-openshift-default-7bcbd59bb6-78l9k" podUID="7f155593-9c15-4514-8c71-aed691e89177" containerName="istio-proxy" containerID="cri-o://a72f86ac39a5d547f711761df557d13d68d385b067ccd544101c711ee7354a7a" gracePeriod=30
Sep 11 03:54:39.679103 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:39.679067    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-4ce893dc-openshift-default-59496b9896-zt44l"]
Sep 11 03:54:39.679481 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:39.679468    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51a74db3-49db-4aa3-94e3-3a6e629cf554" containerName="payload-pre-processing"
Sep 11 03:54:39.679481 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:39.679482    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="51a74db3-49db-4aa3-94e3-3a6e629cf554" containerName="payload-pre-processing"
Sep 11 03:54:39.679555 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:39.679495    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7915b92-7fbe-46dd-b22e-6edf899ca470" containerName="maas-api"
Sep 11 03:54:39.679555 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:39.679501    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7915b92-7fbe-46dd-b22e-6edf899ca470" containerName="maas-api"
Sep 11 03:54:39.679555 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:39.679513    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1fc8c1c7-3ddb-4d48-94fa-f8cc49ae79b8" containerName="payload-processing"
Sep 11 03:54:39.679555 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:39.679518    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="1fc8c1c7-3ddb-4d48-94fa-f8cc49ae79b8" containerName="payload-processing"
Sep 11 03:54:39.679709 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:39.679582    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="a7915b92-7fbe-46dd-b22e-6edf899ca470" containerName="maas-api"
Sep 11 03:54:39.679709 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:39.679590    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="51a74db3-49db-4aa3-94e3-3a6e629cf554" containerName="payload-pre-processing"
Sep 11 03:54:39.679709 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:39.679597    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="1fc8c1c7-3ddb-4d48-94fa-f8cc49ae79b8" containerName="payload-processing"
Sep 11 03:54:39.682645 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:39.682624    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-4ce893dc-openshift-default-59496b9896-zt44l"
Sep 11 03:54:39.684918 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:39.684894    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-4ce893dc-openshift-default-dockercfg-9fdsk\""
Sep 11 03:54:39.698823 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:39.698270    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-4ce893dc-openshift-default-59496b9896-zt44l"]
Sep 11 03:54:39.734188 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:39.734148    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07e19967-b8ed-4adc-a511-2860068a4cba-workload-certs\") pod \"e2e-derive-4ce893dc-openshift-default-59496b9896-zt44l\" (UID: \"07e19967-b8ed-4adc-a511-2860068a4cba\") " pod="openshift-ingress/e2e-derive-4ce893dc-openshift-default-59496b9896-zt44l"
Sep 11 03:54:39.734188 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:39.734186    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07e19967-b8ed-4adc-a511-2860068a4cba-istio-token\") pod \"e2e-derive-4ce893dc-openshift-default-59496b9896-zt44l\" (UID: \"07e19967-b8ed-4adc-a511-2860068a4cba\") " pod="openshift-ingress/e2e-derive-4ce893dc-openshift-default-59496b9896-zt44l"
Sep 11 03:54:39.734390 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:39.734202    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07e19967-b8ed-4adc-a511-2860068a4cba-istiod-ca-cert\") pod \"e2e-derive-4ce893dc-openshift-default-59496b9896-zt44l\" (UID: \"07e19967-b8ed-4adc-a511-2860068a4cba\") " pod="openshift-ingress/e2e-derive-4ce893dc-openshift-default-59496b9896-zt44l"
Sep 11 03:54:39.734390 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:39.734218    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07e19967-b8ed-4adc-a511-2860068a4cba-istio-data\") pod \"e2e-derive-4ce893dc-openshift-default-59496b9896-zt44l\" (UID: \"07e19967-b8ed-4adc-a511-2860068a4cba\") " pod="openshift-ingress/e2e-derive-4ce893dc-openshift-default-59496b9896-zt44l"
Sep 11 03:54:39.734390 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:39.734302    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07e19967-b8ed-4adc-a511-2860068a4cba-credential-socket\") pod \"e2e-derive-4ce893dc-openshift-default-59496b9896-zt44l\" (UID: \"07e19967-b8ed-4adc-a511-2860068a4cba\") " pod="openshift-ingress/e2e-derive-4ce893dc-openshift-default-59496b9896-zt44l"
Sep 11 03:54:39.734390 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:39.734354    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07e19967-b8ed-4adc-a511-2860068a4cba-istio-envoy\") pod \"e2e-derive-4ce893dc-openshift-default-59496b9896-zt44l\" (UID: \"07e19967-b8ed-4adc-a511-2860068a4cba\") " pod="openshift-ingress/e2e-derive-4ce893dc-openshift-default-59496b9896-zt44l"
Sep 11 03:54:39.734390 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:39.734372    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9289j\" (UniqueName: \"kubernetes.io/projected/07e19967-b8ed-4adc-a511-2860068a4cba-kube-api-access-9289j\") pod \"e2e-derive-4ce893dc-openshift-default-59496b9896-zt44l\" (UID: \"07e19967-b8ed-4adc-a511-2860068a4cba\") " pod="openshift-ingress/e2e-derive-4ce893dc-openshift-default-59496b9896-zt44l"
Sep 11 03:54:39.734545 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:39.734405    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07e19967-b8ed-4adc-a511-2860068a4cba-istio-podinfo\") pod \"e2e-derive-4ce893dc-openshift-default-59496b9896-zt44l\" (UID: \"07e19967-b8ed-4adc-a511-2860068a4cba\") " pod="openshift-ingress/e2e-derive-4ce893dc-openshift-default-59496b9896-zt44l"
Sep 11 03:54:39.734545 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:39.734441    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07e19967-b8ed-4adc-a511-2860068a4cba-workload-socket\") pod \"e2e-derive-4ce893dc-openshift-default-59496b9896-zt44l\" (UID: \"07e19967-b8ed-4adc-a511-2860068a4cba\") " pod="openshift-ingress/e2e-derive-4ce893dc-openshift-default-59496b9896-zt44l"
Sep 11 03:54:39.835797 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:39.835749    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07e19967-b8ed-4adc-a511-2860068a4cba-istio-podinfo\") pod \"e2e-derive-4ce893dc-openshift-default-59496b9896-zt44l\" (UID: \"07e19967-b8ed-4adc-a511-2860068a4cba\") " pod="openshift-ingress/e2e-derive-4ce893dc-openshift-default-59496b9896-zt44l"
Sep 11 03:54:39.835972 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:39.835821    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07e19967-b8ed-4adc-a511-2860068a4cba-workload-socket\") pod \"e2e-derive-4ce893dc-openshift-default-59496b9896-zt44l\" (UID: \"07e19967-b8ed-4adc-a511-2860068a4cba\") " pod="openshift-ingress/e2e-derive-4ce893dc-openshift-default-59496b9896-zt44l"
Sep 11 03:54:39.835972 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:39.835881    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07e19967-b8ed-4adc-a511-2860068a4cba-workload-certs\") pod \"e2e-derive-4ce893dc-openshift-default-59496b9896-zt44l\" (UID: \"07e19967-b8ed-4adc-a511-2860068a4cba\") " pod="openshift-ingress/e2e-derive-4ce893dc-openshift-default-59496b9896-zt44l"
Sep 11 03:54:39.835972 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:39.835903    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07e19967-b8ed-4adc-a511-2860068a4cba-istio-token\") pod \"e2e-derive-4ce893dc-openshift-default-59496b9896-zt44l\" (UID: \"07e19967-b8ed-4adc-a511-2860068a4cba\") " pod="openshift-ingress/e2e-derive-4ce893dc-openshift-default-59496b9896-zt44l"
Sep 11 03:54:39.835972 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:39.835926    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07e19967-b8ed-4adc-a511-2860068a4cba-istiod-ca-cert\") pod \"e2e-derive-4ce893dc-openshift-default-59496b9896-zt44l\" (UID: \"07e19967-b8ed-4adc-a511-2860068a4cba\") " pod="openshift-ingress/e2e-derive-4ce893dc-openshift-default-59496b9896-zt44l"
Sep 11 03:54:39.835972 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:39.835949    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07e19967-b8ed-4adc-a511-2860068a4cba-istio-data\") pod \"e2e-derive-4ce893dc-openshift-default-59496b9896-zt44l\" (UID: \"07e19967-b8ed-4adc-a511-2860068a4cba\") " pod="openshift-ingress/e2e-derive-4ce893dc-openshift-default-59496b9896-zt44l"
Sep 11 03:54:39.836262 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:39.835991    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07e19967-b8ed-4adc-a511-2860068a4cba-credential-socket\") pod \"e2e-derive-4ce893dc-openshift-default-59496b9896-zt44l\" (UID: \"07e19967-b8ed-4adc-a511-2860068a4cba\") " pod="openshift-ingress/e2e-derive-4ce893dc-openshift-default-59496b9896-zt44l"
Sep 11 03:54:39.836262 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:39.836048    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07e19967-b8ed-4adc-a511-2860068a4cba-istio-envoy\") pod \"e2e-derive-4ce893dc-openshift-default-59496b9896-zt44l\" (UID: \"07e19967-b8ed-4adc-a511-2860068a4cba\") " pod="openshift-ingress/e2e-derive-4ce893dc-openshift-default-59496b9896-zt44l"
Sep 11 03:54:39.836262 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:39.836073    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9289j\" (UniqueName: \"kubernetes.io/projected/07e19967-b8ed-4adc-a511-2860068a4cba-kube-api-access-9289j\") pod \"e2e-derive-4ce893dc-openshift-default-59496b9896-zt44l\" (UID: \"07e19967-b8ed-4adc-a511-2860068a4cba\") " pod="openshift-ingress/e2e-derive-4ce893dc-openshift-default-59496b9896-zt44l"
Sep 11 03:54:39.836408 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:39.836301    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07e19967-b8ed-4adc-a511-2860068a4cba-workload-socket\") pod \"e2e-derive-4ce893dc-openshift-default-59496b9896-zt44l\" (UID: \"07e19967-b8ed-4adc-a511-2860068a4cba\") " pod="openshift-ingress/e2e-derive-4ce893dc-openshift-default-59496b9896-zt44l"
Sep 11 03:54:39.836408 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:39.836339    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07e19967-b8ed-4adc-a511-2860068a4cba-istio-data\") pod \"e2e-derive-4ce893dc-openshift-default-59496b9896-zt44l\" (UID: \"07e19967-b8ed-4adc-a511-2860068a4cba\") " pod="openshift-ingress/e2e-derive-4ce893dc-openshift-default-59496b9896-zt44l"
Sep 11 03:54:39.836479 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:39.836424    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07e19967-b8ed-4adc-a511-2860068a4cba-workload-certs\") pod \"e2e-derive-4ce893dc-openshift-default-59496b9896-zt44l\" (UID: \"07e19967-b8ed-4adc-a511-2860068a4cba\") " pod="openshift-ingress/e2e-derive-4ce893dc-openshift-default-59496b9896-zt44l"
Sep 11 03:54:39.836587 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:39.836566    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07e19967-b8ed-4adc-a511-2860068a4cba-credential-socket\") pod \"e2e-derive-4ce893dc-openshift-default-59496b9896-zt44l\" (UID: \"07e19967-b8ed-4adc-a511-2860068a4cba\") " pod="openshift-ingress/e2e-derive-4ce893dc-openshift-default-59496b9896-zt44l"
Sep 11 03:54:39.836713 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:39.836692    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07e19967-b8ed-4adc-a511-2860068a4cba-istiod-ca-cert\") pod \"e2e-derive-4ce893dc-openshift-default-59496b9896-zt44l\" (UID: \"07e19967-b8ed-4adc-a511-2860068a4cba\") " pod="openshift-ingress/e2e-derive-4ce893dc-openshift-default-59496b9896-zt44l"
Sep 11 03:54:39.838225 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:39.838205    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07e19967-b8ed-4adc-a511-2860068a4cba-istio-envoy\") pod \"e2e-derive-4ce893dc-openshift-default-59496b9896-zt44l\" (UID: \"07e19967-b8ed-4adc-a511-2860068a4cba\") " pod="openshift-ingress/e2e-derive-4ce893dc-openshift-default-59496b9896-zt44l"
Sep 11 03:54:39.838390 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:39.838370    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07e19967-b8ed-4adc-a511-2860068a4cba-istio-podinfo\") pod \"e2e-derive-4ce893dc-openshift-default-59496b9896-zt44l\" (UID: \"07e19967-b8ed-4adc-a511-2860068a4cba\") " pod="openshift-ingress/e2e-derive-4ce893dc-openshift-default-59496b9896-zt44l"
Sep 11 03:54:39.844325 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:39.844300    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07e19967-b8ed-4adc-a511-2860068a4cba-istio-token\") pod \"e2e-derive-4ce893dc-openshift-default-59496b9896-zt44l\" (UID: \"07e19967-b8ed-4adc-a511-2860068a4cba\") " pod="openshift-ingress/e2e-derive-4ce893dc-openshift-default-59496b9896-zt44l"
Sep 11 03:54:39.844493 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:39.844469    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9289j\" (UniqueName: \"kubernetes.io/projected/07e19967-b8ed-4adc-a511-2860068a4cba-kube-api-access-9289j\") pod \"e2e-derive-4ce893dc-openshift-default-59496b9896-zt44l\" (UID: \"07e19967-b8ed-4adc-a511-2860068a4cba\") " pod="openshift-ingress/e2e-derive-4ce893dc-openshift-default-59496b9896-zt44l"
Sep 11 03:54:39.992594 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:39.992496    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-4ce893dc-openshift-default-59496b9896-zt44l"
Sep 11 03:54:40.294236 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:40.294141    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-4ce893dc-openshift-default-59496b9896-zt44l"]
Sep 11 03:54:40.297270 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:54:40.297240    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07e19967_b8ed_4adc_a511_2860068a4cba.slice/crio-0a5f0dc2d053c302a7bfae60c3dca971f47ceeede1e49ea741deebc03a72d70a WatchSource:0}: Error finding container 0a5f0dc2d053c302a7bfae60c3dca971f47ceeede1e49ea741deebc03a72d70a: Status 404 returned error can't find the container with id 0a5f0dc2d053c302a7bfae60c3dca971f47ceeede1e49ea741deebc03a72d70a
Sep 11 03:54:40.299781 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:40.299743    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 03:54:40.299864 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:40.299816    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 03:54:40.299864 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:40.299852    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 03:54:40.467594 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:40.467553    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1fc8c1c7-3ddb-4d48-94fa-f8cc49ae79b8" path="/var/lib/kubelet/pods/1fc8c1c7-3ddb-4d48-94fa-f8cc49ae79b8/volumes"
Sep 11 03:54:40.468100 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:40.468083    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="51a74db3-49db-4aa3-94e3-3a6e629cf554" path="/var/lib/kubelet/pods/51a74db3-49db-4aa3-94e3-3a6e629cf554/volumes"
Sep 11 03:54:40.468587 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:40.468573    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a7915b92-7fbe-46dd-b22e-6edf899ca470" path="/var/lib/kubelet/pods/a7915b92-7fbe-46dd-b22e-6edf899ca470/volumes"
Sep 11 03:54:40.552810 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:40.552648    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-4ce893dc-openshift-default-59496b9896-zt44l" event={"ID":"07e19967-b8ed-4adc-a511-2860068a4cba","Type":"ContainerStarted","Data":"f67845faedd703089715f952ea463568dd8709613243ba2f566415ed4ea8acd2"}
Sep 11 03:54:40.552810 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:40.552739    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-4ce893dc-openshift-default-59496b9896-zt44l" event={"ID":"07e19967-b8ed-4adc-a511-2860068a4cba","Type":"ContainerStarted","Data":"0a5f0dc2d053c302a7bfae60c3dca971f47ceeede1e49ea741deebc03a72d70a"}
Sep 11 03:54:40.570815 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:40.570755    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-4ce893dc-openshift-default-59496b9896-zt44l" podStartSLOduration=1.570733454 podStartE2EDuration="1.570733454s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 03:54:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:54:40.570244418 +0000 UTC m=+868.728730741" watchObservedRunningTime="2026-09-11 03:54:40.570733454 +0000 UTC m=+868.729219777"
Sep 11 03:54:40.993630 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:40.993589    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-4ce893dc-openshift-default-59496b9896-zt44l"
Sep 11 03:54:41.999276 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:41.999246    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-4ce893dc-openshift-default-59496b9896-zt44l"
Sep 11 03:54:42.563708 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:42.563676    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-4ce893dc-openshift-default-59496b9896-zt44l"
Sep 11 03:54:42.564729 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:42.564710    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-4ce893dc-openshift-default-59496b9896-zt44l"
Sep 11 03:54:44.144001 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:44.143977    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b65989d0-openshift-default-7bcbd59bb6-78l9k"
Sep 11 03:54:44.179981 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:44.179899    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f155593-9c15-4514-8c71-aed691e89177-istiod-ca-cert\") pod \"7f155593-9c15-4514-8c71-aed691e89177\" (UID: \"7f155593-9c15-4514-8c71-aed691e89177\") "
Sep 11 03:54:44.179981 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:44.179953    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f155593-9c15-4514-8c71-aed691e89177-workload-certs\") pod \"7f155593-9c15-4514-8c71-aed691e89177\" (UID: \"7f155593-9c15-4514-8c71-aed691e89177\") "
Sep 11 03:54:44.180213 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:44.179991    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f155593-9c15-4514-8c71-aed691e89177-istio-podinfo\") pod \"7f155593-9c15-4514-8c71-aed691e89177\" (UID: \"7f155593-9c15-4514-8c71-aed691e89177\") "
Sep 11 03:54:44.180213 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:44.180047    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f155593-9c15-4514-8c71-aed691e89177-istio-token\") pod \"7f155593-9c15-4514-8c71-aed691e89177\" (UID: \"7f155593-9c15-4514-8c71-aed691e89177\") "
Sep 11 03:54:44.180213 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:44.180078    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f155593-9c15-4514-8c71-aed691e89177-workload-socket\") pod \"7f155593-9c15-4514-8c71-aed691e89177\" (UID: \"7f155593-9c15-4514-8c71-aed691e89177\") "
Sep 11 03:54:44.180213 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:44.180118    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f155593-9c15-4514-8c71-aed691e89177-credential-socket\") pod \"7f155593-9c15-4514-8c71-aed691e89177\" (UID: \"7f155593-9c15-4514-8c71-aed691e89177\") "
Sep 11 03:54:44.180213 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:44.180144    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f155593-9c15-4514-8c71-aed691e89177-istio-data\") pod \"7f155593-9c15-4514-8c71-aed691e89177\" (UID: \"7f155593-9c15-4514-8c71-aed691e89177\") "
Sep 11 03:54:44.180213 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:44.180183    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jg9nb\" (UniqueName: \"kubernetes.io/projected/7f155593-9c15-4514-8c71-aed691e89177-kube-api-access-jg9nb\") pod \"7f155593-9c15-4514-8c71-aed691e89177\" (UID: \"7f155593-9c15-4514-8c71-aed691e89177\") "
Sep 11 03:54:44.180518 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:44.180238    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f155593-9c15-4514-8c71-aed691e89177-istio-envoy\") pod \"7f155593-9c15-4514-8c71-aed691e89177\" (UID: \"7f155593-9c15-4514-8c71-aed691e89177\") "
Sep 11 03:54:44.180518 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:44.180290    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f155593-9c15-4514-8c71-aed691e89177-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "7f155593-9c15-4514-8c71-aed691e89177" (UID: "7f155593-9c15-4514-8c71-aed691e89177"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 03:54:44.180518 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:44.180309    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f155593-9c15-4514-8c71-aed691e89177-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "7f155593-9c15-4514-8c71-aed691e89177" (UID: "7f155593-9c15-4514-8c71-aed691e89177"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 03:54:44.180727 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:44.180519    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f155593-9c15-4514-8c71-aed691e89177-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "7f155593-9c15-4514-8c71-aed691e89177" (UID: "7f155593-9c15-4514-8c71-aed691e89177"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 03:54:44.180727 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:44.180590    2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f155593-9c15-4514-8c71-aed691e89177-credential-socket\") on node \"ip-10-0-138-121.ec2.internal\" DevicePath \"\""
Sep 11 03:54:44.180727 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:44.180610    2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f155593-9c15-4514-8c71-aed691e89177-istiod-ca-cert\") on node \"ip-10-0-138-121.ec2.internal\" DevicePath \"\""
Sep 11 03:54:44.180727 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:44.180626    2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f155593-9c15-4514-8c71-aed691e89177-workload-certs\") on node \"ip-10-0-138-121.ec2.internal\" DevicePath \"\""
Sep 11 03:54:44.180948 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:44.180726    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f155593-9c15-4514-8c71-aed691e89177-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "7f155593-9c15-4514-8c71-aed691e89177" (UID: "7f155593-9c15-4514-8c71-aed691e89177"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 03:54:44.180948 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:44.180912    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f155593-9c15-4514-8c71-aed691e89177-istio-data" (OuterVolumeSpecName: "istio-data") pod "7f155593-9c15-4514-8c71-aed691e89177" (UID: "7f155593-9c15-4514-8c71-aed691e89177"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 03:54:44.182605 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:44.182577    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f155593-9c15-4514-8c71-aed691e89177-kube-api-access-jg9nb" (OuterVolumeSpecName: "kube-api-access-jg9nb") pod "7f155593-9c15-4514-8c71-aed691e89177" (UID: "7f155593-9c15-4514-8c71-aed691e89177"). InnerVolumeSpecName "kube-api-access-jg9nb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 03:54:44.182789 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:44.182769    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7f155593-9c15-4514-8c71-aed691e89177-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "7f155593-9c15-4514-8c71-aed691e89177" (UID: "7f155593-9c15-4514-8c71-aed691e89177"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 03:54:44.182905 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:44.182885    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f155593-9c15-4514-8c71-aed691e89177-istio-token" (OuterVolumeSpecName: "istio-token") pod "7f155593-9c15-4514-8c71-aed691e89177" (UID: "7f155593-9c15-4514-8c71-aed691e89177"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 03:54:44.182952 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:44.182929    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f155593-9c15-4514-8c71-aed691e89177-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "7f155593-9c15-4514-8c71-aed691e89177" (UID: "7f155593-9c15-4514-8c71-aed691e89177"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 03:54:44.288228 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:44.281876    2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f155593-9c15-4514-8c71-aed691e89177-istio-podinfo\") on node \"ip-10-0-138-121.ec2.internal\" DevicePath \"\""
Sep 11 03:54:44.288228 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:44.288123    2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f155593-9c15-4514-8c71-aed691e89177-istio-token\") on node \"ip-10-0-138-121.ec2.internal\" DevicePath \"\""
Sep 11 03:54:44.288228 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:44.288141    2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f155593-9c15-4514-8c71-aed691e89177-workload-socket\") on node \"ip-10-0-138-121.ec2.internal\" DevicePath \"\""
Sep 11 03:54:44.288228 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:44.288158    2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f155593-9c15-4514-8c71-aed691e89177-istio-data\") on node \"ip-10-0-138-121.ec2.internal\" DevicePath \"\""
Sep 11 03:54:44.288228 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:44.288180    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jg9nb\" (UniqueName: \"kubernetes.io/projected/7f155593-9c15-4514-8c71-aed691e89177-kube-api-access-jg9nb\") on node \"ip-10-0-138-121.ec2.internal\" DevicePath \"\""
Sep 11 03:54:44.288228 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:44.288194    2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f155593-9c15-4514-8c71-aed691e89177-istio-envoy\") on node \"ip-10-0-138-121.ec2.internal\" DevicePath \"\""
Sep 11 03:54:44.493947 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:44.493913    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-4ce893dc-55496949d8-66znc"]
Sep 11 03:54:44.494414 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:44.494374    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f155593-9c15-4514-8c71-aed691e89177" containerName="istio-proxy"
Sep 11 03:54:44.494414 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:44.494394    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f155593-9c15-4514-8c71-aed691e89177" containerName="istio-proxy"
Sep 11 03:54:44.494542 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:44.494498    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="7f155593-9c15-4514-8c71-aed691e89177" containerName="istio-proxy"
Sep 11 03:54:44.494542 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:44.494514    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="7f155593-9c15-4514-8c71-aed691e89177" containerName="istio-proxy"
Sep 11 03:54:44.497886 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:44.497864    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4ce893dc-55496949d8-66znc"
Sep 11 03:54:44.500495 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:44.500171    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 11 03:54:44.500495 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:44.500185    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-fcpfs\""
Sep 11 03:54:44.500495 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:44.500222    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-4ce893dc-serving-cert\""
Sep 11 03:54:44.500495 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:44.500238    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 11 03:54:44.500495 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:44.500185    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 03:54:44.506774 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:44.506724    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-4ce893dc-55496949d8-66znc"]
Sep 11 03:54:44.576242 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:44.576206    2580 generic.go:356] "Generic (PLEG): container finished" podID="7f155593-9c15-4514-8c71-aed691e89177" containerID="a72f86ac39a5d547f711761df557d13d68d385b067ccd544101c711ee7354a7a" exitCode=0
Sep 11 03:54:44.576380 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:44.576277    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b65989d0-openshift-default-7bcbd59bb6-78l9k"
Sep 11 03:54:44.576380 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:44.576297    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b65989d0-openshift-default-7bcbd59bb6-78l9k" event={"ID":"7f155593-9c15-4514-8c71-aed691e89177","Type":"ContainerDied","Data":"a72f86ac39a5d547f711761df557d13d68d385b067ccd544101c711ee7354a7a"}
Sep 11 03:54:44.576380 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:44.576358    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b65989d0-openshift-default-7bcbd59bb6-78l9k" event={"ID":"7f155593-9c15-4514-8c71-aed691e89177","Type":"ContainerDied","Data":"187e5b11989aaae6d70ce077a73749c0c83755ce673022904e70adad47ad08ad"}
Sep 11 03:54:44.576512 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:44.576387    2580 scope.go:117] "RemoveContainer" containerID="a72f86ac39a5d547f711761df557d13d68d385b067ccd544101c711ee7354a7a"
Sep 11 03:54:44.585359 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:44.585330    2580 scope.go:117] "RemoveContainer" containerID="213aadda92dc57306d32c258c3336abf6dfadf9efde88cc2c68d9132a9f39e3d"
Sep 11 03:54:44.590509 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:44.590486    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e8324da1-e47d-4d1f-8307-c39ceb847d8f-maas-api-tls\") pod \"maas-api-e2e-derive-4ce893dc-55496949d8-66znc\" (UID: \"e8324da1-e47d-4d1f-8307-c39ceb847d8f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-4ce893dc-55496949d8-66znc"
Sep 11 03:54:44.590623 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:44.590529    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8324da1-e47d-4d1f-8307-c39ceb847d8f-openshift-service-ca\") pod \"maas-api-e2e-derive-4ce893dc-55496949d8-66znc\" (UID: \"e8324da1-e47d-4d1f-8307-c39ceb847d8f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-4ce893dc-55496949d8-66znc"
Sep 11 03:54:44.590703 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:44.590637    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmw4s\" (UniqueName: \"kubernetes.io/projected/e8324da1-e47d-4d1f-8307-c39ceb847d8f-kube-api-access-lmw4s\") pod \"maas-api-e2e-derive-4ce893dc-55496949d8-66znc\" (UID: \"e8324da1-e47d-4d1f-8307-c39ceb847d8f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-4ce893dc-55496949d8-66znc"
Sep 11 03:54:44.590756 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:44.590704    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8324da1-e47d-4d1f-8307-c39ceb847d8f-metrics-certs\") pod \"maas-api-e2e-derive-4ce893dc-55496949d8-66znc\" (UID: \"e8324da1-e47d-4d1f-8307-c39ceb847d8f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-4ce893dc-55496949d8-66znc"
Sep 11 03:54:44.593124 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:44.593100    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-b65989d0-openshift-default-7bcbd59bb6-78l9k"]
Sep 11 03:54:44.594506 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:44.594491    2580 scope.go:117] "RemoveContainer" containerID="a72f86ac39a5d547f711761df557d13d68d385b067ccd544101c711ee7354a7a"
Sep 11 03:54:44.594884 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:54:44.594867    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a72f86ac39a5d547f711761df557d13d68d385b067ccd544101c711ee7354a7a\": container with ID starting with a72f86ac39a5d547f711761df557d13d68d385b067ccd544101c711ee7354a7a not found: ID does not exist" containerID="a72f86ac39a5d547f711761df557d13d68d385b067ccd544101c711ee7354a7a"
Sep 11 03:54:44.594943 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:44.594893    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a72f86ac39a5d547f711761df557d13d68d385b067ccd544101c711ee7354a7a"} err="failed to get container status \"a72f86ac39a5d547f711761df557d13d68d385b067ccd544101c711ee7354a7a\": rpc error: code = NotFound desc = could not find container \"a72f86ac39a5d547f711761df557d13d68d385b067ccd544101c711ee7354a7a\": container with ID starting with a72f86ac39a5d547f711761df557d13d68d385b067ccd544101c711ee7354a7a not found: ID does not exist"
Sep 11 03:54:44.594943 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:44.594911    2580 scope.go:117] "RemoveContainer" containerID="213aadda92dc57306d32c258c3336abf6dfadf9efde88cc2c68d9132a9f39e3d"
Sep 11 03:54:44.595144 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:54:44.595129    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"213aadda92dc57306d32c258c3336abf6dfadf9efde88cc2c68d9132a9f39e3d\": container with ID starting with 213aadda92dc57306d32c258c3336abf6dfadf9efde88cc2c68d9132a9f39e3d not found: ID does not exist" containerID="213aadda92dc57306d32c258c3336abf6dfadf9efde88cc2c68d9132a9f39e3d"
Sep 11 03:54:44.595183 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:44.595148    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"213aadda92dc57306d32c258c3336abf6dfadf9efde88cc2c68d9132a9f39e3d"} err="failed to get container status \"213aadda92dc57306d32c258c3336abf6dfadf9efde88cc2c68d9132a9f39e3d\": rpc error: code = NotFound desc = could not find container \"213aadda92dc57306d32c258c3336abf6dfadf9efde88cc2c68d9132a9f39e3d\": container with ID starting with 213aadda92dc57306d32c258c3336abf6dfadf9efde88cc2c68d9132a9f39e3d not found: ID does not exist"
Sep 11 03:54:44.599134 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:44.599084    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-b65989d0-openshift-default-7bcbd59bb6-78l9k"]
Sep 11 03:54:44.692160 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:44.692117    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e8324da1-e47d-4d1f-8307-c39ceb847d8f-maas-api-tls\") pod \"maas-api-e2e-derive-4ce893dc-55496949d8-66znc\" (UID: \"e8324da1-e47d-4d1f-8307-c39ceb847d8f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-4ce893dc-55496949d8-66znc"
Sep 11 03:54:44.692160 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:44.692165    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8324da1-e47d-4d1f-8307-c39ceb847d8f-openshift-service-ca\") pod \"maas-api-e2e-derive-4ce893dc-55496949d8-66znc\" (UID: \"e8324da1-e47d-4d1f-8307-c39ceb847d8f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-4ce893dc-55496949d8-66znc"
Sep 11 03:54:44.692434 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:44.692214    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmw4s\" (UniqueName: \"kubernetes.io/projected/e8324da1-e47d-4d1f-8307-c39ceb847d8f-kube-api-access-lmw4s\") pod \"maas-api-e2e-derive-4ce893dc-55496949d8-66znc\" (UID: \"e8324da1-e47d-4d1f-8307-c39ceb847d8f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-4ce893dc-55496949d8-66znc"
Sep 11 03:54:44.692434 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:54:44.692277    2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-4ce893dc-serving-cert: secret "maas-api-e2e-derive-4ce893dc-serving-cert" not found
Sep 11 03:54:44.692434 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:54:44.692346    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e8324da1-e47d-4d1f-8307-c39ceb847d8f-maas-api-tls podName:e8324da1-e47d-4d1f-8307-c39ceb847d8f nodeName:}" failed. No retries permitted until 2026-09-11 03:54:45.19232373 +0000 UTC m=+873.350810052 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e8324da1-e47d-4d1f-8307-c39ceb847d8f-maas-api-tls") pod "maas-api-e2e-derive-4ce893dc-55496949d8-66znc" (UID: "e8324da1-e47d-4d1f-8307-c39ceb847d8f") : secret "maas-api-e2e-derive-4ce893dc-serving-cert" not found
Sep 11 03:54:44.692434 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:44.692273    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8324da1-e47d-4d1f-8307-c39ceb847d8f-metrics-certs\") pod \"maas-api-e2e-derive-4ce893dc-55496949d8-66znc\" (UID: \"e8324da1-e47d-4d1f-8307-c39ceb847d8f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-4ce893dc-55496949d8-66znc"
Sep 11 03:54:44.693024 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:44.693001    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8324da1-e47d-4d1f-8307-c39ceb847d8f-openshift-service-ca\") pod \"maas-api-e2e-derive-4ce893dc-55496949d8-66znc\" (UID: \"e8324da1-e47d-4d1f-8307-c39ceb847d8f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-4ce893dc-55496949d8-66znc"
Sep 11 03:54:44.694700 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:44.694654    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8324da1-e47d-4d1f-8307-c39ceb847d8f-metrics-certs\") pod \"maas-api-e2e-derive-4ce893dc-55496949d8-66znc\" (UID: \"e8324da1-e47d-4d1f-8307-c39ceb847d8f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-4ce893dc-55496949d8-66znc"
Sep 11 03:54:44.700055 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:44.700036    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmw4s\" (UniqueName: \"kubernetes.io/projected/e8324da1-e47d-4d1f-8307-c39ceb847d8f-kube-api-access-lmw4s\") pod \"maas-api-e2e-derive-4ce893dc-55496949d8-66znc\" (UID: \"e8324da1-e47d-4d1f-8307-c39ceb847d8f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-4ce893dc-55496949d8-66znc"
Sep 11 03:54:45.060994 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:45.060960    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-4ce893dc-5cb6566566-vwnql"]
Sep 11 03:54:45.061409 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:45.061395    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f155593-9c15-4514-8c71-aed691e89177" containerName="istio-proxy"
Sep 11 03:54:45.061457 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:45.061411    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f155593-9c15-4514-8c71-aed691e89177" containerName="istio-proxy"
Sep 11 03:54:45.065825 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:45.065803    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-4ce893dc-5cb6566566-vwnql"
Sep 11 03:54:45.067940 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:45.067918    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-4ce893dc-dockercfg-xvtk7\""
Sep 11 03:54:45.068039 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:45.068003    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-4ce893dc\""
Sep 11 03:54:45.074211 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:45.074186    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-4ce893dc-5cb6566566-vwnql"]
Sep 11 03:54:45.095012 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:45.094982    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-4ce893dc-5458888db5-q24xq"]
Sep 11 03:54:45.098352 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:45.098334    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-4ce893dc-5458888db5-q24xq"
Sep 11 03:54:45.107539 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:45.107512    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-4ce893dc-5458888db5-q24xq"]
Sep 11 03:54:45.198474 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:45.198438    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bvzt\" (UniqueName: \"kubernetes.io/projected/c1a33cdd-52f8-47ed-adbf-3bbf0017f616-kube-api-access-6bvzt\") pod \"payload-pre-processing-e2e-derive-4ce893dc-5cb6566566-vwnql\" (UID: \"c1a33cdd-52f8-47ed-adbf-3bbf0017f616\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-4ce893dc-5cb6566566-vwnql"
Sep 11 03:54:45.198928 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:45.198506    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e8324da1-e47d-4d1f-8307-c39ceb847d8f-maas-api-tls\") pod \"maas-api-e2e-derive-4ce893dc-55496949d8-66znc\" (UID: \"e8324da1-e47d-4d1f-8307-c39ceb847d8f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-4ce893dc-55496949d8-66znc"
Sep 11 03:54:45.198928 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:45.198541    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1a33cdd-52f8-47ed-adbf-3bbf0017f616-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-4ce893dc-5cb6566566-vwnql\" (UID: \"c1a33cdd-52f8-47ed-adbf-3bbf0017f616\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-4ce893dc-5cb6566566-vwnql"
Sep 11 03:54:45.198928 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:45.198629    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7ghv\" (UniqueName: \"kubernetes.io/projected/473f4335-1974-4765-a1ac-da185e3ffe37-kube-api-access-v7ghv\") pod \"payload-processing-e2e-derive-4ce893dc-5458888db5-q24xq\" (UID: \"473f4335-1974-4765-a1ac-da185e3ffe37\") " pod="openshift-ingress/payload-processing-e2e-derive-4ce893dc-5458888db5-q24xq"
Sep 11 03:54:45.198928 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:45.198754    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/473f4335-1974-4765-a1ac-da185e3ffe37-plugins-config-volume\") pod \"payload-processing-e2e-derive-4ce893dc-5458888db5-q24xq\" (UID: \"473f4335-1974-4765-a1ac-da185e3ffe37\") " pod="openshift-ingress/payload-processing-e2e-derive-4ce893dc-5458888db5-q24xq"
Sep 11 03:54:45.200843 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:45.200826    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e8324da1-e47d-4d1f-8307-c39ceb847d8f-maas-api-tls\") pod \"maas-api-e2e-derive-4ce893dc-55496949d8-66znc\" (UID: \"e8324da1-e47d-4d1f-8307-c39ceb847d8f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-4ce893dc-55496949d8-66znc"
Sep 11 03:54:45.299575 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:45.299535    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7ghv\" (UniqueName: \"kubernetes.io/projected/473f4335-1974-4765-a1ac-da185e3ffe37-kube-api-access-v7ghv\") pod \"payload-processing-e2e-derive-4ce893dc-5458888db5-q24xq\" (UID: \"473f4335-1974-4765-a1ac-da185e3ffe37\") " pod="openshift-ingress/payload-processing-e2e-derive-4ce893dc-5458888db5-q24xq"
Sep 11 03:54:45.299801 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:45.299607    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/473f4335-1974-4765-a1ac-da185e3ffe37-plugins-config-volume\") pod \"payload-processing-e2e-derive-4ce893dc-5458888db5-q24xq\" (UID: \"473f4335-1974-4765-a1ac-da185e3ffe37\") " pod="openshift-ingress/payload-processing-e2e-derive-4ce893dc-5458888db5-q24xq"
Sep 11 03:54:45.299801 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:45.299637    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bvzt\" (UniqueName: \"kubernetes.io/projected/c1a33cdd-52f8-47ed-adbf-3bbf0017f616-kube-api-access-6bvzt\") pod \"payload-pre-processing-e2e-derive-4ce893dc-5cb6566566-vwnql\" (UID: \"c1a33cdd-52f8-47ed-adbf-3bbf0017f616\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-4ce893dc-5cb6566566-vwnql"
Sep 11 03:54:45.299801 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:45.299704    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1a33cdd-52f8-47ed-adbf-3bbf0017f616-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-4ce893dc-5cb6566566-vwnql\" (UID: \"c1a33cdd-52f8-47ed-adbf-3bbf0017f616\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-4ce893dc-5cb6566566-vwnql"
Sep 11 03:54:45.300346 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:45.300325    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/473f4335-1974-4765-a1ac-da185e3ffe37-plugins-config-volume\") pod \"payload-processing-e2e-derive-4ce893dc-5458888db5-q24xq\" (UID: \"473f4335-1974-4765-a1ac-da185e3ffe37\") " pod="openshift-ingress/payload-processing-e2e-derive-4ce893dc-5458888db5-q24xq"
Sep 11 03:54:45.300418 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:45.300334    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1a33cdd-52f8-47ed-adbf-3bbf0017f616-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-4ce893dc-5cb6566566-vwnql\" (UID: \"c1a33cdd-52f8-47ed-adbf-3bbf0017f616\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-4ce893dc-5cb6566566-vwnql"
Sep 11 03:54:45.307034 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:45.307009    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bvzt\" (UniqueName: \"kubernetes.io/projected/c1a33cdd-52f8-47ed-adbf-3bbf0017f616-kube-api-access-6bvzt\") pod \"payload-pre-processing-e2e-derive-4ce893dc-5cb6566566-vwnql\" (UID: \"c1a33cdd-52f8-47ed-adbf-3bbf0017f616\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-4ce893dc-5cb6566566-vwnql"
Sep 11 03:54:45.307129 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:45.307051    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7ghv\" (UniqueName: \"kubernetes.io/projected/473f4335-1974-4765-a1ac-da185e3ffe37-kube-api-access-v7ghv\") pod \"payload-processing-e2e-derive-4ce893dc-5458888db5-q24xq\" (UID: \"473f4335-1974-4765-a1ac-da185e3ffe37\") " pod="openshift-ingress/payload-processing-e2e-derive-4ce893dc-5458888db5-q24xq"
Sep 11 03:54:45.377158 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:45.377048    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-4ce893dc-5cb6566566-vwnql"
Sep 11 03:54:45.410056 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:45.410022    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-4ce893dc-5458888db5-q24xq"
Sep 11 03:54:45.411192 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:45.411156    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4ce893dc-55496949d8-66znc"
Sep 11 03:54:45.648650 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:45.648624    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-4ce893dc-5cb6566566-vwnql"]
Sep 11 03:54:45.650785 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:54:45.650754    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1a33cdd_52f8_47ed_adbf_3bbf0017f616.slice/crio-b40b0058de6e8b654c236086ce9fa09cb677a7eef71019f00fbb324ea82044a0 WatchSource:0}: Error finding container b40b0058de6e8b654c236086ce9fa09cb677a7eef71019f00fbb324ea82044a0: Status 404 returned error can't find the container with id b40b0058de6e8b654c236086ce9fa09cb677a7eef71019f00fbb324ea82044a0
Sep 11 03:54:45.693733 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:45.693707    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-4ce893dc-5458888db5-q24xq"]
Sep 11 03:54:45.695451 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:54:45.695422    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod473f4335_1974_4765_a1ac_da185e3ffe37.slice/crio-4ab57dd4c26b6b4369a1df092525c4702034b96e07cd993825fe801746248278 WatchSource:0}: Error finding container 4ab57dd4c26b6b4369a1df092525c4702034b96e07cd993825fe801746248278: Status 404 returned error can't find the container with id 4ab57dd4c26b6b4369a1df092525c4702034b96e07cd993825fe801746248278
Sep 11 03:54:45.711175 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:45.711090    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-4ce893dc-55496949d8-66znc"]
Sep 11 03:54:45.716644 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:54:45.716617    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8324da1_e47d_4d1f_8307_c39ceb847d8f.slice/crio-d51ddd7bf8d802b6d7ee9936192f532fc2f3898e7b1b4300bcef97a4b819aa1d WatchSource:0}: Error finding container d51ddd7bf8d802b6d7ee9936192f532fc2f3898e7b1b4300bcef97a4b819aa1d: Status 404 returned error can't find the container with id d51ddd7bf8d802b6d7ee9936192f532fc2f3898e7b1b4300bcef97a4b819aa1d
Sep 11 03:54:46.466036 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:46.465955    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7f155593-9c15-4514-8c71-aed691e89177" path="/var/lib/kubelet/pods/7f155593-9c15-4514-8c71-aed691e89177/volumes"
Sep 11 03:54:46.589908 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:46.589875    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4ce893dc-55496949d8-66znc" event={"ID":"e8324da1-e47d-4d1f-8307-c39ceb847d8f","Type":"ContainerStarted","Data":"0973d94198f2c8522bdb830788aa14960dd09e9a4e436d3aed76d3f04cf2a95b"}
Sep 11 03:54:46.590118 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:46.589923    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4ce893dc-55496949d8-66znc" event={"ID":"e8324da1-e47d-4d1f-8307-c39ceb847d8f","Type":"ContainerStarted","Data":"d51ddd7bf8d802b6d7ee9936192f532fc2f3898e7b1b4300bcef97a4b819aa1d"}
Sep 11 03:54:46.590118 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:46.589964    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4ce893dc-55496949d8-66znc"
Sep 11 03:54:46.591489 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:46.591461    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-4ce893dc-5458888db5-q24xq" event={"ID":"473f4335-1974-4765-a1ac-da185e3ffe37","Type":"ContainerStarted","Data":"1cdd7450feaf9ab5929bae4fa763b326086f364523b7dddef6acaf14024e87ca"}
Sep 11 03:54:46.591622 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:46.591511    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-4ce893dc-5458888db5-q24xq" event={"ID":"473f4335-1974-4765-a1ac-da185e3ffe37","Type":"ContainerStarted","Data":"4ab57dd4c26b6b4369a1df092525c4702034b96e07cd993825fe801746248278"}
Sep 11 03:54:46.591622 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:46.591538    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-4ce893dc-5458888db5-q24xq"
Sep 11 03:54:46.593214 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:46.593195    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-4ce893dc-5cb6566566-vwnql" event={"ID":"c1a33cdd-52f8-47ed-adbf-3bbf0017f616","Type":"ContainerStarted","Data":"da2b715fd91c838c02345fb163b2efaa1d57a0d2a8b7023f4b5816899420082b"}
Sep 11 03:54:46.593311 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:46.593234    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-4ce893dc-5cb6566566-vwnql" event={"ID":"c1a33cdd-52f8-47ed-adbf-3bbf0017f616","Type":"ContainerStarted","Data":"b40b0058de6e8b654c236086ce9fa09cb677a7eef71019f00fbb324ea82044a0"}
Sep 11 03:54:46.593311 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:46.593272    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-4ce893dc-5cb6566566-vwnql"
Sep 11 03:54:46.605208 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:46.605168    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4ce893dc-55496949d8-66znc" podStartSLOduration=2.148099086 podStartE2EDuration="2.605153169s" totalImagesPullingTime="457.054083ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 03:54:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:54:46.603752001 +0000 UTC m=+874.762238329" watchObservedRunningTime="2026-09-11 03:54:46.605153169 +0000 UTC m=+874.763639492"
Sep 11 03:54:46.618743 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:46.618641    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-4ce893dc-5cb6566566-vwnql" podStartSLOduration=1.6185990220000002 podStartE2EDuration="1.618599022s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 03:54:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:54:46.616181371 +0000 UTC m=+874.774667722" watchObservedRunningTime="2026-09-11 03:54:46.618599022 +0000 UTC m=+874.777085346"
Sep 11 03:54:46.630691 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:46.630631    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-4ce893dc-5458888db5-q24xq" podStartSLOduration=1.630615422 podStartE2EDuration="1.630615422s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 03:54:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:54:46.630119724 +0000 UTC m=+874.788606048" watchObservedRunningTime="2026-09-11 03:54:46.630615422 +0000 UTC m=+874.789101744"
Sep 11 03:54:52.604687 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:52.604645    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4ce893dc-55496949d8-66znc"
Sep 11 03:54:57.602935 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:57.602893    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-4ce893dc-5cb6566566-vwnql"
Sep 11 03:54:57.603391 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:57.603102    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-4ce893dc-5458888db5-q24xq"
Sep 11 03:54:58.605921 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:58.605852    2580 patch_prober.go:28] interesting pod/e2e-derive-4ce893dc-openshift-default-59496b9896-zt44l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 11 03:54:58.606450 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:58.605952    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-4ce893dc-openshift-default-59496b9896-zt44l" podUID="07e19967-b8ed-4adc-a511-2860068a4cba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 11 03:54:59.663356 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:59.663324    2580 generic.go:356] "Generic (PLEG): container finished" podID="07e19967-b8ed-4adc-a511-2860068a4cba" containerID="f67845faedd703089715f952ea463568dd8709613243ba2f566415ed4ea8acd2" exitCode=0
Sep 11 03:54:59.663773 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:59.663404    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-4ce893dc-openshift-default-59496b9896-zt44l" event={"ID":"07e19967-b8ed-4adc-a511-2860068a4cba","Type":"ContainerDied","Data":"f67845faedd703089715f952ea463568dd8709613243ba2f566415ed4ea8acd2"}
Sep 11 03:54:59.663990 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:59.663974    2580 scope.go:117] "RemoveContainer" containerID="f67845faedd703089715f952ea463568dd8709613243ba2f566415ed4ea8acd2"
Sep 11 03:54:59.665286 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:59.665259    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 03:54:59.665363 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:59.665323    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 03:54:59.665363 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:59.665355    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 03:54:59.978519 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:54:59.978433    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-a-08754c30-kserve-6d9bddf55-p4kjr"]
Sep 11 03:55:00.670233 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:00.670197    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-4ce893dc-openshift-default-59496b9896-zt44l" event={"ID":"07e19967-b8ed-4adc-a511-2860068a4cba","Type":"ContainerStarted","Data":"2b67cc6fae2fb32190e053b73d26b7f1032e9b5d0155215556c28af0ee3034c9"}
Sep 11 03:55:00.993389 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:00.993294    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-4ce893dc-openshift-default-59496b9896-zt44l"
Sep 11 03:55:00.993541 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:00.993513    2580 patch_prober.go:28] interesting pod/e2e-derive-4ce893dc-openshift-default-59496b9896-zt44l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body=
Sep 11 03:55:00.993582 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:00.993563    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-4ce893dc-openshift-default-59496b9896-zt44l" podUID="07e19967-b8ed-4adc-a511-2860068a4cba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused"
Sep 11 03:55:01.993136 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:01.993100    2580 patch_prober.go:28] interesting pod/e2e-derive-4ce893dc-openshift-default-59496b9896-zt44l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body=
Sep 11 03:55:01.993551 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:01.993157    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-4ce893dc-openshift-default-59496b9896-zt44l" podUID="07e19967-b8ed-4adc-a511-2860068a4cba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused"
Sep 11 03:55:02.993679 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:02.993637    2580 patch_prober.go:28] interesting pod/e2e-derive-4ce893dc-openshift-default-59496b9896-zt44l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body=
Sep 11 03:55:02.994152 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:02.993720    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-4ce893dc-openshift-default-59496b9896-zt44l" podUID="07e19967-b8ed-4adc-a511-2860068a4cba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused"
Sep 11 03:55:03.993727 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:03.993691    2580 patch_prober.go:28] interesting pod/e2e-derive-4ce893dc-openshift-default-59496b9896-zt44l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body=
Sep 11 03:55:03.994206 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:03.993748    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-4ce893dc-openshift-default-59496b9896-zt44l" podUID="07e19967-b8ed-4adc-a511-2860068a4cba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused"
Sep 11 03:55:04.993993 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:04.993908    2580 patch_prober.go:28] interesting pod/e2e-derive-4ce893dc-openshift-default-59496b9896-zt44l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body=
Sep 11 03:55:04.994355 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:04.993986    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-4ce893dc-openshift-default-59496b9896-zt44l" podUID="07e19967-b8ed-4adc-a511-2860068a4cba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused"
Sep 11 03:55:05.993570 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:05.993531    2580 patch_prober.go:28] interesting pod/e2e-derive-4ce893dc-openshift-default-59496b9896-zt44l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body=
Sep 11 03:55:05.993787 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:05.993601    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-4ce893dc-openshift-default-59496b9896-zt44l" podUID="07e19967-b8ed-4adc-a511-2860068a4cba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused"
Sep 11 03:55:06.997157 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:06.997121    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-4ce893dc-openshift-default-59496b9896-zt44l"
Sep 11 03:55:06.997551 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:06.997408    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-4ce893dc-openshift-default-59496b9896-zt44l"
Sep 11 03:55:06.998082 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:06.998060    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-4ce893dc-openshift-default-59496b9896-zt44l"
Sep 11 03:55:12.284689 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:12.284476    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-4ce893dc-55496949d8-66znc"]
Sep 11 03:55:12.285164 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:12.284781    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4ce893dc-55496949d8-66znc" podUID="e8324da1-e47d-4d1f-8307-c39ceb847d8f" containerName="maas-api" containerID="cri-o://0973d94198f2c8522bdb830788aa14960dd09e9a4e436d3aed76d3f04cf2a95b" gracePeriod=30
Sep 11 03:55:12.336110 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:12.336075    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-4ce893dc-5458888db5-q24xq"]
Sep 11 03:55:12.336316 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:12.336289    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-4ce893dc-5458888db5-q24xq" podUID="473f4335-1974-4765-a1ac-da185e3ffe37" containerName="payload-processing" containerID="cri-o://1cdd7450feaf9ab5929bae4fa763b326086f364523b7dddef6acaf14024e87ca" gracePeriod=30
Sep 11 03:55:12.341201 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:12.341175    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-4ce893dc-5cb6566566-vwnql"]
Sep 11 03:55:12.341587 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:12.341560    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-4ce893dc-5cb6566566-vwnql" podUID="c1a33cdd-52f8-47ed-adbf-3bbf0017f616" containerName="payload-pre-processing" containerID="cri-o://da2b715fd91c838c02345fb163b2efaa1d57a0d2a8b7023f4b5816899420082b" gracePeriod=30
Sep 11 03:55:12.365612 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:12.365580    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j4pgw_28b30dae-a2d3-4251-99e8-04bf5153015b/ovn-acl-logging/0.log"
Sep 11 03:55:12.366020 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:12.365992    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j4pgw_28b30dae-a2d3-4251-99e8-04bf5153015b/ovn-acl-logging/0.log"
Sep 11 03:55:12.569911 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:12.569446    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4ce893dc-55496949d8-66znc"
Sep 11 03:55:12.648387 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:12.648337    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-4ce893dc-5cb6566566-vwnql"
Sep 11 03:55:12.651646 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:12.651618    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-4ce893dc-openshift-default-59496b9896-zt44l"]
Sep 11 03:55:12.652019 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:12.651988    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-4ce893dc-openshift-default-59496b9896-zt44l" podUID="07e19967-b8ed-4adc-a511-2860068a4cba" containerName="istio-proxy" containerID="cri-o://2b67cc6fae2fb32190e053b73d26b7f1032e9b5d0155215556c28af0ee3034c9" gracePeriod=30
Sep 11 03:55:12.664307 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:12.664223    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-4ce893dc-5458888db5-q24xq"
Sep 11 03:55:12.685449 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:12.685405    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8324da1-e47d-4d1f-8307-c39ceb847d8f-openshift-service-ca\") pod \"e8324da1-e47d-4d1f-8307-c39ceb847d8f\" (UID: \"e8324da1-e47d-4d1f-8307-c39ceb847d8f\") "
Sep 11 03:55:12.685637 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:12.685467    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8324da1-e47d-4d1f-8307-c39ceb847d8f-metrics-certs\") pod \"e8324da1-e47d-4d1f-8307-c39ceb847d8f\" (UID: \"e8324da1-e47d-4d1f-8307-c39ceb847d8f\") "
Sep 11 03:55:12.685637 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:12.685587    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e8324da1-e47d-4d1f-8307-c39ceb847d8f-maas-api-tls\") pod \"e8324da1-e47d-4d1f-8307-c39ceb847d8f\" (UID: \"e8324da1-e47d-4d1f-8307-c39ceb847d8f\") "
Sep 11 03:55:12.685637 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:12.685616    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/473f4335-1974-4765-a1ac-da185e3ffe37-plugins-config-volume\") pod \"473f4335-1974-4765-a1ac-da185e3ffe37\" (UID: \"473f4335-1974-4765-a1ac-da185e3ffe37\") "
Sep 11 03:55:12.685841 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:12.685678    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lmw4s\" (UniqueName: \"kubernetes.io/projected/e8324da1-e47d-4d1f-8307-c39ceb847d8f-kube-api-access-lmw4s\") pod \"e8324da1-e47d-4d1f-8307-c39ceb847d8f\" (UID: \"e8324da1-e47d-4d1f-8307-c39ceb847d8f\") "
Sep 11 03:55:12.685841 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:12.685709    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6bvzt\" (UniqueName: \"kubernetes.io/projected/c1a33cdd-52f8-47ed-adbf-3bbf0017f616-kube-api-access-6bvzt\") pod \"c1a33cdd-52f8-47ed-adbf-3bbf0017f616\" (UID: \"c1a33cdd-52f8-47ed-adbf-3bbf0017f616\") "
Sep 11 03:55:12.685841 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:12.685737    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1a33cdd-52f8-47ed-adbf-3bbf0017f616-plugins-config-volume\") pod \"c1a33cdd-52f8-47ed-adbf-3bbf0017f616\" (UID: \"c1a33cdd-52f8-47ed-adbf-3bbf0017f616\") "
Sep 11 03:55:12.685841 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:12.685770    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v7ghv\" (UniqueName: \"kubernetes.io/projected/473f4335-1974-4765-a1ac-da185e3ffe37-kube-api-access-v7ghv\") pod \"473f4335-1974-4765-a1ac-da185e3ffe37\" (UID: \"473f4335-1974-4765-a1ac-da185e3ffe37\") "
Sep 11 03:55:12.686035 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:12.685939    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8324da1-e47d-4d1f-8307-c39ceb847d8f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e8324da1-e47d-4d1f-8307-c39ceb847d8f" (UID: "e8324da1-e47d-4d1f-8307-c39ceb847d8f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 03:55:12.686035 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:12.685999    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/473f4335-1974-4765-a1ac-da185e3ffe37-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "473f4335-1974-4765-a1ac-da185e3ffe37" (UID: "473f4335-1974-4765-a1ac-da185e3ffe37"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 03:55:12.686212 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:12.686189    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/473f4335-1974-4765-a1ac-da185e3ffe37-plugins-config-volume\") on node \"ip-10-0-138-121.ec2.internal\" DevicePath \"\""
Sep 11 03:55:12.686278 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:12.686216    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8324da1-e47d-4d1f-8307-c39ceb847d8f-openshift-service-ca\") on node \"ip-10-0-138-121.ec2.internal\" DevicePath \"\""
Sep 11 03:55:12.686443 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:12.686412    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1a33cdd-52f8-47ed-adbf-3bbf0017f616-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c1a33cdd-52f8-47ed-adbf-3bbf0017f616" (UID: "c1a33cdd-52f8-47ed-adbf-3bbf0017f616"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 03:55:12.688709 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:12.688640    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1a33cdd-52f8-47ed-adbf-3bbf0017f616-kube-api-access-6bvzt" (OuterVolumeSpecName: "kube-api-access-6bvzt") pod "c1a33cdd-52f8-47ed-adbf-3bbf0017f616" (UID: "c1a33cdd-52f8-47ed-adbf-3bbf0017f616"). InnerVolumeSpecName "kube-api-access-6bvzt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 03:55:12.688915 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:12.688794    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8324da1-e47d-4d1f-8307-c39ceb847d8f-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "e8324da1-e47d-4d1f-8307-c39ceb847d8f" (UID: "e8324da1-e47d-4d1f-8307-c39ceb847d8f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 03:55:12.689038 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:12.689007    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8324da1-e47d-4d1f-8307-c39ceb847d8f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e8324da1-e47d-4d1f-8307-c39ceb847d8f" (UID: "e8324da1-e47d-4d1f-8307-c39ceb847d8f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 03:55:12.689309 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:12.689260    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8324da1-e47d-4d1f-8307-c39ceb847d8f-kube-api-access-lmw4s" (OuterVolumeSpecName: "kube-api-access-lmw4s") pod "e8324da1-e47d-4d1f-8307-c39ceb847d8f" (UID: "e8324da1-e47d-4d1f-8307-c39ceb847d8f"). InnerVolumeSpecName "kube-api-access-lmw4s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 03:55:12.689601 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:12.689584    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/473f4335-1974-4765-a1ac-da185e3ffe37-kube-api-access-v7ghv" (OuterVolumeSpecName: "kube-api-access-v7ghv") pod "473f4335-1974-4765-a1ac-da185e3ffe37" (UID: "473f4335-1974-4765-a1ac-da185e3ffe37"). InnerVolumeSpecName "kube-api-access-v7ghv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 03:55:12.767592 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:12.766082    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4ce893dc-55496949d8-66znc"
Sep 11 03:55:12.767592 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:12.765769    2580 generic.go:356] "Generic (PLEG): container finished" podID="e8324da1-e47d-4d1f-8307-c39ceb847d8f" containerID="0973d94198f2c8522bdb830788aa14960dd09e9a4e436d3aed76d3f04cf2a95b" exitCode=0
Sep 11 03:55:12.767592 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:12.766704    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4ce893dc-55496949d8-66znc" event={"ID":"e8324da1-e47d-4d1f-8307-c39ceb847d8f","Type":"ContainerDied","Data":"0973d94198f2c8522bdb830788aa14960dd09e9a4e436d3aed76d3f04cf2a95b"}
Sep 11 03:55:12.767592 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:12.766793    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4ce893dc-55496949d8-66znc" event={"ID":"e8324da1-e47d-4d1f-8307-c39ceb847d8f","Type":"ContainerDied","Data":"d51ddd7bf8d802b6d7ee9936192f532fc2f3898e7b1b4300bcef97a4b819aa1d"}
Sep 11 03:55:12.767592 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:12.766824    2580 scope.go:117] "RemoveContainer" containerID="0973d94198f2c8522bdb830788aa14960dd09e9a4e436d3aed76d3f04cf2a95b"
Sep 11 03:55:12.771499 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:12.771468    2580 generic.go:356] "Generic (PLEG): container finished" podID="473f4335-1974-4765-a1ac-da185e3ffe37" containerID="1cdd7450feaf9ab5929bae4fa763b326086f364523b7dddef6acaf14024e87ca" exitCode=0
Sep 11 03:55:12.771711 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:12.771548    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-4ce893dc-5458888db5-q24xq" event={"ID":"473f4335-1974-4765-a1ac-da185e3ffe37","Type":"ContainerDied","Data":"1cdd7450feaf9ab5929bae4fa763b326086f364523b7dddef6acaf14024e87ca"}
Sep 11 03:55:12.771805 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:12.771731    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-4ce893dc-5458888db5-q24xq" event={"ID":"473f4335-1974-4765-a1ac-da185e3ffe37","Type":"ContainerDied","Data":"4ab57dd4c26b6b4369a1df092525c4702034b96e07cd993825fe801746248278"}
Sep 11 03:55:12.771805 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:12.771561    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-4ce893dc-5458888db5-q24xq"
Sep 11 03:55:12.777032 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:12.776982    2580 generic.go:356] "Generic (PLEG): container finished" podID="c1a33cdd-52f8-47ed-adbf-3bbf0017f616" containerID="da2b715fd91c838c02345fb163b2efaa1d57a0d2a8b7023f4b5816899420082b" exitCode=0
Sep 11 03:55:12.777201 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:12.777179    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-4ce893dc-5cb6566566-vwnql" event={"ID":"c1a33cdd-52f8-47ed-adbf-3bbf0017f616","Type":"ContainerDied","Data":"da2b715fd91c838c02345fb163b2efaa1d57a0d2a8b7023f4b5816899420082b"}
Sep 11 03:55:12.777273 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:12.777239    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-4ce893dc-5cb6566566-vwnql" event={"ID":"c1a33cdd-52f8-47ed-adbf-3bbf0017f616","Type":"ContainerDied","Data":"b40b0058de6e8b654c236086ce9fa09cb677a7eef71019f00fbb324ea82044a0"}
Sep 11 03:55:12.777443 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:12.777426    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-4ce893dc-5cb6566566-vwnql"
Sep 11 03:55:12.781560 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:12.781542    2580 scope.go:117] "RemoveContainer" containerID="0973d94198f2c8522bdb830788aa14960dd09e9a4e436d3aed76d3f04cf2a95b"
Sep 11 03:55:12.782107 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:55:12.781945    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0973d94198f2c8522bdb830788aa14960dd09e9a4e436d3aed76d3f04cf2a95b\": container with ID starting with 0973d94198f2c8522bdb830788aa14960dd09e9a4e436d3aed76d3f04cf2a95b not found: ID does not exist" containerID="0973d94198f2c8522bdb830788aa14960dd09e9a4e436d3aed76d3f04cf2a95b"
Sep 11 03:55:12.782107 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:12.781980    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0973d94198f2c8522bdb830788aa14960dd09e9a4e436d3aed76d3f04cf2a95b"} err="failed to get container status \"0973d94198f2c8522bdb830788aa14960dd09e9a4e436d3aed76d3f04cf2a95b\": rpc error: code = NotFound desc = could not find container \"0973d94198f2c8522bdb830788aa14960dd09e9a4e436d3aed76d3f04cf2a95b\": container with ID starting with 0973d94198f2c8522bdb830788aa14960dd09e9a4e436d3aed76d3f04cf2a95b not found: ID does not exist"
Sep 11 03:55:12.782107 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:12.782003    2580 scope.go:117] "RemoveContainer" containerID="1cdd7450feaf9ab5929bae4fa763b326086f364523b7dddef6acaf14024e87ca"
Sep 11 03:55:12.787348 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:12.787289    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6bvzt\" (UniqueName: \"kubernetes.io/projected/c1a33cdd-52f8-47ed-adbf-3bbf0017f616-kube-api-access-6bvzt\") on node \"ip-10-0-138-121.ec2.internal\" DevicePath \"\""
Sep 11 03:55:12.787443 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:12.787356    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1a33cdd-52f8-47ed-adbf-3bbf0017f616-plugins-config-volume\") on node \"ip-10-0-138-121.ec2.internal\" DevicePath \"\""
Sep 11 03:55:12.787443 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:12.787373    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v7ghv\" (UniqueName: \"kubernetes.io/projected/473f4335-1974-4765-a1ac-da185e3ffe37-kube-api-access-v7ghv\") on node \"ip-10-0-138-121.ec2.internal\" DevicePath \"\""
Sep 11 03:55:12.787443 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:12.787389    2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8324da1-e47d-4d1f-8307-c39ceb847d8f-metrics-certs\") on node \"ip-10-0-138-121.ec2.internal\" DevicePath \"\""
Sep 11 03:55:12.787443 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:12.787402    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e8324da1-e47d-4d1f-8307-c39ceb847d8f-maas-api-tls\") on node \"ip-10-0-138-121.ec2.internal\" DevicePath \"\""
Sep 11 03:55:12.787443 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:12.787415    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lmw4s\" (UniqueName: \"kubernetes.io/projected/e8324da1-e47d-4d1f-8307-c39ceb847d8f-kube-api-access-lmw4s\") on node \"ip-10-0-138-121.ec2.internal\" DevicePath \"\""
Sep 11 03:55:12.792849 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:12.792828    2580 scope.go:117] "RemoveContainer" containerID="1cdd7450feaf9ab5929bae4fa763b326086f364523b7dddef6acaf14024e87ca"
Sep 11 03:55:12.793150 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:55:12.793132    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1cdd7450feaf9ab5929bae4fa763b326086f364523b7dddef6acaf14024e87ca\": container with ID starting with 1cdd7450feaf9ab5929bae4fa763b326086f364523b7dddef6acaf14024e87ca not found: ID does not exist" containerID="1cdd7450feaf9ab5929bae4fa763b326086f364523b7dddef6acaf14024e87ca"
Sep 11 03:55:12.793215 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:12.793158    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1cdd7450feaf9ab5929bae4fa763b326086f364523b7dddef6acaf14024e87ca"} err="failed to get container status \"1cdd7450feaf9ab5929bae4fa763b326086f364523b7dddef6acaf14024e87ca\": rpc error: code = NotFound desc = could not find container \"1cdd7450feaf9ab5929bae4fa763b326086f364523b7dddef6acaf14024e87ca\": container with ID starting with 1cdd7450feaf9ab5929bae4fa763b326086f364523b7dddef6acaf14024e87ca not found: ID does not exist"
Sep 11 03:55:12.793215 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:12.793175    2580 scope.go:117] "RemoveContainer" containerID="da2b715fd91c838c02345fb163b2efaa1d57a0d2a8b7023f4b5816899420082b"
Sep 11 03:55:12.796455 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:12.796433    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-4ce893dc-55496949d8-66znc"]
Sep 11 03:55:12.800078 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:12.800055    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-4ce893dc-55496949d8-66znc"]
Sep 11 03:55:12.802191 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:12.802174    2580 scope.go:117] "RemoveContainer" containerID="da2b715fd91c838c02345fb163b2efaa1d57a0d2a8b7023f4b5816899420082b"
Sep 11 03:55:12.802474 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:55:12.802455    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da2b715fd91c838c02345fb163b2efaa1d57a0d2a8b7023f4b5816899420082b\": container with ID starting with da2b715fd91c838c02345fb163b2efaa1d57a0d2a8b7023f4b5816899420082b not found: ID does not exist" containerID="da2b715fd91c838c02345fb163b2efaa1d57a0d2a8b7023f4b5816899420082b"
Sep 11 03:55:12.802535 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:12.802482    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da2b715fd91c838c02345fb163b2efaa1d57a0d2a8b7023f4b5816899420082b"} err="failed to get container status \"da2b715fd91c838c02345fb163b2efaa1d57a0d2a8b7023f4b5816899420082b\": rpc error: code = NotFound desc = could not find container \"da2b715fd91c838c02345fb163b2efaa1d57a0d2a8b7023f4b5816899420082b\": container with ID starting with da2b715fd91c838c02345fb163b2efaa1d57a0d2a8b7023f4b5816899420082b not found: ID does not exist"
Sep 11 03:55:12.809207 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:12.809186    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-4ce893dc-5cb6566566-vwnql"]
Sep 11 03:55:12.813010 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:12.812986    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-4ce893dc-5cb6566566-vwnql"]
Sep 11 03:55:12.821611 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:12.821587    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-4ce893dc-5458888db5-q24xq"]
Sep 11 03:55:12.827107 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:12.827084    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-4ce893dc-5458888db5-q24xq"]
Sep 11 03:55:14.464417 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:14.464382    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="473f4335-1974-4765-a1ac-da185e3ffe37" path="/var/lib/kubelet/pods/473f4335-1974-4765-a1ac-da185e3ffe37/volumes"
Sep 11 03:55:14.464858 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:14.464837    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c1a33cdd-52f8-47ed-adbf-3bbf0017f616" path="/var/lib/kubelet/pods/c1a33cdd-52f8-47ed-adbf-3bbf0017f616/volumes"
Sep 11 03:55:14.465270 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:14.465257    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e8324da1-e47d-4d1f-8307-c39ceb847d8f" path="/var/lib/kubelet/pods/e8324da1-e47d-4d1f-8307-c39ceb847d8f/volumes"
Sep 11 03:55:14.998362 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:14.998321    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-conflict-a-08754c30-kserve-6d9bddf55-p4kjr" podUID="8a778771-3dcb-4ca4-9a6a-ce05cd87d081" containerName="main" containerID="cri-o://1dd5811c88116fd8798d754366b311c67fc6a50df5fd6c8cbf164148086edf86" gracePeriod=45
Sep 11 03:55:15.090512 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:55:15.090479    2580 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8324da1_e47d_4d1f_8307_c39ceb847d8f.slice/crio-0973d94198f2c8522bdb830788aa14960dd09e9a4e436d3aed76d3f04cf2a95b.scope\": RecentStats: unable to find data in memory cache]"
Sep 11 03:55:15.249137 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:15.249061    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-08754c30-kserve-6d9bddf55-p4kjr"
Sep 11 03:55:15.310136 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:15.310101    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a778771-3dcb-4ca4-9a6a-ce05cd87d081-tls-certs\") pod \"8a778771-3dcb-4ca4-9a6a-ce05cd87d081\" (UID: \"8a778771-3dcb-4ca4-9a6a-ce05cd87d081\") "
Sep 11 03:55:15.310277 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:15.310150    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a778771-3dcb-4ca4-9a6a-ce05cd87d081-dshm\") pod \"8a778771-3dcb-4ca4-9a6a-ce05cd87d081\" (UID: \"8a778771-3dcb-4ca4-9a6a-ce05cd87d081\") "
Sep 11 03:55:15.310277 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:15.310191    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a778771-3dcb-4ca4-9a6a-ce05cd87d081-model-cache\") pod \"8a778771-3dcb-4ca4-9a6a-ce05cd87d081\" (UID: \"8a778771-3dcb-4ca4-9a6a-ce05cd87d081\") "
Sep 11 03:55:15.310277 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:15.310223    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a778771-3dcb-4ca4-9a6a-ce05cd87d081-tmp-dir\") pod \"8a778771-3dcb-4ca4-9a6a-ce05cd87d081\" (UID: \"8a778771-3dcb-4ca4-9a6a-ce05cd87d081\") "
Sep 11 03:55:15.310277 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:15.310271    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zzdgw\" (UniqueName: \"kubernetes.io/projected/8a778771-3dcb-4ca4-9a6a-ce05cd87d081-kube-api-access-zzdgw\") pod \"8a778771-3dcb-4ca4-9a6a-ce05cd87d081\" (UID: \"8a778771-3dcb-4ca4-9a6a-ce05cd87d081\") "
Sep 11 03:55:15.310493 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:15.310358    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a778771-3dcb-4ca4-9a6a-ce05cd87d081-home\") pod \"8a778771-3dcb-4ca4-9a6a-ce05cd87d081\" (UID: \"8a778771-3dcb-4ca4-9a6a-ce05cd87d081\") "
Sep 11 03:55:15.310554 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:15.310538    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a778771-3dcb-4ca4-9a6a-ce05cd87d081-model-cache" (OuterVolumeSpecName: "model-cache") pod "8a778771-3dcb-4ca4-9a6a-ce05cd87d081" (UID: "8a778771-3dcb-4ca4-9a6a-ce05cd87d081"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 03:55:15.310614 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:15.310552    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a778771-3dcb-4ca4-9a6a-ce05cd87d081-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8a778771-3dcb-4ca4-9a6a-ce05cd87d081" (UID: "8a778771-3dcb-4ca4-9a6a-ce05cd87d081"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 03:55:15.310760 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:15.310744    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a778771-3dcb-4ca4-9a6a-ce05cd87d081-model-cache\") on node \"ip-10-0-138-121.ec2.internal\" DevicePath \"\""
Sep 11 03:55:15.310829 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:15.310772    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a778771-3dcb-4ca4-9a6a-ce05cd87d081-tmp-dir\") on node \"ip-10-0-138-121.ec2.internal\" DevicePath \"\""
Sep 11 03:55:15.310829 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:15.310744    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a778771-3dcb-4ca4-9a6a-ce05cd87d081-home" (OuterVolumeSpecName: "home") pod "8a778771-3dcb-4ca4-9a6a-ce05cd87d081" (UID: "8a778771-3dcb-4ca4-9a6a-ce05cd87d081"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 03:55:15.312502 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:15.312482    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a778771-3dcb-4ca4-9a6a-ce05cd87d081-kube-api-access-zzdgw" (OuterVolumeSpecName: "kube-api-access-zzdgw") pod "8a778771-3dcb-4ca4-9a6a-ce05cd87d081" (UID: "8a778771-3dcb-4ca4-9a6a-ce05cd87d081"). InnerVolumeSpecName "kube-api-access-zzdgw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 03:55:15.312781 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:15.312750    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a778771-3dcb-4ca4-9a6a-ce05cd87d081-dshm" (OuterVolumeSpecName: "dshm") pod "8a778771-3dcb-4ca4-9a6a-ce05cd87d081" (UID: "8a778771-3dcb-4ca4-9a6a-ce05cd87d081"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 03:55:15.312847 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:15.312773    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a778771-3dcb-4ca4-9a6a-ce05cd87d081-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8a778771-3dcb-4ca4-9a6a-ce05cd87d081" (UID: "8a778771-3dcb-4ca4-9a6a-ce05cd87d081"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 03:55:15.412084 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:15.412043    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a778771-3dcb-4ca4-9a6a-ce05cd87d081-dshm\") on node \"ip-10-0-138-121.ec2.internal\" DevicePath \"\""
Sep 11 03:55:15.412084 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:15.412080    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zzdgw\" (UniqueName: \"kubernetes.io/projected/8a778771-3dcb-4ca4-9a6a-ce05cd87d081-kube-api-access-zzdgw\") on node \"ip-10-0-138-121.ec2.internal\" DevicePath \"\""
Sep 11 03:55:15.412084 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:15.412092    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a778771-3dcb-4ca4-9a6a-ce05cd87d081-home\") on node \"ip-10-0-138-121.ec2.internal\" DevicePath \"\""
Sep 11 03:55:15.412383 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:15.412104    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a778771-3dcb-4ca4-9a6a-ce05cd87d081-tls-certs\") on node \"ip-10-0-138-121.ec2.internal\" DevicePath \"\""
Sep 11 03:55:15.801469 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:15.801433    2580 generic.go:356] "Generic (PLEG): container finished" podID="8a778771-3dcb-4ca4-9a6a-ce05cd87d081" containerID="1dd5811c88116fd8798d754366b311c67fc6a50df5fd6c8cbf164148086edf86" exitCode=0
Sep 11 03:55:15.801872 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:15.801478    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-08754c30-kserve-6d9bddf55-p4kjr"
Sep 11 03:55:15.801872 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:15.801520    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-08754c30-kserve-6d9bddf55-p4kjr" event={"ID":"8a778771-3dcb-4ca4-9a6a-ce05cd87d081","Type":"ContainerDied","Data":"1dd5811c88116fd8798d754366b311c67fc6a50df5fd6c8cbf164148086edf86"}
Sep 11 03:55:15.801872 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:15.801568    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-08754c30-kserve-6d9bddf55-p4kjr" event={"ID":"8a778771-3dcb-4ca4-9a6a-ce05cd87d081","Type":"ContainerDied","Data":"ef50e66810d29ab678e556466570a1c3442281652fa2fa15d07f80514a4815ae"}
Sep 11 03:55:15.801872 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:15.801588    2580 scope.go:117] "RemoveContainer" containerID="1dd5811c88116fd8798d754366b311c67fc6a50df5fd6c8cbf164148086edf86"
Sep 11 03:55:15.810769 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:15.810749    2580 scope.go:117] "RemoveContainer" containerID="1dd5811c88116fd8798d754366b311c67fc6a50df5fd6c8cbf164148086edf86"
Sep 11 03:55:15.811030 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:55:15.811013    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1dd5811c88116fd8798d754366b311c67fc6a50df5fd6c8cbf164148086edf86\": container with ID starting with 1dd5811c88116fd8798d754366b311c67fc6a50df5fd6c8cbf164148086edf86 not found: ID does not exist" containerID="1dd5811c88116fd8798d754366b311c67fc6a50df5fd6c8cbf164148086edf86"
Sep 11 03:55:15.811079 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:15.811039    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1dd5811c88116fd8798d754366b311c67fc6a50df5fd6c8cbf164148086edf86"} err="failed to get container status \"1dd5811c88116fd8798d754366b311c67fc6a50df5fd6c8cbf164148086edf86\": rpc error: code = NotFound desc = could not find container \"1dd5811c88116fd8798d754366b311c67fc6a50df5fd6c8cbf164148086edf86\": container with ID starting with 1dd5811c88116fd8798d754366b311c67fc6a50df5fd6c8cbf164148086edf86 not found: ID does not exist"
Sep 11 03:55:15.821460 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:15.821435    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-a-08754c30-kserve-6d9bddf55-p4kjr"]
Sep 11 03:55:15.824729 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:15.824707    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-conflict-a-08754c30-kserve-6d9bddf55-p4kjr"]
Sep 11 03:55:16.466264 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:16.466229    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8a778771-3dcb-4ca4-9a6a-ce05cd87d081" path="/var/lib/kubelet/pods/8a778771-3dcb-4ca4-9a6a-ce05cd87d081/volumes"
Sep 11 03:55:18.016917 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:18.016890    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-4ce893dc-openshift-default-59496b9896-zt44l"
Sep 11 03:55:18.137304 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:18.137203    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07e19967-b8ed-4adc-a511-2860068a4cba-istio-token\") pod \"07e19967-b8ed-4adc-a511-2860068a4cba\" (UID: \"07e19967-b8ed-4adc-a511-2860068a4cba\") "
Sep 11 03:55:18.137304 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:18.137250    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07e19967-b8ed-4adc-a511-2860068a4cba-istio-envoy\") pod \"07e19967-b8ed-4adc-a511-2860068a4cba\" (UID: \"07e19967-b8ed-4adc-a511-2860068a4cba\") "
Sep 11 03:55:18.137304 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:18.137279    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07e19967-b8ed-4adc-a511-2860068a4cba-istio-podinfo\") pod \"07e19967-b8ed-4adc-a511-2860068a4cba\" (UID: \"07e19967-b8ed-4adc-a511-2860068a4cba\") "
Sep 11 03:55:18.137603 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:18.137329    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07e19967-b8ed-4adc-a511-2860068a4cba-credential-socket\") pod \"07e19967-b8ed-4adc-a511-2860068a4cba\" (UID: \"07e19967-b8ed-4adc-a511-2860068a4cba\") "
Sep 11 03:55:18.137603 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:18.137396    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07e19967-b8ed-4adc-a511-2860068a4cba-workload-certs\") pod \"07e19967-b8ed-4adc-a511-2860068a4cba\" (UID: \"07e19967-b8ed-4adc-a511-2860068a4cba\") "
Sep 11 03:55:18.137603 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:18.137443    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07e19967-b8ed-4adc-a511-2860068a4cba-istiod-ca-cert\") pod \"07e19967-b8ed-4adc-a511-2860068a4cba\" (UID: \"07e19967-b8ed-4adc-a511-2860068a4cba\") "
Sep 11 03:55:18.137603 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:18.137488    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9289j\" (UniqueName: \"kubernetes.io/projected/07e19967-b8ed-4adc-a511-2860068a4cba-kube-api-access-9289j\") pod \"07e19967-b8ed-4adc-a511-2860068a4cba\" (UID: \"07e19967-b8ed-4adc-a511-2860068a4cba\") "
Sep 11 03:55:18.137603 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:18.137511    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07e19967-b8ed-4adc-a511-2860068a4cba-istio-data\") pod \"07e19967-b8ed-4adc-a511-2860068a4cba\" (UID: \"07e19967-b8ed-4adc-a511-2860068a4cba\") "
Sep 11 03:55:18.137603 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:18.137562    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07e19967-b8ed-4adc-a511-2860068a4cba-workload-socket\") pod \"07e19967-b8ed-4adc-a511-2860068a4cba\" (UID: \"07e19967-b8ed-4adc-a511-2860068a4cba\") "
Sep 11 03:55:18.137950 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:18.137698    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07e19967-b8ed-4adc-a511-2860068a4cba-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "07e19967-b8ed-4adc-a511-2860068a4cba" (UID: "07e19967-b8ed-4adc-a511-2860068a4cba"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 03:55:18.137950 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:18.137845    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07e19967-b8ed-4adc-a511-2860068a4cba-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "07e19967-b8ed-4adc-a511-2860068a4cba" (UID: "07e19967-b8ed-4adc-a511-2860068a4cba"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 03:55:18.137950 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:18.137730    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07e19967-b8ed-4adc-a511-2860068a4cba-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "07e19967-b8ed-4adc-a511-2860068a4cba" (UID: "07e19967-b8ed-4adc-a511-2860068a4cba"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 03:55:18.138112 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:18.137997    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07e19967-b8ed-4adc-a511-2860068a4cba-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "07e19967-b8ed-4adc-a511-2860068a4cba" (UID: "07e19967-b8ed-4adc-a511-2860068a4cba"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 03:55:18.138112 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:18.138039    2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07e19967-b8ed-4adc-a511-2860068a4cba-workload-certs\") on node \"ip-10-0-138-121.ec2.internal\" DevicePath \"\""
Sep 11 03:55:18.138112 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:18.138063    2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07e19967-b8ed-4adc-a511-2860068a4cba-istiod-ca-cert\") on node \"ip-10-0-138-121.ec2.internal\" DevicePath \"\""
Sep 11 03:55:18.138112 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:18.138077    2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07e19967-b8ed-4adc-a511-2860068a4cba-credential-socket\") on node \"ip-10-0-138-121.ec2.internal\" DevicePath \"\""
Sep 11 03:55:18.138317 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:18.138130    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07e19967-b8ed-4adc-a511-2860068a4cba-istio-data" (OuterVolumeSpecName: "istio-data") pod "07e19967-b8ed-4adc-a511-2860068a4cba" (UID: "07e19967-b8ed-4adc-a511-2860068a4cba"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 03:55:18.139616 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:18.139589    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07e19967-b8ed-4adc-a511-2860068a4cba-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "07e19967-b8ed-4adc-a511-2860068a4cba" (UID: "07e19967-b8ed-4adc-a511-2860068a4cba"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 03:55:18.139736 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:18.139624    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/07e19967-b8ed-4adc-a511-2860068a4cba-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "07e19967-b8ed-4adc-a511-2860068a4cba" (UID: "07e19967-b8ed-4adc-a511-2860068a4cba"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 03:55:18.139736 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:18.139719    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07e19967-b8ed-4adc-a511-2860068a4cba-istio-token" (OuterVolumeSpecName: "istio-token") pod "07e19967-b8ed-4adc-a511-2860068a4cba" (UID: "07e19967-b8ed-4adc-a511-2860068a4cba"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 03:55:18.139846 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:18.139821    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07e19967-b8ed-4adc-a511-2860068a4cba-kube-api-access-9289j" (OuterVolumeSpecName: "kube-api-access-9289j") pod "07e19967-b8ed-4adc-a511-2860068a4cba" (UID: "07e19967-b8ed-4adc-a511-2860068a4cba"). InnerVolumeSpecName "kube-api-access-9289j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 03:55:18.238543 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:18.238503    2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07e19967-b8ed-4adc-a511-2860068a4cba-workload-socket\") on node \"ip-10-0-138-121.ec2.internal\" DevicePath \"\""
Sep 11 03:55:18.238543 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:18.238534    2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07e19967-b8ed-4adc-a511-2860068a4cba-istio-token\") on node \"ip-10-0-138-121.ec2.internal\" DevicePath \"\""
Sep 11 03:55:18.238543 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:18.238547    2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07e19967-b8ed-4adc-a511-2860068a4cba-istio-envoy\") on node \"ip-10-0-138-121.ec2.internal\" DevicePath \"\""
Sep 11 03:55:18.238543 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:18.238555    2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07e19967-b8ed-4adc-a511-2860068a4cba-istio-podinfo\") on node \"ip-10-0-138-121.ec2.internal\" DevicePath \"\""
Sep 11 03:55:18.238886 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:18.238563    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9289j\" (UniqueName: \"kubernetes.io/projected/07e19967-b8ed-4adc-a511-2860068a4cba-kube-api-access-9289j\") on node \"ip-10-0-138-121.ec2.internal\" DevicePath \"\""
Sep 11 03:55:18.238886 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:18.238573    2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07e19967-b8ed-4adc-a511-2860068a4cba-istio-data\") on node \"ip-10-0-138-121.ec2.internal\" DevicePath \"\""
Sep 11 03:55:18.826820 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:18.826740    2580 generic.go:356] "Generic (PLEG): container finished" podID="07e19967-b8ed-4adc-a511-2860068a4cba" containerID="2b67cc6fae2fb32190e053b73d26b7f1032e9b5d0155215556c28af0ee3034c9" exitCode=0
Sep 11 03:55:18.826820 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:18.826784    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-4ce893dc-openshift-default-59496b9896-zt44l"
Sep 11 03:55:18.826820 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:18.826798    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-4ce893dc-openshift-default-59496b9896-zt44l" event={"ID":"07e19967-b8ed-4adc-a511-2860068a4cba","Type":"ContainerDied","Data":"2b67cc6fae2fb32190e053b73d26b7f1032e9b5d0155215556c28af0ee3034c9"}
Sep 11 03:55:18.827041 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:18.826842    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-4ce893dc-openshift-default-59496b9896-zt44l" event={"ID":"07e19967-b8ed-4adc-a511-2860068a4cba","Type":"ContainerDied","Data":"0a5f0dc2d053c302a7bfae60c3dca971f47ceeede1e49ea741deebc03a72d70a"}
Sep 11 03:55:18.827041 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:18.826863    2580 scope.go:117] "RemoveContainer" containerID="2b67cc6fae2fb32190e053b73d26b7f1032e9b5d0155215556c28af0ee3034c9"
Sep 11 03:55:18.835941 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:18.835918    2580 scope.go:117] "RemoveContainer" containerID="f67845faedd703089715f952ea463568dd8709613243ba2f566415ed4ea8acd2"
Sep 11 03:55:18.844172 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:18.844150    2580 scope.go:117] "RemoveContainer" containerID="2b67cc6fae2fb32190e053b73d26b7f1032e9b5d0155215556c28af0ee3034c9"
Sep 11 03:55:18.844489 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:55:18.844465    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b67cc6fae2fb32190e053b73d26b7f1032e9b5d0155215556c28af0ee3034c9\": container with ID starting with 2b67cc6fae2fb32190e053b73d26b7f1032e9b5d0155215556c28af0ee3034c9 not found: ID does not exist" containerID="2b67cc6fae2fb32190e053b73d26b7f1032e9b5d0155215556c28af0ee3034c9"
Sep 11 03:55:18.844607 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:18.844496    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b67cc6fae2fb32190e053b73d26b7f1032e9b5d0155215556c28af0ee3034c9"} err="failed to get container status \"2b67cc6fae2fb32190e053b73d26b7f1032e9b5d0155215556c28af0ee3034c9\": rpc error: code = NotFound desc = could not find container \"2b67cc6fae2fb32190e053b73d26b7f1032e9b5d0155215556c28af0ee3034c9\": container with ID starting with 2b67cc6fae2fb32190e053b73d26b7f1032e9b5d0155215556c28af0ee3034c9 not found: ID does not exist"
Sep 11 03:55:18.844607 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:18.844512    2580 scope.go:117] "RemoveContainer" containerID="f67845faedd703089715f952ea463568dd8709613243ba2f566415ed4ea8acd2"
Sep 11 03:55:18.844789 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:55:18.844773    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f67845faedd703089715f952ea463568dd8709613243ba2f566415ed4ea8acd2\": container with ID starting with f67845faedd703089715f952ea463568dd8709613243ba2f566415ed4ea8acd2 not found: ID does not exist" containerID="f67845faedd703089715f952ea463568dd8709613243ba2f566415ed4ea8acd2"
Sep 11 03:55:18.844827 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:18.844792    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f67845faedd703089715f952ea463568dd8709613243ba2f566415ed4ea8acd2"} err="failed to get container status \"f67845faedd703089715f952ea463568dd8709613243ba2f566415ed4ea8acd2\": rpc error: code = NotFound desc = could not find container \"f67845faedd703089715f952ea463568dd8709613243ba2f566415ed4ea8acd2\": container with ID starting with f67845faedd703089715f952ea463568dd8709613243ba2f566415ed4ea8acd2 not found: ID does not exist"
Sep 11 03:55:18.845130 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:18.845112    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-4ce893dc-openshift-default-59496b9896-zt44l"]
Sep 11 03:55:18.848269 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:18.848247    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-4ce893dc-openshift-default-59496b9896-zt44l"]
Sep 11 03:55:20.413045 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:20.413001    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-explicit-459a4c-openshift-default-977b8764d-bsptj"]
Sep 11 03:55:20.413614 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:20.413591    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c1a33cdd-52f8-47ed-adbf-3bbf0017f616" containerName="payload-pre-processing"
Sep 11 03:55:20.413706 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:20.413618    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="c1a33cdd-52f8-47ed-adbf-3bbf0017f616" containerName="payload-pre-processing"
Sep 11 03:55:20.413706 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:20.413641    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="473f4335-1974-4765-a1ac-da185e3ffe37" containerName="payload-processing"
Sep 11 03:55:20.413706 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:20.413652    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="473f4335-1974-4765-a1ac-da185e3ffe37" containerName="payload-processing"
Sep 11 03:55:20.413706 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:20.413682    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8324da1-e47d-4d1f-8307-c39ceb847d8f" containerName="maas-api"
Sep 11 03:55:20.413706 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:20.413691    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8324da1-e47d-4d1f-8307-c39ceb847d8f" containerName="maas-api"
Sep 11 03:55:20.413968 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:20.413714    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="07e19967-b8ed-4adc-a511-2860068a4cba" containerName="istio-proxy"
Sep 11 03:55:20.413968 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:20.413722    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="07e19967-b8ed-4adc-a511-2860068a4cba" containerName="istio-proxy"
Sep 11 03:55:20.413968 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:20.413733    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a778771-3dcb-4ca4-9a6a-ce05cd87d081" containerName="main"
Sep 11 03:55:20.413968 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:20.413742    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a778771-3dcb-4ca4-9a6a-ce05cd87d081" containerName="main"
Sep 11 03:55:20.413968 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:20.413756    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="07e19967-b8ed-4adc-a511-2860068a4cba" containerName="istio-proxy"
Sep 11 03:55:20.413968 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:20.413768    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="07e19967-b8ed-4adc-a511-2860068a4cba" containerName="istio-proxy"
Sep 11 03:55:20.413968 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:20.413872    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="07e19967-b8ed-4adc-a511-2860068a4cba" containerName="istio-proxy"
Sep 11 03:55:20.413968 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:20.413882    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="c1a33cdd-52f8-47ed-adbf-3bbf0017f616" containerName="payload-pre-processing"
Sep 11 03:55:20.413968 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:20.413893    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="473f4335-1974-4765-a1ac-da185e3ffe37" containerName="payload-processing"
Sep 11 03:55:20.413968 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:20.413902    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="e8324da1-e47d-4d1f-8307-c39ceb847d8f" containerName="maas-api"
Sep 11 03:55:20.413968 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:20.413914    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a778771-3dcb-4ca4-9a6a-ce05cd87d081" containerName="main"
Sep 11 03:55:20.413968 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:20.413923    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="07e19967-b8ed-4adc-a511-2860068a4cba" containerName="istio-proxy"
Sep 11 03:55:20.419142 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:20.419119    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-459a4c-openshift-default-977b8764d-bsptj"
Sep 11 03:55:20.421345 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:20.421323    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-explicit-459a4c-openshift-default-dockercfg-z2fff\""
Sep 11 03:55:20.426492 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:20.426468    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-459a4c-openshift-default-977b8764d-bsptj"]
Sep 11 03:55:20.461896 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:20.461855    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0821739-382a-4dac-8ca9-c08841d1031c-istio-token\") pod \"e2e-explicit-459a4c-openshift-default-977b8764d-bsptj\" (UID: \"a0821739-382a-4dac-8ca9-c08841d1031c\") " pod="openshift-ingress/e2e-explicit-459a4c-openshift-default-977b8764d-bsptj"
Sep 11 03:55:20.462061 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:20.461909    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0821739-382a-4dac-8ca9-c08841d1031c-istio-data\") pod \"e2e-explicit-459a4c-openshift-default-977b8764d-bsptj\" (UID: \"a0821739-382a-4dac-8ca9-c08841d1031c\") " pod="openshift-ingress/e2e-explicit-459a4c-openshift-default-977b8764d-bsptj"
Sep 11 03:55:20.462061 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:20.461948    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgl5m\" (UniqueName: \"kubernetes.io/projected/a0821739-382a-4dac-8ca9-c08841d1031c-kube-api-access-jgl5m\") pod \"e2e-explicit-459a4c-openshift-default-977b8764d-bsptj\" (UID: \"a0821739-382a-4dac-8ca9-c08841d1031c\") " pod="openshift-ingress/e2e-explicit-459a4c-openshift-default-977b8764d-bsptj"
Sep 11 03:55:20.462061 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:20.461979    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0821739-382a-4dac-8ca9-c08841d1031c-istio-podinfo\") pod \"e2e-explicit-459a4c-openshift-default-977b8764d-bsptj\" (UID: \"a0821739-382a-4dac-8ca9-c08841d1031c\") " pod="openshift-ingress/e2e-explicit-459a4c-openshift-default-977b8764d-bsptj"
Sep 11 03:55:20.462061 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:20.462035    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0821739-382a-4dac-8ca9-c08841d1031c-credential-socket\") pod \"e2e-explicit-459a4c-openshift-default-977b8764d-bsptj\" (UID: \"a0821739-382a-4dac-8ca9-c08841d1031c\") " pod="openshift-ingress/e2e-explicit-459a4c-openshift-default-977b8764d-bsptj"
Sep 11 03:55:20.462290 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:20.462084    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0821739-382a-4dac-8ca9-c08841d1031c-istio-envoy\") pod \"e2e-explicit-459a4c-openshift-default-977b8764d-bsptj\" (UID: \"a0821739-382a-4dac-8ca9-c08841d1031c\") " pod="openshift-ingress/e2e-explicit-459a4c-openshift-default-977b8764d-bsptj"
Sep 11 03:55:20.462290 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:20.462113    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0821739-382a-4dac-8ca9-c08841d1031c-workload-socket\") pod \"e2e-explicit-459a4c-openshift-default-977b8764d-bsptj\" (UID: \"a0821739-382a-4dac-8ca9-c08841d1031c\") " pod="openshift-ingress/e2e-explicit-459a4c-openshift-default-977b8764d-bsptj"
Sep 11 03:55:20.462290 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:20.462159    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0821739-382a-4dac-8ca9-c08841d1031c-istiod-ca-cert\") pod \"e2e-explicit-459a4c-openshift-default-977b8764d-bsptj\" (UID: \"a0821739-382a-4dac-8ca9-c08841d1031c\") " pod="openshift-ingress/e2e-explicit-459a4c-openshift-default-977b8764d-bsptj"
Sep 11 03:55:20.462290 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:20.462223    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0821739-382a-4dac-8ca9-c08841d1031c-workload-certs\") pod \"e2e-explicit-459a4c-openshift-default-977b8764d-bsptj\" (UID: \"a0821739-382a-4dac-8ca9-c08841d1031c\") " pod="openshift-ingress/e2e-explicit-459a4c-openshift-default-977b8764d-bsptj"
Sep 11 03:55:20.465039 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:20.465015    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="07e19967-b8ed-4adc-a511-2860068a4cba" path="/var/lib/kubelet/pods/07e19967-b8ed-4adc-a511-2860068a4cba/volumes"
Sep 11 03:55:20.562916 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:20.562874    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0821739-382a-4dac-8ca9-c08841d1031c-istiod-ca-cert\") pod \"e2e-explicit-459a4c-openshift-default-977b8764d-bsptj\" (UID: \"a0821739-382a-4dac-8ca9-c08841d1031c\") " pod="openshift-ingress/e2e-explicit-459a4c-openshift-default-977b8764d-bsptj"
Sep 11 03:55:20.563104 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:20.562953    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0821739-382a-4dac-8ca9-c08841d1031c-workload-certs\") pod \"e2e-explicit-459a4c-openshift-default-977b8764d-bsptj\" (UID: \"a0821739-382a-4dac-8ca9-c08841d1031c\") " pod="openshift-ingress/e2e-explicit-459a4c-openshift-default-977b8764d-bsptj"
Sep 11 03:55:20.563104 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:20.563009    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0821739-382a-4dac-8ca9-c08841d1031c-istio-token\") pod \"e2e-explicit-459a4c-openshift-default-977b8764d-bsptj\" (UID: \"a0821739-382a-4dac-8ca9-c08841d1031c\") " pod="openshift-ingress/e2e-explicit-459a4c-openshift-default-977b8764d-bsptj"
Sep 11 03:55:20.563104 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:20.563044    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0821739-382a-4dac-8ca9-c08841d1031c-istio-data\") pod \"e2e-explicit-459a4c-openshift-default-977b8764d-bsptj\" (UID: \"a0821739-382a-4dac-8ca9-c08841d1031c\") " pod="openshift-ingress/e2e-explicit-459a4c-openshift-default-977b8764d-bsptj"
Sep 11 03:55:20.563104 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:20.563084    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgl5m\" (UniqueName: \"kubernetes.io/projected/a0821739-382a-4dac-8ca9-c08841d1031c-kube-api-access-jgl5m\") pod \"e2e-explicit-459a4c-openshift-default-977b8764d-bsptj\" (UID: \"a0821739-382a-4dac-8ca9-c08841d1031c\") " pod="openshift-ingress/e2e-explicit-459a4c-openshift-default-977b8764d-bsptj"
Sep 11 03:55:20.563349 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:20.563123    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0821739-382a-4dac-8ca9-c08841d1031c-istio-podinfo\") pod \"e2e-explicit-459a4c-openshift-default-977b8764d-bsptj\" (UID: \"a0821739-382a-4dac-8ca9-c08841d1031c\") " pod="openshift-ingress/e2e-explicit-459a4c-openshift-default-977b8764d-bsptj"
Sep 11 03:55:20.563349 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:20.563161    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0821739-382a-4dac-8ca9-c08841d1031c-credential-socket\") pod \"e2e-explicit-459a4c-openshift-default-977b8764d-bsptj\" (UID: \"a0821739-382a-4dac-8ca9-c08841d1031c\") " pod="openshift-ingress/e2e-explicit-459a4c-openshift-default-977b8764d-bsptj"
Sep 11 03:55:20.563349 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:20.563259    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0821739-382a-4dac-8ca9-c08841d1031c-istio-envoy\") pod \"e2e-explicit-459a4c-openshift-default-977b8764d-bsptj\" (UID: \"a0821739-382a-4dac-8ca9-c08841d1031c\") " pod="openshift-ingress/e2e-explicit-459a4c-openshift-default-977b8764d-bsptj"
Sep 11 03:55:20.563349 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:20.563295    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0821739-382a-4dac-8ca9-c08841d1031c-workload-socket\") pod \"e2e-explicit-459a4c-openshift-default-977b8764d-bsptj\" (UID: \"a0821739-382a-4dac-8ca9-c08841d1031c\") " pod="openshift-ingress/e2e-explicit-459a4c-openshift-default-977b8764d-bsptj"
Sep 11 03:55:20.563760 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:20.563649    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0821739-382a-4dac-8ca9-c08841d1031c-istiod-ca-cert\") pod \"e2e-explicit-459a4c-openshift-default-977b8764d-bsptj\" (UID: \"a0821739-382a-4dac-8ca9-c08841d1031c\") " pod="openshift-ingress/e2e-explicit-459a4c-openshift-default-977b8764d-bsptj"
Sep 11 03:55:20.563760 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:20.563681    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0821739-382a-4dac-8ca9-c08841d1031c-workload-certs\") pod \"e2e-explicit-459a4c-openshift-default-977b8764d-bsptj\" (UID: \"a0821739-382a-4dac-8ca9-c08841d1031c\") " pod="openshift-ingress/e2e-explicit-459a4c-openshift-default-977b8764d-bsptj"
Sep 11 03:55:20.563760 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:20.563738    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0821739-382a-4dac-8ca9-c08841d1031c-istio-data\") pod \"e2e-explicit-459a4c-openshift-default-977b8764d-bsptj\" (UID: \"a0821739-382a-4dac-8ca9-c08841d1031c\") " pod="openshift-ingress/e2e-explicit-459a4c-openshift-default-977b8764d-bsptj"
Sep 11 03:55:20.563984 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:20.563800    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0821739-382a-4dac-8ca9-c08841d1031c-workload-socket\") pod \"e2e-explicit-459a4c-openshift-default-977b8764d-bsptj\" (UID: \"a0821739-382a-4dac-8ca9-c08841d1031c\") " pod="openshift-ingress/e2e-explicit-459a4c-openshift-default-977b8764d-bsptj"
Sep 11 03:55:20.563984 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:20.563928    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0821739-382a-4dac-8ca9-c08841d1031c-credential-socket\") pod \"e2e-explicit-459a4c-openshift-default-977b8764d-bsptj\" (UID: \"a0821739-382a-4dac-8ca9-c08841d1031c\") " pod="openshift-ingress/e2e-explicit-459a4c-openshift-default-977b8764d-bsptj"
Sep 11 03:55:20.565849 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:20.565824    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0821739-382a-4dac-8ca9-c08841d1031c-istio-envoy\") pod \"e2e-explicit-459a4c-openshift-default-977b8764d-bsptj\" (UID: \"a0821739-382a-4dac-8ca9-c08841d1031c\") " pod="openshift-ingress/e2e-explicit-459a4c-openshift-default-977b8764d-bsptj"
Sep 11 03:55:20.565973 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:20.565955    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0821739-382a-4dac-8ca9-c08841d1031c-istio-podinfo\") pod \"e2e-explicit-459a4c-openshift-default-977b8764d-bsptj\" (UID: \"a0821739-382a-4dac-8ca9-c08841d1031c\") " pod="openshift-ingress/e2e-explicit-459a4c-openshift-default-977b8764d-bsptj"
Sep 11 03:55:20.570221 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:20.570190    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0821739-382a-4dac-8ca9-c08841d1031c-istio-token\") pod \"e2e-explicit-459a4c-openshift-default-977b8764d-bsptj\" (UID: \"a0821739-382a-4dac-8ca9-c08841d1031c\") " pod="openshift-ingress/e2e-explicit-459a4c-openshift-default-977b8764d-bsptj"
Sep 11 03:55:20.570418 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:20.570400    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgl5m\" (UniqueName: \"kubernetes.io/projected/a0821739-382a-4dac-8ca9-c08841d1031c-kube-api-access-jgl5m\") pod \"e2e-explicit-459a4c-openshift-default-977b8764d-bsptj\" (UID: \"a0821739-382a-4dac-8ca9-c08841d1031c\") " pod="openshift-ingress/e2e-explicit-459a4c-openshift-default-977b8764d-bsptj"
Sep 11 03:55:20.730517 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:20.730426    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-459a4c-openshift-default-977b8764d-bsptj"
Sep 11 03:55:21.006936 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:21.006856    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-459a4c-openshift-default-977b8764d-bsptj"]
Sep 11 03:55:21.010421 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:55:21.010392    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0821739_382a_4dac_8ca9_c08841d1031c.slice/crio-bad55ffad8869fcfa4806e229ef3cd58b8e2cffe1d2cf1bfa762db22467c26b5 WatchSource:0}: Error finding container bad55ffad8869fcfa4806e229ef3cd58b8e2cffe1d2cf1bfa762db22467c26b5: Status 404 returned error can't find the container with id bad55ffad8869fcfa4806e229ef3cd58b8e2cffe1d2cf1bfa762db22467c26b5
Sep 11 03:55:21.012445 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:21.012412    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 03:55:21.012538 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:21.012479    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 03:55:21.012538 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:21.012508    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 03:55:21.842944 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:21.842904    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-459a4c-openshift-default-977b8764d-bsptj" event={"ID":"a0821739-382a-4dac-8ca9-c08841d1031c","Type":"ContainerStarted","Data":"c10ebc83988e735cfe38f6c076e73fb8830bbface90741e24ab314624b9aff1f"}
Sep 11 03:55:21.843349 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:21.842965    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-459a4c-openshift-default-977b8764d-bsptj" event={"ID":"a0821739-382a-4dac-8ca9-c08841d1031c","Type":"ContainerStarted","Data":"bad55ffad8869fcfa4806e229ef3cd58b8e2cffe1d2cf1bfa762db22467c26b5"}
Sep 11 03:55:21.864138 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:21.864084    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-explicit-459a4c-openshift-default-977b8764d-bsptj" podStartSLOduration=1.864068308 podStartE2EDuration="1.864068308s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 03:55:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:55:21.862244084 +0000 UTC m=+910.020730409" watchObservedRunningTime="2026-09-11 03:55:21.864068308 +0000 UTC m=+910.022554630"
Sep 11 03:55:22.731408 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:22.731368    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-459a4c-openshift-default-977b8764d-bsptj"
Sep 11 03:55:22.736845 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:22.736813    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-459a4c-openshift-default-977b8764d-bsptj"
Sep 11 03:55:22.848468 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:22.848436    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-459a4c-openshift-default-977b8764d-bsptj"
Sep 11 03:55:22.849594 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:22.849572    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-459a4c-openshift-default-977b8764d-bsptj"
Sep 11 03:55:25.137937 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:55:25.137904    2580 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8324da1_e47d_4d1f_8307_c39ceb847d8f.slice/crio-0973d94198f2c8522bdb830788aa14960dd09e9a4e436d3aed76d3f04cf2a95b.scope\": RecentStats: unable to find data in memory cache]"
Sep 11 03:55:25.311074 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:25.311041    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-459a4c-9fffb4c8d-9tvlh"]
Sep 11 03:55:25.315255 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:25.315226    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-459a4c-9fffb4c8d-9tvlh"
Sep 11 03:55:25.317716 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:25.317695    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-explicit-459a4c-serving-cert\""
Sep 11 03:55:25.317805 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:25.317694    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 03:55:25.318437 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:25.318419    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 11 03:55:25.318536 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:25.318433    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-fcpfs\""
Sep 11 03:55:25.318536 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:25.318466    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 11 03:55:25.322554 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:25.322527    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-459a4c-9fffb4c8d-9tvlh"]
Sep 11 03:55:25.411101 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:25.411067    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-spswm\" (UniqueName: \"kubernetes.io/projected/fd5ae1ec-e90c-422e-84b7-540932f427dc-kube-api-access-spswm\") pod \"maas-api-e2e-explicit-459a4c-9fffb4c8d-9tvlh\" (UID: \"fd5ae1ec-e90c-422e-84b7-540932f427dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-459a4c-9fffb4c8d-9tvlh"
Sep 11 03:55:25.411276 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:25.411156    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fd5ae1ec-e90c-422e-84b7-540932f427dc-maas-api-tls\") pod \"maas-api-e2e-explicit-459a4c-9fffb4c8d-9tvlh\" (UID: \"fd5ae1ec-e90c-422e-84b7-540932f427dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-459a4c-9fffb4c8d-9tvlh"
Sep 11 03:55:25.411276 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:25.411205    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd5ae1ec-e90c-422e-84b7-540932f427dc-openshift-service-ca\") pod \"maas-api-e2e-explicit-459a4c-9fffb4c8d-9tvlh\" (UID: \"fd5ae1ec-e90c-422e-84b7-540932f427dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-459a4c-9fffb4c8d-9tvlh"
Sep 11 03:55:25.411276 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:25.411259    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd5ae1ec-e90c-422e-84b7-540932f427dc-metrics-certs\") pod \"maas-api-e2e-explicit-459a4c-9fffb4c8d-9tvlh\" (UID: \"fd5ae1ec-e90c-422e-84b7-540932f427dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-459a4c-9fffb4c8d-9tvlh"
Sep 11 03:55:25.512550 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:25.512512    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fd5ae1ec-e90c-422e-84b7-540932f427dc-maas-api-tls\") pod \"maas-api-e2e-explicit-459a4c-9fffb4c8d-9tvlh\" (UID: \"fd5ae1ec-e90c-422e-84b7-540932f427dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-459a4c-9fffb4c8d-9tvlh"
Sep 11 03:55:25.512738 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:25.512562    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd5ae1ec-e90c-422e-84b7-540932f427dc-openshift-service-ca\") pod \"maas-api-e2e-explicit-459a4c-9fffb4c8d-9tvlh\" (UID: \"fd5ae1ec-e90c-422e-84b7-540932f427dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-459a4c-9fffb4c8d-9tvlh"
Sep 11 03:55:25.512738 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:25.512597    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd5ae1ec-e90c-422e-84b7-540932f427dc-metrics-certs\") pod \"maas-api-e2e-explicit-459a4c-9fffb4c8d-9tvlh\" (UID: \"fd5ae1ec-e90c-422e-84b7-540932f427dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-459a4c-9fffb4c8d-9tvlh"
Sep 11 03:55:25.512738 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:25.512657    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spswm\" (UniqueName: \"kubernetes.io/projected/fd5ae1ec-e90c-422e-84b7-540932f427dc-kube-api-access-spswm\") pod \"maas-api-e2e-explicit-459a4c-9fffb4c8d-9tvlh\" (UID: \"fd5ae1ec-e90c-422e-84b7-540932f427dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-459a4c-9fffb4c8d-9tvlh"
Sep 11 03:55:25.512738 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:55:25.512703    2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-explicit-459a4c-serving-cert: secret "maas-api-e2e-explicit-459a4c-serving-cert" not found
Sep 11 03:55:25.512952 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:55:25.512781    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd5ae1ec-e90c-422e-84b7-540932f427dc-maas-api-tls podName:fd5ae1ec-e90c-422e-84b7-540932f427dc nodeName:}" failed. No retries permitted until 2026-09-11 03:55:26.012759605 +0000 UTC m=+914.171245930 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/fd5ae1ec-e90c-422e-84b7-540932f427dc-maas-api-tls") pod "maas-api-e2e-explicit-459a4c-9fffb4c8d-9tvlh" (UID: "fd5ae1ec-e90c-422e-84b7-540932f427dc") : secret "maas-api-e2e-explicit-459a4c-serving-cert" not found
Sep 11 03:55:25.513337 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:25.513319    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd5ae1ec-e90c-422e-84b7-540932f427dc-openshift-service-ca\") pod \"maas-api-e2e-explicit-459a4c-9fffb4c8d-9tvlh\" (UID: \"fd5ae1ec-e90c-422e-84b7-540932f427dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-459a4c-9fffb4c8d-9tvlh"
Sep 11 03:55:25.515176 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:25.515156    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd5ae1ec-e90c-422e-84b7-540932f427dc-metrics-certs\") pod \"maas-api-e2e-explicit-459a4c-9fffb4c8d-9tvlh\" (UID: \"fd5ae1ec-e90c-422e-84b7-540932f427dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-459a4c-9fffb4c8d-9tvlh"
Sep 11 03:55:25.523955 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:25.523923    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-spswm\" (UniqueName: \"kubernetes.io/projected/fd5ae1ec-e90c-422e-84b7-540932f427dc-kube-api-access-spswm\") pod \"maas-api-e2e-explicit-459a4c-9fffb4c8d-9tvlh\" (UID: \"fd5ae1ec-e90c-422e-84b7-540932f427dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-459a4c-9fffb4c8d-9tvlh"
Sep 11 03:55:26.004166 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:26.004133    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-459a4c-66448b8677-b7c55"]
Sep 11 03:55:26.007830 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:26.007807    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-459a4c-66448b8677-b7c55"
Sep 11 03:55:26.010064 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:26.010043    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-459a4c\""
Sep 11 03:55:26.010163 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:26.010089    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-459a4c-dockercfg-gpl6d\""
Sep 11 03:55:26.018263 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:26.018241    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-459a4c-66448b8677-b7c55"]
Sep 11 03:55:26.018468 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:26.018445    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fd5ae1ec-e90c-422e-84b7-540932f427dc-maas-api-tls\") pod \"maas-api-e2e-explicit-459a4c-9fffb4c8d-9tvlh\" (UID: \"fd5ae1ec-e90c-422e-84b7-540932f427dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-459a4c-9fffb4c8d-9tvlh"
Sep 11 03:55:26.020816 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:26.020791    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fd5ae1ec-e90c-422e-84b7-540932f427dc-maas-api-tls\") pod \"maas-api-e2e-explicit-459a4c-9fffb4c8d-9tvlh\" (UID: \"fd5ae1ec-e90c-422e-84b7-540932f427dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-459a4c-9fffb4c8d-9tvlh"
Sep 11 03:55:26.119649 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:26.119605    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc8faf14-960d-4837-8579-7b7abab4489f-plugins-config-volume\") pod \"payload-processing-e2e-explicit-459a4c-66448b8677-b7c55\" (UID: \"fc8faf14-960d-4837-8579-7b7abab4489f\") " pod="openshift-ingress/payload-processing-e2e-explicit-459a4c-66448b8677-b7c55"
Sep 11 03:55:26.119823 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:26.119692    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhpkk\" (UniqueName: \"kubernetes.io/projected/fc8faf14-960d-4837-8579-7b7abab4489f-kube-api-access-mhpkk\") pod \"payload-processing-e2e-explicit-459a4c-66448b8677-b7c55\" (UID: \"fc8faf14-960d-4837-8579-7b7abab4489f\") " pod="openshift-ingress/payload-processing-e2e-explicit-459a4c-66448b8677-b7c55"
Sep 11 03:55:26.220304 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:26.220261    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc8faf14-960d-4837-8579-7b7abab4489f-plugins-config-volume\") pod \"payload-processing-e2e-explicit-459a4c-66448b8677-b7c55\" (UID: \"fc8faf14-960d-4837-8579-7b7abab4489f\") " pod="openshift-ingress/payload-processing-e2e-explicit-459a4c-66448b8677-b7c55"
Sep 11 03:55:26.220736 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:26.220323    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhpkk\" (UniqueName: \"kubernetes.io/projected/fc8faf14-960d-4837-8579-7b7abab4489f-kube-api-access-mhpkk\") pod \"payload-processing-e2e-explicit-459a4c-66448b8677-b7c55\" (UID: \"fc8faf14-960d-4837-8579-7b7abab4489f\") " pod="openshift-ingress/payload-processing-e2e-explicit-459a4c-66448b8677-b7c55"
Sep 11 03:55:26.221027 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:26.221004    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc8faf14-960d-4837-8579-7b7abab4489f-plugins-config-volume\") pod \"payload-processing-e2e-explicit-459a4c-66448b8677-b7c55\" (UID: \"fc8faf14-960d-4837-8579-7b7abab4489f\") " pod="openshift-ingress/payload-processing-e2e-explicit-459a4c-66448b8677-b7c55"
Sep 11 03:55:26.227145 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:26.227120    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhpkk\" (UniqueName: \"kubernetes.io/projected/fc8faf14-960d-4837-8579-7b7abab4489f-kube-api-access-mhpkk\") pod \"payload-processing-e2e-explicit-459a4c-66448b8677-b7c55\" (UID: \"fc8faf14-960d-4837-8579-7b7abab4489f\") " pod="openshift-ingress/payload-processing-e2e-explicit-459a4c-66448b8677-b7c55"
Sep 11 03:55:26.230008 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:26.229988    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-459a4c-9fffb4c8d-9tvlh"
Sep 11 03:55:26.319045 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:26.319013    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-459a4c-66448b8677-b7c55"
Sep 11 03:55:26.502599 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:26.502570    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-459a4c-9fffb4c8d-9tvlh"]
Sep 11 03:55:26.504498 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:55:26.504465    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd5ae1ec_e90c_422e_84b7_540932f427dc.slice/crio-5c8d67befbf6070d7338212cc8908fa4baa41efd5505209585aa5596d6d7df44 WatchSource:0}: Error finding container 5c8d67befbf6070d7338212cc8908fa4baa41efd5505209585aa5596d6d7df44: Status 404 returned error can't find the container with id 5c8d67befbf6070d7338212cc8908fa4baa41efd5505209585aa5596d6d7df44
Sep 11 03:55:26.592717 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:26.592689    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-459a4c-66448b8677-b7c55"]
Sep 11 03:55:26.594536 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:55:26.594502    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc8faf14_960d_4837_8579_7b7abab4489f.slice/crio-29d081e800dd4a3d346a8cb9ac6ae425f7a49468c540b81e2d5caf728fc1a734 WatchSource:0}: Error finding container 29d081e800dd4a3d346a8cb9ac6ae425f7a49468c540b81e2d5caf728fc1a734: Status 404 returned error can't find the container with id 29d081e800dd4a3d346a8cb9ac6ae425f7a49468c540b81e2d5caf728fc1a734
Sep 11 03:55:26.871370 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:26.871263    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-459a4c-66448b8677-b7c55" event={"ID":"fc8faf14-960d-4837-8579-7b7abab4489f","Type":"ContainerStarted","Data":"a975c47bd2ceeff3f2a1da0594c4ac86f06f422c2f79ecf9707fee7dcb1f6fac"}
Sep 11 03:55:26.871370 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:26.871324    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-459a4c-66448b8677-b7c55" event={"ID":"fc8faf14-960d-4837-8579-7b7abab4489f","Type":"ContainerStarted","Data":"29d081e800dd4a3d346a8cb9ac6ae425f7a49468c540b81e2d5caf728fc1a734"}
Sep 11 03:55:26.871370 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:26.871370    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-explicit-459a4c-66448b8677-b7c55"
Sep 11 03:55:26.872621 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:26.872596    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-459a4c-9fffb4c8d-9tvlh" event={"ID":"fd5ae1ec-e90c-422e-84b7-540932f427dc","Type":"ContainerStarted","Data":"5c8d67befbf6070d7338212cc8908fa4baa41efd5505209585aa5596d6d7df44"}
Sep 11 03:55:26.888449 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:26.888401    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-explicit-459a4c-66448b8677-b7c55" podStartSLOduration=1.888384588 podStartE2EDuration="1.888384588s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 03:55:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:55:26.887639361 +0000 UTC m=+915.046125685" watchObservedRunningTime="2026-09-11 03:55:26.888384588 +0000 UTC m=+915.046870910"
Sep 11 03:55:27.020111 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:55:27.020080    2580 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8324da1_e47d_4d1f_8307_c39ceb847d8f.slice/crio-0973d94198f2c8522bdb830788aa14960dd09e9a4e436d3aed76d3f04cf2a95b.scope\": RecentStats: unable to find data in memory cache]"
Sep 11 03:55:27.880426 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:27.880381    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-459a4c-9fffb4c8d-9tvlh" event={"ID":"fd5ae1ec-e90c-422e-84b7-540932f427dc","Type":"ContainerStarted","Data":"0a79df6c6e5458b0137532264dd590f9807fc4bb7efb65492764fe4f05ef176b"}
Sep 11 03:55:27.897134 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:27.897074    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-459a4c-9fffb4c8d-9tvlh" podStartSLOduration=2.453211551 podStartE2EDuration="2.897059794s" totalImagesPullingTime="443.848243ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 03:55:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:55:27.895185226 +0000 UTC m=+916.053671576" watchObservedRunningTime="2026-09-11 03:55:27.897059794 +0000 UTC m=+916.055546117"
Sep 11 03:55:28.885762 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:28.885734    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-459a4c-9fffb4c8d-9tvlh"
Sep 11 03:55:34.706230 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:55:34.706193    2580 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8324da1_e47d_4d1f_8307_c39ceb847d8f.slice/crio-0973d94198f2c8522bdb830788aa14960dd09e9a4e436d3aed76d3f04cf2a95b.scope\": RecentStats: unable to find data in memory cache]"
Sep 11 03:55:34.706712 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:55:34.706232    2580 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8324da1_e47d_4d1f_8307_c39ceb847d8f.slice/crio-0973d94198f2c8522bdb830788aa14960dd09e9a4e436d3aed76d3f04cf2a95b.scope\": RecentStats: unable to find data in memory cache]"
Sep 11 03:55:34.896471 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:34.896441    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-459a4c-9fffb4c8d-9tvlh"
Sep 11 03:55:35.146992 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:55:35.146908    2580 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8324da1_e47d_4d1f_8307_c39ceb847d8f.slice/crio-0973d94198f2c8522bdb830788aa14960dd09e9a4e436d3aed76d3f04cf2a95b.scope\": RecentStats: unable to find data in memory cache]"
Sep 11 03:55:37.882828 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:37.882794    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-explicit-459a4c-66448b8677-b7c55"
Sep 11 03:55:41.955542 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:41.955511    2580 generic.go:356] "Generic (PLEG): container finished" podID="a0821739-382a-4dac-8ca9-c08841d1031c" containerID="c10ebc83988e735cfe38f6c076e73fb8830bbface90741e24ab314624b9aff1f" exitCode=0
Sep 11 03:55:41.955982 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:41.955583    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-459a4c-openshift-default-977b8764d-bsptj" event={"ID":"a0821739-382a-4dac-8ca9-c08841d1031c","Type":"ContainerDied","Data":"c10ebc83988e735cfe38f6c076e73fb8830bbface90741e24ab314624b9aff1f"}
Sep 11 03:55:41.956029 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:41.955999    2580 scope.go:117] "RemoveContainer" containerID="c10ebc83988e735cfe38f6c076e73fb8830bbface90741e24ab314624b9aff1f"
Sep 11 03:55:41.957109 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:41.957077    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 03:55:41.957173 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:41.957146    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 03:55:41.957211 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:41.957190    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 03:55:42.073872 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:55:42.073841    2580 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8324da1_e47d_4d1f_8307_c39ceb847d8f.slice/crio-0973d94198f2c8522bdb830788aa14960dd09e9a4e436d3aed76d3f04cf2a95b.scope\": RecentStats: unable to find data in memory cache]"
Sep 11 03:55:42.967254 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:42.967214    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-459a4c-openshift-default-977b8764d-bsptj" event={"ID":"a0821739-382a-4dac-8ca9-c08841d1031c","Type":"ContainerStarted","Data":"469c2d3c8ef9d28a684ca6b2a127efb2c23df0adc1f9e694a63a917734a4ca24"}
Sep 11 03:55:43.731259 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:43.731225    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-459a4c-openshift-default-977b8764d-bsptj"
Sep 11 03:55:43.731463 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:43.731440    2580 patch_prober.go:28] interesting pod/e2e-explicit-459a4c-openshift-default-977b8764d-bsptj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Sep 11 03:55:43.731529 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:43.731490    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-459a4c-openshift-default-977b8764d-bsptj" podUID="a0821739-382a-4dac-8ca9-c08841d1031c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Sep 11 03:55:44.731487 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:44.731448    2580 patch_prober.go:28] interesting pod/e2e-explicit-459a4c-openshift-default-977b8764d-bsptj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Sep 11 03:55:44.731907 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:44.731507    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-459a4c-openshift-default-977b8764d-bsptj" podUID="a0821739-382a-4dac-8ca9-c08841d1031c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Sep 11 03:55:45.159412 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:55:45.159362    2580 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8324da1_e47d_4d1f_8307_c39ceb847d8f.slice/crio-0973d94198f2c8522bdb830788aa14960dd09e9a4e436d3aed76d3f04cf2a95b.scope\": RecentStats: unable to find data in memory cache]"
Sep 11 03:55:45.731014 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:45.730975    2580 patch_prober.go:28] interesting pod/e2e-explicit-459a4c-openshift-default-977b8764d-bsptj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Sep 11 03:55:45.731196 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:45.731034    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-459a4c-openshift-default-977b8764d-bsptj" podUID="a0821739-382a-4dac-8ca9-c08841d1031c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Sep 11 03:55:46.731022 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:46.730984    2580 patch_prober.go:28] interesting pod/e2e-explicit-459a4c-openshift-default-977b8764d-bsptj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Sep 11 03:55:46.731496 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:46.731047    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-459a4c-openshift-default-977b8764d-bsptj" podUID="a0821739-382a-4dac-8ca9-c08841d1031c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Sep 11 03:55:47.731252 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:47.731215    2580 patch_prober.go:28] interesting pod/e2e-explicit-459a4c-openshift-default-977b8764d-bsptj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Sep 11 03:55:47.731644 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:47.731276    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-459a4c-openshift-default-977b8764d-bsptj" podUID="a0821739-382a-4dac-8ca9-c08841d1031c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Sep 11 03:55:48.735287 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:48.735251    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-459a4c-openshift-default-977b8764d-bsptj"
Sep 11 03:55:48.735738 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:48.735522    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-459a4c-openshift-default-977b8764d-bsptj"
Sep 11 03:55:48.736133 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:55:48.736118    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-459a4c-openshift-default-977b8764d-bsptj"
Sep 11 03:55:55.214432 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:55:55.214390    2580 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8324da1_e47d_4d1f_8307_c39ceb847d8f.slice/crio-0973d94198f2c8522bdb830788aa14960dd09e9a4e436d3aed76d3f04cf2a95b.scope\": RecentStats: unable to find data in memory cache]"
Sep 11 03:55:57.019296 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:55:57.019252    2580 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8324da1_e47d_4d1f_8307_c39ceb847d8f.slice/crio-0973d94198f2c8522bdb830788aa14960dd09e9a4e436d3aed76d3f04cf2a95b.scope\": RecentStats: unable to find data in memory cache]"
Sep 11 03:56:05.225591 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:56:05.225547    2580 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8324da1_e47d_4d1f_8307_c39ceb847d8f.slice/crio-0973d94198f2c8522bdb830788aa14960dd09e9a4e436d3aed76d3f04cf2a95b.scope\": RecentStats: unable to find data in memory cache]"
Sep 11 03:56:12.072198 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:56:12.072162    2580 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8324da1_e47d_4d1f_8307_c39ceb847d8f.slice/crio-0973d94198f2c8522bdb830788aa14960dd09e9a4e436d3aed76d3f04cf2a95b.scope\": RecentStats: unable to find data in memory cache]"
Sep 11 03:56:27.209524 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:56:27.209480    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-459a4c-9fffb4c8d-9tvlh"]
Sep 11 03:56:27.210058 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:56:27.209731    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-459a4c-9fffb4c8d-9tvlh" podUID="fd5ae1ec-e90c-422e-84b7-540932f427dc" containerName="maas-api" containerID="cri-o://0a79df6c6e5458b0137532264dd590f9807fc4bb7efb65492764fe4f05ef176b" gracePeriod=30
Sep 11 03:56:27.263963 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:56:27.263929    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-459a4c-66448b8677-b7c55"]
Sep 11 03:56:27.264172 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:56:27.264145    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-explicit-459a4c-66448b8677-b7c55" podUID="fc8faf14-960d-4837-8579-7b7abab4489f" containerName="payload-processing" containerID="cri-o://a975c47bd2ceeff3f2a1da0594c4ac86f06f422c2f79ecf9707fee7dcb1f6fac" gracePeriod=30
Sep 11 03:56:27.527389 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:56:27.527358    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-459a4c-9fffb4c8d-9tvlh"
Sep 11 03:56:27.556430 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:56:27.556156    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-459a4c-66448b8677-b7c55"
Sep 11 03:56:27.626111 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:56:27.626076    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mhpkk\" (UniqueName: \"kubernetes.io/projected/fc8faf14-960d-4837-8579-7b7abab4489f-kube-api-access-mhpkk\") pod \"fc8faf14-960d-4837-8579-7b7abab4489f\" (UID: \"fc8faf14-960d-4837-8579-7b7abab4489f\") "
Sep 11 03:56:27.626278 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:56:27.626132    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-spswm\" (UniqueName: \"kubernetes.io/projected/fd5ae1ec-e90c-422e-84b7-540932f427dc-kube-api-access-spswm\") pod \"fd5ae1ec-e90c-422e-84b7-540932f427dc\" (UID: \"fd5ae1ec-e90c-422e-84b7-540932f427dc\") "
Sep 11 03:56:27.626278 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:56:27.626195    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fd5ae1ec-e90c-422e-84b7-540932f427dc-maas-api-tls\") pod \"fd5ae1ec-e90c-422e-84b7-540932f427dc\" (UID: \"fd5ae1ec-e90c-422e-84b7-540932f427dc\") "
Sep 11 03:56:27.626278 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:56:27.626215    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc8faf14-960d-4837-8579-7b7abab4489f-plugins-config-volume\") pod \"fc8faf14-960d-4837-8579-7b7abab4489f\" (UID: \"fc8faf14-960d-4837-8579-7b7abab4489f\") "
Sep 11 03:56:27.626278 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:56:27.626245    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd5ae1ec-e90c-422e-84b7-540932f427dc-openshift-service-ca\") pod \"fd5ae1ec-e90c-422e-84b7-540932f427dc\" (UID: \"fd5ae1ec-e90c-422e-84b7-540932f427dc\") "
Sep 11 03:56:27.626278 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:56:27.626270    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd5ae1ec-e90c-422e-84b7-540932f427dc-metrics-certs\") pod \"fd5ae1ec-e90c-422e-84b7-540932f427dc\" (UID: \"fd5ae1ec-e90c-422e-84b7-540932f427dc\") "
Sep 11 03:56:27.626638 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:56:27.626602    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd5ae1ec-e90c-422e-84b7-540932f427dc-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "fd5ae1ec-e90c-422e-84b7-540932f427dc" (UID: "fd5ae1ec-e90c-422e-84b7-540932f427dc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 03:56:27.626638 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:56:27.626619    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc8faf14-960d-4837-8579-7b7abab4489f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "fc8faf14-960d-4837-8579-7b7abab4489f" (UID: "fc8faf14-960d-4837-8579-7b7abab4489f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 03:56:27.628290 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:56:27.628264    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd5ae1ec-e90c-422e-84b7-540932f427dc-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "fd5ae1ec-e90c-422e-84b7-540932f427dc" (UID: "fd5ae1ec-e90c-422e-84b7-540932f427dc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 03:56:27.628290 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:56:27.628274    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd5ae1ec-e90c-422e-84b7-540932f427dc-kube-api-access-spswm" (OuterVolumeSpecName: "kube-api-access-spswm") pod "fd5ae1ec-e90c-422e-84b7-540932f427dc" (UID: "fd5ae1ec-e90c-422e-84b7-540932f427dc"). InnerVolumeSpecName "kube-api-access-spswm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 03:56:27.628428 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:56:27.628313    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd5ae1ec-e90c-422e-84b7-540932f427dc-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "fd5ae1ec-e90c-422e-84b7-540932f427dc" (UID: "fd5ae1ec-e90c-422e-84b7-540932f427dc"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 03:56:27.628428 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:56:27.628324    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc8faf14-960d-4837-8579-7b7abab4489f-kube-api-access-mhpkk" (OuterVolumeSpecName: "kube-api-access-mhpkk") pod "fc8faf14-960d-4837-8579-7b7abab4489f" (UID: "fc8faf14-960d-4837-8579-7b7abab4489f"). InnerVolumeSpecName "kube-api-access-mhpkk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 03:56:27.727193 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:56:27.727146    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mhpkk\" (UniqueName: \"kubernetes.io/projected/fc8faf14-960d-4837-8579-7b7abab4489f-kube-api-access-mhpkk\") on node \"ip-10-0-138-121.ec2.internal\" DevicePath \"\""
Sep 11 03:56:27.727193 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:56:27.727185    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-spswm\" (UniqueName: \"kubernetes.io/projected/fd5ae1ec-e90c-422e-84b7-540932f427dc-kube-api-access-spswm\") on node \"ip-10-0-138-121.ec2.internal\" DevicePath \"\""
Sep 11 03:56:27.727193 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:56:27.727196    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fd5ae1ec-e90c-422e-84b7-540932f427dc-maas-api-tls\") on node \"ip-10-0-138-121.ec2.internal\" DevicePath \"\""
Sep 11 03:56:27.727193 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:56:27.727204    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc8faf14-960d-4837-8579-7b7abab4489f-plugins-config-volume\") on node \"ip-10-0-138-121.ec2.internal\" DevicePath \"\""
Sep 11 03:56:27.727487 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:56:27.727213    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd5ae1ec-e90c-422e-84b7-540932f427dc-openshift-service-ca\") on node \"ip-10-0-138-121.ec2.internal\" DevicePath \"\""
Sep 11 03:56:27.727487 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:56:27.727221    2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd5ae1ec-e90c-422e-84b7-540932f427dc-metrics-certs\") on node \"ip-10-0-138-121.ec2.internal\" DevicePath \"\""
Sep 11 03:56:28.197838 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:56:28.197801    2580 generic.go:356] "Generic (PLEG): container finished" podID="fc8faf14-960d-4837-8579-7b7abab4489f" containerID="a975c47bd2ceeff3f2a1da0594c4ac86f06f422c2f79ecf9707fee7dcb1f6fac" exitCode=0
Sep 11 03:56:28.198029 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:56:28.197862    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-459a4c-66448b8677-b7c55"
Sep 11 03:56:28.198029 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:56:28.197892    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-459a4c-66448b8677-b7c55" event={"ID":"fc8faf14-960d-4837-8579-7b7abab4489f","Type":"ContainerDied","Data":"a975c47bd2ceeff3f2a1da0594c4ac86f06f422c2f79ecf9707fee7dcb1f6fac"}
Sep 11 03:56:28.198029 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:56:28.197949    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-459a4c-66448b8677-b7c55" event={"ID":"fc8faf14-960d-4837-8579-7b7abab4489f","Type":"ContainerDied","Data":"29d081e800dd4a3d346a8cb9ac6ae425f7a49468c540b81e2d5caf728fc1a734"}
Sep 11 03:56:28.198029 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:56:28.197974    2580 scope.go:117] "RemoveContainer" containerID="a975c47bd2ceeff3f2a1da0594c4ac86f06f422c2f79ecf9707fee7dcb1f6fac"
Sep 11 03:56:28.199367 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:56:28.199343    2580 generic.go:356] "Generic (PLEG): container finished" podID="fd5ae1ec-e90c-422e-84b7-540932f427dc" containerID="0a79df6c6e5458b0137532264dd590f9807fc4bb7efb65492764fe4f05ef176b" exitCode=0
Sep 11 03:56:28.199441 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:56:28.199384    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-459a4c-9fffb4c8d-9tvlh"
Sep 11 03:56:28.199441 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:56:28.199384    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-459a4c-9fffb4c8d-9tvlh" event={"ID":"fd5ae1ec-e90c-422e-84b7-540932f427dc","Type":"ContainerDied","Data":"0a79df6c6e5458b0137532264dd590f9807fc4bb7efb65492764fe4f05ef176b"}
Sep 11 03:56:28.199441 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:56:28.199432    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-459a4c-9fffb4c8d-9tvlh" event={"ID":"fd5ae1ec-e90c-422e-84b7-540932f427dc","Type":"ContainerDied","Data":"5c8d67befbf6070d7338212cc8908fa4baa41efd5505209585aa5596d6d7df44"}
Sep 11 03:56:28.207609 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:56:28.207589    2580 scope.go:117] "RemoveContainer" containerID="a975c47bd2ceeff3f2a1da0594c4ac86f06f422c2f79ecf9707fee7dcb1f6fac"
Sep 11 03:56:28.207937 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:56:28.207918    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a975c47bd2ceeff3f2a1da0594c4ac86f06f422c2f79ecf9707fee7dcb1f6fac\": container with ID starting with a975c47bd2ceeff3f2a1da0594c4ac86f06f422c2f79ecf9707fee7dcb1f6fac not found: ID does not exist" containerID="a975c47bd2ceeff3f2a1da0594c4ac86f06f422c2f79ecf9707fee7dcb1f6fac"
Sep 11 03:56:28.208011 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:56:28.207945    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a975c47bd2ceeff3f2a1da0594c4ac86f06f422c2f79ecf9707fee7dcb1f6fac"} err="failed to get container status \"a975c47bd2ceeff3f2a1da0594c4ac86f06f422c2f79ecf9707fee7dcb1f6fac\": rpc error: code = NotFound desc = could not find container \"a975c47bd2ceeff3f2a1da0594c4ac86f06f422c2f79ecf9707fee7dcb1f6fac\": container with ID starting with a975c47bd2ceeff3f2a1da0594c4ac86f06f422c2f79ecf9707fee7dcb1f6fac not found: ID does not exist"
Sep 11 03:56:28.208011 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:56:28.207960    2580 scope.go:117] "RemoveContainer" containerID="0a79df6c6e5458b0137532264dd590f9807fc4bb7efb65492764fe4f05ef176b"
Sep 11 03:56:28.216504 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:56:28.216330    2580 scope.go:117] "RemoveContainer" containerID="0a79df6c6e5458b0137532264dd590f9807fc4bb7efb65492764fe4f05ef176b"
Sep 11 03:56:28.216763 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:56:28.216633    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a79df6c6e5458b0137532264dd590f9807fc4bb7efb65492764fe4f05ef176b\": container with ID starting with 0a79df6c6e5458b0137532264dd590f9807fc4bb7efb65492764fe4f05ef176b not found: ID does not exist" containerID="0a79df6c6e5458b0137532264dd590f9807fc4bb7efb65492764fe4f05ef176b"
Sep 11 03:56:28.216763 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:56:28.216685    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a79df6c6e5458b0137532264dd590f9807fc4bb7efb65492764fe4f05ef176b"} err="failed to get container status \"0a79df6c6e5458b0137532264dd590f9807fc4bb7efb65492764fe4f05ef176b\": rpc error: code = NotFound desc = could not find container \"0a79df6c6e5458b0137532264dd590f9807fc4bb7efb65492764fe4f05ef176b\": container with ID starting with 0a79df6c6e5458b0137532264dd590f9807fc4bb7efb65492764fe4f05ef176b not found: ID does not exist"
Sep 11 03:56:28.221778 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:56:28.221752    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-459a4c-66448b8677-b7c55"]
Sep 11 03:56:28.225254 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:56:28.224992    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-459a4c-66448b8677-b7c55"]
Sep 11 03:56:28.236081 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:56:28.236058    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-459a4c-9fffb4c8d-9tvlh"]
Sep 11 03:56:28.240065 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:56:28.240046    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-459a4c-9fffb4c8d-9tvlh"]
Sep 11 03:56:28.463806 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:56:28.463726    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fc8faf14-960d-4837-8579-7b7abab4489f" path="/var/lib/kubelet/pods/fc8faf14-960d-4837-8579-7b7abab4489f/volumes"
Sep 11 03:56:28.464156 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:56:28.464144    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fd5ae1ec-e90c-422e-84b7-540932f427dc" path="/var/lib/kubelet/pods/fd5ae1ec-e90c-422e-84b7-540932f427dc/volumes"
Sep 11 03:56:39.057966 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:56:39.057880    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-459a4c-openshift-default-977b8764d-bsptj"]
Sep 11 03:56:39.058440 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:56:39.058113    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-explicit-459a4c-openshift-default-977b8764d-bsptj" podUID="a0821739-382a-4dac-8ca9-c08841d1031c" containerName="istio-proxy" containerID="cri-o://469c2d3c8ef9d28a684ca6b2a127efb2c23df0adc1f9e694a63a917734a4ca24" gracePeriod=30
Sep 11 03:56:39.819142 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:56:39.819105    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-noten-96c7d4-openshift-default-785b65d79d-j8777"]
Sep 11 03:56:39.819624 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:56:39.819608    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd5ae1ec-e90c-422e-84b7-540932f427dc" containerName="maas-api"
Sep 11 03:56:39.819624 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:56:39.819624    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd5ae1ec-e90c-422e-84b7-540932f427dc" containerName="maas-api"
Sep 11 03:56:39.819787 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:56:39.819636    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc8faf14-960d-4837-8579-7b7abab4489f" containerName="payload-processing"
Sep 11 03:56:39.819787 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:56:39.819642    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc8faf14-960d-4837-8579-7b7abab4489f" containerName="payload-processing"
Sep 11 03:56:39.819787 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:56:39.819738    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="fc8faf14-960d-4837-8579-7b7abab4489f" containerName="payload-processing"
Sep 11 03:56:39.819787 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:56:39.819749    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="fd5ae1ec-e90c-422e-84b7-540932f427dc" containerName="maas-api"
Sep 11 03:56:39.823152 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:56:39.823130    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-96c7d4-openshift-default-785b65d79d-j8777"
Sep 11 03:56:39.825409 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:56:39.825378    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-noten-96c7d4-openshift-default-dockercfg-g99xl\""
Sep 11 03:56:39.830606 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:56:39.830578    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-96c7d4-openshift-default-785b65d79d-j8777"]
Sep 11 03:56:39.947365 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:56:39.947320    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13d95f4c-f162-4072-acf6-c2716bdbb893-istio-envoy\") pod \"e2e-noten-96c7d4-openshift-default-785b65d79d-j8777\" (UID: \"13d95f4c-f162-4072-acf6-c2716bdbb893\") " pod="openshift-ingress/e2e-noten-96c7d4-openshift-default-785b65d79d-j8777"
Sep 11 03:56:39.947648 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:56:39.947623    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13d95f4c-f162-4072-acf6-c2716bdbb893-istio-data\") pod \"e2e-noten-96c7d4-openshift-default-785b65d79d-j8777\" (UID: \"13d95f4c-f162-4072-acf6-c2716bdbb893\") " pod="openshift-ingress/e2e-noten-96c7d4-openshift-default-785b65d79d-j8777"
Sep 11 03:56:39.947846 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:56:39.947798    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13d95f4c-f162-4072-acf6-c2716bdbb893-workload-certs\") pod \"e2e-noten-96c7d4-openshift-default-785b65d79d-j8777\" (UID: \"13d95f4c-f162-4072-acf6-c2716bdbb893\") " pod="openshift-ingress/e2e-noten-96c7d4-openshift-default-785b65d79d-j8777"
Sep 11 03:56:39.947978 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:56:39.947962    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66w4k\" (UniqueName: \"kubernetes.io/projected/13d95f4c-f162-4072-acf6-c2716bdbb893-kube-api-access-66w4k\") pod \"e2e-noten-96c7d4-openshift-default-785b65d79d-j8777\" (UID: \"13d95f4c-f162-4072-acf6-c2716bdbb893\") " pod="openshift-ingress/e2e-noten-96c7d4-openshift-default-785b65d79d-j8777"
Sep 11 03:56:39.948108 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:56:39.948094    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/13d95f4c-f162-4072-acf6-c2716bdbb893-istio-token\") pod \"e2e-noten-96c7d4-openshift-default-785b65d79d-j8777\" (UID: \"13d95f4c-f162-4072-acf6-c2716bdbb893\") " pod="openshift-ingress/e2e-noten-96c7d4-openshift-default-785b65d79d-j8777"
Sep 11 03:56:39.948228 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:56:39.948215    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13d95f4c-f162-4072-acf6-c2716bdbb893-credential-socket\") pod \"e2e-noten-96c7d4-openshift-default-785b65d79d-j8777\" (UID: \"13d95f4c-f162-4072-acf6-c2716bdbb893\") " pod="openshift-ingress/e2e-noten-96c7d4-openshift-default-785b65d79d-j8777"
Sep 11 03:56:39.948329 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:56:39.948317    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13d95f4c-f162-4072-acf6-c2716bdbb893-workload-socket\") pod \"e2e-noten-96c7d4-openshift-default-785b65d79d-j8777\" (UID: \"13d95f4c-f162-4072-acf6-c2716bdbb893\") " pod="openshift-ingress/e2e-noten-96c7d4-openshift-default-785b65d79d-j8777"
Sep 11 03:56:39.948452 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:56:39.948440    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13d95f4c-f162-4072-acf6-c2716bdbb893-istiod-ca-cert\") pod \"e2e-noten-96c7d4-openshift-default-785b65d79d-j8777\" (UID: \"13d95f4c-f162-4072-acf6-c2716bdbb893\") " pod="openshift-ingress/e2e-noten-96c7d4-openshift-default-785b65d79d-j8777"
Sep 11 03:56:39.948556 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:56:39.948544    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13d95f4c-f162-4072-acf6-c2716bdbb893-istio-podinfo\") pod \"e2e-noten-96c7d4-openshift-default-785b65d79d-j8777\" (UID: \"13d95f4c-f162-4072-acf6-c2716bdbb893\") " pod="openshift-ingress/e2e-noten-96c7d4-openshift-default-785b65d79d-j8777"
Sep 11 03:56:40.049893 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:56:40.049858    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13d95f4c-f162-4072-acf6-c2716bdbb893-istio-data\") pod \"e2e-noten-96c7d4-openshift-default-785b65d79d-j8777\" (UID: \"13d95f4c-f162-4072-acf6-c2716bdbb893\") " pod="openshift-ingress/e2e-noten-96c7d4-openshift-default-785b65d79d-j8777"
Sep 11 03:56:40.050102 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:56:40.049906    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13d95f4c-f162-4072-acf6-c2716bdbb893-workload-certs\") pod \"e2e-noten-96c7d4-openshift-default-785b65d79d-j8777\" (UID: \"13d95f4c-f162-4072-acf6-c2716bdbb893\") " pod="openshift-ingress/e2e-noten-96c7d4-openshift-default-785b65d79d-j8777"
Sep 11 03:56:40.050102 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:56:40.049936    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66w4k\" (UniqueName: \"kubernetes.io/projected/13d95f4c-f162-4072-acf6-c2716bdbb893-kube-api-access-66w4k\") pod \"e2e-noten-96c7d4-openshift-default-785b65d79d-j8777\" (UID: \"13d95f4c-f162-4072-acf6-c2716bdbb893\") " pod="openshift-ingress/e2e-noten-96c7d4-openshift-default-785b65d79d-j8777"
Sep 11 03:56:40.050102 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:56:40.049983    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/13d95f4c-f162-4072-acf6-c2716bdbb893-istio-token\") pod \"e2e-noten-96c7d4-openshift-default-785b65d79d-j8777\" (UID: \"13d95f4c-f162-4072-acf6-c2716bdbb893\") " pod="openshift-ingress/e2e-noten-96c7d4-openshift-default-785b65d79d-j8777"
Sep 11 03:56:40.050102 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:56:40.050006    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13d95f4c-f162-4072-acf6-c2716bdbb893-credential-socket\") pod \"e2e-noten-96c7d4-openshift-default-785b65d79d-j8777\" (UID: \"13d95f4c-f162-4072-acf6-c2716bdbb893\") " pod="openshift-ingress/e2e-noten-96c7d4-openshift-default-785b65d79d-j8777"
Sep 11 03:56:40.050102 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:56:40.050036    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13d95f4c-f162-4072-acf6-c2716bdbb893-workload-socket\") pod \"e2e-noten-96c7d4-openshift-default-785b65d79d-j8777\" (UID: \"13d95f4c-f162-4072-acf6-c2716bdbb893\") " pod="openshift-ingress/e2e-noten-96c7d4-openshift-default-785b65d79d-j8777"
Sep 11 03:56:40.050102 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:56:40.050092    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13d95f4c-f162-4072-acf6-c2716bdbb893-istiod-ca-cert\") pod \"e2e-noten-96c7d4-openshift-default-785b65d79d-j8777\" (UID: \"13d95f4c-f162-4072-acf6-c2716bdbb893\") " pod="openshift-ingress/e2e-noten-96c7d4-openshift-default-785b65d79d-j8777"
Sep 11 03:56:40.050442 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:56:40.050128    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13d95f4c-f162-4072-acf6-c2716bdbb893-istio-podinfo\") pod \"e2e-noten-96c7d4-openshift-default-785b65d79d-j8777\" (UID: \"13d95f4c-f162-4072-acf6-c2716bdbb893\") " pod="openshift-ingress/e2e-noten-96c7d4-openshift-default-785b65d79d-j8777"
Sep 11 03:56:40.050442 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:56:40.050176    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13d95f4c-f162-4072-acf6-c2716bdbb893-istio-envoy\") pod \"e2e-noten-96c7d4-openshift-default-785b65d79d-j8777\" (UID: \"13d95f4c-f162-4072-acf6-c2716bdbb893\") " pod="openshift-ingress/e2e-noten-96c7d4-openshift-default-785b65d79d-j8777"
Sep 11 03:56:40.050442 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:56:40.050420    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13d95f4c-f162-4072-acf6-c2716bdbb893-istio-data\") pod \"e2e-noten-96c7d4-openshift-default-785b65d79d-j8777\" (UID: \"13d95f4c-f162-4072-acf6-c2716bdbb893\") " pod="openshift-ingress/e2e-noten-96c7d4-openshift-default-785b65d79d-j8777"
Sep 11 03:56:40.050618 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:56:40.050468    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13d95f4c-f162-4072-acf6-c2716bdbb893-workload-certs\") pod \"e2e-noten-96c7d4-openshift-default-785b65d79d-j8777\" (UID: \"13d95f4c-f162-4072-acf6-c2716bdbb893\") " pod="openshift-ingress/e2e-noten-96c7d4-openshift-default-785b65d79d-j8777"
Sep 11 03:56:40.050618 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:56:40.050544    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13d95f4c-f162-4072-acf6-c2716bdbb893-workload-socket\") pod \"e2e-noten-96c7d4-openshift-default-785b65d79d-j8777\" (UID: \"13d95f4c-f162-4072-acf6-c2716bdbb893\") " pod="openshift-ingress/e2e-noten-96c7d4-openshift-default-785b65d79d-j8777"
Sep 11 03:56:40.050761 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:56:40.050683    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13d95f4c-f162-4072-acf6-c2716bdbb893-credential-socket\") pod \"e2e-noten-96c7d4-openshift-default-785b65d79d-j8777\" (UID: \"13d95f4c-f162-4072-acf6-c2716bdbb893\") " pod="openshift-ingress/e2e-noten-96c7d4-openshift-default-785b65d79d-j8777"
Sep 11 03:56:40.051117 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:56:40.051091    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13d95f4c-f162-4072-acf6-c2716bdbb893-istiod-ca-cert\") pod \"e2e-noten-96c7d4-openshift-default-785b65d79d-j8777\" (UID: \"13d95f4c-f162-4072-acf6-c2716bdbb893\") " pod="openshift-ingress/e2e-noten-96c7d4-openshift-default-785b65d79d-j8777"
Sep 11 03:56:40.052616 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:56:40.052585    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13d95f4c-f162-4072-acf6-c2716bdbb893-istio-envoy\") pod \"e2e-noten-96c7d4-openshift-default-785b65d79d-j8777\" (UID: \"13d95f4c-f162-4072-acf6-c2716bdbb893\") " pod="openshift-ingress/e2e-noten-96c7d4-openshift-default-785b65d79d-j8777"
Sep 11 03:56:40.053000 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:56:40.052978    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13d95f4c-f162-4072-acf6-c2716bdbb893-istio-podinfo\") pod \"e2e-noten-96c7d4-openshift-default-785b65d79d-j8777\" (UID: \"13d95f4c-f162-4072-acf6-c2716bdbb893\") " pod="openshift-ingress/e2e-noten-96c7d4-openshift-default-785b65d79d-j8777"
Sep 11 03:56:40.057716 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:56:40.057689    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/13d95f4c-f162-4072-acf6-c2716bdbb893-istio-token\") pod \"e2e-noten-96c7d4-openshift-default-785b65d79d-j8777\" (UID: \"13d95f4c-f162-4072-acf6-c2716bdbb893\") " pod="openshift-ingress/e2e-noten-96c7d4-openshift-default-785b65d79d-j8777"
Sep 11 03:56:40.057957 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:56:40.057939    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-66w4k\" (UniqueName: \"kubernetes.io/projected/13d95f4c-f162-4072-acf6-c2716bdbb893-kube-api-access-66w4k\") pod \"e2e-noten-96c7d4-openshift-default-785b65d79d-j8777\" (UID: \"13d95f4c-f162-4072-acf6-c2716bdbb893\") " pod="openshift-ingress/e2e-noten-96c7d4-openshift-default-785b65d79d-j8777"
Sep 11 03:56:40.135220 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:56:40.135131    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-96c7d4-openshift-default-785b65d79d-j8777"
Sep 11 03:56:40.414139 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:56:40.414113    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-96c7d4-openshift-default-785b65d79d-j8777"]
Sep 11 03:56:40.415957 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:56:40.415920    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13d95f4c_f162_4072_acf6_c2716bdbb893.slice/crio-4c1575dd55cd0f7c37fb29dee8fa128d32418ea4ff9bafe8b31a86197987a7ba WatchSource:0}: Error finding container 4c1575dd55cd0f7c37fb29dee8fa128d32418ea4ff9bafe8b31a86197987a7ba: Status 404 returned error can't find the container with id 4c1575dd55cd0f7c37fb29dee8fa128d32418ea4ff9bafe8b31a86197987a7ba
Sep 11 03:56:40.418267 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:56:40.418238    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 03:56:40.418376 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:56:40.418315    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 03:56:40.418376 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:56:40.418359    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 03:56:41.270579 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:56:41.270542    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-96c7d4-openshift-default-785b65d79d-j8777" event={"ID":"13d95f4c-f162-4072-acf6-c2716bdbb893","Type":"ContainerStarted","Data":"f824a43573fec36fb42af01cfd9e8dc1ac4edb147e729ca1f2db91072f001d22"}
Sep 11 03:56:41.271040 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:56:41.270595    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-96c7d4-openshift-default-785b65d79d-j8777" event={"ID":"13d95f4c-f162-4072-acf6-c2716bdbb893","Type":"ContainerStarted","Data":"4c1575dd55cd0f7c37fb29dee8fa128d32418ea4ff9bafe8b31a86197987a7ba"}
Sep 11 03:56:41.288936 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:56:41.288878    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-noten-96c7d4-openshift-default-785b65d79d-j8777" podStartSLOduration=2.288859004 podStartE2EDuration="2.288859004s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 03:56:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:56:41.288265833 +0000 UTC m=+989.446752155" watchObservedRunningTime="2026-09-11 03:56:41.288859004 +0000 UTC m=+989.447345327"
Sep 11 03:56:42.136119 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:56:42.136075    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-noten-96c7d4-openshift-default-785b65d79d-j8777"
Sep 11 03:56:42.141551 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:56:42.141525    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-noten-96c7d4-openshift-default-785b65d79d-j8777"
Sep 11 03:56:42.275267 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:56:42.275234    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-noten-96c7d4-openshift-default-785b65d79d-j8777"
Sep 11 03:56:42.276358 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:56:42.276336    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-noten-96c7d4-openshift-default-785b65d79d-j8777"
Sep 11 03:56:44.286537 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:56:44.286508    2580 generic.go:356] "Generic (PLEG): container finished" podID="a0821739-382a-4dac-8ca9-c08841d1031c" containerID="469c2d3c8ef9d28a684ca6b2a127efb2c23df0adc1f9e694a63a917734a4ca24" exitCode=0
Sep 11 03:56:44.286904 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:56:44.286580    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-459a4c-openshift-default-977b8764d-bsptj" event={"ID":"a0821739-382a-4dac-8ca9-c08841d1031c","Type":"ContainerDied","Data":"469c2d3c8ef9d28a684ca6b2a127efb2c23df0adc1f9e694a63a917734a4ca24"}
Sep 11 03:56:44.286904 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:56:44.286649    2580 scope.go:117] "RemoveContainer" containerID="c10ebc83988e735cfe38f6c076e73fb8830bbface90741e24ab314624b9aff1f"
Sep 11 03:56:44.301143 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:56:44.301121    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c10ebc83988e735cfe38f6c076e73fb8830bbface90741e24ab314624b9aff1f\": container with ID starting with c10ebc83988e735cfe38f6c076e73fb8830bbface90741e24ab314624b9aff1f not found: ID does not exist" containerID="c10ebc83988e735cfe38f6c076e73fb8830bbface90741e24ab314624b9aff1f"
Sep 11 03:56:44.304469 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:56:44.304450    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-459a4c-openshift-default-977b8764d-bsptj"
Sep 11 03:56:44.396404 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:56:44.396308    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0821739-382a-4dac-8ca9-c08841d1031c-istiod-ca-cert\") pod \"a0821739-382a-4dac-8ca9-c08841d1031c\" (UID: \"a0821739-382a-4dac-8ca9-c08841d1031c\") "
Sep 11 03:56:44.396404 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:56:44.396383    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0821739-382a-4dac-8ca9-c08841d1031c-istio-envoy\") pod \"a0821739-382a-4dac-8ca9-c08841d1031c\" (UID: \"a0821739-382a-4dac-8ca9-c08841d1031c\") "
Sep 11 03:56:44.396404 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:56:44.396403    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0821739-382a-4dac-8ca9-c08841d1031c-workload-certs\") pod \"a0821739-382a-4dac-8ca9-c08841d1031c\" (UID: \"a0821739-382a-4dac-8ca9-c08841d1031c\") "
Sep 11 03:56:44.396731 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:56:44.396429    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0821739-382a-4dac-8ca9-c08841d1031c-istio-podinfo\") pod \"a0821739-382a-4dac-8ca9-c08841d1031c\" (UID: \"a0821739-382a-4dac-8ca9-c08841d1031c\") "
Sep 11 03:56:44.396731 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:56:44.396501    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0821739-382a-4dac-8ca9-c08841d1031c-istio-data\") pod \"a0821739-382a-4dac-8ca9-c08841d1031c\" (UID: \"a0821739-382a-4dac-8ca9-c08841d1031c\") "
Sep 11 03:56:44.396731 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:56:44.396529    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0821739-382a-4dac-8ca9-c08841d1031c-credential-socket\") pod \"a0821739-382a-4dac-8ca9-c08841d1031c\" (UID: \"a0821739-382a-4dac-8ca9-c08841d1031c\") "
Sep 11 03:56:44.396731 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:56:44.396548    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0821739-382a-4dac-8ca9-c08841d1031c-workload-socket\") pod \"a0821739-382a-4dac-8ca9-c08841d1031c\" (UID: \"a0821739-382a-4dac-8ca9-c08841d1031c\") "
Sep 11 03:56:44.396731 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:56:44.396593    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jgl5m\" (UniqueName: \"kubernetes.io/projected/a0821739-382a-4dac-8ca9-c08841d1031c-kube-api-access-jgl5m\") pod \"a0821739-382a-4dac-8ca9-c08841d1031c\" (UID: \"a0821739-382a-4dac-8ca9-c08841d1031c\") "
Sep 11 03:56:44.396731 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:56:44.396640    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0821739-382a-4dac-8ca9-c08841d1031c-istio-token\") pod \"a0821739-382a-4dac-8ca9-c08841d1031c\" (UID: \"a0821739-382a-4dac-8ca9-c08841d1031c\") "
Sep 11 03:56:44.397045 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:56:44.396755    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0821739-382a-4dac-8ca9-c08841d1031c-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "a0821739-382a-4dac-8ca9-c08841d1031c" (UID: "a0821739-382a-4dac-8ca9-c08841d1031c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 03:56:44.397045 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:56:44.396826    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0821739-382a-4dac-8ca9-c08841d1031c-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "a0821739-382a-4dac-8ca9-c08841d1031c" (UID: "a0821739-382a-4dac-8ca9-c08841d1031c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 03:56:44.397045 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:56:44.396843    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0821739-382a-4dac-8ca9-c08841d1031c-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "a0821739-382a-4dac-8ca9-c08841d1031c" (UID: "a0821739-382a-4dac-8ca9-c08841d1031c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 03:56:44.397045 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:56:44.396939    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0821739-382a-4dac-8ca9-c08841d1031c-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "a0821739-382a-4dac-8ca9-c08841d1031c" (UID: "a0821739-382a-4dac-8ca9-c08841d1031c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 03:56:44.397278 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:56:44.397101    2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0821739-382a-4dac-8ca9-c08841d1031c-workload-certs\") on node \"ip-10-0-138-121.ec2.internal\" DevicePath \"\""
Sep 11 03:56:44.397278 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:56:44.397125    2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0821739-382a-4dac-8ca9-c08841d1031c-credential-socket\") on node \"ip-10-0-138-121.ec2.internal\" DevicePath \"\""
Sep 11 03:56:44.397278 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:56:44.397142    2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0821739-382a-4dac-8ca9-c08841d1031c-workload-socket\") on node \"ip-10-0-138-121.ec2.internal\" DevicePath \"\""
Sep 11 03:56:44.397278 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:56:44.397159    2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0821739-382a-4dac-8ca9-c08841d1031c-istiod-ca-cert\") on node \"ip-10-0-138-121.ec2.internal\" DevicePath \"\""
Sep 11 03:56:44.397484 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:56:44.397443    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0821739-382a-4dac-8ca9-c08841d1031c-istio-data" (OuterVolumeSpecName: "istio-data") pod "a0821739-382a-4dac-8ca9-c08841d1031c" (UID: "a0821739-382a-4dac-8ca9-c08841d1031c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 03:56:44.398919 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:56:44.398890    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0821739-382a-4dac-8ca9-c08841d1031c-kube-api-access-jgl5m" (OuterVolumeSpecName: "kube-api-access-jgl5m") pod "a0821739-382a-4dac-8ca9-c08841d1031c" (UID: "a0821739-382a-4dac-8ca9-c08841d1031c"). InnerVolumeSpecName "kube-api-access-jgl5m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 03:56:44.399009 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:56:44.398915    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0821739-382a-4dac-8ca9-c08841d1031c-istio-token" (OuterVolumeSpecName: "istio-token") pod "a0821739-382a-4dac-8ca9-c08841d1031c" (UID: "a0821739-382a-4dac-8ca9-c08841d1031c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 03:56:44.399084 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:56:44.399066    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a0821739-382a-4dac-8ca9-c08841d1031c-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "a0821739-382a-4dac-8ca9-c08841d1031c" (UID: "a0821739-382a-4dac-8ca9-c08841d1031c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 03:56:44.399125 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:56:44.399071    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0821739-382a-4dac-8ca9-c08841d1031c-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "a0821739-382a-4dac-8ca9-c08841d1031c" (UID: "a0821739-382a-4dac-8ca9-c08841d1031c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 03:56:44.498462 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:56:44.498423    2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0821739-382a-4dac-8ca9-c08841d1031c-istio-data\") on node \"ip-10-0-138-121.ec2.internal\" DevicePath \"\""
Sep 11 03:56:44.498462 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:56:44.498455    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jgl5m\" (UniqueName: \"kubernetes.io/projected/a0821739-382a-4dac-8ca9-c08841d1031c-kube-api-access-jgl5m\") on node \"ip-10-0-138-121.ec2.internal\" DevicePath \"\""
Sep 11 03:56:44.498462 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:56:44.498470    2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0821739-382a-4dac-8ca9-c08841d1031c-istio-token\") on node \"ip-10-0-138-121.ec2.internal\" DevicePath \"\""
Sep 11 03:56:44.498698 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:56:44.498478    2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0821739-382a-4dac-8ca9-c08841d1031c-istio-envoy\") on node \"ip-10-0-138-121.ec2.internal\" DevicePath \"\""
Sep 11 03:56:44.498698 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:56:44.498487    2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0821739-382a-4dac-8ca9-c08841d1031c-istio-podinfo\") on node \"ip-10-0-138-121.ec2.internal\" DevicePath \"\""
Sep 11 03:56:45.292633 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:56:45.292600    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-459a4c-openshift-default-977b8764d-bsptj"
Sep 11 03:56:45.293120 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:56:45.292639    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-459a4c-openshift-default-977b8764d-bsptj" event={"ID":"a0821739-382a-4dac-8ca9-c08841d1031c","Type":"ContainerDied","Data":"bad55ffad8869fcfa4806e229ef3cd58b8e2cffe1d2cf1bfa762db22467c26b5"}
Sep 11 03:56:45.293120 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:56:45.292723    2580 scope.go:117] "RemoveContainer" containerID="469c2d3c8ef9d28a684ca6b2a127efb2c23df0adc1f9e694a63a917734a4ca24"
Sep 11 03:56:45.308845 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:56:45.308812    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-459a4c-openshift-default-977b8764d-bsptj"]
Sep 11 03:56:45.312326 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:56:45.312302    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-explicit-459a4c-openshift-default-977b8764d-bsptj"]
Sep 11 03:56:46.465090 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:56:46.465050    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a0821739-382a-4dac-8ca9-c08841d1031c" path="/var/lib/kubelet/pods/a0821739-382a-4dac-8ca9-c08841d1031c/volumes"
Sep 11 03:56:47.339313 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:56:47.339278    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-4c62f1-8468fc7f57-x72s8"]
Sep 11 03:56:47.339657 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:56:47.339622    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-4c62f1-8468fc7f57-x72s8" podUID="cc8734e6-1130-4626-9e55-1d10cf9492b7" containerName="payload-processing" containerID="cri-o://747348d41b9cc0424056168c352b64909ff3946a7ae9c5b37a1829e3b4ff00e8" gracePeriod=30
Sep 11 03:56:47.604600 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:56:47.604538    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-4c62f1-8468fc7f57-x72s8"
Sep 11 03:56:47.732977 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:56:47.732939    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc8734e6-1130-4626-9e55-1d10cf9492b7-plugins-config-volume\") pod \"cc8734e6-1130-4626-9e55-1d10cf9492b7\" (UID: \"cc8734e6-1130-4626-9e55-1d10cf9492b7\") "
Sep 11 03:56:47.733180 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:56:47.733067    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pcb4d\" (UniqueName: \"kubernetes.io/projected/cc8734e6-1130-4626-9e55-1d10cf9492b7-kube-api-access-pcb4d\") pod \"cc8734e6-1130-4626-9e55-1d10cf9492b7\" (UID: \"cc8734e6-1130-4626-9e55-1d10cf9492b7\") "
Sep 11 03:56:47.733393 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:56:47.733366    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc8734e6-1130-4626-9e55-1d10cf9492b7-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "cc8734e6-1130-4626-9e55-1d10cf9492b7" (UID: "cc8734e6-1130-4626-9e55-1d10cf9492b7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 03:56:47.735088 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:56:47.735069    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc8734e6-1130-4626-9e55-1d10cf9492b7-kube-api-access-pcb4d" (OuterVolumeSpecName: "kube-api-access-pcb4d") pod "cc8734e6-1130-4626-9e55-1d10cf9492b7" (UID: "cc8734e6-1130-4626-9e55-1d10cf9492b7"). InnerVolumeSpecName "kube-api-access-pcb4d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 03:56:47.834477 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:56:47.834437    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc8734e6-1130-4626-9e55-1d10cf9492b7-plugins-config-volume\") on node \"ip-10-0-138-121.ec2.internal\" DevicePath \"\""
Sep 11 03:56:47.834477 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:56:47.834469    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pcb4d\" (UniqueName: \"kubernetes.io/projected/cc8734e6-1130-4626-9e55-1d10cf9492b7-kube-api-access-pcb4d\") on node \"ip-10-0-138-121.ec2.internal\" DevicePath \"\""
Sep 11 03:56:48.311285 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:56:48.311248    2580 generic.go:356] "Generic (PLEG): container finished" podID="cc8734e6-1130-4626-9e55-1d10cf9492b7" containerID="747348d41b9cc0424056168c352b64909ff3946a7ae9c5b37a1829e3b4ff00e8" exitCode=0
Sep 11 03:56:48.311484 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:56:48.311305    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-4c62f1-8468fc7f57-x72s8"
Sep 11 03:56:48.311484 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:56:48.311332    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-4c62f1-8468fc7f57-x72s8" event={"ID":"cc8734e6-1130-4626-9e55-1d10cf9492b7","Type":"ContainerDied","Data":"747348d41b9cc0424056168c352b64909ff3946a7ae9c5b37a1829e3b4ff00e8"}
Sep 11 03:56:48.311484 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:56:48.311388    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-4c62f1-8468fc7f57-x72s8" event={"ID":"cc8734e6-1130-4626-9e55-1d10cf9492b7","Type":"ContainerDied","Data":"429627c4ed68117fce4ea85529bd50f5199dc6685457624c886201dc7cae3b24"}
Sep 11 03:56:48.311484 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:56:48.311410    2580 scope.go:117] "RemoveContainer" containerID="747348d41b9cc0424056168c352b64909ff3946a7ae9c5b37a1829e3b4ff00e8"
Sep 11 03:56:48.320165 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:56:48.320144    2580 scope.go:117] "RemoveContainer" containerID="747348d41b9cc0424056168c352b64909ff3946a7ae9c5b37a1829e3b4ff00e8"
Sep 11 03:56:48.320437 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:56:48.320415    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"747348d41b9cc0424056168c352b64909ff3946a7ae9c5b37a1829e3b4ff00e8\": container with ID starting with 747348d41b9cc0424056168c352b64909ff3946a7ae9c5b37a1829e3b4ff00e8 not found: ID does not exist" containerID="747348d41b9cc0424056168c352b64909ff3946a7ae9c5b37a1829e3b4ff00e8"
Sep 11 03:56:48.320510 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:56:48.320445    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"747348d41b9cc0424056168c352b64909ff3946a7ae9c5b37a1829e3b4ff00e8"} err="failed to get container status \"747348d41b9cc0424056168c352b64909ff3946a7ae9c5b37a1829e3b4ff00e8\": rpc error: code = NotFound desc = could not find container \"747348d41b9cc0424056168c352b64909ff3946a7ae9c5b37a1829e3b4ff00e8\": container with ID starting with 747348d41b9cc0424056168c352b64909ff3946a7ae9c5b37a1829e3b4ff00e8 not found: ID does not exist"
Sep 11 03:56:48.331082 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:56:48.331053    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-4c62f1-8468fc7f57-x72s8"]
Sep 11 03:56:48.339260 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:56:48.339237    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-4c62f1-8468fc7f57-x72s8"]
Sep 11 03:56:48.464263 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:56:48.464228    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cc8734e6-1130-4626-9e55-1d10cf9492b7" path="/var/lib/kubelet/pods/cc8734e6-1130-4626-9e55-1d10cf9492b7/volumes"
Sep 11 03:56:49.634514 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:56:49.634439    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-4f6c0189-openshift-default-7cd8f458d4-jswgm"]
Sep 11 03:56:49.635040 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:56:49.634965    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a0821739-382a-4dac-8ca9-c08841d1031c" containerName="istio-proxy"
Sep 11 03:56:49.635040 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:56:49.634979    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="a0821739-382a-4dac-8ca9-c08841d1031c" containerName="istio-proxy"
Sep 11 03:56:49.635040 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:56:49.634996    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc8734e6-1130-4626-9e55-1d10cf9492b7" containerName="payload-processing"
Sep 11 03:56:49.635040 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:56:49.635002    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc8734e6-1130-4626-9e55-1d10cf9492b7" containerName="payload-processing"
Sep 11 03:56:49.635040 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:56:49.635014    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a0821739-382a-4dac-8ca9-c08841d1031c" containerName="istio-proxy"
Sep 11 03:56:49.635040 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:56:49.635021    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="a0821739-382a-4dac-8ca9-c08841d1031c" containerName="istio-proxy"
Sep 11 03:56:49.635262 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:56:49.635077    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="a0821739-382a-4dac-8ca9-c08841d1031c" containerName="istio-proxy"
Sep 11 03:56:49.635262 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:56:49.635084    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="a0821739-382a-4dac-8ca9-c08841d1031c" containerName="istio-proxy"
Sep 11 03:56:49.635262 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:56:49.635092    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="cc8734e6-1130-4626-9e55-1d10cf9492b7" containerName="payload-processing"
Sep 11 03:56:49.639075 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:56:49.639054    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-4f6c0189-openshift-default-7cd8f458d4-jswgm"
Sep 11 03:56:49.641438 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:56:49.641413    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-4f6c0189-openshift-default-dockercfg-vwwtw\""
Sep 11 03:56:49.648533 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:56:49.648508    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-4f6c0189-openshift-default-7cd8f458d4-jswgm"]
Sep 11 03:56:49.753468 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:56:49.753424    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18a0c038-ebff-4b9e-8920-fddc1f84bdf5-workload-socket\") pod \"e2e-mt-4f6c0189-openshift-default-7cd8f458d4-jswgm\" (UID: \"18a0c038-ebff-4b9e-8920-fddc1f84bdf5\") " pod="openshift-ingress/e2e-mt-4f6c0189-openshift-default-7cd8f458d4-jswgm"
Sep 11 03:56:49.753710 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:56:49.753494    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18a0c038-ebff-4b9e-8920-fddc1f84bdf5-istio-envoy\") pod \"e2e-mt-4f6c0189-openshift-default-7cd8f458d4-jswgm\" (UID: \"18a0c038-ebff-4b9e-8920-fddc1f84bdf5\") " pod="openshift-ingress/e2e-mt-4f6c0189-openshift-default-7cd8f458d4-jswgm"
Sep 11 03:56:49.753710 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:56:49.753568    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18a0c038-ebff-4b9e-8920-fddc1f84bdf5-workload-certs\") pod \"e2e-mt-4f6c0189-openshift-default-7cd8f458d4-jswgm\" (UID: \"18a0c038-ebff-4b9e-8920-fddc1f84bdf5\") " pod="openshift-ingress/e2e-mt-4f6c0189-openshift-default-7cd8f458d4-jswgm"
Sep 11 03:56:49.753710 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:56:49.753590    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18a0c038-ebff-4b9e-8920-fddc1f84bdf5-istio-data\") pod \"e2e-mt-4f6c0189-openshift-default-7cd8f458d4-jswgm\" (UID: \"18a0c038-ebff-4b9e-8920-fddc1f84bdf5\") " pod="openshift-ingress/e2e-mt-4f6c0189-openshift-default-7cd8f458d4-jswgm"
Sep 11 03:56:49.753710 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:56:49.753617    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18a0c038-ebff-4b9e-8920-fddc1f84bdf5-istiod-ca-cert\") pod \"e2e-mt-4f6c0189-openshift-default-7cd8f458d4-jswgm\" (UID: \"18a0c038-ebff-4b9e-8920-fddc1f84bdf5\") " pod="openshift-ingress/e2e-mt-4f6c0189-openshift-default-7cd8f458d4-jswgm"
Sep 11 03:56:49.753710 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:56:49.753646    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18a0c038-ebff-4b9e-8920-fddc1f84bdf5-credential-socket\") pod \"e2e-mt-4f6c0189-openshift-default-7cd8f458d4-jswgm\" (UID: \"18a0c038-ebff-4b9e-8920-fddc1f84bdf5\") " pod="openshift-ingress/e2e-mt-4f6c0189-openshift-default-7cd8f458d4-jswgm"
Sep 11 03:56:49.753995 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:56:49.753790    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18a0c038-ebff-4b9e-8920-fddc1f84bdf5-istio-token\") pod \"e2e-mt-4f6c0189-openshift-default-7cd8f458d4-jswgm\" (UID: \"18a0c038-ebff-4b9e-8920-fddc1f84bdf5\") " pod="openshift-ingress/e2e-mt-4f6c0189-openshift-default-7cd8f458d4-jswgm"
Sep 11 03:56:49.753995 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:56:49.753851    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18a0c038-ebff-4b9e-8920-fddc1f84bdf5-istio-podinfo\") pod \"e2e-mt-4f6c0189-openshift-default-7cd8f458d4-jswgm\" (UID: \"18a0c038-ebff-4b9e-8920-fddc1f84bdf5\") " pod="openshift-ingress/e2e-mt-4f6c0189-openshift-default-7cd8f458d4-jswgm"
Sep 11 03:56:49.753995 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:56:49.753893    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkr4q\" (UniqueName: \"kubernetes.io/projected/18a0c038-ebff-4b9e-8920-fddc1f84bdf5-kube-api-access-qkr4q\") pod \"e2e-mt-4f6c0189-openshift-default-7cd8f458d4-jswgm\" (UID: \"18a0c038-ebff-4b9e-8920-fddc1f84bdf5\") " pod="openshift-ingress/e2e-mt-4f6c0189-openshift-default-7cd8f458d4-jswgm"
Sep 11 03:56:49.855135 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:56:49.855089    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18a0c038-ebff-4b9e-8920-fddc1f84bdf5-workload-certs\") pod \"e2e-mt-4f6c0189-openshift-default-7cd8f458d4-jswgm\" (UID: \"18a0c038-ebff-4b9e-8920-fddc1f84bdf5\") " pod="openshift-ingress/e2e-mt-4f6c0189-openshift-default-7cd8f458d4-jswgm"
Sep 11 03:56:49.855316 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:56:49.855141    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18a0c038-ebff-4b9e-8920-fddc1f84bdf5-istio-data\") pod \"e2e-mt-4f6c0189-openshift-default-7cd8f458d4-jswgm\" (UID: \"18a0c038-ebff-4b9e-8920-fddc1f84bdf5\") " pod="openshift-ingress/e2e-mt-4f6c0189-openshift-default-7cd8f458d4-jswgm"
Sep 11 03:56:49.855316 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:56:49.855175    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18a0c038-ebff-4b9e-8920-fddc1f84bdf5-istiod-ca-cert\") pod \"e2e-mt-4f6c0189-openshift-default-7cd8f458d4-jswgm\" (UID: \"18a0c038-ebff-4b9e-8920-fddc1f84bdf5\") " pod="openshift-ingress/e2e-mt-4f6c0189-openshift-default-7cd8f458d4-jswgm"
Sep 11 03:56:49.855316 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:56:49.855206    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18a0c038-ebff-4b9e-8920-fddc1f84bdf5-credential-socket\") pod \"e2e-mt-4f6c0189-openshift-default-7cd8f458d4-jswgm\" (UID: \"18a0c038-ebff-4b9e-8920-fddc1f84bdf5\") " pod="openshift-ingress/e2e-mt-4f6c0189-openshift-default-7cd8f458d4-jswgm"
Sep 11 03:56:49.855316 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:56:49.855257    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18a0c038-ebff-4b9e-8920-fddc1f84bdf5-istio-token\") pod \"e2e-mt-4f6c0189-openshift-default-7cd8f458d4-jswgm\" (UID: \"18a0c038-ebff-4b9e-8920-fddc1f84bdf5\") " pod="openshift-ingress/e2e-mt-4f6c0189-openshift-default-7cd8f458d4-jswgm"
Sep 11 03:56:49.855316 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:56:49.855297    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18a0c038-ebff-4b9e-8920-fddc1f84bdf5-istio-podinfo\") pod \"e2e-mt-4f6c0189-openshift-default-7cd8f458d4-jswgm\" (UID: \"18a0c038-ebff-4b9e-8920-fddc1f84bdf5\") " pod="openshift-ingress/e2e-mt-4f6c0189-openshift-default-7cd8f458d4-jswgm"
Sep 11 03:56:49.855599 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:56:49.855324    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkr4q\" (UniqueName: \"kubernetes.io/projected/18a0c038-ebff-4b9e-8920-fddc1f84bdf5-kube-api-access-qkr4q\") pod \"e2e-mt-4f6c0189-openshift-default-7cd8f458d4-jswgm\" (UID: \"18a0c038-ebff-4b9e-8920-fddc1f84bdf5\") " pod="openshift-ingress/e2e-mt-4f6c0189-openshift-default-7cd8f458d4-jswgm"
Sep 11 03:56:49.855599 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:56:49.855434    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18a0c038-ebff-4b9e-8920-fddc1f84bdf5-workload-socket\") pod \"e2e-mt-4f6c0189-openshift-default-7cd8f458d4-jswgm\" (UID: \"18a0c038-ebff-4b9e-8920-fddc1f84bdf5\") " pod="openshift-ingress/e2e-mt-4f6c0189-openshift-default-7cd8f458d4-jswgm"
Sep 11 03:56:49.855599 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:56:49.855486    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18a0c038-ebff-4b9e-8920-fddc1f84bdf5-istio-envoy\") pod \"e2e-mt-4f6c0189-openshift-default-7cd8f458d4-jswgm\" (UID: \"18a0c038-ebff-4b9e-8920-fddc1f84bdf5\") " pod="openshift-ingress/e2e-mt-4f6c0189-openshift-default-7cd8f458d4-jswgm"
Sep 11 03:56:49.855599 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:56:49.855547    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18a0c038-ebff-4b9e-8920-fddc1f84bdf5-workload-certs\") pod \"e2e-mt-4f6c0189-openshift-default-7cd8f458d4-jswgm\" (UID: \"18a0c038-ebff-4b9e-8920-fddc1f84bdf5\") " pod="openshift-ingress/e2e-mt-4f6c0189-openshift-default-7cd8f458d4-jswgm"
Sep 11 03:56:49.855599 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:56:49.855554    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18a0c038-ebff-4b9e-8920-fddc1f84bdf5-credential-socket\") pod \"e2e-mt-4f6c0189-openshift-default-7cd8f458d4-jswgm\" (UID: \"18a0c038-ebff-4b9e-8920-fddc1f84bdf5\") " pod="openshift-ingress/e2e-mt-4f6c0189-openshift-default-7cd8f458d4-jswgm"
Sep 11 03:56:49.855888 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:56:49.855630    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18a0c038-ebff-4b9e-8920-fddc1f84bdf5-istio-data\") pod \"e2e-mt-4f6c0189-openshift-default-7cd8f458d4-jswgm\" (UID: \"18a0c038-ebff-4b9e-8920-fddc1f84bdf5\") " pod="openshift-ingress/e2e-mt-4f6c0189-openshift-default-7cd8f458d4-jswgm"
Sep 11 03:56:49.855947 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:56:49.855897    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18a0c038-ebff-4b9e-8920-fddc1f84bdf5-workload-socket\") pod \"e2e-mt-4f6c0189-openshift-default-7cd8f458d4-jswgm\" (UID: \"18a0c038-ebff-4b9e-8920-fddc1f84bdf5\") " pod="openshift-ingress/e2e-mt-4f6c0189-openshift-default-7cd8f458d4-jswgm"
Sep 11 03:56:49.856040 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:56:49.856021    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18a0c038-ebff-4b9e-8920-fddc1f84bdf5-istiod-ca-cert\") pod \"e2e-mt-4f6c0189-openshift-default-7cd8f458d4-jswgm\" (UID: \"18a0c038-ebff-4b9e-8920-fddc1f84bdf5\") " pod="openshift-ingress/e2e-mt-4f6c0189-openshift-default-7cd8f458d4-jswgm"
Sep 11 03:56:49.857819 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:56:49.857795    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18a0c038-ebff-4b9e-8920-fddc1f84bdf5-istio-envoy\") pod \"e2e-mt-4f6c0189-openshift-default-7cd8f458d4-jswgm\" (UID: \"18a0c038-ebff-4b9e-8920-fddc1f84bdf5\") " pod="openshift-ingress/e2e-mt-4f6c0189-openshift-default-7cd8f458d4-jswgm"
Sep 11 03:56:49.857930 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:56:49.857910    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18a0c038-ebff-4b9e-8920-fddc1f84bdf5-istio-podinfo\") pod \"e2e-mt-4f6c0189-openshift-default-7cd8f458d4-jswgm\" (UID: \"18a0c038-ebff-4b9e-8920-fddc1f84bdf5\") " pod="openshift-ingress/e2e-mt-4f6c0189-openshift-default-7cd8f458d4-jswgm"
Sep 11 03:56:49.862866 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:56:49.862841    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18a0c038-ebff-4b9e-8920-fddc1f84bdf5-istio-token\") pod \"e2e-mt-4f6c0189-openshift-default-7cd8f458d4-jswgm\" (UID: \"18a0c038-ebff-4b9e-8920-fddc1f84bdf5\") " pod="openshift-ingress/e2e-mt-4f6c0189-openshift-default-7cd8f458d4-jswgm"
Sep 11 03:56:49.863020 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:56:49.863002    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkr4q\" (UniqueName: \"kubernetes.io/projected/18a0c038-ebff-4b9e-8920-fddc1f84bdf5-kube-api-access-qkr4q\") pod \"e2e-mt-4f6c0189-openshift-default-7cd8f458d4-jswgm\" (UID: \"18a0c038-ebff-4b9e-8920-fddc1f84bdf5\") " pod="openshift-ingress/e2e-mt-4f6c0189-openshift-default-7cd8f458d4-jswgm"
Sep 11 03:56:49.950443 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:56:49.950402    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-4f6c0189-openshift-default-7cd8f458d4-jswgm"
Sep 11 03:56:50.233827 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:56:50.233778    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-4f6c0189-openshift-default-7cd8f458d4-jswgm"]
Sep 11 03:56:50.236188 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:56:50.236155    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18a0c038_ebff_4b9e_8920_fddc1f84bdf5.slice/crio-b875cfb83cd72569b275b37e6eb1f6aa96971397c7efaa1f7a7523534222d52e WatchSource:0}: Error finding container b875cfb83cd72569b275b37e6eb1f6aa96971397c7efaa1f7a7523534222d52e: Status 404 returned error can't find the container with id b875cfb83cd72569b275b37e6eb1f6aa96971397c7efaa1f7a7523534222d52e
Sep 11 03:56:50.238434 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:56:50.238398    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 03:56:50.238528 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:56:50.238465    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 03:56:50.238528 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:56:50.238514    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 03:56:50.323815 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:56:50.323776    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-4f6c0189-openshift-default-7cd8f458d4-jswgm" event={"ID":"18a0c038-ebff-4b9e-8920-fddc1f84bdf5","Type":"ContainerStarted","Data":"e06ca8f5831ea7968932ac00de5ad3d174dba32951942b7e4cdd7003113cba1e"}
Sep 11 03:56:50.323997 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:56:50.323846    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-4f6c0189-openshift-default-7cd8f458d4-jswgm" event={"ID":"18a0c038-ebff-4b9e-8920-fddc1f84bdf5","Type":"ContainerStarted","Data":"b875cfb83cd72569b275b37e6eb1f6aa96971397c7efaa1f7a7523534222d52e"}
Sep 11 03:56:50.342320 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:56:50.342261    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-4f6c0189-openshift-default-7cd8f458d4-jswgm" podStartSLOduration=1.342242541 podStartE2EDuration="1.342242541s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 03:56:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:56:50.339652882 +0000 UTC m=+998.498139206" watchObservedRunningTime="2026-09-11 03:56:50.342242541 +0000 UTC m=+998.500728865"
Sep 11 03:56:50.951361 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:56:50.951316    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-4f6c0189-openshift-default-7cd8f458d4-jswgm"
Sep 11 03:56:51.955936 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:56:51.955901    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-4f6c0189-openshift-default-7cd8f458d4-jswgm"
Sep 11 03:56:52.335093 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:56:52.334997    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-4f6c0189-openshift-default-7cd8f458d4-jswgm"
Sep 11 03:56:52.335942 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:56:52.335924    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-4f6c0189-openshift-default-7cd8f458d4-jswgm"
Sep 11 03:57:05.334106 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:05.334066    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-8efac4-5df6f86d78-8xm9m"]
Sep 11 03:57:05.334579 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:05.334422    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8efac4-5df6f86d78-8xm9m" podUID="094495b3-e2a6-413d-b2d3-6251ad5667bd" containerName="payload-pre-processing" containerID="cri-o://4b7b83ebb63a0850376438b3661871b033754dfee85f697b2062e5704c4695bd" gracePeriod=30
Sep 11 03:57:05.614577 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:05.614549    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8efac4-5df6f86d78-8xm9m"
Sep 11 03:57:05.726992 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:05.726955    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/094495b3-e2a6-413d-b2d3-6251ad5667bd-plugins-config-volume\") pod \"094495b3-e2a6-413d-b2d3-6251ad5667bd\" (UID: \"094495b3-e2a6-413d-b2d3-6251ad5667bd\") "
Sep 11 03:57:05.727162 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:05.727061    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rbzjv\" (UniqueName: \"kubernetes.io/projected/094495b3-e2a6-413d-b2d3-6251ad5667bd-kube-api-access-rbzjv\") pod \"094495b3-e2a6-413d-b2d3-6251ad5667bd\" (UID: \"094495b3-e2a6-413d-b2d3-6251ad5667bd\") "
Sep 11 03:57:05.727337 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:05.727310    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/094495b3-e2a6-413d-b2d3-6251ad5667bd-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "094495b3-e2a6-413d-b2d3-6251ad5667bd" (UID: "094495b3-e2a6-413d-b2d3-6251ad5667bd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 03:57:05.729251 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:05.729217    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/094495b3-e2a6-413d-b2d3-6251ad5667bd-kube-api-access-rbzjv" (OuterVolumeSpecName: "kube-api-access-rbzjv") pod "094495b3-e2a6-413d-b2d3-6251ad5667bd" (UID: "094495b3-e2a6-413d-b2d3-6251ad5667bd"). InnerVolumeSpecName "kube-api-access-rbzjv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 03:57:05.828762 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:05.828721    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rbzjv\" (UniqueName: \"kubernetes.io/projected/094495b3-e2a6-413d-b2d3-6251ad5667bd-kube-api-access-rbzjv\") on node \"ip-10-0-138-121.ec2.internal\" DevicePath \"\""
Sep 11 03:57:05.828762 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:05.828764    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/094495b3-e2a6-413d-b2d3-6251ad5667bd-plugins-config-volume\") on node \"ip-10-0-138-121.ec2.internal\" DevicePath \"\""
Sep 11 03:57:06.417392 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:06.417357    2580 generic.go:356] "Generic (PLEG): container finished" podID="094495b3-e2a6-413d-b2d3-6251ad5667bd" containerID="4b7b83ebb63a0850376438b3661871b033754dfee85f697b2062e5704c4695bd" exitCode=0
Sep 11 03:57:06.419856 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:06.417398    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8efac4-5df6f86d78-8xm9m"
Sep 11 03:57:06.419856 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:06.417434    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8efac4-5df6f86d78-8xm9m" event={"ID":"094495b3-e2a6-413d-b2d3-6251ad5667bd","Type":"ContainerDied","Data":"4b7b83ebb63a0850376438b3661871b033754dfee85f697b2062e5704c4695bd"}
Sep 11 03:57:06.419856 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:06.417493    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8efac4-5df6f86d78-8xm9m" event={"ID":"094495b3-e2a6-413d-b2d3-6251ad5667bd","Type":"ContainerDied","Data":"73411ca29cc2713afb22d2ba1ec08a280bcce4537c009f3d7c2533b5ab4dbf62"}
Sep 11 03:57:06.419856 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:06.417521    2580 scope.go:117] "RemoveContainer" containerID="4b7b83ebb63a0850376438b3661871b033754dfee85f697b2062e5704c4695bd"
Sep 11 03:57:06.426980 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:06.426948    2580 scope.go:117] "RemoveContainer" containerID="4b7b83ebb63a0850376438b3661871b033754dfee85f697b2062e5704c4695bd"
Sep 11 03:57:06.427320 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:57:06.427301    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b7b83ebb63a0850376438b3661871b033754dfee85f697b2062e5704c4695bd\": container with ID starting with 4b7b83ebb63a0850376438b3661871b033754dfee85f697b2062e5704c4695bd not found: ID does not exist" containerID="4b7b83ebb63a0850376438b3661871b033754dfee85f697b2062e5704c4695bd"
Sep 11 03:57:06.427367 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:06.427329    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b7b83ebb63a0850376438b3661871b033754dfee85f697b2062e5704c4695bd"} err="failed to get container status \"4b7b83ebb63a0850376438b3661871b033754dfee85f697b2062e5704c4695bd\": rpc error: code = NotFound desc = could not find container \"4b7b83ebb63a0850376438b3661871b033754dfee85f697b2062e5704c4695bd\": container with ID starting with 4b7b83ebb63a0850376438b3661871b033754dfee85f697b2062e5704c4695bd not found: ID does not exist"
Sep 11 03:57:06.438213 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:06.438188    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-8efac4-5df6f86d78-8xm9m"]
Sep 11 03:57:06.441550 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:06.441521    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-8efac4-5df6f86d78-8xm9m"]
Sep 11 03:57:06.465736 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:06.465697    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="094495b3-e2a6-413d-b2d3-6251ad5667bd" path="/var/lib/kubelet/pods/094495b3-e2a6-413d-b2d3-6251ad5667bd/volumes"
Sep 11 03:57:07.336822 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:07.336786    2580 patch_prober.go:28] interesting pod/e2e-mt-4f6c0189-openshift-default-7cd8f458d4-jswgm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Sep 11 03:57:07.337007 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:07.336841    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-4f6c0189-openshift-default-7cd8f458d4-jswgm" podUID="18a0c038-ebff-4b9e-8920-fddc1f84bdf5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Sep 11 03:57:07.423932 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:07.423902    2580 generic.go:356] "Generic (PLEG): container finished" podID="18a0c038-ebff-4b9e-8920-fddc1f84bdf5" containerID="e06ca8f5831ea7968932ac00de5ad3d174dba32951942b7e4cdd7003113cba1e" exitCode=0
Sep 11 03:57:07.424326 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:07.423979    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-4f6c0189-openshift-default-7cd8f458d4-jswgm" event={"ID":"18a0c038-ebff-4b9e-8920-fddc1f84bdf5","Type":"ContainerDied","Data":"e06ca8f5831ea7968932ac00de5ad3d174dba32951942b7e4cdd7003113cba1e"}
Sep 11 03:57:07.424466 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:07.424449    2580 scope.go:117] "RemoveContainer" containerID="e06ca8f5831ea7968932ac00de5ad3d174dba32951942b7e4cdd7003113cba1e"
Sep 11 03:57:07.425519 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:07.425494    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 03:57:07.425578 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:07.425565    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 03:57:07.425614 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:07.425597    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 03:57:08.431847 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:08.431813    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-4f6c0189-openshift-default-7cd8f458d4-jswgm" event={"ID":"18a0c038-ebff-4b9e-8920-fddc1f84bdf5","Type":"ContainerStarted","Data":"8e4074e1fb61e23ab653f13343154b1b32d152e63ea566d06acdaf33bd018e58"}
Sep 11 03:57:08.951162 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:08.951124    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-4f6c0189-openshift-default-7cd8f458d4-jswgm"
Sep 11 03:57:08.951490 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:08.951373    2580 patch_prober.go:28] interesting pod/e2e-mt-4f6c0189-openshift-default-7cd8f458d4-jswgm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Sep 11 03:57:08.951701 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:08.951643    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-4f6c0189-openshift-default-7cd8f458d4-jswgm" podUID="18a0c038-ebff-4b9e-8920-fddc1f84bdf5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Sep 11 03:57:09.951388 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:09.951343    2580 patch_prober.go:28] interesting pod/e2e-mt-4f6c0189-openshift-default-7cd8f458d4-jswgm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Sep 11 03:57:09.952073 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:09.952035    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-4f6c0189-openshift-default-7cd8f458d4-jswgm" podUID="18a0c038-ebff-4b9e-8920-fddc1f84bdf5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Sep 11 03:57:10.951501 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:10.951458    2580 patch_prober.go:28] interesting pod/e2e-mt-4f6c0189-openshift-default-7cd8f458d4-jswgm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Sep 11 03:57:10.952177 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:10.952139    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-4f6c0189-openshift-default-7cd8f458d4-jswgm" podUID="18a0c038-ebff-4b9e-8920-fddc1f84bdf5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Sep 11 03:57:11.951794 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:11.951754    2580 patch_prober.go:28] interesting pod/e2e-mt-4f6c0189-openshift-default-7cd8f458d4-jswgm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Sep 11 03:57:11.952379 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:11.952346    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-4f6c0189-openshift-default-7cd8f458d4-jswgm" podUID="18a0c038-ebff-4b9e-8920-fddc1f84bdf5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Sep 11 03:57:11.999209 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:11.999173    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-8efac4-openshift-default-66557bb4f8-dl5z8"]
Sep 11 03:57:11.999712 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:11.999639    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-8efac4-openshift-default-66557bb4f8-dl5z8" podUID="bfe5ee73-4698-420d-9729-3153ab5656fa" containerName="istio-proxy" containerID="cri-o://552b8541e0a1981a238d9d627a711fa51e4d4c069425d744101e8ebd7b7ad04e" gracePeriod=30
Sep 11 03:57:12.289430 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:12.289340    2580 patch_prober.go:28] interesting pod/e2e-inf-b-8efac4-openshift-default-66557bb4f8-dl5z8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 11 03:57:12.289430 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:12.289393    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-8efac4-openshift-default-66557bb4f8-dl5z8" podUID="bfe5ee73-4698-420d-9729-3153ab5656fa" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 11 03:57:12.950938 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:12.950902    2580 patch_prober.go:28] interesting pod/e2e-mt-4f6c0189-openshift-default-7cd8f458d4-jswgm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Sep 11 03:57:12.951288 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:12.951255    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-4f6c0189-openshift-default-7cd8f458d4-jswgm" podUID="18a0c038-ebff-4b9e-8920-fddc1f84bdf5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Sep 11 03:57:13.955823 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:13.955792    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-4f6c0189-openshift-default-7cd8f458d4-jswgm"
Sep 11 03:57:13.956215 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:13.956127    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-4f6c0189-openshift-default-7cd8f458d4-jswgm"
Sep 11 03:57:13.956924 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:13.956907    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-4f6c0189-openshift-default-7cd8f458d4-jswgm"
Sep 11 03:57:17.275510 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:17.275441    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-8efac4-openshift-default-66557bb4f8-dl5z8"
Sep 11 03:57:17.345873 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:17.345589    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfe5ee73-4698-420d-9729-3153ab5656fa-istio-envoy\") pod \"bfe5ee73-4698-420d-9729-3153ab5656fa\" (UID: \"bfe5ee73-4698-420d-9729-3153ab5656fa\") "
Sep 11 03:57:17.345873 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:17.345640    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfe5ee73-4698-420d-9729-3153ab5656fa-istio-data\") pod \"bfe5ee73-4698-420d-9729-3153ab5656fa\" (UID: \"bfe5ee73-4698-420d-9729-3153ab5656fa\") "
Sep 11 03:57:17.351650 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:17.346425    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfe5ee73-4698-420d-9729-3153ab5656fa-istio-data" (OuterVolumeSpecName: "istio-data") pod "bfe5ee73-4698-420d-9729-3153ab5656fa" (UID: "bfe5ee73-4698-420d-9729-3153ab5656fa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 03:57:17.351650 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:17.348449    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfe5ee73-4698-420d-9729-3153ab5656fa-credential-socket\") pod \"bfe5ee73-4698-420d-9729-3153ab5656fa\" (UID: \"bfe5ee73-4698-420d-9729-3153ab5656fa\") "
Sep 11 03:57:17.351650 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:17.348504    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfe5ee73-4698-420d-9729-3153ab5656fa-workload-socket\") pod \"bfe5ee73-4698-420d-9729-3153ab5656fa\" (UID: \"bfe5ee73-4698-420d-9729-3153ab5656fa\") "
Sep 11 03:57:17.351650 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:17.348554    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfe5ee73-4698-420d-9729-3153ab5656fa-istio-podinfo\") pod \"bfe5ee73-4698-420d-9729-3153ab5656fa\" (UID: \"bfe5ee73-4698-420d-9729-3153ab5656fa\") "
Sep 11 03:57:17.351650 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:17.348627    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bfe5ee73-4698-420d-9729-3153ab5656fa-istio-token\") pod \"bfe5ee73-4698-420d-9729-3153ab5656fa\" (UID: \"bfe5ee73-4698-420d-9729-3153ab5656fa\") "
Sep 11 03:57:17.351650 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:17.348654    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfe5ee73-4698-420d-9729-3153ab5656fa-istiod-ca-cert\") pod \"bfe5ee73-4698-420d-9729-3153ab5656fa\" (UID: \"bfe5ee73-4698-420d-9729-3153ab5656fa\") "
Sep 11 03:57:17.351650 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:17.348763    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tp2fp\" (UniqueName: \"kubernetes.io/projected/bfe5ee73-4698-420d-9729-3153ab5656fa-kube-api-access-tp2fp\") pod \"bfe5ee73-4698-420d-9729-3153ab5656fa\" (UID: \"bfe5ee73-4698-420d-9729-3153ab5656fa\") "
Sep 11 03:57:17.351650 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:17.348800    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfe5ee73-4698-420d-9729-3153ab5656fa-workload-certs\") pod \"bfe5ee73-4698-420d-9729-3153ab5656fa\" (UID: \"bfe5ee73-4698-420d-9729-3153ab5656fa\") "
Sep 11 03:57:17.351650 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:17.349127    2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfe5ee73-4698-420d-9729-3153ab5656fa-istio-data\") on node \"ip-10-0-138-121.ec2.internal\" DevicePath \"\""
Sep 11 03:57:17.351650 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:17.349751    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bfe5ee73-4698-420d-9729-3153ab5656fa-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "bfe5ee73-4698-420d-9729-3153ab5656fa" (UID: "bfe5ee73-4698-420d-9729-3153ab5656fa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 03:57:17.351650 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:17.349993    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfe5ee73-4698-420d-9729-3153ab5656fa-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "bfe5ee73-4698-420d-9729-3153ab5656fa" (UID: "bfe5ee73-4698-420d-9729-3153ab5656fa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 03:57:17.351650 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:17.350041    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfe5ee73-4698-420d-9729-3153ab5656fa-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "bfe5ee73-4698-420d-9729-3153ab5656fa" (UID: "bfe5ee73-4698-420d-9729-3153ab5656fa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 03:57:17.351650 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:17.350107    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfe5ee73-4698-420d-9729-3153ab5656fa-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "bfe5ee73-4698-420d-9729-3153ab5656fa" (UID: "bfe5ee73-4698-420d-9729-3153ab5656fa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 03:57:17.351650 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:17.350439    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfe5ee73-4698-420d-9729-3153ab5656fa-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "bfe5ee73-4698-420d-9729-3153ab5656fa" (UID: "bfe5ee73-4698-420d-9729-3153ab5656fa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 03:57:17.352446 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:17.352364    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfe5ee73-4698-420d-9729-3153ab5656fa-istio-token" (OuterVolumeSpecName: "istio-token") pod "bfe5ee73-4698-420d-9729-3153ab5656fa" (UID: "bfe5ee73-4698-420d-9729-3153ab5656fa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 03:57:17.352601 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:17.352564    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfe5ee73-4698-420d-9729-3153ab5656fa-kube-api-access-tp2fp" (OuterVolumeSpecName: "kube-api-access-tp2fp") pod "bfe5ee73-4698-420d-9729-3153ab5656fa" (UID: "bfe5ee73-4698-420d-9729-3153ab5656fa"). InnerVolumeSpecName "kube-api-access-tp2fp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 03:57:17.355599 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:17.355561    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bfe5ee73-4698-420d-9729-3153ab5656fa-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "bfe5ee73-4698-420d-9729-3153ab5656fa" (UID: "bfe5ee73-4698-420d-9729-3153ab5656fa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 03:57:17.450282 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:17.450242    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tp2fp\" (UniqueName: \"kubernetes.io/projected/bfe5ee73-4698-420d-9729-3153ab5656fa-kube-api-access-tp2fp\") on node \"ip-10-0-138-121.ec2.internal\" DevicePath \"\""
Sep 11 03:57:17.450282 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:17.450276    2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfe5ee73-4698-420d-9729-3153ab5656fa-workload-certs\") on node \"ip-10-0-138-121.ec2.internal\" DevicePath \"\""
Sep 11 03:57:17.450282 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:17.450289    2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfe5ee73-4698-420d-9729-3153ab5656fa-istio-envoy\") on node \"ip-10-0-138-121.ec2.internal\" DevicePath \"\""
Sep 11 03:57:17.450527 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:17.450297    2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfe5ee73-4698-420d-9729-3153ab5656fa-credential-socket\") on node \"ip-10-0-138-121.ec2.internal\" DevicePath \"\""
Sep 11 03:57:17.450527 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:17.450305    2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfe5ee73-4698-420d-9729-3153ab5656fa-workload-socket\") on node \"ip-10-0-138-121.ec2.internal\" DevicePath \"\""
Sep 11 03:57:17.450527 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:17.450315    2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfe5ee73-4698-420d-9729-3153ab5656fa-istio-podinfo\") on node \"ip-10-0-138-121.ec2.internal\" DevicePath \"\""
Sep 11 03:57:17.450527 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:17.450324    2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bfe5ee73-4698-420d-9729-3153ab5656fa-istio-token\") on node \"ip-10-0-138-121.ec2.internal\" DevicePath \"\""
Sep 11 03:57:17.450527 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:17.450331    2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfe5ee73-4698-420d-9729-3153ab5656fa-istiod-ca-cert\") on node \"ip-10-0-138-121.ec2.internal\" DevicePath \"\""
Sep 11 03:57:17.484220 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:17.484183    2580 generic.go:356] "Generic (PLEG): container finished" podID="bfe5ee73-4698-420d-9729-3153ab5656fa" containerID="552b8541e0a1981a238d9d627a711fa51e4d4c069425d744101e8ebd7b7ad04e" exitCode=0
Sep 11 03:57:17.484398 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:17.484233    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-8efac4-openshift-default-66557bb4f8-dl5z8"
Sep 11 03:57:17.484398 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:17.484263    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-8efac4-openshift-default-66557bb4f8-dl5z8" event={"ID":"bfe5ee73-4698-420d-9729-3153ab5656fa","Type":"ContainerDied","Data":"552b8541e0a1981a238d9d627a711fa51e4d4c069425d744101e8ebd7b7ad04e"}
Sep 11 03:57:17.484398 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:17.484313    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-8efac4-openshift-default-66557bb4f8-dl5z8" event={"ID":"bfe5ee73-4698-420d-9729-3153ab5656fa","Type":"ContainerDied","Data":"129d106a6c6da6bbf23ea239901dc07537b9307f0e78fb292b44f7a47a98b2df"}
Sep 11 03:57:17.484398 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:17.484344    2580 scope.go:117] "RemoveContainer" containerID="552b8541e0a1981a238d9d627a711fa51e4d4c069425d744101e8ebd7b7ad04e"
Sep 11 03:57:17.493542 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:17.493521    2580 scope.go:117] "RemoveContainer" containerID="1856f2ca275da8b5a2f861e2ee2827b84fc738a8867abf2c136fb088d94e9b0e"
Sep 11 03:57:17.501677 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:17.501638    2580 scope.go:117] "RemoveContainer" containerID="552b8541e0a1981a238d9d627a711fa51e4d4c069425d744101e8ebd7b7ad04e"
Sep 11 03:57:17.502014 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:57:17.501993    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"552b8541e0a1981a238d9d627a711fa51e4d4c069425d744101e8ebd7b7ad04e\": container with ID starting with 552b8541e0a1981a238d9d627a711fa51e4d4c069425d744101e8ebd7b7ad04e not found: ID does not exist" containerID="552b8541e0a1981a238d9d627a711fa51e4d4c069425d744101e8ebd7b7ad04e"
Sep 11 03:57:17.502100 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:17.502019    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"552b8541e0a1981a238d9d627a711fa51e4d4c069425d744101e8ebd7b7ad04e"} err="failed to get container status \"552b8541e0a1981a238d9d627a711fa51e4d4c069425d744101e8ebd7b7ad04e\": rpc error: code = NotFound desc = could not find container \"552b8541e0a1981a238d9d627a711fa51e4d4c069425d744101e8ebd7b7ad04e\": container with ID starting with 552b8541e0a1981a238d9d627a711fa51e4d4c069425d744101e8ebd7b7ad04e not found: ID does not exist"
Sep 11 03:57:17.502100 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:17.502034    2580 scope.go:117] "RemoveContainer" containerID="1856f2ca275da8b5a2f861e2ee2827b84fc738a8867abf2c136fb088d94e9b0e"
Sep 11 03:57:17.502328 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:57:17.502299    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1856f2ca275da8b5a2f861e2ee2827b84fc738a8867abf2c136fb088d94e9b0e\": container with ID starting with 1856f2ca275da8b5a2f861e2ee2827b84fc738a8867abf2c136fb088d94e9b0e not found: ID does not exist" containerID="1856f2ca275da8b5a2f861e2ee2827b84fc738a8867abf2c136fb088d94e9b0e"
Sep 11 03:57:17.502460 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:17.502326    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1856f2ca275da8b5a2f861e2ee2827b84fc738a8867abf2c136fb088d94e9b0e"} err="failed to get container status \"1856f2ca275da8b5a2f861e2ee2827b84fc738a8867abf2c136fb088d94e9b0e\": rpc error: code = NotFound desc = could not find container \"1856f2ca275da8b5a2f861e2ee2827b84fc738a8867abf2c136fb088d94e9b0e\": container with ID starting with 1856f2ca275da8b5a2f861e2ee2827b84fc738a8867abf2c136fb088d94e9b0e not found: ID does not exist"
Sep 11 03:57:17.504989 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:17.504966    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-8efac4-openshift-default-66557bb4f8-dl5z8"]
Sep 11 03:57:17.508624 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:17.508606    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-8efac4-openshift-default-66557bb4f8-dl5z8"]
Sep 11 03:57:17.978645 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:17.978608    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-b4d3cf-559484c998-cq9hg"]
Sep 11 03:57:17.979013 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:17.978998    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bfe5ee73-4698-420d-9729-3153ab5656fa" containerName="istio-proxy"
Sep 11 03:57:17.979013 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:17.979014    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="bfe5ee73-4698-420d-9729-3153ab5656fa" containerName="istio-proxy"
Sep 11 03:57:17.979105 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:17.979041    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="094495b3-e2a6-413d-b2d3-6251ad5667bd" containerName="payload-pre-processing"
Sep 11 03:57:17.979105 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:17.979047    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="094495b3-e2a6-413d-b2d3-6251ad5667bd" containerName="payload-pre-processing"
Sep 11 03:57:17.979105 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:17.979104    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="bfe5ee73-4698-420d-9729-3153ab5656fa" containerName="istio-proxy"
Sep 11 03:57:17.979212 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:17.979112    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="bfe5ee73-4698-420d-9729-3153ab5656fa" containerName="istio-proxy"
Sep 11 03:57:17.979212 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:17.979119    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="094495b3-e2a6-413d-b2d3-6251ad5667bd" containerName="payload-pre-processing"
Sep 11 03:57:17.983146 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:17.983125    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-b4d3cf-559484c998-cq9hg"
Sep 11 03:57:17.985301 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:17.985278    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-b4d3cf-dockercfg-bwf7q\""
Sep 11 03:57:17.985301 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:17.985296    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-b4d3cf\""
Sep 11 03:57:17.992187 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:17.992160    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-b4d3cf-559484c998-cq9hg"]
Sep 11 03:57:18.057237 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:18.057200    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2870913c-68b6-4753-9352-ee0547316428-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-b4d3cf-559484c998-cq9hg\" (UID: \"2870913c-68b6-4753-9352-ee0547316428\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-b4d3cf-559484c998-cq9hg"
Sep 11 03:57:18.057450 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:18.057337    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ck85\" (UniqueName: \"kubernetes.io/projected/2870913c-68b6-4753-9352-ee0547316428-kube-api-access-2ck85\") pod \"payload-pre-processing-e2e-shared-a-w3-b4d3cf-559484c998-cq9hg\" (UID: \"2870913c-68b6-4753-9352-ee0547316428\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-b4d3cf-559484c998-cq9hg"
Sep 11 03:57:18.111629 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:18.111594    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-b4d3cf-5bf969dcf-ggj5v"]
Sep 11 03:57:18.112005 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:18.111992    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bfe5ee73-4698-420d-9729-3153ab5656fa" containerName="istio-proxy"
Sep 11 03:57:18.112056 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:18.112006    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="bfe5ee73-4698-420d-9729-3153ab5656fa" containerName="istio-proxy"
Sep 11 03:57:18.115159 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:18.115141    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-b4d3cf-5bf969dcf-ggj5v"
Sep 11 03:57:18.123896 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:18.123870    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-b4d3cf-5bf969dcf-ggj5v"]
Sep 11 03:57:18.158151 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:18.158115    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ck85\" (UniqueName: \"kubernetes.io/projected/2870913c-68b6-4753-9352-ee0547316428-kube-api-access-2ck85\") pod \"payload-pre-processing-e2e-shared-a-w3-b4d3cf-559484c998-cq9hg\" (UID: \"2870913c-68b6-4753-9352-ee0547316428\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-b4d3cf-559484c998-cq9hg"
Sep 11 03:57:18.158327 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:18.158214    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2870913c-68b6-4753-9352-ee0547316428-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-b4d3cf-559484c998-cq9hg\" (UID: \"2870913c-68b6-4753-9352-ee0547316428\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-b4d3cf-559484c998-cq9hg"
Sep 11 03:57:18.158933 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:18.158912    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2870913c-68b6-4753-9352-ee0547316428-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-b4d3cf-559484c998-cq9hg\" (UID: \"2870913c-68b6-4753-9352-ee0547316428\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-b4d3cf-559484c998-cq9hg"
Sep 11 03:57:18.165511 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:18.165487    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ck85\" (UniqueName: \"kubernetes.io/projected/2870913c-68b6-4753-9352-ee0547316428-kube-api-access-2ck85\") pod \"payload-pre-processing-e2e-shared-a-w3-b4d3cf-559484c998-cq9hg\" (UID: \"2870913c-68b6-4753-9352-ee0547316428\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-b4d3cf-559484c998-cq9hg"
Sep 11 03:57:18.259009 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:18.258908    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/390e944a-c647-4015-9aa9-10e3a374795b-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-b4d3cf-5bf969dcf-ggj5v\" (UID: \"390e944a-c647-4015-9aa9-10e3a374795b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-b4d3cf-5bf969dcf-ggj5v"
Sep 11 03:57:18.259009 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:18.258975    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7brk\" (UniqueName: \"kubernetes.io/projected/390e944a-c647-4015-9aa9-10e3a374795b-kube-api-access-h7brk\") pod \"payload-processing-e2e-shared-a-w3-b4d3cf-5bf969dcf-ggj5v\" (UID: \"390e944a-c647-4015-9aa9-10e3a374795b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-b4d3cf-5bf969dcf-ggj5v"
Sep 11 03:57:18.294105 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:18.294076    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-b4d3cf-559484c998-cq9hg"
Sep 11 03:57:18.360561 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:18.360522    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/390e944a-c647-4015-9aa9-10e3a374795b-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-b4d3cf-5bf969dcf-ggj5v\" (UID: \"390e944a-c647-4015-9aa9-10e3a374795b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-b4d3cf-5bf969dcf-ggj5v"
Sep 11 03:57:18.360730 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:18.360640    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7brk\" (UniqueName: \"kubernetes.io/projected/390e944a-c647-4015-9aa9-10e3a374795b-kube-api-access-h7brk\") pod \"payload-processing-e2e-shared-a-w3-b4d3cf-5bf969dcf-ggj5v\" (UID: \"390e944a-c647-4015-9aa9-10e3a374795b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-b4d3cf-5bf969dcf-ggj5v"
Sep 11 03:57:18.361446 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:18.361419    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/390e944a-c647-4015-9aa9-10e3a374795b-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-b4d3cf-5bf969dcf-ggj5v\" (UID: \"390e944a-c647-4015-9aa9-10e3a374795b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-b4d3cf-5bf969dcf-ggj5v"
Sep 11 03:57:18.369143 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:18.369086    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7brk\" (UniqueName: \"kubernetes.io/projected/390e944a-c647-4015-9aa9-10e3a374795b-kube-api-access-h7brk\") pod \"payload-processing-e2e-shared-a-w3-b4d3cf-5bf969dcf-ggj5v\" (UID: \"390e944a-c647-4015-9aa9-10e3a374795b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-b4d3cf-5bf969dcf-ggj5v"
Sep 11 03:57:18.426835 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:18.426801    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-b4d3cf-5bf969dcf-ggj5v"
Sep 11 03:57:18.475037 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:18.475000    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bfe5ee73-4698-420d-9729-3153ab5656fa" path="/var/lib/kubelet/pods/bfe5ee73-4698-420d-9729-3153ab5656fa/volumes"
Sep 11 03:57:18.563721 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:18.563621    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-b4d3cf-559484c998-cq9hg"]
Sep 11 03:57:18.566655 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:57:18.566627    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2870913c_68b6_4753_9352_ee0547316428.slice/crio-125d96c8ced7ab6122d168f34be14d79600df5be4a55143501d8083334b2146c WatchSource:0}: Error finding container 125d96c8ced7ab6122d168f34be14d79600df5be4a55143501d8083334b2146c: Status 404 returned error can't find the container with id 125d96c8ced7ab6122d168f34be14d79600df5be4a55143501d8083334b2146c
Sep 11 03:57:18.702297 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:18.702271    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-b4d3cf-5bf969dcf-ggj5v"]
Sep 11 03:57:18.703896 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:57:18.703872    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod390e944a_c647_4015_9aa9_10e3a374795b.slice/crio-b7c842c8a4dae77b34fadba684bfe83d1a8a028d76deb65576078a7ed247f45a WatchSource:0}: Error finding container b7c842c8a4dae77b34fadba684bfe83d1a8a028d76deb65576078a7ed247f45a: Status 404 returned error can't find the container with id b7c842c8a4dae77b34fadba684bfe83d1a8a028d76deb65576078a7ed247f45a
Sep 11 03:57:19.502399 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:19.502358    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-b4d3cf-5bf969dcf-ggj5v" event={"ID":"390e944a-c647-4015-9aa9-10e3a374795b","Type":"ContainerStarted","Data":"59ab76ed2e5c096f2a89a41b6bbaea125b3cf03c03673f9f2ff2273df413b6e3"}
Sep 11 03:57:19.502922 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:19.502419    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-b4d3cf-5bf969dcf-ggj5v" event={"ID":"390e944a-c647-4015-9aa9-10e3a374795b","Type":"ContainerStarted","Data":"b7c842c8a4dae77b34fadba684bfe83d1a8a028d76deb65576078a7ed247f45a"}
Sep 11 03:57:19.502922 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:19.502448    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-b4d3cf-5bf969dcf-ggj5v"
Sep 11 03:57:19.503931 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:19.503901    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-b4d3cf-559484c998-cq9hg" event={"ID":"2870913c-68b6-4753-9352-ee0547316428","Type":"ContainerStarted","Data":"de571050b673cffa0248dea75d6c1dfcb7a7e3cf042ac4769d5654ddb7ba571b"}
Sep 11 03:57:19.504055 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:19.503951    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-b4d3cf-559484c998-cq9hg" event={"ID":"2870913c-68b6-4753-9352-ee0547316428","Type":"ContainerStarted","Data":"125d96c8ced7ab6122d168f34be14d79600df5be4a55143501d8083334b2146c"}
Sep 11 03:57:19.504055 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:19.503988    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-b4d3cf-559484c998-cq9hg"
Sep 11 03:57:19.517241 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:19.517196    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-b4d3cf-5bf969dcf-ggj5v" podStartSLOduration=1.517181614 podStartE2EDuration="1.517181614s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 03:57:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:57:19.516053226 +0000 UTC m=+1027.674539549" watchObservedRunningTime="2026-09-11 03:57:19.517181614 +0000 UTC m=+1027.675667937"
Sep 11 03:57:19.531170 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:19.531125    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-b4d3cf-559484c998-cq9hg" podStartSLOduration=2.5311102930000002 podStartE2EDuration="2.531110293s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 03:57:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:57:19.530188435 +0000 UTC m=+1027.688674759" watchObservedRunningTime="2026-09-11 03:57:19.531110293 +0000 UTC m=+1027.689596616"
Sep 11 03:57:30.512562 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:30.512532    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-b4d3cf-5bf969dcf-ggj5v"
Sep 11 03:57:30.513061 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:30.512602    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-b4d3cf-559484c998-cq9hg"
Sep 11 03:57:30.821315 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:30.821234    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-4f6c0189-openshift-default-7cd8f458d4-jswgm"]
Sep 11 03:57:30.821526 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:30.821499    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-4f6c0189-openshift-default-7cd8f458d4-jswgm" podUID="18a0c038-ebff-4b9e-8920-fddc1f84bdf5" containerName="istio-proxy" containerID="cri-o://8e4074e1fb61e23ab653f13343154b1b32d152e63ea566d06acdaf33bd018e58" gracePeriod=30
Sep 11 03:57:36.083581 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:36.083509    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-4f6c0189-openshift-default-7cd8f458d4-jswgm"
Sep 11 03:57:36.126077 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:36.126049    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18a0c038-ebff-4b9e-8920-fddc1f84bdf5-istio-podinfo\") pod \"18a0c038-ebff-4b9e-8920-fddc1f84bdf5\" (UID: \"18a0c038-ebff-4b9e-8920-fddc1f84bdf5\") "
Sep 11 03:57:36.126257 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:36.126105    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18a0c038-ebff-4b9e-8920-fddc1f84bdf5-istio-data\") pod \"18a0c038-ebff-4b9e-8920-fddc1f84bdf5\" (UID: \"18a0c038-ebff-4b9e-8920-fddc1f84bdf5\") "
Sep 11 03:57:36.126257 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:36.126146    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18a0c038-ebff-4b9e-8920-fddc1f84bdf5-workload-socket\") pod \"18a0c038-ebff-4b9e-8920-fddc1f84bdf5\" (UID: \"18a0c038-ebff-4b9e-8920-fddc1f84bdf5\") "
Sep 11 03:57:36.126257 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:36.126173    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18a0c038-ebff-4b9e-8920-fddc1f84bdf5-istio-envoy\") pod \"18a0c038-ebff-4b9e-8920-fddc1f84bdf5\" (UID: \"18a0c038-ebff-4b9e-8920-fddc1f84bdf5\") "
Sep 11 03:57:36.126257 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:36.126216    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qkr4q\" (UniqueName: \"kubernetes.io/projected/18a0c038-ebff-4b9e-8920-fddc1f84bdf5-kube-api-access-qkr4q\") pod \"18a0c038-ebff-4b9e-8920-fddc1f84bdf5\" (UID: \"18a0c038-ebff-4b9e-8920-fddc1f84bdf5\") "
Sep 11 03:57:36.126257 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:36.126244    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18a0c038-ebff-4b9e-8920-fddc1f84bdf5-workload-certs\") pod \"18a0c038-ebff-4b9e-8920-fddc1f84bdf5\" (UID: \"18a0c038-ebff-4b9e-8920-fddc1f84bdf5\") "
Sep 11 03:57:36.126519 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:36.126267    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18a0c038-ebff-4b9e-8920-fddc1f84bdf5-istiod-ca-cert\") pod \"18a0c038-ebff-4b9e-8920-fddc1f84bdf5\" (UID: \"18a0c038-ebff-4b9e-8920-fddc1f84bdf5\") "
Sep 11 03:57:36.126519 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:36.126303    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18a0c038-ebff-4b9e-8920-fddc1f84bdf5-istio-token\") pod \"18a0c038-ebff-4b9e-8920-fddc1f84bdf5\" (UID: \"18a0c038-ebff-4b9e-8920-fddc1f84bdf5\") "
Sep 11 03:57:36.126519 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:36.126353    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18a0c038-ebff-4b9e-8920-fddc1f84bdf5-credential-socket\") pod \"18a0c038-ebff-4b9e-8920-fddc1f84bdf5\" (UID: \"18a0c038-ebff-4b9e-8920-fddc1f84bdf5\") "
Sep 11 03:57:36.126519 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:36.126494    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18a0c038-ebff-4b9e-8920-fddc1f84bdf5-istio-data" (OuterVolumeSpecName: "istio-data") pod "18a0c038-ebff-4b9e-8920-fddc1f84bdf5" (UID: "18a0c038-ebff-4b9e-8920-fddc1f84bdf5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 03:57:36.126761 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:36.126510    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18a0c038-ebff-4b9e-8920-fddc1f84bdf5-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "18a0c038-ebff-4b9e-8920-fddc1f84bdf5" (UID: "18a0c038-ebff-4b9e-8920-fddc1f84bdf5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 03:57:36.126761 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:36.126717    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18a0c038-ebff-4b9e-8920-fddc1f84bdf5-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "18a0c038-ebff-4b9e-8920-fddc1f84bdf5" (UID: "18a0c038-ebff-4b9e-8920-fddc1f84bdf5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 03:57:36.126761 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:36.126723    2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18a0c038-ebff-4b9e-8920-fddc1f84bdf5-workload-certs\") on node \"ip-10-0-138-121.ec2.internal\" DevicePath \"\""
Sep 11 03:57:36.126761 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:36.126757    2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18a0c038-ebff-4b9e-8920-fddc1f84bdf5-istio-data\") on node \"ip-10-0-138-121.ec2.internal\" DevicePath \"\""
Sep 11 03:57:36.126963 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:36.126795    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18a0c038-ebff-4b9e-8920-fddc1f84bdf5-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "18a0c038-ebff-4b9e-8920-fddc1f84bdf5" (UID: "18a0c038-ebff-4b9e-8920-fddc1f84bdf5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 03:57:36.126963 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:36.126874    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18a0c038-ebff-4b9e-8920-fddc1f84bdf5-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "18a0c038-ebff-4b9e-8920-fddc1f84bdf5" (UID: "18a0c038-ebff-4b9e-8920-fddc1f84bdf5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 03:57:36.128387 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:36.128357    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/18a0c038-ebff-4b9e-8920-fddc1f84bdf5-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "18a0c038-ebff-4b9e-8920-fddc1f84bdf5" (UID: "18a0c038-ebff-4b9e-8920-fddc1f84bdf5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 03:57:36.128609 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:36.128583    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18a0c038-ebff-4b9e-8920-fddc1f84bdf5-kube-api-access-qkr4q" (OuterVolumeSpecName: "kube-api-access-qkr4q") pod "18a0c038-ebff-4b9e-8920-fddc1f84bdf5" (UID: "18a0c038-ebff-4b9e-8920-fddc1f84bdf5"). InnerVolumeSpecName "kube-api-access-qkr4q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 03:57:36.128721 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:36.128650    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18a0c038-ebff-4b9e-8920-fddc1f84bdf5-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "18a0c038-ebff-4b9e-8920-fddc1f84bdf5" (UID: "18a0c038-ebff-4b9e-8920-fddc1f84bdf5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 03:57:36.128887 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:36.128867    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18a0c038-ebff-4b9e-8920-fddc1f84bdf5-istio-token" (OuterVolumeSpecName: "istio-token") pod "18a0c038-ebff-4b9e-8920-fddc1f84bdf5" (UID: "18a0c038-ebff-4b9e-8920-fddc1f84bdf5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 03:57:36.227814 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:36.227781    2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18a0c038-ebff-4b9e-8920-fddc1f84bdf5-workload-socket\") on node \"ip-10-0-138-121.ec2.internal\" DevicePath \"\""
Sep 11 03:57:36.227814 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:36.227810    2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18a0c038-ebff-4b9e-8920-fddc1f84bdf5-istio-envoy\") on node \"ip-10-0-138-121.ec2.internal\" DevicePath \"\""
Sep 11 03:57:36.227814 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:36.227821    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qkr4q\" (UniqueName: \"kubernetes.io/projected/18a0c038-ebff-4b9e-8920-fddc1f84bdf5-kube-api-access-qkr4q\") on node \"ip-10-0-138-121.ec2.internal\" DevicePath \"\""
Sep 11 03:57:36.228035 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:36.227831    2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18a0c038-ebff-4b9e-8920-fddc1f84bdf5-istiod-ca-cert\") on node \"ip-10-0-138-121.ec2.internal\" DevicePath \"\""
Sep 11 03:57:36.228035 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:36.227841    2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18a0c038-ebff-4b9e-8920-fddc1f84bdf5-istio-token\") on node \"ip-10-0-138-121.ec2.internal\" DevicePath \"\""
Sep 11 03:57:36.228035 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:36.227849    2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18a0c038-ebff-4b9e-8920-fddc1f84bdf5-credential-socket\") on node \"ip-10-0-138-121.ec2.internal\" DevicePath \"\""
Sep 11 03:57:36.228035 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:36.227862    2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18a0c038-ebff-4b9e-8920-fddc1f84bdf5-istio-podinfo\") on node \"ip-10-0-138-121.ec2.internal\" DevicePath \"\""
Sep 11 03:57:36.233414 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:36.233382    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-dd5b30-cbb7b64f-dx4z8"]
Sep 11 03:57:36.233887 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:36.233867    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="18a0c038-ebff-4b9e-8920-fddc1f84bdf5" containerName="istio-proxy"
Sep 11 03:57:36.234024 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:36.233889    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="18a0c038-ebff-4b9e-8920-fddc1f84bdf5" containerName="istio-proxy"
Sep 11 03:57:36.234024 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:36.233917    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="18a0c038-ebff-4b9e-8920-fddc1f84bdf5" containerName="istio-proxy"
Sep 11 03:57:36.234024 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:36.233925    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="18a0c038-ebff-4b9e-8920-fddc1f84bdf5" containerName="istio-proxy"
Sep 11 03:57:36.234024 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:36.234011    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="18a0c038-ebff-4b9e-8920-fddc1f84bdf5" containerName="istio-proxy"
Sep 11 03:57:36.234024 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:36.234025    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="18a0c038-ebff-4b9e-8920-fddc1f84bdf5" containerName="istio-proxy"
Sep 11 03:57:36.237378 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:36.237358    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-dd5b30-cbb7b64f-dx4z8"
Sep 11 03:57:36.239747 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:36.239655    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 11 03:57:36.239747 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:36.239709    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 03:57:36.239928 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:36.239779    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 11 03:57:36.239928 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:36.239846    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-fcpfs\""
Sep 11 03:57:36.240042 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:36.239978    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-dd5b30-serving-cert\""
Sep 11 03:57:36.246872 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:36.246850    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-dd5b30-cbb7b64f-dx4z8"]
Sep 11 03:57:36.328501 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:36.328464    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6pl4\" (UniqueName: \"kubernetes.io/projected/cc7227b2-19bf-4d54-b0b3-ce4de2c8a35b-kube-api-access-h6pl4\") pod \"maas-api-e2e-shared-b-w3-dd5b30-cbb7b64f-dx4z8\" (UID: \"cc7227b2-19bf-4d54-b0b3-ce4de2c8a35b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-dd5b30-cbb7b64f-dx4z8"
Sep 11 03:57:36.328683 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:36.328532    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc7227b2-19bf-4d54-b0b3-ce4de2c8a35b-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-dd5b30-cbb7b64f-dx4z8\" (UID: \"cc7227b2-19bf-4d54-b0b3-ce4de2c8a35b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-dd5b30-cbb7b64f-dx4z8"
Sep 11 03:57:36.328683 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:36.328583    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc7227b2-19bf-4d54-b0b3-ce4de2c8a35b-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-dd5b30-cbb7b64f-dx4z8\" (UID: \"cc7227b2-19bf-4d54-b0b3-ce4de2c8a35b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-dd5b30-cbb7b64f-dx4z8"
Sep 11 03:57:36.328683 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:36.328641    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc7227b2-19bf-4d54-b0b3-ce4de2c8a35b-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-dd5b30-cbb7b64f-dx4z8\" (UID: \"cc7227b2-19bf-4d54-b0b3-ce4de2c8a35b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-dd5b30-cbb7b64f-dx4z8"
Sep 11 03:57:36.429832 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:36.429796    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc7227b2-19bf-4d54-b0b3-ce4de2c8a35b-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-dd5b30-cbb7b64f-dx4z8\" (UID: \"cc7227b2-19bf-4d54-b0b3-ce4de2c8a35b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-dd5b30-cbb7b64f-dx4z8"
Sep 11 03:57:36.430039 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:36.429845    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc7227b2-19bf-4d54-b0b3-ce4de2c8a35b-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-dd5b30-cbb7b64f-dx4z8\" (UID: \"cc7227b2-19bf-4d54-b0b3-ce4de2c8a35b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-dd5b30-cbb7b64f-dx4z8"
Sep 11 03:57:36.430039 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:36.429896    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6pl4\" (UniqueName: \"kubernetes.io/projected/cc7227b2-19bf-4d54-b0b3-ce4de2c8a35b-kube-api-access-h6pl4\") pod \"maas-api-e2e-shared-b-w3-dd5b30-cbb7b64f-dx4z8\" (UID: \"cc7227b2-19bf-4d54-b0b3-ce4de2c8a35b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-dd5b30-cbb7b64f-dx4z8"
Sep 11 03:57:36.430039 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:36.429932    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc7227b2-19bf-4d54-b0b3-ce4de2c8a35b-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-dd5b30-cbb7b64f-dx4z8\" (UID: \"cc7227b2-19bf-4d54-b0b3-ce4de2c8a35b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-dd5b30-cbb7b64f-dx4z8"
Sep 11 03:57:36.430440 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:36.430409    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc7227b2-19bf-4d54-b0b3-ce4de2c8a35b-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-dd5b30-cbb7b64f-dx4z8\" (UID: \"cc7227b2-19bf-4d54-b0b3-ce4de2c8a35b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-dd5b30-cbb7b64f-dx4z8"
Sep 11 03:57:36.432284 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:36.432260    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc7227b2-19bf-4d54-b0b3-ce4de2c8a35b-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-dd5b30-cbb7b64f-dx4z8\" (UID: \"cc7227b2-19bf-4d54-b0b3-ce4de2c8a35b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-dd5b30-cbb7b64f-dx4z8"
Sep 11 03:57:36.432362 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:36.432318    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc7227b2-19bf-4d54-b0b3-ce4de2c8a35b-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-dd5b30-cbb7b64f-dx4z8\" (UID: \"cc7227b2-19bf-4d54-b0b3-ce4de2c8a35b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-dd5b30-cbb7b64f-dx4z8"
Sep 11 03:57:36.438551 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:36.438525    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6pl4\" (UniqueName: \"kubernetes.io/projected/cc7227b2-19bf-4d54-b0b3-ce4de2c8a35b-kube-api-access-h6pl4\") pod \"maas-api-e2e-shared-b-w3-dd5b30-cbb7b64f-dx4z8\" (UID: \"cc7227b2-19bf-4d54-b0b3-ce4de2c8a35b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-dd5b30-cbb7b64f-dx4z8"
Sep 11 03:57:36.550839 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:36.550808    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-dd5b30-cbb7b64f-dx4z8"
Sep 11 03:57:36.599176 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:36.599142    2580 generic.go:356] "Generic (PLEG): container finished" podID="18a0c038-ebff-4b9e-8920-fddc1f84bdf5" containerID="8e4074e1fb61e23ab653f13343154b1b32d152e63ea566d06acdaf33bd018e58" exitCode=0
Sep 11 03:57:36.599330 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:36.599197    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-4f6c0189-openshift-default-7cd8f458d4-jswgm"
Sep 11 03:57:36.599330 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:36.599240    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-4f6c0189-openshift-default-7cd8f458d4-jswgm" event={"ID":"18a0c038-ebff-4b9e-8920-fddc1f84bdf5","Type":"ContainerDied","Data":"8e4074e1fb61e23ab653f13343154b1b32d152e63ea566d06acdaf33bd018e58"}
Sep 11 03:57:36.599330 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:36.599309    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-4f6c0189-openshift-default-7cd8f458d4-jswgm" event={"ID":"18a0c038-ebff-4b9e-8920-fddc1f84bdf5","Type":"ContainerDied","Data":"b875cfb83cd72569b275b37e6eb1f6aa96971397c7efaa1f7a7523534222d52e"}
Sep 11 03:57:36.599491 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:36.599341    2580 scope.go:117] "RemoveContainer" containerID="8e4074e1fb61e23ab653f13343154b1b32d152e63ea566d06acdaf33bd018e58"
Sep 11 03:57:36.612967 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:36.612942    2580 scope.go:117] "RemoveContainer" containerID="e06ca8f5831ea7968932ac00de5ad3d174dba32951942b7e4cdd7003113cba1e"
Sep 11 03:57:36.617466 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:36.617442    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-4f6c0189-openshift-default-7cd8f458d4-jswgm"]
Sep 11 03:57:36.623890 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:36.623866    2580 scope.go:117] "RemoveContainer" containerID="8e4074e1fb61e23ab653f13343154b1b32d152e63ea566d06acdaf33bd018e58"
Sep 11 03:57:36.624231 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:57:36.624210    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e4074e1fb61e23ab653f13343154b1b32d152e63ea566d06acdaf33bd018e58\": container with ID starting with 8e4074e1fb61e23ab653f13343154b1b32d152e63ea566d06acdaf33bd018e58 not found: ID does not exist" containerID="8e4074e1fb61e23ab653f13343154b1b32d152e63ea566d06acdaf33bd018e58"
Sep 11 03:57:36.624307 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:36.624240    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e4074e1fb61e23ab653f13343154b1b32d152e63ea566d06acdaf33bd018e58"} err="failed to get container status \"8e4074e1fb61e23ab653f13343154b1b32d152e63ea566d06acdaf33bd018e58\": rpc error: code = NotFound desc = could not find container \"8e4074e1fb61e23ab653f13343154b1b32d152e63ea566d06acdaf33bd018e58\": container with ID starting with 8e4074e1fb61e23ab653f13343154b1b32d152e63ea566d06acdaf33bd018e58 not found: ID does not exist"
Sep 11 03:57:36.624307 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:36.624261    2580 scope.go:117] "RemoveContainer" containerID="e06ca8f5831ea7968932ac00de5ad3d174dba32951942b7e4cdd7003113cba1e"
Sep 11 03:57:36.624441 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:36.624420    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-4f6c0189-openshift-default-7cd8f458d4-jswgm"]
Sep 11 03:57:36.624566 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:57:36.624550    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e06ca8f5831ea7968932ac00de5ad3d174dba32951942b7e4cdd7003113cba1e\": container with ID starting with e06ca8f5831ea7968932ac00de5ad3d174dba32951942b7e4cdd7003113cba1e not found: ID does not exist" containerID="e06ca8f5831ea7968932ac00de5ad3d174dba32951942b7e4cdd7003113cba1e"
Sep 11 03:57:36.624613 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:36.624572    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e06ca8f5831ea7968932ac00de5ad3d174dba32951942b7e4cdd7003113cba1e"} err="failed to get container status \"e06ca8f5831ea7968932ac00de5ad3d174dba32951942b7e4cdd7003113cba1e\": rpc error: code = NotFound desc = could not find container \"e06ca8f5831ea7968932ac00de5ad3d174dba32951942b7e4cdd7003113cba1e\": container with ID starting with e06ca8f5831ea7968932ac00de5ad3d174dba32951942b7e4cdd7003113cba1e not found: ID does not exist"
Sep 11 03:57:36.828132 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:36.828106    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-dd5b30-cbb7b64f-dx4z8"]
Sep 11 03:57:36.830931 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:57:36.830901    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc7227b2_19bf_4d54_b0b3_ce4de2c8a35b.slice/crio-a480973ec5afe95a3a348a7e65dd5988cb9fc06fa6b2f4aa6b2b4e6c1bca9994 WatchSource:0}: Error finding container a480973ec5afe95a3a348a7e65dd5988cb9fc06fa6b2f4aa6b2b4e6c1bca9994: Status 404 returned error can't find the container with id a480973ec5afe95a3a348a7e65dd5988cb9fc06fa6b2f4aa6b2b4e6c1bca9994
Sep 11 03:57:37.040948 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:37.040901    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-dd5b30-59cd644494-hqhn2"]
Sep 11 03:57:37.045838 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:37.045816    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-dd5b30-59cd644494-hqhn2"
Sep 11 03:57:37.048095 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:37.048077    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-dd5b30-dockercfg-c67g2\""
Sep 11 03:57:37.048187 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:37.048139    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-dd5b30\""
Sep 11 03:57:37.055098 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:37.055044    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-dd5b30-59cd644494-hqhn2"]
Sep 11 03:57:37.134903 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:37.134803    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwglw\" (UniqueName: \"kubernetes.io/projected/0c012db4-89ca-40c3-a7e6-42681955e252-kube-api-access-lwglw\") pod \"payload-pre-processing-e2e-shared-b-w3-dd5b30-59cd644494-hqhn2\" (UID: \"0c012db4-89ca-40c3-a7e6-42681955e252\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-dd5b30-59cd644494-hqhn2"
Sep 11 03:57:37.135288 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:37.134906    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c012db4-89ca-40c3-a7e6-42681955e252-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-dd5b30-59cd644494-hqhn2\" (UID: \"0c012db4-89ca-40c3-a7e6-42681955e252\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-dd5b30-59cd644494-hqhn2"
Sep 11 03:57:37.236193 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:37.236149    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwglw\" (UniqueName: \"kubernetes.io/projected/0c012db4-89ca-40c3-a7e6-42681955e252-kube-api-access-lwglw\") pod \"payload-pre-processing-e2e-shared-b-w3-dd5b30-59cd644494-hqhn2\" (UID: \"0c012db4-89ca-40c3-a7e6-42681955e252\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-dd5b30-59cd644494-hqhn2"
Sep 11 03:57:37.236399 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:37.236224    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c012db4-89ca-40c3-a7e6-42681955e252-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-dd5b30-59cd644494-hqhn2\" (UID: \"0c012db4-89ca-40c3-a7e6-42681955e252\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-dd5b30-59cd644494-hqhn2"
Sep 11 03:57:37.236994 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:37.236970    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c012db4-89ca-40c3-a7e6-42681955e252-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-dd5b30-59cd644494-hqhn2\" (UID: \"0c012db4-89ca-40c3-a7e6-42681955e252\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-dd5b30-59cd644494-hqhn2"
Sep 11 03:57:37.244890 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:37.244863    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwglw\" (UniqueName: \"kubernetes.io/projected/0c012db4-89ca-40c3-a7e6-42681955e252-kube-api-access-lwglw\") pod \"payload-pre-processing-e2e-shared-b-w3-dd5b30-59cd644494-hqhn2\" (UID: \"0c012db4-89ca-40c3-a7e6-42681955e252\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-dd5b30-59cd644494-hqhn2"
Sep 11 03:57:37.359506 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:37.359466    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-dd5b30-59cd644494-hqhn2"
Sep 11 03:57:37.607911 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:37.607875    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-dd5b30-cbb7b64f-dx4z8" event={"ID":"cc7227b2-19bf-4d54-b0b3-ce4de2c8a35b","Type":"ContainerStarted","Data":"2d1154dcbc9db4f5369bda9471dffcc96340426af7b7bc5365a77aed11ad9bbb"}
Sep 11 03:57:37.608098 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:37.607926    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-dd5b30-cbb7b64f-dx4z8" event={"ID":"cc7227b2-19bf-4d54-b0b3-ce4de2c8a35b","Type":"ContainerStarted","Data":"a480973ec5afe95a3a348a7e65dd5988cb9fc06fa6b2f4aa6b2b4e6c1bca9994"}
Sep 11 03:57:37.608098 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:37.607993    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-dd5b30-cbb7b64f-dx4z8"
Sep 11 03:57:37.637309 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:37.637258    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-dd5b30-cbb7b64f-dx4z8" podStartSLOduration=1.048007179 podStartE2EDuration="1.637239494s" totalImagesPullingTime="589.232315ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 03:57:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:57:37.636224682 +0000 UTC m=+1045.794711017" watchObservedRunningTime="2026-09-11 03:57:37.637239494 +0000 UTC m=+1045.795725816"
Sep 11 03:57:37.652077 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:37.652051    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-dd5b30-59cd644494-hqhn2"]
Sep 11 03:57:37.655117 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:57:37.655084    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c012db4_89ca_40c3_a7e6_42681955e252.slice/crio-f188ba559075c0f640953a374e6c514e21645068e32b61b6db4210e01c2cf6f4 WatchSource:0}: Error finding container f188ba559075c0f640953a374e6c514e21645068e32b61b6db4210e01c2cf6f4: Status 404 returned error can't find the container with id f188ba559075c0f640953a374e6c514e21645068e32b61b6db4210e01c2cf6f4
Sep 11 03:57:38.466080 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:38.466048    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="18a0c038-ebff-4b9e-8920-fddc1f84bdf5" path="/var/lib/kubelet/pods/18a0c038-ebff-4b9e-8920-fddc1f84bdf5/volumes"
Sep 11 03:57:38.620084 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:38.620042    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-dd5b30-59cd644494-hqhn2" event={"ID":"0c012db4-89ca-40c3-a7e6-42681955e252","Type":"ContainerStarted","Data":"1738ccd7e17c9b55cdb29910ee41bebe0165d1b86a12e34c469e4d3985fb262a"}
Sep 11 03:57:38.620279 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:38.620135    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-dd5b30-59cd644494-hqhn2" event={"ID":"0c012db4-89ca-40c3-a7e6-42681955e252","Type":"ContainerStarted","Data":"f188ba559075c0f640953a374e6c514e21645068e32b61b6db4210e01c2cf6f4"}
Sep 11 03:57:38.620344 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:38.620304    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-dd5b30-59cd644494-hqhn2"
Sep 11 03:57:38.638739 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:38.638681    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-dd5b30-59cd644494-hqhn2" podStartSLOduration=1.638649953 podStartE2EDuration="1.638649953s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 03:57:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:57:38.636527395 +0000 UTC m=+1046.795013724" watchObservedRunningTime="2026-09-11 03:57:38.638649953 +0000 UTC m=+1046.797136275"
Sep 11 03:57:43.625318 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:43.625285    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-dd5b30-cbb7b64f-dx4z8"
Sep 11 03:57:43.808990 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:43.808950    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-96c7d4-openshift-default-785b65d79d-j8777"]
Sep 11 03:57:43.809313 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:43.809255    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-noten-96c7d4-openshift-default-785b65d79d-j8777" podUID="13d95f4c-f162-4072-acf6-c2716bdbb893" containerName="istio-proxy" containerID="cri-o://f824a43573fec36fb42af01cfd9e8dc1ac4edb147e729ca1f2db91072f001d22" gracePeriod=30
Sep 11 03:57:49.049582 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:49.049556    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-96c7d4-openshift-default-785b65d79d-j8777"
Sep 11 03:57:49.156849 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:49.156813    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-66w4k\" (UniqueName: \"kubernetes.io/projected/13d95f4c-f162-4072-acf6-c2716bdbb893-kube-api-access-66w4k\") pod \"13d95f4c-f162-4072-acf6-c2716bdbb893\" (UID: \"13d95f4c-f162-4072-acf6-c2716bdbb893\") "
Sep 11 03:57:49.157028 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:49.156859    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13d95f4c-f162-4072-acf6-c2716bdbb893-istiod-ca-cert\") pod \"13d95f4c-f162-4072-acf6-c2716bdbb893\" (UID: \"13d95f4c-f162-4072-acf6-c2716bdbb893\") "
Sep 11 03:57:49.157028 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:49.156939    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13d95f4c-f162-4072-acf6-c2716bdbb893-istio-envoy\") pod \"13d95f4c-f162-4072-acf6-c2716bdbb893\" (UID: \"13d95f4c-f162-4072-acf6-c2716bdbb893\") "
Sep 11 03:57:49.157028 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:49.156966    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/13d95f4c-f162-4072-acf6-c2716bdbb893-istio-token\") pod \"13d95f4c-f162-4072-acf6-c2716bdbb893\" (UID: \"13d95f4c-f162-4072-acf6-c2716bdbb893\") "
Sep 11 03:57:49.157028 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:49.156995    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13d95f4c-f162-4072-acf6-c2716bdbb893-istio-data\") pod \"13d95f4c-f162-4072-acf6-c2716bdbb893\" (UID: \"13d95f4c-f162-4072-acf6-c2716bdbb893\") "
Sep 11 03:57:49.157279 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:49.157037    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13d95f4c-f162-4072-acf6-c2716bdbb893-workload-certs\") pod \"13d95f4c-f162-4072-acf6-c2716bdbb893\" (UID: \"13d95f4c-f162-4072-acf6-c2716bdbb893\") "
Sep 11 03:57:49.157279 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:49.157067    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13d95f4c-f162-4072-acf6-c2716bdbb893-credential-socket\") pod \"13d95f4c-f162-4072-acf6-c2716bdbb893\" (UID: \"13d95f4c-f162-4072-acf6-c2716bdbb893\") "
Sep 11 03:57:49.157279 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:49.157096    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13d95f4c-f162-4072-acf6-c2716bdbb893-workload-socket\") pod \"13d95f4c-f162-4072-acf6-c2716bdbb893\" (UID: \"13d95f4c-f162-4072-acf6-c2716bdbb893\") "
Sep 11 03:57:49.157279 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:49.157131    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13d95f4c-f162-4072-acf6-c2716bdbb893-istio-podinfo\") pod \"13d95f4c-f162-4072-acf6-c2716bdbb893\" (UID: \"13d95f4c-f162-4072-acf6-c2716bdbb893\") "
Sep 11 03:57:49.157484 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:49.157337    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13d95f4c-f162-4072-acf6-c2716bdbb893-istio-data" (OuterVolumeSpecName: "istio-data") pod "13d95f4c-f162-4072-acf6-c2716bdbb893" (UID: "13d95f4c-f162-4072-acf6-c2716bdbb893"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 03:57:49.157484 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:49.157361    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13d95f4c-f162-4072-acf6-c2716bdbb893-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "13d95f4c-f162-4072-acf6-c2716bdbb893" (UID: "13d95f4c-f162-4072-acf6-c2716bdbb893"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 03:57:49.157588 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:49.157487    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13d95f4c-f162-4072-acf6-c2716bdbb893-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "13d95f4c-f162-4072-acf6-c2716bdbb893" (UID: "13d95f4c-f162-4072-acf6-c2716bdbb893"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 03:57:49.157588 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:49.157516    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13d95f4c-f162-4072-acf6-c2716bdbb893-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "13d95f4c-f162-4072-acf6-c2716bdbb893" (UID: "13d95f4c-f162-4072-acf6-c2716bdbb893"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 03:57:49.157588 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:49.157572    2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13d95f4c-f162-4072-acf6-c2716bdbb893-istio-data\") on node \"ip-10-0-138-121.ec2.internal\" DevicePath \"\""
Sep 11 03:57:49.157756 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:49.157591    2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13d95f4c-f162-4072-acf6-c2716bdbb893-istiod-ca-cert\") on node \"ip-10-0-138-121.ec2.internal\" DevicePath \"\""
Sep 11 03:57:49.157756 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:49.157606    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13d95f4c-f162-4072-acf6-c2716bdbb893-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "13d95f4c-f162-4072-acf6-c2716bdbb893" (UID: "13d95f4c-f162-4072-acf6-c2716bdbb893"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 03:57:49.159386 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:49.159362    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/13d95f4c-f162-4072-acf6-c2716bdbb893-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "13d95f4c-f162-4072-acf6-c2716bdbb893" (UID: "13d95f4c-f162-4072-acf6-c2716bdbb893"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 03:57:49.159588 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:49.159557    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13d95f4c-f162-4072-acf6-c2716bdbb893-istio-token" (OuterVolumeSpecName: "istio-token") pod "13d95f4c-f162-4072-acf6-c2716bdbb893" (UID: "13d95f4c-f162-4072-acf6-c2716bdbb893"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 03:57:49.159690 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:49.159571    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13d95f4c-f162-4072-acf6-c2716bdbb893-kube-api-access-66w4k" (OuterVolumeSpecName: "kube-api-access-66w4k") pod "13d95f4c-f162-4072-acf6-c2716bdbb893" (UID: "13d95f4c-f162-4072-acf6-c2716bdbb893"). InnerVolumeSpecName "kube-api-access-66w4k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 03:57:49.159739 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:49.159727    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13d95f4c-f162-4072-acf6-c2716bdbb893-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "13d95f4c-f162-4072-acf6-c2716bdbb893" (UID: "13d95f4c-f162-4072-acf6-c2716bdbb893"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 03:57:49.258611 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:49.258573    2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13d95f4c-f162-4072-acf6-c2716bdbb893-istio-envoy\") on node \"ip-10-0-138-121.ec2.internal\" DevicePath \"\""
Sep 11 03:57:49.258611 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:49.258605    2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/13d95f4c-f162-4072-acf6-c2716bdbb893-istio-token\") on node \"ip-10-0-138-121.ec2.internal\" DevicePath \"\""
Sep 11 03:57:49.258611 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:49.258614    2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13d95f4c-f162-4072-acf6-c2716bdbb893-workload-certs\") on node \"ip-10-0-138-121.ec2.internal\" DevicePath \"\""
Sep 11 03:57:49.258857 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:49.258621    2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13d95f4c-f162-4072-acf6-c2716bdbb893-credential-socket\") on node \"ip-10-0-138-121.ec2.internal\" DevicePath \"\""
Sep 11 03:57:49.258857 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:49.258631    2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13d95f4c-f162-4072-acf6-c2716bdbb893-workload-socket\") on node \"ip-10-0-138-121.ec2.internal\" DevicePath \"\""
Sep 11 03:57:49.258857 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:49.258640    2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13d95f4c-f162-4072-acf6-c2716bdbb893-istio-podinfo\") on node \"ip-10-0-138-121.ec2.internal\" DevicePath \"\""
Sep 11 03:57:49.258857 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:49.258648    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-66w4k\" (UniqueName: \"kubernetes.io/projected/13d95f4c-f162-4072-acf6-c2716bdbb893-kube-api-access-66w4k\") on node \"ip-10-0-138-121.ec2.internal\" DevicePath \"\""
Sep 11 03:57:49.632096 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:49.632015    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-dd5b30-59cd644494-hqhn2"
Sep 11 03:57:49.691827 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:49.691787    2580 generic.go:356] "Generic (PLEG): container finished" podID="13d95f4c-f162-4072-acf6-c2716bdbb893" containerID="f824a43573fec36fb42af01cfd9e8dc1ac4edb147e729ca1f2db91072f001d22" exitCode=0
Sep 11 03:57:49.691984 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:49.691839    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-96c7d4-openshift-default-785b65d79d-j8777"
Sep 11 03:57:49.691984 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:49.691855    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-96c7d4-openshift-default-785b65d79d-j8777" event={"ID":"13d95f4c-f162-4072-acf6-c2716bdbb893","Type":"ContainerDied","Data":"f824a43573fec36fb42af01cfd9e8dc1ac4edb147e729ca1f2db91072f001d22"}
Sep 11 03:57:49.691984 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:49.691932    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-96c7d4-openshift-default-785b65d79d-j8777" event={"ID":"13d95f4c-f162-4072-acf6-c2716bdbb893","Type":"ContainerDied","Data":"4c1575dd55cd0f7c37fb29dee8fa128d32418ea4ff9bafe8b31a86197987a7ba"}
Sep 11 03:57:49.691984 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:49.691961    2580 scope.go:117] "RemoveContainer" containerID="f824a43573fec36fb42af01cfd9e8dc1ac4edb147e729ca1f2db91072f001d22"
Sep 11 03:57:49.703629 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:49.703592    2580 scope.go:117] "RemoveContainer" containerID="f824a43573fec36fb42af01cfd9e8dc1ac4edb147e729ca1f2db91072f001d22"
Sep 11 03:57:49.703995 ip-10-0-138-121 kubenswrapper[2580]: E0911 03:57:49.703966    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f824a43573fec36fb42af01cfd9e8dc1ac4edb147e729ca1f2db91072f001d22\": container with ID starting with f824a43573fec36fb42af01cfd9e8dc1ac4edb147e729ca1f2db91072f001d22 not found: ID does not exist" containerID="f824a43573fec36fb42af01cfd9e8dc1ac4edb147e729ca1f2db91072f001d22"
Sep 11 03:57:49.704052 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:49.704010    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f824a43573fec36fb42af01cfd9e8dc1ac4edb147e729ca1f2db91072f001d22"} err="failed to get container status \"f824a43573fec36fb42af01cfd9e8dc1ac4edb147e729ca1f2db91072f001d22\": rpc error: code = NotFound desc = could not find container \"f824a43573fec36fb42af01cfd9e8dc1ac4edb147e729ca1f2db91072f001d22\": container with ID starting with f824a43573fec36fb42af01cfd9e8dc1ac4edb147e729ca1f2db91072f001d22 not found: ID does not exist"
Sep 11 03:57:49.714367 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:49.714346    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-96c7d4-openshift-default-785b65d79d-j8777"]
Sep 11 03:57:49.719551 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:49.719530    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-noten-96c7d4-openshift-default-785b65d79d-j8777"]
Sep 11 03:57:50.464491 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:50.464459    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="13d95f4c-f162-4072-acf6-c2716bdbb893" path="/var/lib/kubelet/pods/13d95f4c-f162-4072-acf6-c2716bdbb893/volumes"
Sep 11 03:57:50.501894 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:50.501855    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-20ae9a-554444f99-7jp2g"]
Sep 11 03:57:50.502438 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:50.502418    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13d95f4c-f162-4072-acf6-c2716bdbb893" containerName="istio-proxy"
Sep 11 03:57:50.502553 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:50.502439    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="13d95f4c-f162-4072-acf6-c2716bdbb893" containerName="istio-proxy"
Sep 11 03:57:50.502618 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:50.502562    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="13d95f4c-f162-4072-acf6-c2716bdbb893" containerName="istio-proxy"
Sep 11 03:57:50.507462 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:50.507440    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-20ae9a-554444f99-7jp2g"
Sep 11 03:57:50.509524 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:50.509502    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-20ae9a-serving-cert\""
Sep 11 03:57:50.518297 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:50.518270    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-20ae9a-554444f99-7jp2g"]
Sep 11 03:57:50.573782 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:50.573748    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f77e9215-d1b1-4911-bf22-b3b6e1aba5af-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-20ae9a-554444f99-7jp2g\" (UID: \"f77e9215-d1b1-4911-bf22-b3b6e1aba5af\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-20ae9a-554444f99-7jp2g"
Sep 11 03:57:50.573954 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:50.573800    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f77e9215-d1b1-4911-bf22-b3b6e1aba5af-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-20ae9a-554444f99-7jp2g\" (UID: \"f77e9215-d1b1-4911-bf22-b3b6e1aba5af\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-20ae9a-554444f99-7jp2g"
Sep 11 03:57:50.573954 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:50.573829    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f77e9215-d1b1-4911-bf22-b3b6e1aba5af-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-20ae9a-554444f99-7jp2g\" (UID: \"f77e9215-d1b1-4911-bf22-b3b6e1aba5af\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-20ae9a-554444f99-7jp2g"
Sep 11 03:57:50.573954 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:50.573864    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmhdl\" (UniqueName: \"kubernetes.io/projected/f77e9215-d1b1-4911-bf22-b3b6e1aba5af-kube-api-access-cmhdl\") pod \"maas-api-e2e-shared-a-w2-20ae9a-554444f99-7jp2g\" (UID: \"f77e9215-d1b1-4911-bf22-b3b6e1aba5af\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-20ae9a-554444f99-7jp2g"
Sep 11 03:57:50.675183 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:50.675144    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f77e9215-d1b1-4911-bf22-b3b6e1aba5af-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-20ae9a-554444f99-7jp2g\" (UID: \"f77e9215-d1b1-4911-bf22-b3b6e1aba5af\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-20ae9a-554444f99-7jp2g"
Sep 11 03:57:50.675366 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:50.675201    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmhdl\" (UniqueName: \"kubernetes.io/projected/f77e9215-d1b1-4911-bf22-b3b6e1aba5af-kube-api-access-cmhdl\") pod \"maas-api-e2e-shared-a-w2-20ae9a-554444f99-7jp2g\" (UID: \"f77e9215-d1b1-4911-bf22-b3b6e1aba5af\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-20ae9a-554444f99-7jp2g"
Sep 11 03:57:50.675366 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:50.675259    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f77e9215-d1b1-4911-bf22-b3b6e1aba5af-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-20ae9a-554444f99-7jp2g\" (UID: \"f77e9215-d1b1-4911-bf22-b3b6e1aba5af\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-20ae9a-554444f99-7jp2g"
Sep 11 03:57:50.675366 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:50.675308    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f77e9215-d1b1-4911-bf22-b3b6e1aba5af-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-20ae9a-554444f99-7jp2g\" (UID: \"f77e9215-d1b1-4911-bf22-b3b6e1aba5af\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-20ae9a-554444f99-7jp2g"
Sep 11 03:57:50.675954 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:50.675917    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f77e9215-d1b1-4911-bf22-b3b6e1aba5af-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-20ae9a-554444f99-7jp2g\" (UID: \"f77e9215-d1b1-4911-bf22-b3b6e1aba5af\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-20ae9a-554444f99-7jp2g"
Sep 11 03:57:50.677849 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:50.677822    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f77e9215-d1b1-4911-bf22-b3b6e1aba5af-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-20ae9a-554444f99-7jp2g\" (UID: \"f77e9215-d1b1-4911-bf22-b3b6e1aba5af\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-20ae9a-554444f99-7jp2g"
Sep 11 03:57:50.677976 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:50.677950    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f77e9215-d1b1-4911-bf22-b3b6e1aba5af-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-20ae9a-554444f99-7jp2g\" (UID: \"f77e9215-d1b1-4911-bf22-b3b6e1aba5af\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-20ae9a-554444f99-7jp2g"
Sep 11 03:57:50.682932 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:50.682908    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmhdl\" (UniqueName: \"kubernetes.io/projected/f77e9215-d1b1-4911-bf22-b3b6e1aba5af-kube-api-access-cmhdl\") pod \"maas-api-e2e-shared-a-w2-20ae9a-554444f99-7jp2g\" (UID: \"f77e9215-d1b1-4911-bf22-b3b6e1aba5af\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-20ae9a-554444f99-7jp2g"
Sep 11 03:57:50.819986 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:50.819901    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-20ae9a-554444f99-7jp2g"
Sep 11 03:57:51.077231 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:51.077147    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-20ae9a-87496ffbb-pf2hj"]
Sep 11 03:57:51.081002 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:51.080981    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-20ae9a-87496ffbb-pf2hj"
Sep 11 03:57:51.083262 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:51.083244    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-20ae9a\""
Sep 11 03:57:51.083262 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:51.083253    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-20ae9a-dockercfg-pjqlm\""
Sep 11 03:57:51.091170 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:51.091145    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-20ae9a-87496ffbb-pf2hj"]
Sep 11 03:57:51.094529 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:51.094502    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-20ae9a-554444f99-7jp2g"]
Sep 11 03:57:51.098687 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:57:51.098609    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf77e9215_d1b1_4911_bf22_b3b6e1aba5af.slice/crio-5793c30f64fa8f409cab15ecff35a268c63c592244a0e588e16036dd5ab858a6 WatchSource:0}: Error finding container 5793c30f64fa8f409cab15ecff35a268c63c592244a0e588e16036dd5ab858a6: Status 404 returned error can't find the container with id 5793c30f64fa8f409cab15ecff35a268c63c592244a0e588e16036dd5ab858a6
Sep 11 03:57:51.180056 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:51.180018    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dbb75e41-d406-478a-8d2e-ae3b63efe5ba-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-20ae9a-87496ffbb-pf2hj\" (UID: \"dbb75e41-d406-478a-8d2e-ae3b63efe5ba\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-20ae9a-87496ffbb-pf2hj"
Sep 11 03:57:51.180224 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:51.180105    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfpcn\" (UniqueName: \"kubernetes.io/projected/dbb75e41-d406-478a-8d2e-ae3b63efe5ba-kube-api-access-qfpcn\") pod \"payload-pre-processing-e2e-shared-a-w2-20ae9a-87496ffbb-pf2hj\" (UID: \"dbb75e41-d406-478a-8d2e-ae3b63efe5ba\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-20ae9a-87496ffbb-pf2hj"
Sep 11 03:57:51.281532 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:51.281491    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfpcn\" (UniqueName: \"kubernetes.io/projected/dbb75e41-d406-478a-8d2e-ae3b63efe5ba-kube-api-access-qfpcn\") pod \"payload-pre-processing-e2e-shared-a-w2-20ae9a-87496ffbb-pf2hj\" (UID: \"dbb75e41-d406-478a-8d2e-ae3b63efe5ba\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-20ae9a-87496ffbb-pf2hj"
Sep 11 03:57:51.281726 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:51.281596    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dbb75e41-d406-478a-8d2e-ae3b63efe5ba-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-20ae9a-87496ffbb-pf2hj\" (UID: \"dbb75e41-d406-478a-8d2e-ae3b63efe5ba\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-20ae9a-87496ffbb-pf2hj"
Sep 11 03:57:51.282260 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:51.282241    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dbb75e41-d406-478a-8d2e-ae3b63efe5ba-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-20ae9a-87496ffbb-pf2hj\" (UID: \"dbb75e41-d406-478a-8d2e-ae3b63efe5ba\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-20ae9a-87496ffbb-pf2hj"
Sep 11 03:57:51.288784 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:51.288755    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfpcn\" (UniqueName: \"kubernetes.io/projected/dbb75e41-d406-478a-8d2e-ae3b63efe5ba-kube-api-access-qfpcn\") pod \"payload-pre-processing-e2e-shared-a-w2-20ae9a-87496ffbb-pf2hj\" (UID: \"dbb75e41-d406-478a-8d2e-ae3b63efe5ba\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-20ae9a-87496ffbb-pf2hj"
Sep 11 03:57:51.392853 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:51.392750    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-20ae9a-87496ffbb-pf2hj"
Sep 11 03:57:51.665456 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:51.665428    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-20ae9a-87496ffbb-pf2hj"]
Sep 11 03:57:51.667486 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:57:51.667457    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbb75e41_d406_478a_8d2e_ae3b63efe5ba.slice/crio-92693acc8f54459de04f83cf45aa5bcc1d4deec2cbc99a583f1db9da9e1c0bfc WatchSource:0}: Error finding container 92693acc8f54459de04f83cf45aa5bcc1d4deec2cbc99a583f1db9da9e1c0bfc: Status 404 returned error can't find the container with id 92693acc8f54459de04f83cf45aa5bcc1d4deec2cbc99a583f1db9da9e1c0bfc
Sep 11 03:57:51.703800 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:51.703773    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-20ae9a-554444f99-7jp2g" event={"ID":"f77e9215-d1b1-4911-bf22-b3b6e1aba5af","Type":"ContainerStarted","Data":"5793c30f64fa8f409cab15ecff35a268c63c592244a0e588e16036dd5ab858a6"}
Sep 11 03:57:51.705131 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:51.705108    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-20ae9a-87496ffbb-pf2hj" event={"ID":"dbb75e41-d406-478a-8d2e-ae3b63efe5ba","Type":"ContainerStarted","Data":"92693acc8f54459de04f83cf45aa5bcc1d4deec2cbc99a583f1db9da9e1c0bfc"}
Sep 11 03:57:52.711451 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:52.711416    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-20ae9a-87496ffbb-pf2hj" event={"ID":"dbb75e41-d406-478a-8d2e-ae3b63efe5ba","Type":"ContainerStarted","Data":"8e191c59eb1d5146b2dbb05e8063732e75f75ed26e2fcd33e495ac981507c985"}
Sep 11 03:57:52.711895 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:52.711483    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-20ae9a-87496ffbb-pf2hj"
Sep 11 03:57:52.712976 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:52.712953    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-20ae9a-554444f99-7jp2g" event={"ID":"f77e9215-d1b1-4911-bf22-b3b6e1aba5af","Type":"ContainerStarted","Data":"a28d5eae24d1a94850fba2cb21b7ae79828fa4a8730d7559f43181f63930b5fb"}
Sep 11 03:57:52.713097 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:52.713078    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-20ae9a-554444f99-7jp2g"
Sep 11 03:57:52.726604 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:52.726559    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-20ae9a-87496ffbb-pf2hj" podStartSLOduration=1.726543207 podStartE2EDuration="1.726543207s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 03:57:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:57:52.726006802 +0000 UTC m=+1060.884493125" watchObservedRunningTime="2026-09-11 03:57:52.726543207 +0000 UTC m=+1060.885029530"
Sep 11 03:57:52.742790 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:52.742744    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-20ae9a-554444f99-7jp2g" podStartSLOduration=2.143251995 podStartE2EDuration="2.74272496s" totalImagesPullingTime="599.472965ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 03:57:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:57:52.740808572 +0000 UTC m=+1060.899294896" watchObservedRunningTime="2026-09-11 03:57:52.74272496 +0000 UTC m=+1060.901211312"
Sep 11 03:57:58.723346 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:57:58.723311    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-20ae9a-554444f99-7jp2g"
Sep 11 03:58:03.720522 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:58:03.720491    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-20ae9a-87496ffbb-pf2hj"
Sep 11 03:58:13.967899 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:58:13.967809    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-7dc787-openshift-default-78765f97b-vhwr6"]
Sep 11 03:58:13.971429 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:58:13.971400    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-7dc787-openshift-default-78765f97b-vhwr6"
Sep 11 03:58:13.973987 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:58:13.973961    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-7dc787-openshift-default-dockercfg-d8cw9\""
Sep 11 03:58:13.983797 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:58:13.983772    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-7dc787-openshift-default-78765f97b-vhwr6"]
Sep 11 03:58:13.999367 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:58:13.999335    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0595e62e-1ef0-4ca3-a28e-0753787602f1-istio-envoy\") pod \"e2e-shared-b-w2-7dc787-openshift-default-78765f97b-vhwr6\" (UID: \"0595e62e-1ef0-4ca3-a28e-0753787602f1\") " pod="openshift-ingress/e2e-shared-b-w2-7dc787-openshift-default-78765f97b-vhwr6"
Sep 11 03:58:13.999509 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:58:13.999382    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0595e62e-1ef0-4ca3-a28e-0753787602f1-workload-certs\") pod \"e2e-shared-b-w2-7dc787-openshift-default-78765f97b-vhwr6\" (UID: \"0595e62e-1ef0-4ca3-a28e-0753787602f1\") " pod="openshift-ingress/e2e-shared-b-w2-7dc787-openshift-default-78765f97b-vhwr6"
Sep 11 03:58:13.999509 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:58:13.999449    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0595e62e-1ef0-4ca3-a28e-0753787602f1-workload-socket\") pod \"e2e-shared-b-w2-7dc787-openshift-default-78765f97b-vhwr6\" (UID: \"0595e62e-1ef0-4ca3-a28e-0753787602f1\") " pod="openshift-ingress/e2e-shared-b-w2-7dc787-openshift-default-78765f97b-vhwr6"
Sep 11 03:58:13.999509 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:58:13.999501    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvng9\" (UniqueName: \"kubernetes.io/projected/0595e62e-1ef0-4ca3-a28e-0753787602f1-kube-api-access-zvng9\") pod \"e2e-shared-b-w2-7dc787-openshift-default-78765f97b-vhwr6\" (UID: \"0595e62e-1ef0-4ca3-a28e-0753787602f1\") " pod="openshift-ingress/e2e-shared-b-w2-7dc787-openshift-default-78765f97b-vhwr6"
Sep 11 03:58:13.999706 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:58:13.999573    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0595e62e-1ef0-4ca3-a28e-0753787602f1-istio-podinfo\") pod \"e2e-shared-b-w2-7dc787-openshift-default-78765f97b-vhwr6\" (UID: \"0595e62e-1ef0-4ca3-a28e-0753787602f1\") " pod="openshift-ingress/e2e-shared-b-w2-7dc787-openshift-default-78765f97b-vhwr6"
Sep 11 03:58:13.999706 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:58:13.999602    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0595e62e-1ef0-4ca3-a28e-0753787602f1-istio-token\") pod \"e2e-shared-b-w2-7dc787-openshift-default-78765f97b-vhwr6\" (UID: \"0595e62e-1ef0-4ca3-a28e-0753787602f1\") " pod="openshift-ingress/e2e-shared-b-w2-7dc787-openshift-default-78765f97b-vhwr6"
Sep 11 03:58:13.999706 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:58:13.999628    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0595e62e-1ef0-4ca3-a28e-0753787602f1-istiod-ca-cert\") pod \"e2e-shared-b-w2-7dc787-openshift-default-78765f97b-vhwr6\" (UID: \"0595e62e-1ef0-4ca3-a28e-0753787602f1\") " pod="openshift-ingress/e2e-shared-b-w2-7dc787-openshift-default-78765f97b-vhwr6"
Sep 11 03:58:13.999843 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:58:13.999750    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0595e62e-1ef0-4ca3-a28e-0753787602f1-istio-data\") pod \"e2e-shared-b-w2-7dc787-openshift-default-78765f97b-vhwr6\" (UID: \"0595e62e-1ef0-4ca3-a28e-0753787602f1\") " pod="openshift-ingress/e2e-shared-b-w2-7dc787-openshift-default-78765f97b-vhwr6"
Sep 11 03:58:13.999843 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:58:13.999797    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0595e62e-1ef0-4ca3-a28e-0753787602f1-credential-socket\") pod \"e2e-shared-b-w2-7dc787-openshift-default-78765f97b-vhwr6\" (UID: \"0595e62e-1ef0-4ca3-a28e-0753787602f1\") " pod="openshift-ingress/e2e-shared-b-w2-7dc787-openshift-default-78765f97b-vhwr6"
Sep 11 03:58:14.100352 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:58:14.100317    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0595e62e-1ef0-4ca3-a28e-0753787602f1-istio-data\") pod \"e2e-shared-b-w2-7dc787-openshift-default-78765f97b-vhwr6\" (UID: \"0595e62e-1ef0-4ca3-a28e-0753787602f1\") " pod="openshift-ingress/e2e-shared-b-w2-7dc787-openshift-default-78765f97b-vhwr6"
Sep 11 03:58:14.100352 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:58:14.100354    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0595e62e-1ef0-4ca3-a28e-0753787602f1-credential-socket\") pod \"e2e-shared-b-w2-7dc787-openshift-default-78765f97b-vhwr6\" (UID: \"0595e62e-1ef0-4ca3-a28e-0753787602f1\") " pod="openshift-ingress/e2e-shared-b-w2-7dc787-openshift-default-78765f97b-vhwr6"
Sep 11 03:58:14.100596 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:58:14.100389    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0595e62e-1ef0-4ca3-a28e-0753787602f1-istio-envoy\") pod \"e2e-shared-b-w2-7dc787-openshift-default-78765f97b-vhwr6\" (UID: \"0595e62e-1ef0-4ca3-a28e-0753787602f1\") " pod="openshift-ingress/e2e-shared-b-w2-7dc787-openshift-default-78765f97b-vhwr6"
Sep 11 03:58:14.100596 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:58:14.100414    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0595e62e-1ef0-4ca3-a28e-0753787602f1-workload-certs\") pod \"e2e-shared-b-w2-7dc787-openshift-default-78765f97b-vhwr6\" (UID: \"0595e62e-1ef0-4ca3-a28e-0753787602f1\") " pod="openshift-ingress/e2e-shared-b-w2-7dc787-openshift-default-78765f97b-vhwr6"
Sep 11 03:58:14.100596 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:58:14.100441    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0595e62e-1ef0-4ca3-a28e-0753787602f1-workload-socket\") pod \"e2e-shared-b-w2-7dc787-openshift-default-78765f97b-vhwr6\" (UID: \"0595e62e-1ef0-4ca3-a28e-0753787602f1\") " pod="openshift-ingress/e2e-shared-b-w2-7dc787-openshift-default-78765f97b-vhwr6"
Sep 11 03:58:14.100596 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:58:14.100463    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvng9\" (UniqueName: \"kubernetes.io/projected/0595e62e-1ef0-4ca3-a28e-0753787602f1-kube-api-access-zvng9\") pod \"e2e-shared-b-w2-7dc787-openshift-default-78765f97b-vhwr6\" (UID: \"0595e62e-1ef0-4ca3-a28e-0753787602f1\") " pod="openshift-ingress/e2e-shared-b-w2-7dc787-openshift-default-78765f97b-vhwr6"
Sep 11 03:58:14.100596 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:58:14.100499    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0595e62e-1ef0-4ca3-a28e-0753787602f1-istio-podinfo\") pod \"e2e-shared-b-w2-7dc787-openshift-default-78765f97b-vhwr6\" (UID: \"0595e62e-1ef0-4ca3-a28e-0753787602f1\") " pod="openshift-ingress/e2e-shared-b-w2-7dc787-openshift-default-78765f97b-vhwr6"
Sep 11 03:58:14.100596 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:58:14.100527    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0595e62e-1ef0-4ca3-a28e-0753787602f1-istio-token\") pod \"e2e-shared-b-w2-7dc787-openshift-default-78765f97b-vhwr6\" (UID: \"0595e62e-1ef0-4ca3-a28e-0753787602f1\") " pod="openshift-ingress/e2e-shared-b-w2-7dc787-openshift-default-78765f97b-vhwr6"
Sep 11 03:58:14.100596 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:58:14.100549    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0595e62e-1ef0-4ca3-a28e-0753787602f1-istiod-ca-cert\") pod \"e2e-shared-b-w2-7dc787-openshift-default-78765f97b-vhwr6\" (UID: \"0595e62e-1ef0-4ca3-a28e-0753787602f1\") " pod="openshift-ingress/e2e-shared-b-w2-7dc787-openshift-default-78765f97b-vhwr6"
Sep 11 03:58:14.100955 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:58:14.100778    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0595e62e-1ef0-4ca3-a28e-0753787602f1-istio-data\") pod \"e2e-shared-b-w2-7dc787-openshift-default-78765f97b-vhwr6\" (UID: \"0595e62e-1ef0-4ca3-a28e-0753787602f1\") " pod="openshift-ingress/e2e-shared-b-w2-7dc787-openshift-default-78765f97b-vhwr6"
Sep 11 03:58:14.100955 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:58:14.100840    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0595e62e-1ef0-4ca3-a28e-0753787602f1-workload-certs\") pod \"e2e-shared-b-w2-7dc787-openshift-default-78765f97b-vhwr6\" (UID: \"0595e62e-1ef0-4ca3-a28e-0753787602f1\") " pod="openshift-ingress/e2e-shared-b-w2-7dc787-openshift-default-78765f97b-vhwr6"
Sep 11 03:58:14.100955 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:58:14.100870    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0595e62e-1ef0-4ca3-a28e-0753787602f1-workload-socket\") pod \"e2e-shared-b-w2-7dc787-openshift-default-78765f97b-vhwr6\" (UID: \"0595e62e-1ef0-4ca3-a28e-0753787602f1\") " pod="openshift-ingress/e2e-shared-b-w2-7dc787-openshift-default-78765f97b-vhwr6"
Sep 11 03:58:14.101140 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:58:14.101117    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0595e62e-1ef0-4ca3-a28e-0753787602f1-credential-socket\") pod \"e2e-shared-b-w2-7dc787-openshift-default-78765f97b-vhwr6\" (UID: \"0595e62e-1ef0-4ca3-a28e-0753787602f1\") " pod="openshift-ingress/e2e-shared-b-w2-7dc787-openshift-default-78765f97b-vhwr6"
Sep 11 03:58:14.101268 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:58:14.101207    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0595e62e-1ef0-4ca3-a28e-0753787602f1-istiod-ca-cert\") pod \"e2e-shared-b-w2-7dc787-openshift-default-78765f97b-vhwr6\" (UID: \"0595e62e-1ef0-4ca3-a28e-0753787602f1\") " pod="openshift-ingress/e2e-shared-b-w2-7dc787-openshift-default-78765f97b-vhwr6"
Sep 11 03:58:14.102833 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:58:14.102814    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0595e62e-1ef0-4ca3-a28e-0753787602f1-istio-envoy\") pod \"e2e-shared-b-w2-7dc787-openshift-default-78765f97b-vhwr6\" (UID: \"0595e62e-1ef0-4ca3-a28e-0753787602f1\") " pod="openshift-ingress/e2e-shared-b-w2-7dc787-openshift-default-78765f97b-vhwr6"
Sep 11 03:58:14.103107 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:58:14.103088    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0595e62e-1ef0-4ca3-a28e-0753787602f1-istio-podinfo\") pod \"e2e-shared-b-w2-7dc787-openshift-default-78765f97b-vhwr6\" (UID: \"0595e62e-1ef0-4ca3-a28e-0753787602f1\") " pod="openshift-ingress/e2e-shared-b-w2-7dc787-openshift-default-78765f97b-vhwr6"
Sep 11 03:58:14.108839 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:58:14.108818    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0595e62e-1ef0-4ca3-a28e-0753787602f1-istio-token\") pod \"e2e-shared-b-w2-7dc787-openshift-default-78765f97b-vhwr6\" (UID: \"0595e62e-1ef0-4ca3-a28e-0753787602f1\") " pod="openshift-ingress/e2e-shared-b-w2-7dc787-openshift-default-78765f97b-vhwr6"
Sep 11 03:58:14.109042 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:58:14.109022    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvng9\" (UniqueName: \"kubernetes.io/projected/0595e62e-1ef0-4ca3-a28e-0753787602f1-kube-api-access-zvng9\") pod \"e2e-shared-b-w2-7dc787-openshift-default-78765f97b-vhwr6\" (UID: \"0595e62e-1ef0-4ca3-a28e-0753787602f1\") " pod="openshift-ingress/e2e-shared-b-w2-7dc787-openshift-default-78765f97b-vhwr6"
Sep 11 03:58:14.282756 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:58:14.282644    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-7dc787-openshift-default-78765f97b-vhwr6"
Sep 11 03:58:14.556579 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:58:14.556549    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-7dc787-openshift-default-78765f97b-vhwr6"]
Sep 11 03:58:14.559047 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:58:14.559021    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0595e62e_1ef0_4ca3_a28e_0753787602f1.slice/crio-ced06bcb05db9404807db7fda3a2b765aaf8ea5de2a1cc63791bdfab578a52cf WatchSource:0}: Error finding container ced06bcb05db9404807db7fda3a2b765aaf8ea5de2a1cc63791bdfab578a52cf: Status 404 returned error can't find the container with id ced06bcb05db9404807db7fda3a2b765aaf8ea5de2a1cc63791bdfab578a52cf
Sep 11 03:58:14.562038 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:58:14.562001    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 03:58:14.562113 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:58:14.562083    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 03:58:14.562158 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:58:14.562122    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 03:58:14.825896 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:58:14.825710    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-7dc787-openshift-default-78765f97b-vhwr6" event={"ID":"0595e62e-1ef0-4ca3-a28e-0753787602f1","Type":"ContainerStarted","Data":"29bb1d52146d34beaa58e3190b9a5b43a9b013c87640a67ae6dd5c3cd1816f75"}
Sep 11 03:58:14.825896 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:58:14.825758    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-7dc787-openshift-default-78765f97b-vhwr6" event={"ID":"0595e62e-1ef0-4ca3-a28e-0753787602f1","Type":"ContainerStarted","Data":"ced06bcb05db9404807db7fda3a2b765aaf8ea5de2a1cc63791bdfab578a52cf"}
Sep 11 03:58:14.844281 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:58:14.844229    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-7dc787-openshift-default-78765f97b-vhwr6" podStartSLOduration=1.8442147690000001 podStartE2EDuration="1.844214769s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 03:58:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:58:14.842091365 +0000 UTC m=+1083.000577700" watchObservedRunningTime="2026-09-11 03:58:14.844214769 +0000 UTC m=+1083.002701091"
Sep 11 03:58:15.283756 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:58:15.283722    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-7dc787-openshift-default-78765f97b-vhwr6"
Sep 11 03:58:15.288953 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:58:15.288926    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-7dc787-openshift-default-78765f97b-vhwr6"
Sep 11 03:58:15.830610 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:58:15.830579    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-7dc787-openshift-default-78765f97b-vhwr6"
Sep 11 03:58:15.831536 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:58:15.831519    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-7dc787-openshift-default-78765f97b-vhwr6"
Sep 11 03:58:34.931224 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:58:34.931191    2580 generic.go:356] "Generic (PLEG): container finished" podID="0595e62e-1ef0-4ca3-a28e-0753787602f1" containerID="29bb1d52146d34beaa58e3190b9a5b43a9b013c87640a67ae6dd5c3cd1816f75" exitCode=0
Sep 11 03:58:34.931671 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:58:34.931266    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-7dc787-openshift-default-78765f97b-vhwr6" event={"ID":"0595e62e-1ef0-4ca3-a28e-0753787602f1","Type":"ContainerDied","Data":"29bb1d52146d34beaa58e3190b9a5b43a9b013c87640a67ae6dd5c3cd1816f75"}
Sep 11 03:58:34.931725 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:58:34.931693    2580 scope.go:117] "RemoveContainer" containerID="29bb1d52146d34beaa58e3190b9a5b43a9b013c87640a67ae6dd5c3cd1816f75"
Sep 11 03:58:34.932861 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:58:34.932829    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 03:58:34.932967 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:58:34.932897    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 03:58:34.932967 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:58:34.932943    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 03:58:35.937715 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:58:35.937656    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-7dc787-openshift-default-78765f97b-vhwr6" event={"ID":"0595e62e-1ef0-4ca3-a28e-0753787602f1","Type":"ContainerStarted","Data":"d9e8fdaffb8e4734984afe11531416c5acb13c8700ad36a8d8daf685cf668b6f"}
Sep 11 03:58:36.283237 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:58:36.283136    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-7dc787-openshift-default-78765f97b-vhwr6"
Sep 11 03:58:36.283431 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:58:36.283363    2580 patch_prober.go:28] interesting pod/e2e-shared-b-w2-7dc787-openshift-default-78765f97b-vhwr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Sep 11 03:58:36.283504 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:58:36.283420    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-7dc787-openshift-default-78765f97b-vhwr6" podUID="0595e62e-1ef0-4ca3-a28e-0753787602f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Sep 11 03:58:37.283353 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:58:37.283313    2580 patch_prober.go:28] interesting pod/e2e-shared-b-w2-7dc787-openshift-default-78765f97b-vhwr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Sep 11 03:58:37.283750 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:58:37.283371    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-7dc787-openshift-default-78765f97b-vhwr6" podUID="0595e62e-1ef0-4ca3-a28e-0753787602f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Sep 11 03:58:38.283837 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:58:38.283796    2580 patch_prober.go:28] interesting pod/e2e-shared-b-w2-7dc787-openshift-default-78765f97b-vhwr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Sep 11 03:58:38.284227 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:58:38.283862    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-7dc787-openshift-default-78765f97b-vhwr6" podUID="0595e62e-1ef0-4ca3-a28e-0753787602f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Sep 11 03:58:39.283346 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:58:39.283307    2580 patch_prober.go:28] interesting pod/e2e-shared-b-w2-7dc787-openshift-default-78765f97b-vhwr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Sep 11 03:58:39.283534 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:58:39.283365    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-7dc787-openshift-default-78765f97b-vhwr6" podUID="0595e62e-1ef0-4ca3-a28e-0753787602f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Sep 11 03:58:40.283391 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:58:40.283348    2580 patch_prober.go:28] interesting pod/e2e-shared-b-w2-7dc787-openshift-default-78765f97b-vhwr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Sep 11 03:58:40.283821 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:58:40.283452    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-7dc787-openshift-default-78765f97b-vhwr6" podUID="0595e62e-1ef0-4ca3-a28e-0753787602f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Sep 11 03:58:41.350781 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:58:41.350746    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-7dc787-openshift-default-78765f97b-vhwr6"
Sep 11 03:58:41.351304 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:58:41.351283    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-7dc787-openshift-default-78765f97b-vhwr6"
Sep 11 03:58:41.352129 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:58:41.352107    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-7dc787-openshift-default-78765f97b-vhwr6"
Sep 11 03:58:43.594451 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:58:43.594413    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-dd5b30/auth-test-model-dd5b30-kserve-b5b6bc859-7gd9l"]
Sep 11 03:58:43.598021 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:58:43.598001    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-dd5b30/auth-test-model-dd5b30-kserve-b5b6bc859-7gd9l"
Sep 11 03:58:43.601304 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:58:43.601284    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-dd5b30\"/\"openshift-service-ca.crt\""
Sep 11 03:58:43.601546 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:58:43.601526    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-dd5b30\"/\"default-dockercfg-lvpcb\""
Sep 11 03:58:43.601690 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:58:43.601605    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-dd5b30\"/\"kube-root-ca.crt\""
Sep 11 03:58:43.601758 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:58:43.601728    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-dd5b30\"/\"auth-test-model-dd5b30-kserve-self-signed-certs\""
Sep 11 03:58:43.620508 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:58:43.620478    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-dd5b30/auth-test-model-dd5b30-kserve-b5b6bc859-7gd9l"]
Sep 11 03:58:43.702573 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:58:43.702520    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5738b326-7db0-463d-9d63-a98dc9f51a78-tmp-dir\") pod \"auth-test-model-dd5b30-kserve-b5b6bc859-7gd9l\" (UID: \"5738b326-7db0-463d-9d63-a98dc9f51a78\") " pod="ai-tenant-e2e-shared-b-w3-dd5b30/auth-test-model-dd5b30-kserve-b5b6bc859-7gd9l"
Sep 11 03:58:43.702777 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:58:43.702580    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5738b326-7db0-463d-9d63-a98dc9f51a78-model-cache\") pod \"auth-test-model-dd5b30-kserve-b5b6bc859-7gd9l\" (UID: \"5738b326-7db0-463d-9d63-a98dc9f51a78\") " pod="ai-tenant-e2e-shared-b-w3-dd5b30/auth-test-model-dd5b30-kserve-b5b6bc859-7gd9l"
Sep 11 03:58:43.702777 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:58:43.702688    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qz6fv\" (UniqueName: \"kubernetes.io/projected/5738b326-7db0-463d-9d63-a98dc9f51a78-kube-api-access-qz6fv\") pod \"auth-test-model-dd5b30-kserve-b5b6bc859-7gd9l\" (UID: \"5738b326-7db0-463d-9d63-a98dc9f51a78\") " pod="ai-tenant-e2e-shared-b-w3-dd5b30/auth-test-model-dd5b30-kserve-b5b6bc859-7gd9l"
Sep 11 03:58:43.702777 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:58:43.702719    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5738b326-7db0-463d-9d63-a98dc9f51a78-dshm\") pod \"auth-test-model-dd5b30-kserve-b5b6bc859-7gd9l\" (UID: \"5738b326-7db0-463d-9d63-a98dc9f51a78\") " pod="ai-tenant-e2e-shared-b-w3-dd5b30/auth-test-model-dd5b30-kserve-b5b6bc859-7gd9l"
Sep 11 03:58:43.702915 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:58:43.702849    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5738b326-7db0-463d-9d63-a98dc9f51a78-tls-certs\") pod \"auth-test-model-dd5b30-kserve-b5b6bc859-7gd9l\" (UID: \"5738b326-7db0-463d-9d63-a98dc9f51a78\") " pod="ai-tenant-e2e-shared-b-w3-dd5b30/auth-test-model-dd5b30-kserve-b5b6bc859-7gd9l"
Sep 11 03:58:43.702915 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:58:43.702877    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5738b326-7db0-463d-9d63-a98dc9f51a78-home\") pod \"auth-test-model-dd5b30-kserve-b5b6bc859-7gd9l\" (UID: \"5738b326-7db0-463d-9d63-a98dc9f51a78\") " pod="ai-tenant-e2e-shared-b-w3-dd5b30/auth-test-model-dd5b30-kserve-b5b6bc859-7gd9l"
Sep 11 03:58:43.804099 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:58:43.804063    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5738b326-7db0-463d-9d63-a98dc9f51a78-tls-certs\") pod \"auth-test-model-dd5b30-kserve-b5b6bc859-7gd9l\" (UID: \"5738b326-7db0-463d-9d63-a98dc9f51a78\") " pod="ai-tenant-e2e-shared-b-w3-dd5b30/auth-test-model-dd5b30-kserve-b5b6bc859-7gd9l"
Sep 11 03:58:43.804099 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:58:43.804101    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5738b326-7db0-463d-9d63-a98dc9f51a78-home\") pod \"auth-test-model-dd5b30-kserve-b5b6bc859-7gd9l\" (UID: \"5738b326-7db0-463d-9d63-a98dc9f51a78\") " pod="ai-tenant-e2e-shared-b-w3-dd5b30/auth-test-model-dd5b30-kserve-b5b6bc859-7gd9l"
Sep 11 03:58:43.804306 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:58:43.804232    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5738b326-7db0-463d-9d63-a98dc9f51a78-tmp-dir\") pod \"auth-test-model-dd5b30-kserve-b5b6bc859-7gd9l\" (UID: \"5738b326-7db0-463d-9d63-a98dc9f51a78\") " pod="ai-tenant-e2e-shared-b-w3-dd5b30/auth-test-model-dd5b30-kserve-b5b6bc859-7gd9l"
Sep 11 03:58:43.804306 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:58:43.804270    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5738b326-7db0-463d-9d63-a98dc9f51a78-model-cache\") pod \"auth-test-model-dd5b30-kserve-b5b6bc859-7gd9l\" (UID: \"5738b326-7db0-463d-9d63-a98dc9f51a78\") " pod="ai-tenant-e2e-shared-b-w3-dd5b30/auth-test-model-dd5b30-kserve-b5b6bc859-7gd9l"
Sep 11 03:58:43.804386 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:58:43.804324    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qz6fv\" (UniqueName: \"kubernetes.io/projected/5738b326-7db0-463d-9d63-a98dc9f51a78-kube-api-access-qz6fv\") pod \"auth-test-model-dd5b30-kserve-b5b6bc859-7gd9l\" (UID: \"5738b326-7db0-463d-9d63-a98dc9f51a78\") " pod="ai-tenant-e2e-shared-b-w3-dd5b30/auth-test-model-dd5b30-kserve-b5b6bc859-7gd9l"
Sep 11 03:58:43.804386 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:58:43.804357    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5738b326-7db0-463d-9d63-a98dc9f51a78-dshm\") pod \"auth-test-model-dd5b30-kserve-b5b6bc859-7gd9l\" (UID: \"5738b326-7db0-463d-9d63-a98dc9f51a78\") " pod="ai-tenant-e2e-shared-b-w3-dd5b30/auth-test-model-dd5b30-kserve-b5b6bc859-7gd9l"
Sep 11 03:58:43.804526 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:58:43.804507    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5738b326-7db0-463d-9d63-a98dc9f51a78-home\") pod \"auth-test-model-dd5b30-kserve-b5b6bc859-7gd9l\" (UID: \"5738b326-7db0-463d-9d63-a98dc9f51a78\") " pod="ai-tenant-e2e-shared-b-w3-dd5b30/auth-test-model-dd5b30-kserve-b5b6bc859-7gd9l"
Sep 11 03:58:43.804592 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:58:43.804545    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5738b326-7db0-463d-9d63-a98dc9f51a78-tmp-dir\") pod \"auth-test-model-dd5b30-kserve-b5b6bc859-7gd9l\" (UID: \"5738b326-7db0-463d-9d63-a98dc9f51a78\") " pod="ai-tenant-e2e-shared-b-w3-dd5b30/auth-test-model-dd5b30-kserve-b5b6bc859-7gd9l"
Sep 11 03:58:43.804753 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:58:43.804715    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5738b326-7db0-463d-9d63-a98dc9f51a78-model-cache\") pod \"auth-test-model-dd5b30-kserve-b5b6bc859-7gd9l\" (UID: \"5738b326-7db0-463d-9d63-a98dc9f51a78\") " pod="ai-tenant-e2e-shared-b-w3-dd5b30/auth-test-model-dd5b30-kserve-b5b6bc859-7gd9l"
Sep 11 03:58:43.806508 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:58:43.806486    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5738b326-7db0-463d-9d63-a98dc9f51a78-dshm\") pod \"auth-test-model-dd5b30-kserve-b5b6bc859-7gd9l\" (UID: \"5738b326-7db0-463d-9d63-a98dc9f51a78\") " pod="ai-tenant-e2e-shared-b-w3-dd5b30/auth-test-model-dd5b30-kserve-b5b6bc859-7gd9l"
Sep 11 03:58:43.806789 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:58:43.806773    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5738b326-7db0-463d-9d63-a98dc9f51a78-tls-certs\") pod \"auth-test-model-dd5b30-kserve-b5b6bc859-7gd9l\" (UID: \"5738b326-7db0-463d-9d63-a98dc9f51a78\") " pod="ai-tenant-e2e-shared-b-w3-dd5b30/auth-test-model-dd5b30-kserve-b5b6bc859-7gd9l"
Sep 11 03:58:43.811401 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:58:43.811379    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qz6fv\" (UniqueName: \"kubernetes.io/projected/5738b326-7db0-463d-9d63-a98dc9f51a78-kube-api-access-qz6fv\") pod \"auth-test-model-dd5b30-kserve-b5b6bc859-7gd9l\" (UID: \"5738b326-7db0-463d-9d63-a98dc9f51a78\") " pod="ai-tenant-e2e-shared-b-w3-dd5b30/auth-test-model-dd5b30-kserve-b5b6bc859-7gd9l"
Sep 11 03:58:43.921071 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:58:43.921032    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-dd5b30/auth-test-model-dd5b30-kserve-b5b6bc859-7gd9l"
Sep 11 03:58:44.189776 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:58:44.189717    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-dd5b30/auth-test-model-dd5b30-kserve-b5b6bc859-7gd9l"]
Sep 11 03:58:44.192235 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:58:44.192205    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5738b326_7db0_463d_9d63_a98dc9f51a78.slice/crio-5eaffe8235d9fd0bb3ca3d8291029ad617b855cd1bf1a477a5a3ebd1dbb09bf1 WatchSource:0}: Error finding container 5eaffe8235d9fd0bb3ca3d8291029ad617b855cd1bf1a477a5a3ebd1dbb09bf1: Status 404 returned error can't find the container with id 5eaffe8235d9fd0bb3ca3d8291029ad617b855cd1bf1a477a5a3ebd1dbb09bf1
Sep 11 03:58:44.993745 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:58:44.993702    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-dd5b30/auth-test-model-dd5b30-kserve-b5b6bc859-7gd9l" event={"ID":"5738b326-7db0-463d-9d63-a98dc9f51a78","Type":"ContainerStarted","Data":"ea3bd5554fd679a43824c6c094ad695bbf8481794533829f036db886ac1f74e3"}
Sep 11 03:58:44.994124 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:58:44.993754    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-dd5b30/auth-test-model-dd5b30-kserve-b5b6bc859-7gd9l" event={"ID":"5738b326-7db0-463d-9d63-a98dc9f51a78","Type":"ContainerStarted","Data":"5eaffe8235d9fd0bb3ca3d8291029ad617b855cd1bf1a477a5a3ebd1dbb09bf1"}
Sep 11 03:58:45.010679 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:58:45.010623    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-dd5b30/auth-test-model-dd5b30-kserve-b5b6bc859-7gd9l" podStartSLOduration=2.010606865 podStartE2EDuration="2.010606865s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 03:58:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:58:45.008275322 +0000 UTC m=+1113.166761649" watchObservedRunningTime="2026-09-11 03:58:45.010606865 +0000 UTC m=+1113.169093187"
Sep 11 03:58:53.921755 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:58:53.921710    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-dd5b30/auth-test-model-dd5b30-kserve-b5b6bc859-7gd9l"
Sep 11 03:58:53.922253 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:58:53.921768    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-dd5b30/auth-test-model-dd5b30-kserve-b5b6bc859-7gd9l"
Sep 11 03:58:53.933944 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:58:53.933915    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-dd5b30/auth-test-model-dd5b30-kserve-b5b6bc859-7gd9l"
Sep 11 03:58:54.050469 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:58:54.050432    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-dd5b30/auth-test-model-dd5b30-kserve-b5b6bc859-7gd9l"
Sep 11 03:58:57.821635 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:58:57.821595    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-89230689-86546466-74csh"]
Sep 11 03:58:57.825348 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:58:57.825331    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-89230689-86546466-74csh"
Sep 11 03:58:57.827746 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:58:57.827726    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-89230689\""
Sep 11 03:58:57.827938 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:58:57.827915    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-89230689-dockercfg-s8pqs\""
Sep 11 03:58:57.833533 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:58:57.833377    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-89230689-86546466-74csh"]
Sep 11 03:58:57.959699 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:58:57.959633    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2t7m\" (UniqueName: \"kubernetes.io/projected/09e15fe0-ee05-47f2-ab91-fd3bb2e45d3b-kube-api-access-g2t7m\") pod \"payload-pre-processing-e2e-mt-89230689-86546466-74csh\" (UID: \"09e15fe0-ee05-47f2-ab91-fd3bb2e45d3b\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-89230689-86546466-74csh"
Sep 11 03:58:57.959882 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:58:57.959759    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09e15fe0-ee05-47f2-ab91-fd3bb2e45d3b-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-89230689-86546466-74csh\" (UID: \"09e15fe0-ee05-47f2-ab91-fd3bb2e45d3b\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-89230689-86546466-74csh"
Sep 11 03:58:58.060636 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:58:58.060599    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2t7m\" (UniqueName: \"kubernetes.io/projected/09e15fe0-ee05-47f2-ab91-fd3bb2e45d3b-kube-api-access-g2t7m\") pod \"payload-pre-processing-e2e-mt-89230689-86546466-74csh\" (UID: \"09e15fe0-ee05-47f2-ab91-fd3bb2e45d3b\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-89230689-86546466-74csh"
Sep 11 03:58:58.060841 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:58:58.060714    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09e15fe0-ee05-47f2-ab91-fd3bb2e45d3b-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-89230689-86546466-74csh\" (UID: \"09e15fe0-ee05-47f2-ab91-fd3bb2e45d3b\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-89230689-86546466-74csh"
Sep 11 03:58:58.061429 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:58:58.061404    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09e15fe0-ee05-47f2-ab91-fd3bb2e45d3b-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-89230689-86546466-74csh\" (UID: \"09e15fe0-ee05-47f2-ab91-fd3bb2e45d3b\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-89230689-86546466-74csh"
Sep 11 03:58:58.068150 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:58:58.068123    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2t7m\" (UniqueName: \"kubernetes.io/projected/09e15fe0-ee05-47f2-ab91-fd3bb2e45d3b-kube-api-access-g2t7m\") pod \"payload-pre-processing-e2e-mt-89230689-86546466-74csh\" (UID: \"09e15fe0-ee05-47f2-ab91-fd3bb2e45d3b\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-89230689-86546466-74csh"
Sep 11 03:58:58.137292 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:58:58.137189    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-89230689-86546466-74csh"
Sep 11 03:58:58.407560 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:58:58.407534    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-89230689-86546466-74csh"]
Sep 11 03:58:58.409370 ip-10-0-138-121 kubenswrapper[2580]: W0911 03:58:58.409342    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09e15fe0_ee05_47f2_ab91_fd3bb2e45d3b.slice/crio-efd103fabad4d6cb00eee00f81da9977e0517c4e33f11da62a12ace202d9031f WatchSource:0}: Error finding container efd103fabad4d6cb00eee00f81da9977e0517c4e33f11da62a12ace202d9031f: Status 404 returned error can't find the container with id efd103fabad4d6cb00eee00f81da9977e0517c4e33f11da62a12ace202d9031f
Sep 11 03:58:59.067567 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:58:59.067527    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-89230689-86546466-74csh" event={"ID":"09e15fe0-ee05-47f2-ab91-fd3bb2e45d3b","Type":"ContainerStarted","Data":"7fb0d5c200942bd6671e69242d6a2802ad1dbc3cd3f3307ef681e3868b5980f3"}
Sep 11 03:58:59.068028 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:58:59.067585    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-89230689-86546466-74csh" event={"ID":"09e15fe0-ee05-47f2-ab91-fd3bb2e45d3b","Type":"ContainerStarted","Data":"efd103fabad4d6cb00eee00f81da9977e0517c4e33f11da62a12ace202d9031f"}
Sep 11 03:58:59.068028 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:58:59.067614    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-89230689-86546466-74csh"
Sep 11 03:58:59.082521 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:58:59.082463    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-89230689-86546466-74csh" podStartSLOduration=2.082445919 podStartE2EDuration="2.082445919s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 03:58:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:58:59.08218707 +0000 UTC m=+1127.240673408" watchObservedRunningTime="2026-09-11 03:58:59.082445919 +0000 UTC m=+1127.240932240"
Sep 11 03:59:10.075138 ip-10-0-138-121 kubenswrapper[2580]: I0911 03:59:10.075108    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-89230689-86546466-74csh"
Sep 11 04:00:00.140547 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:00:00.140506    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-b4d3cf-29818320-n2k46"]
Sep 11 04:00:00.144217 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:00:00.144200    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-7dc787-29818320-95766"]
Sep 11 04:00:00.144334 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:00:00.144312    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-b4d3cf-29818320-n2k46"
Sep 11 04:00:00.147045 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:00:00.147017    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-95ckr\""
Sep 11 04:00:00.147888 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:00:00.147864    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-7dc787-29818320-95766"
Sep 11 04:00:00.155120 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:00:00.152917    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-7dc787-29818320-95766"]
Sep 11 04:00:00.156367 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:00:00.156345    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-b4d3cf-29818320-n2k46"]
Sep 11 04:00:00.161528 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:00:00.161500    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-b4d3cf-29818320-n2k46"
Sep 11 04:00:00.169772 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:00:00.169748    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-7dc787-29818320-95766"
Sep 11 04:00:00.442528 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:00:00.442499    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-b4d3cf-29818320-n2k46"]
Sep 11 04:00:00.444567 ip-10-0-138-121 kubenswrapper[2580]: W0911 04:00:00.444542    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb7e4860_ed01_4069_b9b4_070549d3a531.slice/crio-b2378fae8970be01c40ceef97251c49ceeb5413c3e1a64b4386b0fb46f7f4f42 WatchSource:0}: Error finding container b2378fae8970be01c40ceef97251c49ceeb5413c3e1a64b4386b0fb46f7f4f42: Status 404 returned error can't find the container with id b2378fae8970be01c40ceef97251c49ceeb5413c3e1a64b4386b0fb46f7f4f42
Sep 11 04:00:00.446542 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:00:00.446524    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 04:00:00.457385 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:00:00.457287    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-7dc787-29818320-95766"]
Sep 11 04:00:00.459633 ip-10-0-138-121 kubenswrapper[2580]: W0911 04:00:00.459606    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34e05270_eb2c_4de2_b09d_c9b2b86bc1cf.slice/crio-d8d3212e4125c04bae0cbe0b9849db7bd9365231e375488a25a308d1e8585a08 WatchSource:0}: Error finding container d8d3212e4125c04bae0cbe0b9849db7bd9365231e375488a25a308d1e8585a08: Status 404 returned error can't find the container with id d8d3212e4125c04bae0cbe0b9849db7bd9365231e375488a25a308d1e8585a08
Sep 11 04:00:01.389046 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:00:01.389015    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-b4d3cf-29818320-n2k46" event={"ID":"fb7e4860-ed01-4069-b9b4-070549d3a531","Type":"ContainerStarted","Data":"b2378fae8970be01c40ceef97251c49ceeb5413c3e1a64b4386b0fb46f7f4f42"}
Sep 11 04:00:01.390298 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:00:01.390274    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-7dc787-29818320-95766" event={"ID":"34e05270-eb2c-4de2-b09d-c9b2b86bc1cf","Type":"ContainerStarted","Data":"d8d3212e4125c04bae0cbe0b9849db7bd9365231e375488a25a308d1e8585a08"}
Sep 11 04:00:03.403390 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:00:03.403355    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-7dc787-29818320-95766" event={"ID":"34e05270-eb2c-4de2-b09d-c9b2b86bc1cf","Type":"ContainerStarted","Data":"6c841644724e7f56255d1a72d4a6858282d1a5e439e34a7484cca589d8c907a9"}
Sep 11 04:00:03.404938 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:00:03.404912    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-b4d3cf-29818320-n2k46" event={"ID":"fb7e4860-ed01-4069-b9b4-070549d3a531","Type":"ContainerStarted","Data":"8590ed7d7578ddc517feceb2a3eec0e4138d47d63a9e22650a2b9bdd2387d892"}
Sep 11 04:00:03.417353 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:00:03.417278    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-7dc787-29818320-95766" podStartSLOduration=0.989687568 podStartE2EDuration="3.417262852s" totalImagesPullingTime="2.427575284s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:00:03.415361171 +0000 UTC m=+1191.573847496" watchObservedRunningTime="2026-09-11 04:00:03.417262852 +0000 UTC m=+1191.575749175"
Sep 11 04:00:03.429737 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:00:03.429694    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-b4d3cf-29818320-n2k46" podStartSLOduration=0.994686623 podStartE2EDuration="3.429679372s" totalImagesPullingTime="2.434992749s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:00:03.427379211 +0000 UTC m=+1191.585865535" watchObservedRunningTime="2026-09-11 04:00:03.429679372 +0000 UTC m=+1191.588165686"
Sep 11 04:00:04.411521 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:00:04.411482    2580 generic.go:356] "Generic (PLEG): container finished" podID="34e05270-eb2c-4de2-b09d-c9b2b86bc1cf" containerID="6c841644724e7f56255d1a72d4a6858282d1a5e439e34a7484cca589d8c907a9" exitCode=0
Sep 11 04:00:04.411946 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:00:04.411571    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-7dc787-29818320-95766" event={"ID":"34e05270-eb2c-4de2-b09d-c9b2b86bc1cf","Type":"ContainerDied","Data":"6c841644724e7f56255d1a72d4a6858282d1a5e439e34a7484cca589d8c907a9"}
Sep 11 04:00:04.413111 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:00:04.413090    2580 generic.go:356] "Generic (PLEG): container finished" podID="fb7e4860-ed01-4069-b9b4-070549d3a531" containerID="8590ed7d7578ddc517feceb2a3eec0e4138d47d63a9e22650a2b9bdd2387d892" exitCode=0
Sep 11 04:00:04.413222 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:00:04.413170    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-b4d3cf-29818320-n2k46" event={"ID":"fb7e4860-ed01-4069-b9b4-070549d3a531","Type":"ContainerDied","Data":"8590ed7d7578ddc517feceb2a3eec0e4138d47d63a9e22650a2b9bdd2387d892"}
Sep 11 04:00:05.568837 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:00:05.568807    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-b4d3cf-29818320-n2k46"
Sep 11 04:00:05.600631 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:00:05.600606    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-7dc787-29818320-95766"
Sep 11 04:00:06.424309 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:00:06.424278    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-7dc787-29818320-95766"
Sep 11 04:00:06.424309 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:00:06.424292    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-7dc787-29818320-95766" event={"ID":"34e05270-eb2c-4de2-b09d-c9b2b86bc1cf","Type":"ContainerDied","Data":"d8d3212e4125c04bae0cbe0b9849db7bd9365231e375488a25a308d1e8585a08"}
Sep 11 04:00:06.424543 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:00:06.424342    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d8d3212e4125c04bae0cbe0b9849db7bd9365231e375488a25a308d1e8585a08"
Sep 11 04:00:06.425710 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:00:06.425691    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-b4d3cf-29818320-n2k46"
Sep 11 04:00:06.425843 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:00:06.425691    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-b4d3cf-29818320-n2k46" event={"ID":"fb7e4860-ed01-4069-b9b4-070549d3a531","Type":"ContainerDied","Data":"b2378fae8970be01c40ceef97251c49ceeb5413c3e1a64b4386b0fb46f7f4f42"}
Sep 11 04:00:06.425843 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:00:06.425798    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b2378fae8970be01c40ceef97251c49ceeb5413c3e1a64b4386b0fb46f7f4f42"
Sep 11 04:00:12.402923 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:00:12.402896    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j4pgw_28b30dae-a2d3-4251-99e8-04bf5153015b/ovn-acl-logging/0.log"
Sep 11 04:00:12.404365 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:00:12.404345    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j4pgw_28b30dae-a2d3-4251-99e8-04bf5153015b/ovn-acl-logging/0.log"
Sep 11 04:00:25.123585 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:00:25.123549    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-89230689-86546466-74csh"]
Sep 11 04:00:25.124107 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:00:25.123802    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-89230689-86546466-74csh" podUID="09e15fe0-ee05-47f2-ab91-fd3bb2e45d3b" containerName="payload-pre-processing" containerID="cri-o://7fb0d5c200942bd6671e69242d6a2802ad1dbc3cd3f3307ef681e3868b5980f3" gracePeriod=30
Sep 11 04:00:25.390392 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:00:25.390319    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-89230689-86546466-74csh"
Sep 11 04:00:25.526914 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:00:25.526882    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g2t7m\" (UniqueName: \"kubernetes.io/projected/09e15fe0-ee05-47f2-ab91-fd3bb2e45d3b-kube-api-access-g2t7m\") pod \"09e15fe0-ee05-47f2-ab91-fd3bb2e45d3b\" (UID: \"09e15fe0-ee05-47f2-ab91-fd3bb2e45d3b\") "
Sep 11 04:00:25.527071 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:00:25.526926    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09e15fe0-ee05-47f2-ab91-fd3bb2e45d3b-plugins-config-volume\") pod \"09e15fe0-ee05-47f2-ab91-fd3bb2e45d3b\" (UID: \"09e15fe0-ee05-47f2-ab91-fd3bb2e45d3b\") "
Sep 11 04:00:25.527451 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:00:25.527422    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09e15fe0-ee05-47f2-ab91-fd3bb2e45d3b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "09e15fe0-ee05-47f2-ab91-fd3bb2e45d3b" (UID: "09e15fe0-ee05-47f2-ab91-fd3bb2e45d3b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 04:00:25.529059 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:00:25.529034    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09e15fe0-ee05-47f2-ab91-fd3bb2e45d3b-kube-api-access-g2t7m" (OuterVolumeSpecName: "kube-api-access-g2t7m") pod "09e15fe0-ee05-47f2-ab91-fd3bb2e45d3b" (UID: "09e15fe0-ee05-47f2-ab91-fd3bb2e45d3b"). InnerVolumeSpecName "kube-api-access-g2t7m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 04:00:25.530842 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:00:25.530812    2580 generic.go:356] "Generic (PLEG): container finished" podID="09e15fe0-ee05-47f2-ab91-fd3bb2e45d3b" containerID="7fb0d5c200942bd6671e69242d6a2802ad1dbc3cd3f3307ef681e3868b5980f3" exitCode=0
Sep 11 04:00:25.530948 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:00:25.530862    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-89230689-86546466-74csh"
Sep 11 04:00:25.530948 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:00:25.530878    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-89230689-86546466-74csh" event={"ID":"09e15fe0-ee05-47f2-ab91-fd3bb2e45d3b","Type":"ContainerDied","Data":"7fb0d5c200942bd6671e69242d6a2802ad1dbc3cd3f3307ef681e3868b5980f3"}
Sep 11 04:00:25.530948 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:00:25.530932    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-89230689-86546466-74csh" event={"ID":"09e15fe0-ee05-47f2-ab91-fd3bb2e45d3b","Type":"ContainerDied","Data":"efd103fabad4d6cb00eee00f81da9977e0517c4e33f11da62a12ace202d9031f"}
Sep 11 04:00:25.531069 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:00:25.530962    2580 scope.go:117] "RemoveContainer" containerID="7fb0d5c200942bd6671e69242d6a2802ad1dbc3cd3f3307ef681e3868b5980f3"
Sep 11 04:00:25.542027 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:00:25.542008    2580 scope.go:117] "RemoveContainer" containerID="7fb0d5c200942bd6671e69242d6a2802ad1dbc3cd3f3307ef681e3868b5980f3"
Sep 11 04:00:25.542353 ip-10-0-138-121 kubenswrapper[2580]: E0911 04:00:25.542322    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7fb0d5c200942bd6671e69242d6a2802ad1dbc3cd3f3307ef681e3868b5980f3\": container with ID starting with 7fb0d5c200942bd6671e69242d6a2802ad1dbc3cd3f3307ef681e3868b5980f3 not found: ID does not exist" containerID="7fb0d5c200942bd6671e69242d6a2802ad1dbc3cd3f3307ef681e3868b5980f3"
Sep 11 04:00:25.542428 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:00:25.542362    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7fb0d5c200942bd6671e69242d6a2802ad1dbc3cd3f3307ef681e3868b5980f3"} err="failed to get container status \"7fb0d5c200942bd6671e69242d6a2802ad1dbc3cd3f3307ef681e3868b5980f3\": rpc error: code = NotFound desc = could not find container \"7fb0d5c200942bd6671e69242d6a2802ad1dbc3cd3f3307ef681e3868b5980f3\": container with ID starting with 7fb0d5c200942bd6671e69242d6a2802ad1dbc3cd3f3307ef681e3868b5980f3 not found: ID does not exist"
Sep 11 04:00:25.552264 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:00:25.552240    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-89230689-86546466-74csh"]
Sep 11 04:00:25.556671 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:00:25.556631    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-89230689-86546466-74csh"]
Sep 11 04:00:25.628468 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:00:25.628432    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g2t7m\" (UniqueName: \"kubernetes.io/projected/09e15fe0-ee05-47f2-ab91-fd3bb2e45d3b-kube-api-access-g2t7m\") on node \"ip-10-0-138-121.ec2.internal\" DevicePath \"\""
Sep 11 04:00:25.628468 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:00:25.628464    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09e15fe0-ee05-47f2-ab91-fd3bb2e45d3b-plugins-config-volume\") on node \"ip-10-0-138-121.ec2.internal\" DevicePath \"\""
Sep 11 04:00:26.466055 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:00:26.466018    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="09e15fe0-ee05-47f2-ab91-fd3bb2e45d3b" path="/var/lib/kubelet/pods/09e15fe0-ee05-47f2-ab91-fd3bb2e45d3b/volumes"
Sep 11 04:01:28.174038 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:01:28.174002    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-dd5b30/auth-test-model-dd5b30-kserve-b5b6bc859-7gd9l"]
Sep 11 04:01:43.193804 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:01:43.193757    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-dd5b30/auth-test-model-dd5b30-kserve-b5b6bc859-7gd9l" podUID="5738b326-7db0-463d-9d63-a98dc9f51a78" containerName="main" containerID="cri-o://ea3bd5554fd679a43824c6c094ad695bbf8481794533829f036db886ac1f74e3" gracePeriod=45
Sep 11 04:01:43.440149 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:01:43.440119    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-dd5b30/auth-test-model-dd5b30-kserve-b5b6bc859-7gd9l"
Sep 11 04:01:43.606631 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:01:43.606533    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5738b326-7db0-463d-9d63-a98dc9f51a78-tmp-dir\") pod \"5738b326-7db0-463d-9d63-a98dc9f51a78\" (UID: \"5738b326-7db0-463d-9d63-a98dc9f51a78\") "
Sep 11 04:01:43.606813 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:01:43.606693    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5738b326-7db0-463d-9d63-a98dc9f51a78-tls-certs\") pod \"5738b326-7db0-463d-9d63-a98dc9f51a78\" (UID: \"5738b326-7db0-463d-9d63-a98dc9f51a78\") "
Sep 11 04:01:43.606813 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:01:43.606733    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5738b326-7db0-463d-9d63-a98dc9f51a78-model-cache\") pod \"5738b326-7db0-463d-9d63-a98dc9f51a78\" (UID: \"5738b326-7db0-463d-9d63-a98dc9f51a78\") "
Sep 11 04:01:43.606813 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:01:43.606758    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5738b326-7db0-463d-9d63-a98dc9f51a78-home\") pod \"5738b326-7db0-463d-9d63-a98dc9f51a78\" (UID: \"5738b326-7db0-463d-9d63-a98dc9f51a78\") "
Sep 11 04:01:43.606813 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:01:43.606782    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5738b326-7db0-463d-9d63-a98dc9f51a78-dshm\") pod \"5738b326-7db0-463d-9d63-a98dc9f51a78\" (UID: \"5738b326-7db0-463d-9d63-a98dc9f51a78\") "
Sep 11 04:01:43.607127 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:01:43.606827    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qz6fv\" (UniqueName: \"kubernetes.io/projected/5738b326-7db0-463d-9d63-a98dc9f51a78-kube-api-access-qz6fv\") pod \"5738b326-7db0-463d-9d63-a98dc9f51a78\" (UID: \"5738b326-7db0-463d-9d63-a98dc9f51a78\") "
Sep 11 04:01:43.607127 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:01:43.606865    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5738b326-7db0-463d-9d63-a98dc9f51a78-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5738b326-7db0-463d-9d63-a98dc9f51a78" (UID: "5738b326-7db0-463d-9d63-a98dc9f51a78"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 04:01:43.607127 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:01:43.606954    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5738b326-7db0-463d-9d63-a98dc9f51a78-home" (OuterVolumeSpecName: "home") pod "5738b326-7db0-463d-9d63-a98dc9f51a78" (UID: "5738b326-7db0-463d-9d63-a98dc9f51a78"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 04:01:43.607127 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:01:43.607006    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5738b326-7db0-463d-9d63-a98dc9f51a78-model-cache" (OuterVolumeSpecName: "model-cache") pod "5738b326-7db0-463d-9d63-a98dc9f51a78" (UID: "5738b326-7db0-463d-9d63-a98dc9f51a78"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 04:01:43.607342 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:01:43.607214    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5738b326-7db0-463d-9d63-a98dc9f51a78-model-cache\") on node \"ip-10-0-138-121.ec2.internal\" DevicePath \"\""
Sep 11 04:01:43.607342 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:01:43.607233    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5738b326-7db0-463d-9d63-a98dc9f51a78-home\") on node \"ip-10-0-138-121.ec2.internal\" DevicePath \"\""
Sep 11 04:01:43.607342 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:01:43.607245    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5738b326-7db0-463d-9d63-a98dc9f51a78-tmp-dir\") on node \"ip-10-0-138-121.ec2.internal\" DevicePath \"\""
Sep 11 04:01:43.609044 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:01:43.609020    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5738b326-7db0-463d-9d63-a98dc9f51a78-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5738b326-7db0-463d-9d63-a98dc9f51a78" (UID: "5738b326-7db0-463d-9d63-a98dc9f51a78"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 04:01:43.609143 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:01:43.609042    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5738b326-7db0-463d-9d63-a98dc9f51a78-dshm" (OuterVolumeSpecName: "dshm") pod "5738b326-7db0-463d-9d63-a98dc9f51a78" (UID: "5738b326-7db0-463d-9d63-a98dc9f51a78"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 04:01:43.609143 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:01:43.609052    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5738b326-7db0-463d-9d63-a98dc9f51a78-kube-api-access-qz6fv" (OuterVolumeSpecName: "kube-api-access-qz6fv") pod "5738b326-7db0-463d-9d63-a98dc9f51a78" (UID: "5738b326-7db0-463d-9d63-a98dc9f51a78"). InnerVolumeSpecName "kube-api-access-qz6fv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 04:01:43.707967 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:01:43.707931    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5738b326-7db0-463d-9d63-a98dc9f51a78-dshm\") on node \"ip-10-0-138-121.ec2.internal\" DevicePath \"\""
Sep 11 04:01:43.707967 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:01:43.707959    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qz6fv\" (UniqueName: \"kubernetes.io/projected/5738b326-7db0-463d-9d63-a98dc9f51a78-kube-api-access-qz6fv\") on node \"ip-10-0-138-121.ec2.internal\" DevicePath \"\""
Sep 11 04:01:43.707967 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:01:43.707971    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5738b326-7db0-463d-9d63-a98dc9f51a78-tls-certs\") on node \"ip-10-0-138-121.ec2.internal\" DevicePath \"\""
Sep 11 04:01:43.942707 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:01:43.942643    2580 generic.go:356] "Generic (PLEG): container finished" podID="5738b326-7db0-463d-9d63-a98dc9f51a78" containerID="ea3bd5554fd679a43824c6c094ad695bbf8481794533829f036db886ac1f74e3" exitCode=0
Sep 11 04:01:43.942707 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:01:43.942708    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-dd5b30/auth-test-model-dd5b30-kserve-b5b6bc859-7gd9l"
Sep 11 04:01:43.942914 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:01:43.942733    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-dd5b30/auth-test-model-dd5b30-kserve-b5b6bc859-7gd9l" event={"ID":"5738b326-7db0-463d-9d63-a98dc9f51a78","Type":"ContainerDied","Data":"ea3bd5554fd679a43824c6c094ad695bbf8481794533829f036db886ac1f74e3"}
Sep 11 04:01:43.942914 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:01:43.942780    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-dd5b30/auth-test-model-dd5b30-kserve-b5b6bc859-7gd9l" event={"ID":"5738b326-7db0-463d-9d63-a98dc9f51a78","Type":"ContainerDied","Data":"5eaffe8235d9fd0bb3ca3d8291029ad617b855cd1bf1a477a5a3ebd1dbb09bf1"}
Sep 11 04:01:43.942914 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:01:43.942799    2580 scope.go:117] "RemoveContainer" containerID="ea3bd5554fd679a43824c6c094ad695bbf8481794533829f036db886ac1f74e3"
Sep 11 04:01:43.953131 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:01:43.953106    2580 scope.go:117] "RemoveContainer" containerID="ea3bd5554fd679a43824c6c094ad695bbf8481794533829f036db886ac1f74e3"
Sep 11 04:01:43.953492 ip-10-0-138-121 kubenswrapper[2580]: E0911 04:01:43.953472    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea3bd5554fd679a43824c6c094ad695bbf8481794533829f036db886ac1f74e3\": container with ID starting with ea3bd5554fd679a43824c6c094ad695bbf8481794533829f036db886ac1f74e3 not found: ID does not exist" containerID="ea3bd5554fd679a43824c6c094ad695bbf8481794533829f036db886ac1f74e3"
Sep 11 04:01:43.953556 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:01:43.953501    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea3bd5554fd679a43824c6c094ad695bbf8481794533829f036db886ac1f74e3"} err="failed to get container status \"ea3bd5554fd679a43824c6c094ad695bbf8481794533829f036db886ac1f74e3\": rpc error: code = NotFound desc = could not find container \"ea3bd5554fd679a43824c6c094ad695bbf8481794533829f036db886ac1f74e3\": container with ID starting with ea3bd5554fd679a43824c6c094ad695bbf8481794533829f036db886ac1f74e3 not found: ID does not exist"
Sep 11 04:01:43.965791 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:01:43.965764    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-dd5b30/auth-test-model-dd5b30-kserve-b5b6bc859-7gd9l"]
Sep 11 04:01:43.968467 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:01:43.968447    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-dd5b30/auth-test-model-dd5b30-kserve-b5b6bc859-7gd9l"]
Sep 11 04:01:44.466207 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:01:44.466166    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5738b326-7db0-463d-9d63-a98dc9f51a78" path="/var/lib/kubelet/pods/5738b326-7db0-463d-9d63-a98dc9f51a78/volumes"
Sep 11 04:02:32.635882 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:02:32.635844    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-49518b-778b56d9b6-w5snt"]
Sep 11 04:02:32.636471 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:02:32.636457    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09e15fe0-ee05-47f2-ab91-fd3bb2e45d3b" containerName="payload-pre-processing"
Sep 11 04:02:32.636545 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:02:32.636479    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="09e15fe0-ee05-47f2-ab91-fd3bb2e45d3b" containerName="payload-pre-processing"
Sep 11 04:02:32.636545 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:02:32.636496    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34e05270-eb2c-4de2-b09d-c9b2b86bc1cf" containerName="cleanup"
Sep 11 04:02:32.636545 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:02:32.636505    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="34e05270-eb2c-4de2-b09d-c9b2b86bc1cf" containerName="cleanup"
Sep 11 04:02:32.636545 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:02:32.636523    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb7e4860-ed01-4069-b9b4-070549d3a531" containerName="cleanup"
Sep 11 04:02:32.636545 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:02:32.636534    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb7e4860-ed01-4069-b9b4-070549d3a531" containerName="cleanup"
Sep 11 04:02:32.636838 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:02:32.636550    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5738b326-7db0-463d-9d63-a98dc9f51a78" containerName="main"
Sep 11 04:02:32.636838 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:02:32.636559    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="5738b326-7db0-463d-9d63-a98dc9f51a78" containerName="main"
Sep 11 04:02:32.636838 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:02:32.636682    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="34e05270-eb2c-4de2-b09d-c9b2b86bc1cf" containerName="cleanup"
Sep 11 04:02:32.636838 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:02:32.636695    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="09e15fe0-ee05-47f2-ab91-fd3bb2e45d3b" containerName="payload-pre-processing"
Sep 11 04:02:32.636838 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:02:32.636708    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="5738b326-7db0-463d-9d63-a98dc9f51a78" containerName="main"
Sep 11 04:02:32.636838 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:02:32.636719    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb7e4860-ed01-4069-b9b4-070549d3a531" containerName="cleanup"
Sep 11 04:02:32.641058 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:02:32.641039    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-49518b-778b56d9b6-w5snt"
Sep 11 04:02:32.643836 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:02:32.643816    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-49518b\""
Sep 11 04:02:32.644111 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:02:32.644084    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-49518b-dockercfg-nffm6\""
Sep 11 04:02:32.658886 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:02:32.658837    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-49518b-778b56d9b6-w5snt"]
Sep 11 04:02:32.661072 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:02:32.661040    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-49518b-6dcd48549c-vp88r"]
Sep 11 04:02:32.664764 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:02:32.664741    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-49518b-6dcd48549c-vp88r"
Sep 11 04:02:32.684958 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:02:32.684928    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-49518b-6dcd48549c-vp88r"]
Sep 11 04:02:32.782075 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:02:32.782039    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxnz6\" (UniqueName: \"kubernetes.io/projected/20579439-bcf0-498c-a8ae-5c9988bf5d2f-kube-api-access-gxnz6\") pod \"payload-processing-e2e-api-a-49518b-6dcd48549c-vp88r\" (UID: \"20579439-bcf0-498c-a8ae-5c9988bf5d2f\") " pod="openshift-ingress/payload-processing-e2e-api-a-49518b-6dcd48549c-vp88r"
Sep 11 04:02:32.782237 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:02:32.782100    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qtzd\" (UniqueName: \"kubernetes.io/projected/8b98563f-9815-4506-8301-9f03ac497363-kube-api-access-7qtzd\") pod \"payload-pre-processing-e2e-api-a-49518b-778b56d9b6-w5snt\" (UID: \"8b98563f-9815-4506-8301-9f03ac497363\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-49518b-778b56d9b6-w5snt"
Sep 11 04:02:32.782237 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:02:32.782192    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b98563f-9815-4506-8301-9f03ac497363-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-49518b-778b56d9b6-w5snt\" (UID: \"8b98563f-9815-4506-8301-9f03ac497363\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-49518b-778b56d9b6-w5snt"
Sep 11 04:02:32.782361 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:02:32.782240    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20579439-bcf0-498c-a8ae-5c9988bf5d2f-plugins-config-volume\") pod \"payload-processing-e2e-api-a-49518b-6dcd48549c-vp88r\" (UID: \"20579439-bcf0-498c-a8ae-5c9988bf5d2f\") " pod="openshift-ingress/payload-processing-e2e-api-a-49518b-6dcd48549c-vp88r"
Sep 11 04:02:32.883565 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:02:32.883530    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxnz6\" (UniqueName: \"kubernetes.io/projected/20579439-bcf0-498c-a8ae-5c9988bf5d2f-kube-api-access-gxnz6\") pod \"payload-processing-e2e-api-a-49518b-6dcd48549c-vp88r\" (UID: \"20579439-bcf0-498c-a8ae-5c9988bf5d2f\") " pod="openshift-ingress/payload-processing-e2e-api-a-49518b-6dcd48549c-vp88r"
Sep 11 04:02:32.883773 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:02:32.883712    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qtzd\" (UniqueName: \"kubernetes.io/projected/8b98563f-9815-4506-8301-9f03ac497363-kube-api-access-7qtzd\") pod \"payload-pre-processing-e2e-api-a-49518b-778b56d9b6-w5snt\" (UID: \"8b98563f-9815-4506-8301-9f03ac497363\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-49518b-778b56d9b6-w5snt"
Sep 11 04:02:32.883856 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:02:32.883798    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b98563f-9815-4506-8301-9f03ac497363-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-49518b-778b56d9b6-w5snt\" (UID: \"8b98563f-9815-4506-8301-9f03ac497363\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-49518b-778b56d9b6-w5snt"
Sep 11 04:02:32.883928 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:02:32.883862    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20579439-bcf0-498c-a8ae-5c9988bf5d2f-plugins-config-volume\") pod \"payload-processing-e2e-api-a-49518b-6dcd48549c-vp88r\" (UID: \"20579439-bcf0-498c-a8ae-5c9988bf5d2f\") " pod="openshift-ingress/payload-processing-e2e-api-a-49518b-6dcd48549c-vp88r"
Sep 11 04:02:32.884487 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:02:32.884467    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b98563f-9815-4506-8301-9f03ac497363-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-49518b-778b56d9b6-w5snt\" (UID: \"8b98563f-9815-4506-8301-9f03ac497363\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-49518b-778b56d9b6-w5snt"
Sep 11 04:02:32.884581 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:02:32.884561    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20579439-bcf0-498c-a8ae-5c9988bf5d2f-plugins-config-volume\") pod \"payload-processing-e2e-api-a-49518b-6dcd48549c-vp88r\" (UID: \"20579439-bcf0-498c-a8ae-5c9988bf5d2f\") " pod="openshift-ingress/payload-processing-e2e-api-a-49518b-6dcd48549c-vp88r"
Sep 11 04:02:32.894626 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:02:32.894571    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qtzd\" (UniqueName: \"kubernetes.io/projected/8b98563f-9815-4506-8301-9f03ac497363-kube-api-access-7qtzd\") pod \"payload-pre-processing-e2e-api-a-49518b-778b56d9b6-w5snt\" (UID: \"8b98563f-9815-4506-8301-9f03ac497363\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-49518b-778b56d9b6-w5snt"
Sep 11 04:02:32.894626 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:02:32.894574    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxnz6\" (UniqueName: \"kubernetes.io/projected/20579439-bcf0-498c-a8ae-5c9988bf5d2f-kube-api-access-gxnz6\") pod \"payload-processing-e2e-api-a-49518b-6dcd48549c-vp88r\" (UID: \"20579439-bcf0-498c-a8ae-5c9988bf5d2f\") " pod="openshift-ingress/payload-processing-e2e-api-a-49518b-6dcd48549c-vp88r"
Sep 11 04:02:32.952778 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:02:32.952747    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-49518b-778b56d9b6-w5snt"
Sep 11 04:02:32.978937 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:02:32.978908    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-49518b-6dcd48549c-vp88r"
Sep 11 04:02:33.244822 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:02:33.244798    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-49518b-778b56d9b6-w5snt"]
Sep 11 04:02:33.247024 ip-10-0-138-121 kubenswrapper[2580]: W0911 04:02:33.246994    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b98563f_9815_4506_8301_9f03ac497363.slice/crio-7d44a1c43815afc82737c49e8520125fe39ce271bca71defb02992b5da5cff5e WatchSource:0}: Error finding container 7d44a1c43815afc82737c49e8520125fe39ce271bca71defb02992b5da5cff5e: Status 404 returned error can't find the container with id 7d44a1c43815afc82737c49e8520125fe39ce271bca71defb02992b5da5cff5e
Sep 11 04:02:33.286079 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:02:33.286056    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-49518b-6dcd48549c-vp88r"]
Sep 11 04:02:33.288420 ip-10-0-138-121 kubenswrapper[2580]: W0911 04:02:33.288394    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20579439_bcf0_498c_a8ae_5c9988bf5d2f.slice/crio-e7c2bcdf1caf4da780ab00506ac96b17a48358e51461d1eb755d9ada372c40a4 WatchSource:0}: Error finding container e7c2bcdf1caf4da780ab00506ac96b17a48358e51461d1eb755d9ada372c40a4: Status 404 returned error can't find the container with id e7c2bcdf1caf4da780ab00506ac96b17a48358e51461d1eb755d9ada372c40a4
Sep 11 04:02:34.206654 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:02:34.206615    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-49518b-6dcd48549c-vp88r" event={"ID":"20579439-bcf0-498c-a8ae-5c9988bf5d2f","Type":"ContainerStarted","Data":"a940f04d00eff42754c24dc9a9cfd5512b9bc5f75ef145f3de91f7c17c68138e"}
Sep 11 04:02:34.207138 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:02:34.206725    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-49518b-6dcd48549c-vp88r" event={"ID":"20579439-bcf0-498c-a8ae-5c9988bf5d2f","Type":"ContainerStarted","Data":"e7c2bcdf1caf4da780ab00506ac96b17a48358e51461d1eb755d9ada372c40a4"}
Sep 11 04:02:34.207138 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:02:34.206763    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-49518b-6dcd48549c-vp88r"
Sep 11 04:02:34.208283 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:02:34.208259    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-49518b-778b56d9b6-w5snt" event={"ID":"8b98563f-9815-4506-8301-9f03ac497363","Type":"ContainerStarted","Data":"0015a3f0dd1c8d0ee8baf681c8d03631992e855c9477262ba2847ae5b7aea7f1"}
Sep 11 04:02:34.208416 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:02:34.208304    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-49518b-778b56d9b6-w5snt" event={"ID":"8b98563f-9815-4506-8301-9f03ac497363","Type":"ContainerStarted","Data":"7d44a1c43815afc82737c49e8520125fe39ce271bca71defb02992b5da5cff5e"}
Sep 11 04:02:34.208416 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:02:34.208355    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-49518b-778b56d9b6-w5snt"
Sep 11 04:02:34.252222 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:02:34.252165    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-49518b-778b56d9b6-w5snt" podStartSLOduration=2.252149262 podStartE2EDuration="2.252149262s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:02:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:02:34.250835105 +0000 UTC m=+1342.409321439" watchObservedRunningTime="2026-09-11 04:02:34.252149262 +0000 UTC m=+1342.410635586"
Sep 11 04:02:34.252414 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:02:34.252262    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-49518b-6dcd48549c-vp88r" podStartSLOduration=2.252252748 podStartE2EDuration="2.252252748s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:02:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:02:34.23014357 +0000 UTC m=+1342.388629892" watchObservedRunningTime="2026-09-11 04:02:34.252252748 +0000 UTC m=+1342.410739072"
Sep 11 04:02:45.215863 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:02:45.215788    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-49518b-778b56d9b6-w5snt"
Sep 11 04:02:45.216294 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:02:45.216091    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-49518b-6dcd48549c-vp88r"
Sep 11 04:02:59.106684 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:02:59.106638    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-b61c46-openshift-default-7f74cd5f58-5fl2g"]
Sep 11 04:02:59.110287 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:02:59.110257    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-b61c46-openshift-default-7f74cd5f58-5fl2g"
Sep 11 04:02:59.113272 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:02:59.113253    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-b61c46-openshift-default-dockercfg-f2wmd\""
Sep 11 04:02:59.202705 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:02:59.202641    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-b61c46-openshift-default-7f74cd5f58-5fl2g"]
Sep 11 04:02:59.231607 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:02:59.231574    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4aed45fc-e9d7-4c85-b463-00b90b91cb1b-workload-certs\") pod \"e2e-api-b-b61c46-openshift-default-7f74cd5f58-5fl2g\" (UID: \"4aed45fc-e9d7-4c85-b463-00b90b91cb1b\") " pod="openshift-ingress/e2e-api-b-b61c46-openshift-default-7f74cd5f58-5fl2g"
Sep 11 04:02:59.231795 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:02:59.231656    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4aed45fc-e9d7-4c85-b463-00b90b91cb1b-workload-socket\") pod \"e2e-api-b-b61c46-openshift-default-7f74cd5f58-5fl2g\" (UID: \"4aed45fc-e9d7-4c85-b463-00b90b91cb1b\") " pod="openshift-ingress/e2e-api-b-b61c46-openshift-default-7f74cd5f58-5fl2g"
Sep 11 04:02:59.231795 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:02:59.231712    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wh648\" (UniqueName: \"kubernetes.io/projected/4aed45fc-e9d7-4c85-b463-00b90b91cb1b-kube-api-access-wh648\") pod \"e2e-api-b-b61c46-openshift-default-7f74cd5f58-5fl2g\" (UID: \"4aed45fc-e9d7-4c85-b463-00b90b91cb1b\") " pod="openshift-ingress/e2e-api-b-b61c46-openshift-default-7f74cd5f58-5fl2g"
Sep 11 04:02:59.231795 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:02:59.231741    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4aed45fc-e9d7-4c85-b463-00b90b91cb1b-istio-token\") pod \"e2e-api-b-b61c46-openshift-default-7f74cd5f58-5fl2g\" (UID: \"4aed45fc-e9d7-4c85-b463-00b90b91cb1b\") " pod="openshift-ingress/e2e-api-b-b61c46-openshift-default-7f74cd5f58-5fl2g"
Sep 11 04:02:59.231928 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:02:59.231814    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4aed45fc-e9d7-4c85-b463-00b90b91cb1b-istio-data\") pod \"e2e-api-b-b61c46-openshift-default-7f74cd5f58-5fl2g\" (UID: \"4aed45fc-e9d7-4c85-b463-00b90b91cb1b\") " pod="openshift-ingress/e2e-api-b-b61c46-openshift-default-7f74cd5f58-5fl2g"
Sep 11 04:02:59.231928 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:02:59.231832    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4aed45fc-e9d7-4c85-b463-00b90b91cb1b-istio-podinfo\") pod \"e2e-api-b-b61c46-openshift-default-7f74cd5f58-5fl2g\" (UID: \"4aed45fc-e9d7-4c85-b463-00b90b91cb1b\") " pod="openshift-ingress/e2e-api-b-b61c46-openshift-default-7f74cd5f58-5fl2g"
Sep 11 04:02:59.231928 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:02:59.231859    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4aed45fc-e9d7-4c85-b463-00b90b91cb1b-istiod-ca-cert\") pod \"e2e-api-b-b61c46-openshift-default-7f74cd5f58-5fl2g\" (UID: \"4aed45fc-e9d7-4c85-b463-00b90b91cb1b\") " pod="openshift-ingress/e2e-api-b-b61c46-openshift-default-7f74cd5f58-5fl2g"
Sep 11 04:02:59.232039 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:02:59.231929    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4aed45fc-e9d7-4c85-b463-00b90b91cb1b-istio-envoy\") pod \"e2e-api-b-b61c46-openshift-default-7f74cd5f58-5fl2g\" (UID: \"4aed45fc-e9d7-4c85-b463-00b90b91cb1b\") " pod="openshift-ingress/e2e-api-b-b61c46-openshift-default-7f74cd5f58-5fl2g"
Sep 11 04:02:59.232039 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:02:59.231966    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4aed45fc-e9d7-4c85-b463-00b90b91cb1b-credential-socket\") pod \"e2e-api-b-b61c46-openshift-default-7f74cd5f58-5fl2g\" (UID: \"4aed45fc-e9d7-4c85-b463-00b90b91cb1b\") " pod="openshift-ingress/e2e-api-b-b61c46-openshift-default-7f74cd5f58-5fl2g"
Sep 11 04:02:59.332923 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:02:59.332886    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4aed45fc-e9d7-4c85-b463-00b90b91cb1b-workload-certs\") pod \"e2e-api-b-b61c46-openshift-default-7f74cd5f58-5fl2g\" (UID: \"4aed45fc-e9d7-4c85-b463-00b90b91cb1b\") " pod="openshift-ingress/e2e-api-b-b61c46-openshift-default-7f74cd5f58-5fl2g"
Sep 11 04:02:59.333109 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:02:59.332958    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4aed45fc-e9d7-4c85-b463-00b90b91cb1b-workload-socket\") pod \"e2e-api-b-b61c46-openshift-default-7f74cd5f58-5fl2g\" (UID: \"4aed45fc-e9d7-4c85-b463-00b90b91cb1b\") " pod="openshift-ingress/e2e-api-b-b61c46-openshift-default-7f74cd5f58-5fl2g"
Sep 11 04:02:59.333109 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:02:59.332984    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wh648\" (UniqueName: \"kubernetes.io/projected/4aed45fc-e9d7-4c85-b463-00b90b91cb1b-kube-api-access-wh648\") pod \"e2e-api-b-b61c46-openshift-default-7f74cd5f58-5fl2g\" (UID: \"4aed45fc-e9d7-4c85-b463-00b90b91cb1b\") " pod="openshift-ingress/e2e-api-b-b61c46-openshift-default-7f74cd5f58-5fl2g"
Sep 11 04:02:59.333109 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:02:59.333006    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4aed45fc-e9d7-4c85-b463-00b90b91cb1b-istio-token\") pod \"e2e-api-b-b61c46-openshift-default-7f74cd5f58-5fl2g\" (UID: \"4aed45fc-e9d7-4c85-b463-00b90b91cb1b\") " pod="openshift-ingress/e2e-api-b-b61c46-openshift-default-7f74cd5f58-5fl2g"
Sep 11 04:02:59.333109 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:02:59.333076    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4aed45fc-e9d7-4c85-b463-00b90b91cb1b-istio-data\") pod \"e2e-api-b-b61c46-openshift-default-7f74cd5f58-5fl2g\" (UID: \"4aed45fc-e9d7-4c85-b463-00b90b91cb1b\") " pod="openshift-ingress/e2e-api-b-b61c46-openshift-default-7f74cd5f58-5fl2g"
Sep 11 04:02:59.333109 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:02:59.333099    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4aed45fc-e9d7-4c85-b463-00b90b91cb1b-istio-podinfo\") pod \"e2e-api-b-b61c46-openshift-default-7f74cd5f58-5fl2g\" (UID: \"4aed45fc-e9d7-4c85-b463-00b90b91cb1b\") " pod="openshift-ingress/e2e-api-b-b61c46-openshift-default-7f74cd5f58-5fl2g"
Sep 11 04:02:59.333438 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:02:59.333124    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4aed45fc-e9d7-4c85-b463-00b90b91cb1b-istiod-ca-cert\") pod \"e2e-api-b-b61c46-openshift-default-7f74cd5f58-5fl2g\" (UID: \"4aed45fc-e9d7-4c85-b463-00b90b91cb1b\") " pod="openshift-ingress/e2e-api-b-b61c46-openshift-default-7f74cd5f58-5fl2g"
Sep 11 04:02:59.333438 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:02:59.333163    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4aed45fc-e9d7-4c85-b463-00b90b91cb1b-istio-envoy\") pod \"e2e-api-b-b61c46-openshift-default-7f74cd5f58-5fl2g\" (UID: \"4aed45fc-e9d7-4c85-b463-00b90b91cb1b\") " pod="openshift-ingress/e2e-api-b-b61c46-openshift-default-7f74cd5f58-5fl2g"
Sep 11 04:02:59.333438 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:02:59.333191    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4aed45fc-e9d7-4c85-b463-00b90b91cb1b-credential-socket\") pod \"e2e-api-b-b61c46-openshift-default-7f74cd5f58-5fl2g\" (UID: \"4aed45fc-e9d7-4c85-b463-00b90b91cb1b\") " pod="openshift-ingress/e2e-api-b-b61c46-openshift-default-7f74cd5f58-5fl2g"
Sep 11 04:02:59.333438 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:02:59.333306    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4aed45fc-e9d7-4c85-b463-00b90b91cb1b-workload-certs\") pod \"e2e-api-b-b61c46-openshift-default-7f74cd5f58-5fl2g\" (UID: \"4aed45fc-e9d7-4c85-b463-00b90b91cb1b\") " pod="openshift-ingress/e2e-api-b-b61c46-openshift-default-7f74cd5f58-5fl2g"
Sep 11 04:02:59.333438 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:02:59.333380    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4aed45fc-e9d7-4c85-b463-00b90b91cb1b-workload-socket\") pod \"e2e-api-b-b61c46-openshift-default-7f74cd5f58-5fl2g\" (UID: \"4aed45fc-e9d7-4c85-b463-00b90b91cb1b\") " pod="openshift-ingress/e2e-api-b-b61c46-openshift-default-7f74cd5f58-5fl2g"
Sep 11 04:02:59.333638 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:02:59.333500    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4aed45fc-e9d7-4c85-b463-00b90b91cb1b-istio-data\") pod \"e2e-api-b-b61c46-openshift-default-7f74cd5f58-5fl2g\" (UID: \"4aed45fc-e9d7-4c85-b463-00b90b91cb1b\") " pod="openshift-ingress/e2e-api-b-b61c46-openshift-default-7f74cd5f58-5fl2g"
Sep 11 04:02:59.333638 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:02:59.333597    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4aed45fc-e9d7-4c85-b463-00b90b91cb1b-credential-socket\") pod \"e2e-api-b-b61c46-openshift-default-7f74cd5f58-5fl2g\" (UID: \"4aed45fc-e9d7-4c85-b463-00b90b91cb1b\") " pod="openshift-ingress/e2e-api-b-b61c46-openshift-default-7f74cd5f58-5fl2g"
Sep 11 04:02:59.333992 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:02:59.333974    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4aed45fc-e9d7-4c85-b463-00b90b91cb1b-istiod-ca-cert\") pod \"e2e-api-b-b61c46-openshift-default-7f74cd5f58-5fl2g\" (UID: \"4aed45fc-e9d7-4c85-b463-00b90b91cb1b\") " pod="openshift-ingress/e2e-api-b-b61c46-openshift-default-7f74cd5f58-5fl2g"
Sep 11 04:02:59.335509 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:02:59.335491    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4aed45fc-e9d7-4c85-b463-00b90b91cb1b-istio-envoy\") pod \"e2e-api-b-b61c46-openshift-default-7f74cd5f58-5fl2g\" (UID: \"4aed45fc-e9d7-4c85-b463-00b90b91cb1b\") " pod="openshift-ingress/e2e-api-b-b61c46-openshift-default-7f74cd5f58-5fl2g"
Sep 11 04:02:59.335560 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:02:59.335509    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4aed45fc-e9d7-4c85-b463-00b90b91cb1b-istio-podinfo\") pod \"e2e-api-b-b61c46-openshift-default-7f74cd5f58-5fl2g\" (UID: \"4aed45fc-e9d7-4c85-b463-00b90b91cb1b\") " pod="openshift-ingress/e2e-api-b-b61c46-openshift-default-7f74cd5f58-5fl2g"
Sep 11 04:02:59.362493 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:02:59.362409    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4aed45fc-e9d7-4c85-b463-00b90b91cb1b-istio-token\") pod \"e2e-api-b-b61c46-openshift-default-7f74cd5f58-5fl2g\" (UID: \"4aed45fc-e9d7-4c85-b463-00b90b91cb1b\") " pod="openshift-ingress/e2e-api-b-b61c46-openshift-default-7f74cd5f58-5fl2g"
Sep 11 04:02:59.384345 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:02:59.384308    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wh648\" (UniqueName: \"kubernetes.io/projected/4aed45fc-e9d7-4c85-b463-00b90b91cb1b-kube-api-access-wh648\") pod \"e2e-api-b-b61c46-openshift-default-7f74cd5f58-5fl2g\" (UID: \"4aed45fc-e9d7-4c85-b463-00b90b91cb1b\") " pod="openshift-ingress/e2e-api-b-b61c46-openshift-default-7f74cd5f58-5fl2g"
Sep 11 04:02:59.420824 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:02:59.420794    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-b61c46-openshift-default-7f74cd5f58-5fl2g"
Sep 11 04:02:59.722585 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:02:59.722556    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-b61c46-openshift-default-7f74cd5f58-5fl2g"]
Sep 11 04:02:59.724619 ip-10-0-138-121 kubenswrapper[2580]: W0911 04:02:59.724595    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4aed45fc_e9d7_4c85_b463_00b90b91cb1b.slice/crio-947a8b53bcbb38358d6bb46923b6cd789d0bf36ccc4a8b368526dacf552ee0c1 WatchSource:0}: Error finding container 947a8b53bcbb38358d6bb46923b6cd789d0bf36ccc4a8b368526dacf552ee0c1: Status 404 returned error can't find the container with id 947a8b53bcbb38358d6bb46923b6cd789d0bf36ccc4a8b368526dacf552ee0c1
Sep 11 04:02:59.726847 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:02:59.726815    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 04:02:59.726945 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:02:59.726896    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 04:02:59.727010 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:02:59.726942    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 04:03:00.347940 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:03:00.347909    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b61c46-openshift-default-7f74cd5f58-5fl2g" event={"ID":"4aed45fc-e9d7-4c85-b463-00b90b91cb1b","Type":"ContainerStarted","Data":"c5fcfe2d92a5e2a40ccd96bfc403ccb1a8c31c34aa17acf6fd36a19cc2f21012"}
Sep 11 04:03:00.348309 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:03:00.347957    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b61c46-openshift-default-7f74cd5f58-5fl2g" event={"ID":"4aed45fc-e9d7-4c85-b463-00b90b91cb1b","Type":"ContainerStarted","Data":"947a8b53bcbb38358d6bb46923b6cd789d0bf36ccc4a8b368526dacf552ee0c1"}
Sep 11 04:03:00.386989 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:03:00.386906    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-b61c46-openshift-default-7f74cd5f58-5fl2g" podStartSLOduration=1.386887559 podStartE2EDuration="1.386887559s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:02:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:03:00.385846169 +0000 UTC m=+1368.544332493" watchObservedRunningTime="2026-09-11 04:03:00.386887559 +0000 UTC m=+1368.545373882"
Sep 11 04:03:00.421731 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:03:00.421699    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-b61c46-openshift-default-7f74cd5f58-5fl2g"
Sep 11 04:03:00.427405 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:03:00.427379    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-b61c46-openshift-default-7f74cd5f58-5fl2g"
Sep 11 04:03:01.354504 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:03:01.354474    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-b61c46-openshift-default-7f74cd5f58-5fl2g"
Sep 11 04:03:01.355434 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:03:01.355416    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-b61c46-openshift-default-7f74cd5f58-5fl2g"
Sep 11 04:03:25.493841 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:03:25.493803    2580 generic.go:356] "Generic (PLEG): container finished" podID="4aed45fc-e9d7-4c85-b463-00b90b91cb1b" containerID="c5fcfe2d92a5e2a40ccd96bfc403ccb1a8c31c34aa17acf6fd36a19cc2f21012" exitCode=0
Sep 11 04:03:25.494368 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:03:25.493885    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b61c46-openshift-default-7f74cd5f58-5fl2g" event={"ID":"4aed45fc-e9d7-4c85-b463-00b90b91cb1b","Type":"ContainerDied","Data":"c5fcfe2d92a5e2a40ccd96bfc403ccb1a8c31c34aa17acf6fd36a19cc2f21012"}
Sep 11 04:03:25.494438 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:03:25.494392    2580 scope.go:117] "RemoveContainer" containerID="c5fcfe2d92a5e2a40ccd96bfc403ccb1a8c31c34aa17acf6fd36a19cc2f21012"
Sep 11 04:03:25.495538 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:03:25.495511    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 04:03:25.495648 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:03:25.495589    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 04:03:25.495648 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:03:25.495635    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 04:03:26.501916 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:03:26.501880    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b61c46-openshift-default-7f74cd5f58-5fl2g" event={"ID":"4aed45fc-e9d7-4c85-b463-00b90b91cb1b","Type":"ContainerStarted","Data":"5928efc2718d8e4ec5a1efa912162da9d8e7950b80d59c71dfd92dc5d8411c73"}
Sep 11 04:03:27.421072 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:03:27.421030    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-b61c46-openshift-default-7f74cd5f58-5fl2g"
Sep 11 04:03:27.422346 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:03:27.422316    2580 patch_prober.go:28] interesting pod/e2e-api-b-b61c46-openshift-default-7f74cd5f58-5fl2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Sep 11 04:03:27.422474 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:03:27.422370    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b61c46-openshift-default-7f74cd5f58-5fl2g" podUID="4aed45fc-e9d7-4c85-b463-00b90b91cb1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Sep 11 04:03:28.421736 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:03:28.421700    2580 patch_prober.go:28] interesting pod/e2e-api-b-b61c46-openshift-default-7f74cd5f58-5fl2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Sep 11 04:03:28.422104 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:03:28.421756    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b61c46-openshift-default-7f74cd5f58-5fl2g" podUID="4aed45fc-e9d7-4c85-b463-00b90b91cb1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Sep 11 04:03:29.422270 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:03:29.422231    2580 patch_prober.go:28] interesting pod/e2e-api-b-b61c46-openshift-default-7f74cd5f58-5fl2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Sep 11 04:03:29.422703 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:03:29.422290    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b61c46-openshift-default-7f74cd5f58-5fl2g" podUID="4aed45fc-e9d7-4c85-b463-00b90b91cb1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Sep 11 04:03:30.421954 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:03:30.421916    2580 patch_prober.go:28] interesting pod/e2e-api-b-b61c46-openshift-default-7f74cd5f58-5fl2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Sep 11 04:03:30.422132 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:03:30.421987    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b61c46-openshift-default-7f74cd5f58-5fl2g" podUID="4aed45fc-e9d7-4c85-b463-00b90b91cb1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Sep 11 04:03:31.354705 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:03:31.354638    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-b61c46-openshift-default-7f74cd5f58-5fl2g"
Sep 11 04:03:31.421200 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:03:31.421160    2580 patch_prober.go:28] interesting pod/e2e-api-b-b61c46-openshift-default-7f74cd5f58-5fl2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Sep 11 04:03:31.421376 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:03:31.421223    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b61c46-openshift-default-7f74cd5f58-5fl2g" podUID="4aed45fc-e9d7-4c85-b463-00b90b91cb1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Sep 11 04:03:32.425277 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:03:32.425246    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-b61c46-openshift-default-7f74cd5f58-5fl2g"
Sep 11 04:03:32.426406 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:03:32.426389    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-b61c46-openshift-default-7f74cd5f58-5fl2g"
Sep 11 04:03:46.867864 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:03:46.867828    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-49518b-6dcd48549c-vp88r"]
Sep 11 04:03:46.868679 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:03:46.868601    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-49518b-6dcd48549c-vp88r" podUID="20579439-bcf0-498c-a8ae-5c9988bf5d2f" containerName="payload-processing" containerID="cri-o://a940f04d00eff42754c24dc9a9cfd5512b9bc5f75ef145f3de91f7c17c68138e" gracePeriod=30
Sep 11 04:03:46.871397 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:03:46.871351    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-49518b-778b56d9b6-w5snt"]
Sep 11 04:03:46.871582 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:03:46.871542    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-49518b-778b56d9b6-w5snt" podUID="8b98563f-9815-4506-8301-9f03ac497363" containerName="payload-pre-processing" containerID="cri-o://0015a3f0dd1c8d0ee8baf681c8d03631992e855c9477262ba2847ae5b7aea7f1" gracePeriod=30
Sep 11 04:03:47.132977 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:03:47.132907    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-49518b-778b56d9b6-w5snt"
Sep 11 04:03:47.167863 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:03:47.167838    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-49518b-6dcd48549c-vp88r"
Sep 11 04:03:47.313644 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:03:47.313609    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20579439-bcf0-498c-a8ae-5c9988bf5d2f-plugins-config-volume\") pod \"20579439-bcf0-498c-a8ae-5c9988bf5d2f\" (UID: \"20579439-bcf0-498c-a8ae-5c9988bf5d2f\") "
Sep 11 04:03:47.313838 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:03:47.313674    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7qtzd\" (UniqueName: \"kubernetes.io/projected/8b98563f-9815-4506-8301-9f03ac497363-kube-api-access-7qtzd\") pod \"8b98563f-9815-4506-8301-9f03ac497363\" (UID: \"8b98563f-9815-4506-8301-9f03ac497363\") "
Sep 11 04:03:47.313838 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:03:47.313716    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gxnz6\" (UniqueName: \"kubernetes.io/projected/20579439-bcf0-498c-a8ae-5c9988bf5d2f-kube-api-access-gxnz6\") pod \"20579439-bcf0-498c-a8ae-5c9988bf5d2f\" (UID: \"20579439-bcf0-498c-a8ae-5c9988bf5d2f\") "
Sep 11 04:03:47.313838 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:03:47.313745    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b98563f-9815-4506-8301-9f03ac497363-plugins-config-volume\") pod \"8b98563f-9815-4506-8301-9f03ac497363\" (UID: \"8b98563f-9815-4506-8301-9f03ac497363\") "
Sep 11 04:03:47.314074 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:03:47.313957    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20579439-bcf0-498c-a8ae-5c9988bf5d2f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "20579439-bcf0-498c-a8ae-5c9988bf5d2f" (UID: "20579439-bcf0-498c-a8ae-5c9988bf5d2f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 04:03:47.314139 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:03:47.314108    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20579439-bcf0-498c-a8ae-5c9988bf5d2f-plugins-config-volume\") on node \"ip-10-0-138-121.ec2.internal\" DevicePath \"\""
Sep 11 04:03:47.314195 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:03:47.314152    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b98563f-9815-4506-8301-9f03ac497363-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8b98563f-9815-4506-8301-9f03ac497363" (UID: "8b98563f-9815-4506-8301-9f03ac497363"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 04:03:47.315860 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:03:47.315838    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20579439-bcf0-498c-a8ae-5c9988bf5d2f-kube-api-access-gxnz6" (OuterVolumeSpecName: "kube-api-access-gxnz6") pod "20579439-bcf0-498c-a8ae-5c9988bf5d2f" (UID: "20579439-bcf0-498c-a8ae-5c9988bf5d2f"). InnerVolumeSpecName "kube-api-access-gxnz6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 04:03:47.315946 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:03:47.315867    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b98563f-9815-4506-8301-9f03ac497363-kube-api-access-7qtzd" (OuterVolumeSpecName: "kube-api-access-7qtzd") pod "8b98563f-9815-4506-8301-9f03ac497363" (UID: "8b98563f-9815-4506-8301-9f03ac497363"). InnerVolumeSpecName "kube-api-access-7qtzd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 04:03:47.415613 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:03:47.415579    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b98563f-9815-4506-8301-9f03ac497363-plugins-config-volume\") on node \"ip-10-0-138-121.ec2.internal\" DevicePath \"\""
Sep 11 04:03:47.415613 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:03:47.415609    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7qtzd\" (UniqueName: \"kubernetes.io/projected/8b98563f-9815-4506-8301-9f03ac497363-kube-api-access-7qtzd\") on node \"ip-10-0-138-121.ec2.internal\" DevicePath \"\""
Sep 11 04:03:47.415836 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:03:47.415624    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gxnz6\" (UniqueName: \"kubernetes.io/projected/20579439-bcf0-498c-a8ae-5c9988bf5d2f-kube-api-access-gxnz6\") on node \"ip-10-0-138-121.ec2.internal\" DevicePath \"\""
Sep 11 04:03:47.626300 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:03:47.626264    2580 generic.go:356] "Generic (PLEG): container finished" podID="20579439-bcf0-498c-a8ae-5c9988bf5d2f" containerID="a940f04d00eff42754c24dc9a9cfd5512b9bc5f75ef145f3de91f7c17c68138e" exitCode=0
Sep 11 04:03:47.626487 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:03:47.626327    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-49518b-6dcd48549c-vp88r"
Sep 11 04:03:47.626487 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:03:47.626359    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-49518b-6dcd48549c-vp88r" event={"ID":"20579439-bcf0-498c-a8ae-5c9988bf5d2f","Type":"ContainerDied","Data":"a940f04d00eff42754c24dc9a9cfd5512b9bc5f75ef145f3de91f7c17c68138e"}
Sep 11 04:03:47.626487 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:03:47.626424    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-49518b-6dcd48549c-vp88r" event={"ID":"20579439-bcf0-498c-a8ae-5c9988bf5d2f","Type":"ContainerDied","Data":"e7c2bcdf1caf4da780ab00506ac96b17a48358e51461d1eb755d9ada372c40a4"}
Sep 11 04:03:47.626487 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:03:47.626460    2580 scope.go:117] "RemoveContainer" containerID="a940f04d00eff42754c24dc9a9cfd5512b9bc5f75ef145f3de91f7c17c68138e"
Sep 11 04:03:47.627876 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:03:47.627846    2580 generic.go:356] "Generic (PLEG): container finished" podID="8b98563f-9815-4506-8301-9f03ac497363" containerID="0015a3f0dd1c8d0ee8baf681c8d03631992e855c9477262ba2847ae5b7aea7f1" exitCode=0
Sep 11 04:03:47.628003 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:03:47.627877    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-49518b-778b56d9b6-w5snt"
Sep 11 04:03:47.628003 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:03:47.627930    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-49518b-778b56d9b6-w5snt" event={"ID":"8b98563f-9815-4506-8301-9f03ac497363","Type":"ContainerDied","Data":"0015a3f0dd1c8d0ee8baf681c8d03631992e855c9477262ba2847ae5b7aea7f1"}
Sep 11 04:03:47.628003 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:03:47.627972    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-49518b-778b56d9b6-w5snt" event={"ID":"8b98563f-9815-4506-8301-9f03ac497363","Type":"ContainerDied","Data":"7d44a1c43815afc82737c49e8520125fe39ce271bca71defb02992b5da5cff5e"}
Sep 11 04:03:47.639208 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:03:47.639189    2580 scope.go:117] "RemoveContainer" containerID="a940f04d00eff42754c24dc9a9cfd5512b9bc5f75ef145f3de91f7c17c68138e"
Sep 11 04:03:47.639515 ip-10-0-138-121 kubenswrapper[2580]: E0911 04:03:47.639492    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a940f04d00eff42754c24dc9a9cfd5512b9bc5f75ef145f3de91f7c17c68138e\": container with ID starting with a940f04d00eff42754c24dc9a9cfd5512b9bc5f75ef145f3de91f7c17c68138e not found: ID does not exist" containerID="a940f04d00eff42754c24dc9a9cfd5512b9bc5f75ef145f3de91f7c17c68138e"
Sep 11 04:03:47.639566 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:03:47.639526    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a940f04d00eff42754c24dc9a9cfd5512b9bc5f75ef145f3de91f7c17c68138e"} err="failed to get container status \"a940f04d00eff42754c24dc9a9cfd5512b9bc5f75ef145f3de91f7c17c68138e\": rpc error: code = NotFound desc = could not find container \"a940f04d00eff42754c24dc9a9cfd5512b9bc5f75ef145f3de91f7c17c68138e\": container with ID starting with a940f04d00eff42754c24dc9a9cfd5512b9bc5f75ef145f3de91f7c17c68138e not found: ID does not exist"
Sep 11 04:03:47.639566 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:03:47.639551    2580 scope.go:117] "RemoveContainer" containerID="0015a3f0dd1c8d0ee8baf681c8d03631992e855c9477262ba2847ae5b7aea7f1"
Sep 11 04:03:47.647773 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:03:47.647757    2580 scope.go:117] "RemoveContainer" containerID="0015a3f0dd1c8d0ee8baf681c8d03631992e855c9477262ba2847ae5b7aea7f1"
Sep 11 04:03:47.648028 ip-10-0-138-121 kubenswrapper[2580]: E0911 04:03:47.648013    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0015a3f0dd1c8d0ee8baf681c8d03631992e855c9477262ba2847ae5b7aea7f1\": container with ID starting with 0015a3f0dd1c8d0ee8baf681c8d03631992e855c9477262ba2847ae5b7aea7f1 not found: ID does not exist" containerID="0015a3f0dd1c8d0ee8baf681c8d03631992e855c9477262ba2847ae5b7aea7f1"
Sep 11 04:03:47.648065 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:03:47.648035    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0015a3f0dd1c8d0ee8baf681c8d03631992e855c9477262ba2847ae5b7aea7f1"} err="failed to get container status \"0015a3f0dd1c8d0ee8baf681c8d03631992e855c9477262ba2847ae5b7aea7f1\": rpc error: code = NotFound desc = could not find container \"0015a3f0dd1c8d0ee8baf681c8d03631992e855c9477262ba2847ae5b7aea7f1\": container with ID starting with 0015a3f0dd1c8d0ee8baf681c8d03631992e855c9477262ba2847ae5b7aea7f1 not found: ID does not exist"
Sep 11 04:03:47.651130 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:03:47.651109    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-49518b-6dcd48549c-vp88r"]
Sep 11 04:03:47.655085 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:03:47.655065    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-49518b-6dcd48549c-vp88r"]
Sep 11 04:03:47.665512 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:03:47.665493    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-49518b-778b56d9b6-w5snt"]
Sep 11 04:03:47.667341 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:03:47.667296    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-49518b-778b56d9b6-w5snt"]
Sep 11 04:03:48.464277 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:03:48.464244    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="20579439-bcf0-498c-a8ae-5c9988bf5d2f" path="/var/lib/kubelet/pods/20579439-bcf0-498c-a8ae-5c9988bf5d2f/volumes"
Sep 11 04:03:48.464640 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:03:48.464609    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8b98563f-9815-4506-8301-9f03ac497363" path="/var/lib/kubelet/pods/8b98563f-9815-4506-8301-9f03ac497363/volumes"
Sep 11 04:04:15.819376 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:04:15.819340    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-b61c46-openshift-default-7f74cd5f58-5fl2g"]
Sep 11 04:04:15.819950 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:04:15.819553    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-b61c46-openshift-default-7f74cd5f58-5fl2g" podUID="4aed45fc-e9d7-4c85-b463-00b90b91cb1b" containerName="istio-proxy" containerID="cri-o://5928efc2718d8e4ec5a1efa912162da9d8e7950b80d59c71dfd92dc5d8411c73" gracePeriod=30
Sep 11 04:04:17.427900 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:04:17.427862    2580 patch_prober.go:28] interesting pod/e2e-api-b-b61c46-openshift-default-7f74cd5f58-5fl2g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 11 04:04:17.428389 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:04:17.427937    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-b61c46-openshift-default-7f74cd5f58-5fl2g" podUID="4aed45fc-e9d7-4c85-b463-00b90b91cb1b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 11 04:04:21.072024 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:04:21.071998    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-b61c46-openshift-default-7f74cd5f58-5fl2g"
Sep 11 04:04:21.239283 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:04:21.239245    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4aed45fc-e9d7-4c85-b463-00b90b91cb1b-istio-token\") pod \"4aed45fc-e9d7-4c85-b463-00b90b91cb1b\" (UID: \"4aed45fc-e9d7-4c85-b463-00b90b91cb1b\") "
Sep 11 04:04:21.239489 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:04:21.239294    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4aed45fc-e9d7-4c85-b463-00b90b91cb1b-credential-socket\") pod \"4aed45fc-e9d7-4c85-b463-00b90b91cb1b\" (UID: \"4aed45fc-e9d7-4c85-b463-00b90b91cb1b\") "
Sep 11 04:04:21.239489 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:04:21.239412    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wh648\" (UniqueName: \"kubernetes.io/projected/4aed45fc-e9d7-4c85-b463-00b90b91cb1b-kube-api-access-wh648\") pod \"4aed45fc-e9d7-4c85-b463-00b90b91cb1b\" (UID: \"4aed45fc-e9d7-4c85-b463-00b90b91cb1b\") "
Sep 11 04:04:21.239489 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:04:21.239462    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4aed45fc-e9d7-4c85-b463-00b90b91cb1b-workload-certs\") pod \"4aed45fc-e9d7-4c85-b463-00b90b91cb1b\" (UID: \"4aed45fc-e9d7-4c85-b463-00b90b91cb1b\") "
Sep 11 04:04:21.239690 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:04:21.239494    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4aed45fc-e9d7-4c85-b463-00b90b91cb1b-istio-podinfo\") pod \"4aed45fc-e9d7-4c85-b463-00b90b91cb1b\" (UID: \"4aed45fc-e9d7-4c85-b463-00b90b91cb1b\") "
Sep 11 04:04:21.239690 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:04:21.239514    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4aed45fc-e9d7-4c85-b463-00b90b91cb1b-istiod-ca-cert\") pod \"4aed45fc-e9d7-4c85-b463-00b90b91cb1b\" (UID: \"4aed45fc-e9d7-4c85-b463-00b90b91cb1b\") "
Sep 11 04:04:21.239690 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:04:21.239539    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4aed45fc-e9d7-4c85-b463-00b90b91cb1b-workload-socket\") pod \"4aed45fc-e9d7-4c85-b463-00b90b91cb1b\" (UID: \"4aed45fc-e9d7-4c85-b463-00b90b91cb1b\") "
Sep 11 04:04:21.239690 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:04:21.239553    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4aed45fc-e9d7-4c85-b463-00b90b91cb1b-istio-data\") pod \"4aed45fc-e9d7-4c85-b463-00b90b91cb1b\" (UID: \"4aed45fc-e9d7-4c85-b463-00b90b91cb1b\") "
Sep 11 04:04:21.239690 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:04:21.239581    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4aed45fc-e9d7-4c85-b463-00b90b91cb1b-istio-envoy\") pod \"4aed45fc-e9d7-4c85-b463-00b90b91cb1b\" (UID: \"4aed45fc-e9d7-4c85-b463-00b90b91cb1b\") "
Sep 11 04:04:21.239690 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:04:21.239621    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4aed45fc-e9d7-4c85-b463-00b90b91cb1b-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "4aed45fc-e9d7-4c85-b463-00b90b91cb1b" (UID: "4aed45fc-e9d7-4c85-b463-00b90b91cb1b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 04:04:21.240300 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:04:21.239836    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4aed45fc-e9d7-4c85-b463-00b90b91cb1b-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "4aed45fc-e9d7-4c85-b463-00b90b91cb1b" (UID: "4aed45fc-e9d7-4c85-b463-00b90b91cb1b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 04:04:21.240300 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:04:21.239849    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4aed45fc-e9d7-4c85-b463-00b90b91cb1b-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "4aed45fc-e9d7-4c85-b463-00b90b91cb1b" (UID: "4aed45fc-e9d7-4c85-b463-00b90b91cb1b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 04:04:21.240300 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:04:21.240005    2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4aed45fc-e9d7-4c85-b463-00b90b91cb1b-credential-socket\") on node \"ip-10-0-138-121.ec2.internal\" DevicePath \"\""
Sep 11 04:04:21.240300 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:04:21.240024    2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4aed45fc-e9d7-4c85-b463-00b90b91cb1b-workload-certs\") on node \"ip-10-0-138-121.ec2.internal\" DevicePath \"\""
Sep 11 04:04:21.240300 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:04:21.240039    2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4aed45fc-e9d7-4c85-b463-00b90b91cb1b-workload-socket\") on node \"ip-10-0-138-121.ec2.internal\" DevicePath \"\""
Sep 11 04:04:21.240300 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:04:21.240019    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4aed45fc-e9d7-4c85-b463-00b90b91cb1b-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "4aed45fc-e9d7-4c85-b463-00b90b91cb1b" (UID: "4aed45fc-e9d7-4c85-b463-00b90b91cb1b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 04:04:21.240300 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:04:21.240055    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4aed45fc-e9d7-4c85-b463-00b90b91cb1b-istio-data" (OuterVolumeSpecName: "istio-data") pod "4aed45fc-e9d7-4c85-b463-00b90b91cb1b" (UID: "4aed45fc-e9d7-4c85-b463-00b90b91cb1b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 04:04:21.241678 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:04:21.241629    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4aed45fc-e9d7-4c85-b463-00b90b91cb1b-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "4aed45fc-e9d7-4c85-b463-00b90b91cb1b" (UID: "4aed45fc-e9d7-4c85-b463-00b90b91cb1b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 04:04:21.241785 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:04:21.241718    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4aed45fc-e9d7-4c85-b463-00b90b91cb1b-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "4aed45fc-e9d7-4c85-b463-00b90b91cb1b" (UID: "4aed45fc-e9d7-4c85-b463-00b90b91cb1b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 04:04:21.241927 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:04:21.241912    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4aed45fc-e9d7-4c85-b463-00b90b91cb1b-istio-token" (OuterVolumeSpecName: "istio-token") pod "4aed45fc-e9d7-4c85-b463-00b90b91cb1b" (UID: "4aed45fc-e9d7-4c85-b463-00b90b91cb1b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 04:04:21.241970 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:04:21.241952    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4aed45fc-e9d7-4c85-b463-00b90b91cb1b-kube-api-access-wh648" (OuterVolumeSpecName: "kube-api-access-wh648") pod "4aed45fc-e9d7-4c85-b463-00b90b91cb1b" (UID: "4aed45fc-e9d7-4c85-b463-00b90b91cb1b"). InnerVolumeSpecName "kube-api-access-wh648". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 04:04:21.340767 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:04:21.340731    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wh648\" (UniqueName: \"kubernetes.io/projected/4aed45fc-e9d7-4c85-b463-00b90b91cb1b-kube-api-access-wh648\") on node \"ip-10-0-138-121.ec2.internal\" DevicePath \"\""
Sep 11 04:04:21.340767 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:04:21.340762    2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4aed45fc-e9d7-4c85-b463-00b90b91cb1b-istio-podinfo\") on node \"ip-10-0-138-121.ec2.internal\" DevicePath \"\""
Sep 11 04:04:21.340968 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:04:21.340776    2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4aed45fc-e9d7-4c85-b463-00b90b91cb1b-istiod-ca-cert\") on node \"ip-10-0-138-121.ec2.internal\" DevicePath \"\""
Sep 11 04:04:21.340968 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:04:21.340787    2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4aed45fc-e9d7-4c85-b463-00b90b91cb1b-istio-data\") on node \"ip-10-0-138-121.ec2.internal\" DevicePath \"\""
Sep 11 04:04:21.340968 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:04:21.340798    2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4aed45fc-e9d7-4c85-b463-00b90b91cb1b-istio-envoy\") on node \"ip-10-0-138-121.ec2.internal\" DevicePath \"\""
Sep 11 04:04:21.340968 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:04:21.340808    2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4aed45fc-e9d7-4c85-b463-00b90b91cb1b-istio-token\") on node \"ip-10-0-138-121.ec2.internal\" DevicePath \"\""
Sep 11 04:04:21.811640 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:04:21.811605    2580 generic.go:356] "Generic (PLEG): container finished" podID="4aed45fc-e9d7-4c85-b463-00b90b91cb1b" containerID="5928efc2718d8e4ec5a1efa912162da9d8e7950b80d59c71dfd92dc5d8411c73" exitCode=0
Sep 11 04:04:21.811834 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:04:21.811654    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-b61c46-openshift-default-7f74cd5f58-5fl2g"
Sep 11 04:04:21.811834 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:04:21.811689    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b61c46-openshift-default-7f74cd5f58-5fl2g" event={"ID":"4aed45fc-e9d7-4c85-b463-00b90b91cb1b","Type":"ContainerDied","Data":"5928efc2718d8e4ec5a1efa912162da9d8e7950b80d59c71dfd92dc5d8411c73"}
Sep 11 04:04:21.811834 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:04:21.811750    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b61c46-openshift-default-7f74cd5f58-5fl2g" event={"ID":"4aed45fc-e9d7-4c85-b463-00b90b91cb1b","Type":"ContainerDied","Data":"947a8b53bcbb38358d6bb46923b6cd789d0bf36ccc4a8b368526dacf552ee0c1"}
Sep 11 04:04:21.811834 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:04:21.811781    2580 scope.go:117] "RemoveContainer" containerID="5928efc2718d8e4ec5a1efa912162da9d8e7950b80d59c71dfd92dc5d8411c73"
Sep 11 04:04:21.821076 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:04:21.821056    2580 scope.go:117] "RemoveContainer" containerID="c5fcfe2d92a5e2a40ccd96bfc403ccb1a8c31c34aa17acf6fd36a19cc2f21012"
Sep 11 04:04:21.829148 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:04:21.829130    2580 scope.go:117] "RemoveContainer" containerID="5928efc2718d8e4ec5a1efa912162da9d8e7950b80d59c71dfd92dc5d8411c73"
Sep 11 04:04:21.829388 ip-10-0-138-121 kubenswrapper[2580]: E0911 04:04:21.829372    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5928efc2718d8e4ec5a1efa912162da9d8e7950b80d59c71dfd92dc5d8411c73\": container with ID starting with 5928efc2718d8e4ec5a1efa912162da9d8e7950b80d59c71dfd92dc5d8411c73 not found: ID does not exist" containerID="5928efc2718d8e4ec5a1efa912162da9d8e7950b80d59c71dfd92dc5d8411c73"
Sep 11 04:04:21.829444 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:04:21.829396    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5928efc2718d8e4ec5a1efa912162da9d8e7950b80d59c71dfd92dc5d8411c73"} err="failed to get container status \"5928efc2718d8e4ec5a1efa912162da9d8e7950b80d59c71dfd92dc5d8411c73\": rpc error: code = NotFound desc = could not find container \"5928efc2718d8e4ec5a1efa912162da9d8e7950b80d59c71dfd92dc5d8411c73\": container with ID starting with 5928efc2718d8e4ec5a1efa912162da9d8e7950b80d59c71dfd92dc5d8411c73 not found: ID does not exist"
Sep 11 04:04:21.829444 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:04:21.829411    2580 scope.go:117] "RemoveContainer" containerID="c5fcfe2d92a5e2a40ccd96bfc403ccb1a8c31c34aa17acf6fd36a19cc2f21012"
Sep 11 04:04:21.829631 ip-10-0-138-121 kubenswrapper[2580]: E0911 04:04:21.829614    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5fcfe2d92a5e2a40ccd96bfc403ccb1a8c31c34aa17acf6fd36a19cc2f21012\": container with ID starting with c5fcfe2d92a5e2a40ccd96bfc403ccb1a8c31c34aa17acf6fd36a19cc2f21012 not found: ID does not exist" containerID="c5fcfe2d92a5e2a40ccd96bfc403ccb1a8c31c34aa17acf6fd36a19cc2f21012"
Sep 11 04:04:21.829693 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:04:21.829635    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5fcfe2d92a5e2a40ccd96bfc403ccb1a8c31c34aa17acf6fd36a19cc2f21012"} err="failed to get container status \"c5fcfe2d92a5e2a40ccd96bfc403ccb1a8c31c34aa17acf6fd36a19cc2f21012\": rpc error: code = NotFound desc = could not find container \"c5fcfe2d92a5e2a40ccd96bfc403ccb1a8c31c34aa17acf6fd36a19cc2f21012\": container with ID starting with c5fcfe2d92a5e2a40ccd96bfc403ccb1a8c31c34aa17acf6fd36a19cc2f21012 not found: ID does not exist"
Sep 11 04:04:21.832860 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:04:21.832837    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-b61c46-openshift-default-7f74cd5f58-5fl2g"]
Sep 11 04:04:21.836797 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:04:21.836775    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-b61c46-openshift-default-7f74cd5f58-5fl2g"]
Sep 11 04:04:22.464049 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:04:22.464012    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4aed45fc-e9d7-4c85-b463-00b90b91cb1b" path="/var/lib/kubelet/pods/4aed45fc-e9d7-4c85-b463-00b90b91cb1b/volumes"
Sep 11 04:04:26.600382 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:04:26.600348    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-20ae9a-554444f99-7jp2g"]
Sep 11 04:04:26.600863 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:04:26.600594    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-20ae9a-554444f99-7jp2g" podUID="f77e9215-d1b1-4911-bf22-b3b6e1aba5af" containerName="maas-api" containerID="cri-o://a28d5eae24d1a94850fba2cb21b7ae79828fa4a8730d7559f43181f63930b5fb" gracePeriod=30
Sep 11 04:04:26.659434 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:04:26.659391    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-20ae9a-87496ffbb-pf2hj"]
Sep 11 04:04:26.659716 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:04:26.659684    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-20ae9a-87496ffbb-pf2hj" podUID="dbb75e41-d406-478a-8d2e-ae3b63efe5ba" containerName="payload-pre-processing" containerID="cri-o://8e191c59eb1d5146b2dbb05e8063732e75f75ed26e2fcd33e495ac981507c985" gracePeriod=30
Sep 11 04:04:26.841717 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:04:26.841686    2580 generic.go:356] "Generic (PLEG): container finished" podID="f77e9215-d1b1-4911-bf22-b3b6e1aba5af" containerID="a28d5eae24d1a94850fba2cb21b7ae79828fa4a8730d7559f43181f63930b5fb" exitCode=0
Sep 11 04:04:26.841863 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:04:26.841771    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-20ae9a-554444f99-7jp2g" event={"ID":"f77e9215-d1b1-4911-bf22-b3b6e1aba5af","Type":"ContainerDied","Data":"a28d5eae24d1a94850fba2cb21b7ae79828fa4a8730d7559f43181f63930b5fb"}
Sep 11 04:04:26.843700 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:04:26.843646    2580 generic.go:356] "Generic (PLEG): container finished" podID="dbb75e41-d406-478a-8d2e-ae3b63efe5ba" containerID="8e191c59eb1d5146b2dbb05e8063732e75f75ed26e2fcd33e495ac981507c985" exitCode=0
Sep 11 04:04:26.843808 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:04:26.843701    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-20ae9a-87496ffbb-pf2hj" event={"ID":"dbb75e41-d406-478a-8d2e-ae3b63efe5ba","Type":"ContainerDied","Data":"8e191c59eb1d5146b2dbb05e8063732e75f75ed26e2fcd33e495ac981507c985"}
Sep 11 04:04:26.928644 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:04:26.928616    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-20ae9a-554444f99-7jp2g"
Sep 11 04:04:26.960045 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:04:26.960011    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-20ae9a-87496ffbb-pf2hj"
Sep 11 04:04:27.098249 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:04:27.098211    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f77e9215-d1b1-4911-bf22-b3b6e1aba5af-metrics-certs\") pod \"f77e9215-d1b1-4911-bf22-b3b6e1aba5af\" (UID: \"f77e9215-d1b1-4911-bf22-b3b6e1aba5af\") "
Sep 11 04:04:27.098249 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:04:27.098254    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f77e9215-d1b1-4911-bf22-b3b6e1aba5af-maas-api-tls\") pod \"f77e9215-d1b1-4911-bf22-b3b6e1aba5af\" (UID: \"f77e9215-d1b1-4911-bf22-b3b6e1aba5af\") "
Sep 11 04:04:27.098505 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:04:27.098280    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dbb75e41-d406-478a-8d2e-ae3b63efe5ba-plugins-config-volume\") pod \"dbb75e41-d406-478a-8d2e-ae3b63efe5ba\" (UID: \"dbb75e41-d406-478a-8d2e-ae3b63efe5ba\") "
Sep 11 04:04:27.098505 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:04:27.098308    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cmhdl\" (UniqueName: \"kubernetes.io/projected/f77e9215-d1b1-4911-bf22-b3b6e1aba5af-kube-api-access-cmhdl\") pod \"f77e9215-d1b1-4911-bf22-b3b6e1aba5af\" (UID: \"f77e9215-d1b1-4911-bf22-b3b6e1aba5af\") "
Sep 11 04:04:27.098505 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:04:27.098329    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f77e9215-d1b1-4911-bf22-b3b6e1aba5af-openshift-service-ca\") pod \"f77e9215-d1b1-4911-bf22-b3b6e1aba5af\" (UID: \"f77e9215-d1b1-4911-bf22-b3b6e1aba5af\") "
Sep 11 04:04:27.098505 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:04:27.098388    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qfpcn\" (UniqueName: \"kubernetes.io/projected/dbb75e41-d406-478a-8d2e-ae3b63efe5ba-kube-api-access-qfpcn\") pod \"dbb75e41-d406-478a-8d2e-ae3b63efe5ba\" (UID: \"dbb75e41-d406-478a-8d2e-ae3b63efe5ba\") "
Sep 11 04:04:27.098763 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:04:27.098733    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbb75e41-d406-478a-8d2e-ae3b63efe5ba-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "dbb75e41-d406-478a-8d2e-ae3b63efe5ba" (UID: "dbb75e41-d406-478a-8d2e-ae3b63efe5ba"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 04:04:27.098840 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:04:27.098776    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f77e9215-d1b1-4911-bf22-b3b6e1aba5af-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "f77e9215-d1b1-4911-bf22-b3b6e1aba5af" (UID: "f77e9215-d1b1-4911-bf22-b3b6e1aba5af"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 04:04:27.100637 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:04:27.100608    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f77e9215-d1b1-4911-bf22-b3b6e1aba5af-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "f77e9215-d1b1-4911-bf22-b3b6e1aba5af" (UID: "f77e9215-d1b1-4911-bf22-b3b6e1aba5af"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 04:04:27.100637 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:04:27.100619    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbb75e41-d406-478a-8d2e-ae3b63efe5ba-kube-api-access-qfpcn" (OuterVolumeSpecName: "kube-api-access-qfpcn") pod "dbb75e41-d406-478a-8d2e-ae3b63efe5ba" (UID: "dbb75e41-d406-478a-8d2e-ae3b63efe5ba"). InnerVolumeSpecName "kube-api-access-qfpcn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 04:04:27.100637 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:04:27.100621    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f77e9215-d1b1-4911-bf22-b3b6e1aba5af-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "f77e9215-d1b1-4911-bf22-b3b6e1aba5af" (UID: "f77e9215-d1b1-4911-bf22-b3b6e1aba5af"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 04:04:27.100828 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:04:27.100653    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f77e9215-d1b1-4911-bf22-b3b6e1aba5af-kube-api-access-cmhdl" (OuterVolumeSpecName: "kube-api-access-cmhdl") pod "f77e9215-d1b1-4911-bf22-b3b6e1aba5af" (UID: "f77e9215-d1b1-4911-bf22-b3b6e1aba5af"). InnerVolumeSpecName "kube-api-access-cmhdl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 04:04:27.199313 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:04:27.199276    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qfpcn\" (UniqueName: \"kubernetes.io/projected/dbb75e41-d406-478a-8d2e-ae3b63efe5ba-kube-api-access-qfpcn\") on node \"ip-10-0-138-121.ec2.internal\" DevicePath \"\""
Sep 11 04:04:27.199313 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:04:27.199308    2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f77e9215-d1b1-4911-bf22-b3b6e1aba5af-metrics-certs\") on node \"ip-10-0-138-121.ec2.internal\" DevicePath \"\""
Sep 11 04:04:27.199313 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:04:27.199319    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f77e9215-d1b1-4911-bf22-b3b6e1aba5af-maas-api-tls\") on node \"ip-10-0-138-121.ec2.internal\" DevicePath \"\""
Sep 11 04:04:27.199555 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:04:27.199327    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dbb75e41-d406-478a-8d2e-ae3b63efe5ba-plugins-config-volume\") on node \"ip-10-0-138-121.ec2.internal\" DevicePath \"\""
Sep 11 04:04:27.199555 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:04:27.199338    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cmhdl\" (UniqueName: \"kubernetes.io/projected/f77e9215-d1b1-4911-bf22-b3b6e1aba5af-kube-api-access-cmhdl\") on node \"ip-10-0-138-121.ec2.internal\" DevicePath \"\""
Sep 11 04:04:27.199555 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:04:27.199348    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f77e9215-d1b1-4911-bf22-b3b6e1aba5af-openshift-service-ca\") on node \"ip-10-0-138-121.ec2.internal\" DevicePath \"\""
Sep 11 04:04:27.850736 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:04:27.850698    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-20ae9a-87496ffbb-pf2hj"
Sep 11 04:04:27.851169 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:04:27.850700    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-20ae9a-87496ffbb-pf2hj" event={"ID":"dbb75e41-d406-478a-8d2e-ae3b63efe5ba","Type":"ContainerDied","Data":"92693acc8f54459de04f83cf45aa5bcc1d4deec2cbc99a583f1db9da9e1c0bfc"}
Sep 11 04:04:27.851169 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:04:27.850842    2580 scope.go:117] "RemoveContainer" containerID="8e191c59eb1d5146b2dbb05e8063732e75f75ed26e2fcd33e495ac981507c985"
Sep 11 04:04:27.852808 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:04:27.852334    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-20ae9a-554444f99-7jp2g" event={"ID":"f77e9215-d1b1-4911-bf22-b3b6e1aba5af","Type":"ContainerDied","Data":"5793c30f64fa8f409cab15ecff35a268c63c592244a0e588e16036dd5ab858a6"}
Sep 11 04:04:27.852808 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:04:27.852358    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-20ae9a-554444f99-7jp2g"
Sep 11 04:04:27.863266 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:04:27.862688    2580 scope.go:117] "RemoveContainer" containerID="a28d5eae24d1a94850fba2cb21b7ae79828fa4a8730d7559f43181f63930b5fb"
Sep 11 04:04:27.880719 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:04:27.880692    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-20ae9a-554444f99-7jp2g"]
Sep 11 04:04:27.884245 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:04:27.884222    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-20ae9a-554444f99-7jp2g"]
Sep 11 04:04:27.893367 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:04:27.893344    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-20ae9a-87496ffbb-pf2hj"]
Sep 11 04:04:27.897390 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:04:27.897369    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-20ae9a-87496ffbb-pf2hj"]
Sep 11 04:04:28.464169 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:04:28.464136    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dbb75e41-d406-478a-8d2e-ae3b63efe5ba" path="/var/lib/kubelet/pods/dbb75e41-d406-478a-8d2e-ae3b63efe5ba/volumes"
Sep 11 04:04:28.464551 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:04:28.464538    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f77e9215-d1b1-4911-bf22-b3b6e1aba5af" path="/var/lib/kubelet/pods/f77e9215-d1b1-4911-bf22-b3b6e1aba5af/volumes"
Sep 11 04:04:44.028535 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:04:44.028498    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-7dc787-29818320-95766"]
Sep 11 04:04:44.030296 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:04:44.030273    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-7dc787-29818320-95766"]
Sep 11 04:04:44.464367 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:04:44.464338    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="34e05270-eb2c-4de2-b09d-c9b2b86bc1cf" path="/var/lib/kubelet/pods/34e05270-eb2c-4de2-b09d-c9b2b86bc1cf/volumes"
Sep 11 04:04:50.770202 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:04:50.770169    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-7dc787-openshift-default-78765f97b-vhwr6"]
Sep 11 04:04:50.770582 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:04:50.770451    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-7dc787-openshift-default-78765f97b-vhwr6" podUID="0595e62e-1ef0-4ca3-a28e-0753787602f1" containerName="istio-proxy" containerID="cri-o://d9e8fdaffb8e4734984afe11531416c5acb13c8700ad36a8d8daf685cf668b6f" gracePeriod=30
Sep 11 04:04:56.000089 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:04:56.000056    2580 generic.go:356] "Generic (PLEG): container finished" podID="0595e62e-1ef0-4ca3-a28e-0753787602f1" containerID="d9e8fdaffb8e4734984afe11531416c5acb13c8700ad36a8d8daf685cf668b6f" exitCode=0
Sep 11 04:04:56.000523 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:04:56.000114    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-7dc787-openshift-default-78765f97b-vhwr6" event={"ID":"0595e62e-1ef0-4ca3-a28e-0753787602f1","Type":"ContainerDied","Data":"d9e8fdaffb8e4734984afe11531416c5acb13c8700ad36a8d8daf685cf668b6f"}
Sep 11 04:04:56.000523 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:04:56.000184    2580 scope.go:117] "RemoveContainer" containerID="29bb1d52146d34beaa58e3190b9a5b43a9b013c87640a67ae6dd5c3cd1816f75"
Sep 11 04:04:56.034895 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:04:56.034845    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-7dc787-openshift-default-78765f97b-vhwr6"
Sep 11 04:04:56.157711 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:04:56.157644    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0595e62e-1ef0-4ca3-a28e-0753787602f1-workload-certs\") pod \"0595e62e-1ef0-4ca3-a28e-0753787602f1\" (UID: \"0595e62e-1ef0-4ca3-a28e-0753787602f1\") "
Sep 11 04:04:56.157711 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:04:56.157711    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0595e62e-1ef0-4ca3-a28e-0753787602f1-credential-socket\") pod \"0595e62e-1ef0-4ca3-a28e-0753787602f1\" (UID: \"0595e62e-1ef0-4ca3-a28e-0753787602f1\") "
Sep 11 04:04:56.157959 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:04:56.157736    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0595e62e-1ef0-4ca3-a28e-0753787602f1-istio-podinfo\") pod \"0595e62e-1ef0-4ca3-a28e-0753787602f1\" (UID: \"0595e62e-1ef0-4ca3-a28e-0753787602f1\") "
Sep 11 04:04:56.157959 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:04:56.157758    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0595e62e-1ef0-4ca3-a28e-0753787602f1-istio-token\") pod \"0595e62e-1ef0-4ca3-a28e-0753787602f1\" (UID: \"0595e62e-1ef0-4ca3-a28e-0753787602f1\") "
Sep 11 04:04:56.157959 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:04:56.157797    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0595e62e-1ef0-4ca3-a28e-0753787602f1-workload-socket\") pod \"0595e62e-1ef0-4ca3-a28e-0753787602f1\" (UID: \"0595e62e-1ef0-4ca3-a28e-0753787602f1\") "
Sep 11 04:04:56.157959 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:04:56.157844    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0595e62e-1ef0-4ca3-a28e-0753787602f1-istio-data\") pod \"0595e62e-1ef0-4ca3-a28e-0753787602f1\" (UID: \"0595e62e-1ef0-4ca3-a28e-0753787602f1\") "
Sep 11 04:04:56.157959 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:04:56.157874    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0595e62e-1ef0-4ca3-a28e-0753787602f1-istiod-ca-cert\") pod \"0595e62e-1ef0-4ca3-a28e-0753787602f1\" (UID: \"0595e62e-1ef0-4ca3-a28e-0753787602f1\") "
Sep 11 04:04:56.157959 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:04:56.157905    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0595e62e-1ef0-4ca3-a28e-0753787602f1-istio-envoy\") pod \"0595e62e-1ef0-4ca3-a28e-0753787602f1\" (UID: \"0595e62e-1ef0-4ca3-a28e-0753787602f1\") "
Sep 11 04:04:56.157959 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:04:56.157939    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zvng9\" (UniqueName: \"kubernetes.io/projected/0595e62e-1ef0-4ca3-a28e-0753787602f1-kube-api-access-zvng9\") pod \"0595e62e-1ef0-4ca3-a28e-0753787602f1\" (UID: \"0595e62e-1ef0-4ca3-a28e-0753787602f1\") "
Sep 11 04:04:56.158351 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:04:56.157965    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0595e62e-1ef0-4ca3-a28e-0753787602f1-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "0595e62e-1ef0-4ca3-a28e-0753787602f1" (UID: "0595e62e-1ef0-4ca3-a28e-0753787602f1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 04:04:56.158351 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:04:56.158046    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0595e62e-1ef0-4ca3-a28e-0753787602f1-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "0595e62e-1ef0-4ca3-a28e-0753787602f1" (UID: "0595e62e-1ef0-4ca3-a28e-0753787602f1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 04:04:56.158351 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:04:56.158179    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0595e62e-1ef0-4ca3-a28e-0753787602f1-istio-data" (OuterVolumeSpecName: "istio-data") pod "0595e62e-1ef0-4ca3-a28e-0753787602f1" (UID: "0595e62e-1ef0-4ca3-a28e-0753787602f1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 04:04:56.158351 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:04:56.158317    2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0595e62e-1ef0-4ca3-a28e-0753787602f1-istio-data\") on node \"ip-10-0-138-121.ec2.internal\" DevicePath \"\""
Sep 11 04:04:56.158351 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:04:56.158339    2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0595e62e-1ef0-4ca3-a28e-0753787602f1-workload-certs\") on node \"ip-10-0-138-121.ec2.internal\" DevicePath \"\""
Sep 11 04:04:56.158620 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:04:56.158361    2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0595e62e-1ef0-4ca3-a28e-0753787602f1-credential-socket\") on node \"ip-10-0-138-121.ec2.internal\" DevicePath \"\""
Sep 11 04:04:56.158620 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:04:56.158326    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0595e62e-1ef0-4ca3-a28e-0753787602f1-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "0595e62e-1ef0-4ca3-a28e-0753787602f1" (UID: "0595e62e-1ef0-4ca3-a28e-0753787602f1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 04:04:56.158620 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:04:56.158411    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0595e62e-1ef0-4ca3-a28e-0753787602f1-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "0595e62e-1ef0-4ca3-a28e-0753787602f1" (UID: "0595e62e-1ef0-4ca3-a28e-0753787602f1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 04:04:56.160260 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:04:56.160230    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0595e62e-1ef0-4ca3-a28e-0753787602f1-istio-token" (OuterVolumeSpecName: "istio-token") pod "0595e62e-1ef0-4ca3-a28e-0753787602f1" (UID: "0595e62e-1ef0-4ca3-a28e-0753787602f1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 04:04:56.160460 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:04:56.160434    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0595e62e-1ef0-4ca3-a28e-0753787602f1-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "0595e62e-1ef0-4ca3-a28e-0753787602f1" (UID: "0595e62e-1ef0-4ca3-a28e-0753787602f1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 04:04:56.160460 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:04:56.160455    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0595e62e-1ef0-4ca3-a28e-0753787602f1-kube-api-access-zvng9" (OuterVolumeSpecName: "kube-api-access-zvng9") pod "0595e62e-1ef0-4ca3-a28e-0753787602f1" (UID: "0595e62e-1ef0-4ca3-a28e-0753787602f1"). InnerVolumeSpecName "kube-api-access-zvng9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 04:04:56.160629 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:04:56.160436    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0595e62e-1ef0-4ca3-a28e-0753787602f1-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "0595e62e-1ef0-4ca3-a28e-0753787602f1" (UID: "0595e62e-1ef0-4ca3-a28e-0753787602f1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 04:04:56.258860 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:04:56.258826    2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0595e62e-1ef0-4ca3-a28e-0753787602f1-istiod-ca-cert\") on node \"ip-10-0-138-121.ec2.internal\" DevicePath \"\""
Sep 11 04:04:56.258860 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:04:56.258852    2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0595e62e-1ef0-4ca3-a28e-0753787602f1-istio-envoy\") on node \"ip-10-0-138-121.ec2.internal\" DevicePath \"\""
Sep 11 04:04:56.258860 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:04:56.258861    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zvng9\" (UniqueName: \"kubernetes.io/projected/0595e62e-1ef0-4ca3-a28e-0753787602f1-kube-api-access-zvng9\") on node \"ip-10-0-138-121.ec2.internal\" DevicePath \"\""
Sep 11 04:04:56.259084 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:04:56.258871    2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0595e62e-1ef0-4ca3-a28e-0753787602f1-istio-podinfo\") on node \"ip-10-0-138-121.ec2.internal\" DevicePath \"\""
Sep 11 04:04:56.259084 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:04:56.258879    2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0595e62e-1ef0-4ca3-a28e-0753787602f1-istio-token\") on node \"ip-10-0-138-121.ec2.internal\" DevicePath \"\""
Sep 11 04:04:56.259084 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:04:56.258888    2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0595e62e-1ef0-4ca3-a28e-0753787602f1-workload-socket\") on node \"ip-10-0-138-121.ec2.internal\" DevicePath \"\""
Sep 11 04:04:57.006696 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:04:57.006651    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-7dc787-openshift-default-78765f97b-vhwr6"
Sep 11 04:04:57.007090 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:04:57.006649    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-7dc787-openshift-default-78765f97b-vhwr6" event={"ID":"0595e62e-1ef0-4ca3-a28e-0753787602f1","Type":"ContainerDied","Data":"ced06bcb05db9404807db7fda3a2b765aaf8ea5de2a1cc63791bdfab578a52cf"}
Sep 11 04:04:57.007090 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:04:57.006784    2580 scope.go:117] "RemoveContainer" containerID="d9e8fdaffb8e4734984afe11531416c5acb13c8700ad36a8d8daf685cf668b6f"
Sep 11 04:04:57.023699 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:04:57.023654    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-7dc787-openshift-default-78765f97b-vhwr6"]
Sep 11 04:04:57.026770 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:04:57.026745    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-7dc787-openshift-default-78765f97b-vhwr6"]
Sep 11 04:04:57.301070 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:04:57.300979    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-4cb558-84665c6b56-wqdpk"]
Sep 11 04:04:57.301374 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:04:57.301352    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0595e62e-1ef0-4ca3-a28e-0753787602f1" containerName="istio-proxy"
Sep 11 04:04:57.301374 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:04:57.301368    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="0595e62e-1ef0-4ca3-a28e-0753787602f1" containerName="istio-proxy"
Sep 11 04:04:57.301374 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:04:57.301379    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b98563f-9815-4506-8301-9f03ac497363" containerName="payload-pre-processing"
Sep 11 04:04:57.301620 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:04:57.301385    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b98563f-9815-4506-8301-9f03ac497363" containerName="payload-pre-processing"
Sep 11 04:04:57.301620 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:04:57.301395    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4aed45fc-e9d7-4c85-b463-00b90b91cb1b" containerName="istio-proxy"
Sep 11 04:04:57.301620 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:04:57.301402    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="4aed45fc-e9d7-4c85-b463-00b90b91cb1b" containerName="istio-proxy"
Sep 11 04:04:57.301620 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:04:57.301414    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="20579439-bcf0-498c-a8ae-5c9988bf5d2f" containerName="payload-processing"
Sep 11 04:04:57.301620 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:04:57.301420    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="20579439-bcf0-498c-a8ae-5c9988bf5d2f" containerName="payload-processing"
Sep 11 04:04:57.301620 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:04:57.301437    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4aed45fc-e9d7-4c85-b463-00b90b91cb1b" containerName="istio-proxy"
Sep 11 04:04:57.301620 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:04:57.301445    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="4aed45fc-e9d7-4c85-b463-00b90b91cb1b" containerName="istio-proxy"
Sep 11 04:04:57.301620 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:04:57.301459    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dbb75e41-d406-478a-8d2e-ae3b63efe5ba" containerName="payload-pre-processing"
Sep 11 04:04:57.301620 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:04:57.301464    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="dbb75e41-d406-478a-8d2e-ae3b63efe5ba" containerName="payload-pre-processing"
Sep 11 04:04:57.301620 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:04:57.301473    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f77e9215-d1b1-4911-bf22-b3b6e1aba5af" containerName="maas-api"
Sep 11 04:04:57.301620 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:04:57.301477    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="f77e9215-d1b1-4911-bf22-b3b6e1aba5af" containerName="maas-api"
Sep 11 04:04:57.301620 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:04:57.301541    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="20579439-bcf0-498c-a8ae-5c9988bf5d2f" containerName="payload-processing"
Sep 11 04:04:57.301620 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:04:57.301549    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="dbb75e41-d406-478a-8d2e-ae3b63efe5ba" containerName="payload-pre-processing"
Sep 11 04:04:57.301620 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:04:57.301555    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="4aed45fc-e9d7-4c85-b463-00b90b91cb1b" containerName="istio-proxy"
Sep 11 04:04:57.301620 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:04:57.301563    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="f77e9215-d1b1-4911-bf22-b3b6e1aba5af" containerName="maas-api"
Sep 11 04:04:57.301620 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:04:57.301569    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="8b98563f-9815-4506-8301-9f03ac497363" containerName="payload-pre-processing"
Sep 11 04:04:57.301620 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:04:57.301576    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="0595e62e-1ef0-4ca3-a28e-0753787602f1" containerName="istio-proxy"
Sep 11 04:04:57.301620 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:04:57.301584    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="0595e62e-1ef0-4ca3-a28e-0753787602f1" containerName="istio-proxy"
Sep 11 04:04:57.306315 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:04:57.306294    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4cb558-84665c6b56-wqdpk"
Sep 11 04:04:57.308472 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:04:57.308452    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-4cb558-serving-cert\""
Sep 11 04:04:57.316036 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:04:57.316015    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-4cb558-84665c6b56-wqdpk"]
Sep 11 04:04:57.369268 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:04:57.369235    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71f813c3-4021-4dbc-a261-c86af71f8478-maas-api-tls\") pod \"maas-api-e2e-ipp-a-4cb558-84665c6b56-wqdpk\" (UID: \"71f813c3-4021-4dbc-a261-c86af71f8478\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4cb558-84665c6b56-wqdpk"
Sep 11 04:04:57.369424 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:04:57.369281    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/71f813c3-4021-4dbc-a261-c86af71f8478-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-4cb558-84665c6b56-wqdpk\" (UID: \"71f813c3-4021-4dbc-a261-c86af71f8478\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4cb558-84665c6b56-wqdpk"
Sep 11 04:04:57.369424 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:04:57.369307    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71f813c3-4021-4dbc-a261-c86af71f8478-metrics-certs\") pod \"maas-api-e2e-ipp-a-4cb558-84665c6b56-wqdpk\" (UID: \"71f813c3-4021-4dbc-a261-c86af71f8478\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4cb558-84665c6b56-wqdpk"
Sep 11 04:04:57.369424 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:04:57.369380    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfkbc\" (UniqueName: \"kubernetes.io/projected/71f813c3-4021-4dbc-a261-c86af71f8478-kube-api-access-jfkbc\") pod \"maas-api-e2e-ipp-a-4cb558-84665c6b56-wqdpk\" (UID: \"71f813c3-4021-4dbc-a261-c86af71f8478\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4cb558-84665c6b56-wqdpk"
Sep 11 04:04:57.470711 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:04:57.470653    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71f813c3-4021-4dbc-a261-c86af71f8478-maas-api-tls\") pod \"maas-api-e2e-ipp-a-4cb558-84665c6b56-wqdpk\" (UID: \"71f813c3-4021-4dbc-a261-c86af71f8478\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4cb558-84665c6b56-wqdpk"
Sep 11 04:04:57.470886 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:04:57.470723    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/71f813c3-4021-4dbc-a261-c86af71f8478-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-4cb558-84665c6b56-wqdpk\" (UID: \"71f813c3-4021-4dbc-a261-c86af71f8478\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4cb558-84665c6b56-wqdpk"
Sep 11 04:04:57.470886 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:04:57.470749    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71f813c3-4021-4dbc-a261-c86af71f8478-metrics-certs\") pod \"maas-api-e2e-ipp-a-4cb558-84665c6b56-wqdpk\" (UID: \"71f813c3-4021-4dbc-a261-c86af71f8478\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4cb558-84665c6b56-wqdpk"
Sep 11 04:04:57.470886 ip-10-0-138-121 kubenswrapper[2580]: E0911 04:04:57.470815    2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-4cb558-serving-cert: secret "maas-api-e2e-ipp-a-4cb558-serving-cert" not found
Sep 11 04:04:57.471039 ip-10-0-138-121 kubenswrapper[2580]: E0911 04:04:57.470900    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71f813c3-4021-4dbc-a261-c86af71f8478-maas-api-tls podName:71f813c3-4021-4dbc-a261-c86af71f8478 nodeName:}" failed. No retries permitted until 2026-09-11 04:04:57.970878112 +0000 UTC m=+1486.129364433 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/71f813c3-4021-4dbc-a261-c86af71f8478-maas-api-tls") pod "maas-api-e2e-ipp-a-4cb558-84665c6b56-wqdpk" (UID: "71f813c3-4021-4dbc-a261-c86af71f8478") : secret "maas-api-e2e-ipp-a-4cb558-serving-cert" not found
Sep 11 04:04:57.471039 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:04:57.470825    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfkbc\" (UniqueName: \"kubernetes.io/projected/71f813c3-4021-4dbc-a261-c86af71f8478-kube-api-access-jfkbc\") pod \"maas-api-e2e-ipp-a-4cb558-84665c6b56-wqdpk\" (UID: \"71f813c3-4021-4dbc-a261-c86af71f8478\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4cb558-84665c6b56-wqdpk"
Sep 11 04:04:57.471370 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:04:57.471351    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/71f813c3-4021-4dbc-a261-c86af71f8478-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-4cb558-84665c6b56-wqdpk\" (UID: \"71f813c3-4021-4dbc-a261-c86af71f8478\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4cb558-84665c6b56-wqdpk"
Sep 11 04:04:57.473460 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:04:57.473435    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71f813c3-4021-4dbc-a261-c86af71f8478-metrics-certs\") pod \"maas-api-e2e-ipp-a-4cb558-84665c6b56-wqdpk\" (UID: \"71f813c3-4021-4dbc-a261-c86af71f8478\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4cb558-84665c6b56-wqdpk"
Sep 11 04:04:57.480507 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:04:57.480482    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfkbc\" (UniqueName: \"kubernetes.io/projected/71f813c3-4021-4dbc-a261-c86af71f8478-kube-api-access-jfkbc\") pod \"maas-api-e2e-ipp-a-4cb558-84665c6b56-wqdpk\" (UID: \"71f813c3-4021-4dbc-a261-c86af71f8478\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4cb558-84665c6b56-wqdpk"
Sep 11 04:04:57.888585 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:04:57.888542    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-4cb558-6fcbd7474b-crdl2"]
Sep 11 04:04:57.889036 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:04:57.889018    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0595e62e-1ef0-4ca3-a28e-0753787602f1" containerName="istio-proxy"
Sep 11 04:04:57.889136 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:04:57.889038    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="0595e62e-1ef0-4ca3-a28e-0753787602f1" containerName="istio-proxy"
Sep 11 04:04:57.889195 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:04:57.889166    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="4aed45fc-e9d7-4c85-b463-00b90b91cb1b" containerName="istio-proxy"
Sep 11 04:04:57.892398 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:04:57.892376    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4cb558-6fcbd7474b-crdl2"
Sep 11 04:04:57.894591 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:04:57.894561    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-4cb558-dockercfg-f2m29\""
Sep 11 04:04:57.894855 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:04:57.894838    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-4cb558\""
Sep 11 04:04:57.900347 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:04:57.900144    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-4cb558-6fcbd7474b-crdl2"]
Sep 11 04:04:57.921632 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:04:57.921601    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-4cb558-6d89c46799-s6l4v"]
Sep 11 04:04:57.925013 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:04:57.924998    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-4cb558-6d89c46799-s6l4v"
Sep 11 04:04:57.935362 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:04:57.935339    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-4cb558-6d89c46799-s6l4v"]
Sep 11 04:04:57.976256 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:04:57.976221    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4aca6dec-dc7a-4afc-b894-5393423f6313-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-4cb558-6fcbd7474b-crdl2\" (UID: \"4aca6dec-dc7a-4afc-b894-5393423f6313\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4cb558-6fcbd7474b-crdl2"
Sep 11 04:04:57.976398 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:04:57.976327    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71f813c3-4021-4dbc-a261-c86af71f8478-maas-api-tls\") pod \"maas-api-e2e-ipp-a-4cb558-84665c6b56-wqdpk\" (UID: \"71f813c3-4021-4dbc-a261-c86af71f8478\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4cb558-84665c6b56-wqdpk"
Sep 11 04:04:57.976398 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:04:57.976359    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hltlr\" (UniqueName: \"kubernetes.io/projected/4aca6dec-dc7a-4afc-b894-5393423f6313-kube-api-access-hltlr\") pod \"payload-pre-processing-e2e-ipp-a-4cb558-6fcbd7474b-crdl2\" (UID: \"4aca6dec-dc7a-4afc-b894-5393423f6313\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4cb558-6fcbd7474b-crdl2"
Sep 11 04:04:57.978975 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:04:57.978948    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71f813c3-4021-4dbc-a261-c86af71f8478-maas-api-tls\") pod \"maas-api-e2e-ipp-a-4cb558-84665c6b56-wqdpk\" (UID: \"71f813c3-4021-4dbc-a261-c86af71f8478\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4cb558-84665c6b56-wqdpk"
Sep 11 04:04:58.077315 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:04:58.077279    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hltlr\" (UniqueName: \"kubernetes.io/projected/4aca6dec-dc7a-4afc-b894-5393423f6313-kube-api-access-hltlr\") pod \"payload-pre-processing-e2e-ipp-a-4cb558-6fcbd7474b-crdl2\" (UID: \"4aca6dec-dc7a-4afc-b894-5393423f6313\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4cb558-6fcbd7474b-crdl2"
Sep 11 04:04:58.077692 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:04:58.077353    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e580a41-dba0-4a29-8476-54ca64cd8af4-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-4cb558-6d89c46799-s6l4v\" (UID: \"6e580a41-dba0-4a29-8476-54ca64cd8af4\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-4cb558-6d89c46799-s6l4v"
Sep 11 04:04:58.077692 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:04:58.077385    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4aca6dec-dc7a-4afc-b894-5393423f6313-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-4cb558-6fcbd7474b-crdl2\" (UID: \"4aca6dec-dc7a-4afc-b894-5393423f6313\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4cb558-6fcbd7474b-crdl2"
Sep 11 04:04:58.077692 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:04:58.077431    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwp98\" (UniqueName: \"kubernetes.io/projected/6e580a41-dba0-4a29-8476-54ca64cd8af4-kube-api-access-xwp98\") pod \"payload-processing-e2e-ipp-a-4cb558-6d89c46799-s6l4v\" (UID: \"6e580a41-dba0-4a29-8476-54ca64cd8af4\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-4cb558-6d89c46799-s6l4v"
Sep 11 04:04:58.078084 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:04:58.078063    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4aca6dec-dc7a-4afc-b894-5393423f6313-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-4cb558-6fcbd7474b-crdl2\" (UID: \"4aca6dec-dc7a-4afc-b894-5393423f6313\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4cb558-6fcbd7474b-crdl2"
Sep 11 04:04:58.088544 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:04:58.088516    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hltlr\" (UniqueName: \"kubernetes.io/projected/4aca6dec-dc7a-4afc-b894-5393423f6313-kube-api-access-hltlr\") pod \"payload-pre-processing-e2e-ipp-a-4cb558-6fcbd7474b-crdl2\" (UID: \"4aca6dec-dc7a-4afc-b894-5393423f6313\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4cb558-6fcbd7474b-crdl2"
Sep 11 04:04:58.178231 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:04:58.178191    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwp98\" (UniqueName: \"kubernetes.io/projected/6e580a41-dba0-4a29-8476-54ca64cd8af4-kube-api-access-xwp98\") pod \"payload-processing-e2e-ipp-a-4cb558-6d89c46799-s6l4v\" (UID: \"6e580a41-dba0-4a29-8476-54ca64cd8af4\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-4cb558-6d89c46799-s6l4v"
Sep 11 04:04:58.178393 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:04:58.178296    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e580a41-dba0-4a29-8476-54ca64cd8af4-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-4cb558-6d89c46799-s6l4v\" (UID: \"6e580a41-dba0-4a29-8476-54ca64cd8af4\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-4cb558-6d89c46799-s6l4v"
Sep 11 04:04:58.179001 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:04:58.178979    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e580a41-dba0-4a29-8476-54ca64cd8af4-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-4cb558-6d89c46799-s6l4v\" (UID: \"6e580a41-dba0-4a29-8476-54ca64cd8af4\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-4cb558-6d89c46799-s6l4v"
Sep 11 04:04:58.185984 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:04:58.185962    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwp98\" (UniqueName: \"kubernetes.io/projected/6e580a41-dba0-4a29-8476-54ca64cd8af4-kube-api-access-xwp98\") pod \"payload-processing-e2e-ipp-a-4cb558-6d89c46799-s6l4v\" (UID: \"6e580a41-dba0-4a29-8476-54ca64cd8af4\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-4cb558-6d89c46799-s6l4v"
Sep 11 04:04:58.204930 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:04:58.204899    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4cb558-6fcbd7474b-crdl2"
Sep 11 04:04:58.219148 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:04:58.219114    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4cb558-84665c6b56-wqdpk"
Sep 11 04:04:58.236113 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:04:58.236079    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-4cb558-6d89c46799-s6l4v"
Sep 11 04:04:58.465479 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:04:58.465387    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0595e62e-1ef0-4ca3-a28e-0753787602f1" path="/var/lib/kubelet/pods/0595e62e-1ef0-4ca3-a28e-0753787602f1/volumes"
Sep 11 04:04:58.479847 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:04:58.479818    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-4cb558-6fcbd7474b-crdl2"]
Sep 11 04:04:58.482643 ip-10-0-138-121 kubenswrapper[2580]: W0911 04:04:58.482616    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4aca6dec_dc7a_4afc_b894_5393423f6313.slice/crio-e0bd75e08ebc9480f8908be9851e17285cb74077c125b6e93267dc6ed5d957f1 WatchSource:0}: Error finding container e0bd75e08ebc9480f8908be9851e17285cb74077c125b6e93267dc6ed5d957f1: Status 404 returned error can't find the container with id e0bd75e08ebc9480f8908be9851e17285cb74077c125b6e93267dc6ed5d957f1
Sep 11 04:04:58.513429 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:04:58.513401    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-4cb558-84665c6b56-wqdpk"]
Sep 11 04:04:58.518965 ip-10-0-138-121 kubenswrapper[2580]: W0911 04:04:58.518941    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71f813c3_4021_4dbc_a261_c86af71f8478.slice/crio-47c488f4f2b74a30a564e831dd1ab54d2604a7ff2af55a2808053cfda130b312 WatchSource:0}: Error finding container 47c488f4f2b74a30a564e831dd1ab54d2604a7ff2af55a2808053cfda130b312: Status 404 returned error can't find the container with id 47c488f4f2b74a30a564e831dd1ab54d2604a7ff2af55a2808053cfda130b312
Sep 11 04:04:58.526533 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:04:58.526509    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-4cb558-6d89c46799-s6l4v"]
Sep 11 04:04:58.526913 ip-10-0-138-121 kubenswrapper[2580]: W0911 04:04:58.526884    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e580a41_dba0_4a29_8476_54ca64cd8af4.slice/crio-df40233cbe360a67569089ef1308daf7b022b6eac213f20bb6cf554e3ba000cd WatchSource:0}: Error finding container df40233cbe360a67569089ef1308daf7b022b6eac213f20bb6cf554e3ba000cd: Status 404 returned error can't find the container with id df40233cbe360a67569089ef1308daf7b022b6eac213f20bb6cf554e3ba000cd
Sep 11 04:04:59.020731 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:04:59.020689    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-4cb558-6d89c46799-s6l4v" event={"ID":"6e580a41-dba0-4a29-8476-54ca64cd8af4","Type":"ContainerStarted","Data":"2f82a0803ea98b3cfbd21c19a62f06d21db56b71cd679ec9361b1901f96ee6be"}
Sep 11 04:04:59.020926 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:04:59.020780    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-4cb558-6d89c46799-s6l4v" event={"ID":"6e580a41-dba0-4a29-8476-54ca64cd8af4","Type":"ContainerStarted","Data":"df40233cbe360a67569089ef1308daf7b022b6eac213f20bb6cf554e3ba000cd"}
Sep 11 04:04:59.020926 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:04:59.020906    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-4cb558-6d89c46799-s6l4v"
Sep 11 04:04:59.022307 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:04:59.022281    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4cb558-6fcbd7474b-crdl2" event={"ID":"4aca6dec-dc7a-4afc-b894-5393423f6313","Type":"ContainerStarted","Data":"65a7793b8a2bd50013849f7c987121a6e0d1f31d2bbb97caf4dbf7e4317c11cb"}
Sep 11 04:04:59.022436 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:04:59.022324    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4cb558-6fcbd7474b-crdl2" event={"ID":"4aca6dec-dc7a-4afc-b894-5393423f6313","Type":"ContainerStarted","Data":"e0bd75e08ebc9480f8908be9851e17285cb74077c125b6e93267dc6ed5d957f1"}
Sep 11 04:04:59.022436 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:04:59.022351    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4cb558-6fcbd7474b-crdl2"
Sep 11 04:04:59.023427 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:04:59.023403    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4cb558-84665c6b56-wqdpk" event={"ID":"71f813c3-4021-4dbc-a261-c86af71f8478","Type":"ContainerStarted","Data":"47c488f4f2b74a30a564e831dd1ab54d2604a7ff2af55a2808053cfda130b312"}
Sep 11 04:04:59.037476 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:04:59.037424    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-4cb558-6d89c46799-s6l4v" podStartSLOduration=2.037406968 podStartE2EDuration="2.037406968s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:04:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:04:59.034847341 +0000 UTC m=+1487.193333674" watchObservedRunningTime="2026-09-11 04:04:59.037406968 +0000 UTC m=+1487.195893294"
Sep 11 04:04:59.050136 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:04:59.050095    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4cb558-6fcbd7474b-crdl2" podStartSLOduration=2.050082901 podStartE2EDuration="2.050082901s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:04:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:04:59.049646414 +0000 UTC m=+1487.208132750" watchObservedRunningTime="2026-09-11 04:04:59.050082901 +0000 UTC m=+1487.208569224"
Sep 11 04:05:00.029804 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:05:00.029770    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4cb558-84665c6b56-wqdpk" event={"ID":"71f813c3-4021-4dbc-a261-c86af71f8478","Type":"ContainerStarted","Data":"b3ee8c44990de819c4a9d6f867e365f371f16e622529451c1e8ec36bf7c24bd4"}
Sep 11 04:05:00.030210 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:05:00.030027    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4cb558-84665c6b56-wqdpk"
Sep 11 04:05:00.047731 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:05:00.047681    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4cb558-84665c6b56-wqdpk" podStartSLOduration=2.469830725 podStartE2EDuration="3.047655099s" totalImagesPullingTime="577.824374ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:04:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:05:00.044850859 +0000 UTC m=+1488.203337181" watchObservedRunningTime="2026-09-11 04:05:00.047655099 +0000 UTC m=+1488.206141460"
Sep 11 04:05:06.044147 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:05:06.044118    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4cb558-84665c6b56-wqdpk"
Sep 11 04:05:10.032071 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:05:10.032042    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-4cb558-6d89c46799-s6l4v"
Sep 11 04:05:10.032504 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:05:10.032111    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4cb558-6fcbd7474b-crdl2"
Sep 11 04:05:12.437260 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:05:12.437230    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j4pgw_28b30dae-a2d3-4251-99e8-04bf5153015b/ovn-acl-logging/0.log"
Sep 11 04:05:12.439462 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:05:12.439441    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j4pgw_28b30dae-a2d3-4251-99e8-04bf5153015b/ovn-acl-logging/0.log"
Sep 11 04:06:19.084175 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:06:19.084144    2580 scope.go:117] "RemoveContainer" containerID="6c841644724e7f56255d1a72d4a6858282d1a5e439e34a7484cca589d8c907a9"
Sep 11 04:06:35.883511 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:06:35.883481    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-4cb558-6d89c46799-s6l4v_6e580a41-dba0-4a29-8476-54ca64cd8af4/payload-processing/0.log"
Sep 11 04:06:36.708123 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:06:36.708094    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-ebb9b5-openshift-default-67b599f46-g2j6g"]
Sep 11 04:06:36.713283 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:06:36.713261    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-ebb9b5-openshift-default-67b599f46-g2j6g"
Sep 11 04:06:36.715551 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:06:36.715529    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-cleanup-ebb9b5-openshift-default-dockercfg-p5r2k\""
Sep 11 04:06:36.722923 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:06:36.722899    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-ebb9b5-openshift-default-67b599f46-g2j6g"]
Sep 11 04:06:36.825654 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:06:36.825603    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea9c266c-bd2e-450d-9eb2-7c5d145e9ce3-istio-data\") pod \"e2e-ipp-cleanup-ebb9b5-openshift-default-67b599f46-g2j6g\" (UID: \"ea9c266c-bd2e-450d-9eb2-7c5d145e9ce3\") " pod="openshift-ingress/e2e-ipp-cleanup-ebb9b5-openshift-default-67b599f46-g2j6g"
Sep 11 04:06:36.825871 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:06:36.825693    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea9c266c-bd2e-450d-9eb2-7c5d145e9ce3-workload-socket\") pod \"e2e-ipp-cleanup-ebb9b5-openshift-default-67b599f46-g2j6g\" (UID: \"ea9c266c-bd2e-450d-9eb2-7c5d145e9ce3\") " pod="openshift-ingress/e2e-ipp-cleanup-ebb9b5-openshift-default-67b599f46-g2j6g"
Sep 11 04:06:36.825871 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:06:36.825751    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea9c266c-bd2e-450d-9eb2-7c5d145e9ce3-istio-podinfo\") pod \"e2e-ipp-cleanup-ebb9b5-openshift-default-67b599f46-g2j6g\" (UID: \"ea9c266c-bd2e-450d-9eb2-7c5d145e9ce3\") " pod="openshift-ingress/e2e-ipp-cleanup-ebb9b5-openshift-default-67b599f46-g2j6g"
Sep 11 04:06:36.825986 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:06:36.825859    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea9c266c-bd2e-450d-9eb2-7c5d145e9ce3-istio-token\") pod \"e2e-ipp-cleanup-ebb9b5-openshift-default-67b599f46-g2j6g\" (UID: \"ea9c266c-bd2e-450d-9eb2-7c5d145e9ce3\") " pod="openshift-ingress/e2e-ipp-cleanup-ebb9b5-openshift-default-67b599f46-g2j6g"
Sep 11 04:06:36.825986 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:06:36.825928    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea9c266c-bd2e-450d-9eb2-7c5d145e9ce3-istiod-ca-cert\") pod \"e2e-ipp-cleanup-ebb9b5-openshift-default-67b599f46-g2j6g\" (UID: \"ea9c266c-bd2e-450d-9eb2-7c5d145e9ce3\") " pod="openshift-ingress/e2e-ipp-cleanup-ebb9b5-openshift-default-67b599f46-g2j6g"
Sep 11 04:06:36.825986 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:06:36.825968    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dq22d\" (UniqueName: \"kubernetes.io/projected/ea9c266c-bd2e-450d-9eb2-7c5d145e9ce3-kube-api-access-dq22d\") pod \"e2e-ipp-cleanup-ebb9b5-openshift-default-67b599f46-g2j6g\" (UID: \"ea9c266c-bd2e-450d-9eb2-7c5d145e9ce3\") " pod="openshift-ingress/e2e-ipp-cleanup-ebb9b5-openshift-default-67b599f46-g2j6g"
Sep 11 04:06:36.826129 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:06:36.826014    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea9c266c-bd2e-450d-9eb2-7c5d145e9ce3-istio-envoy\") pod \"e2e-ipp-cleanup-ebb9b5-openshift-default-67b599f46-g2j6g\" (UID: \"ea9c266c-bd2e-450d-9eb2-7c5d145e9ce3\") " pod="openshift-ingress/e2e-ipp-cleanup-ebb9b5-openshift-default-67b599f46-g2j6g"
Sep 11 04:06:36.826129 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:06:36.826048    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea9c266c-bd2e-450d-9eb2-7c5d145e9ce3-credential-socket\") pod \"e2e-ipp-cleanup-ebb9b5-openshift-default-67b599f46-g2j6g\" (UID: \"ea9c266c-bd2e-450d-9eb2-7c5d145e9ce3\") " pod="openshift-ingress/e2e-ipp-cleanup-ebb9b5-openshift-default-67b599f46-g2j6g"
Sep 11 04:06:36.826129 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:06:36.826083    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea9c266c-bd2e-450d-9eb2-7c5d145e9ce3-workload-certs\") pod \"e2e-ipp-cleanup-ebb9b5-openshift-default-67b599f46-g2j6g\" (UID: \"ea9c266c-bd2e-450d-9eb2-7c5d145e9ce3\") " pod="openshift-ingress/e2e-ipp-cleanup-ebb9b5-openshift-default-67b599f46-g2j6g"
Sep 11 04:06:36.926776 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:06:36.926741    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea9c266c-bd2e-450d-9eb2-7c5d145e9ce3-istio-token\") pod \"e2e-ipp-cleanup-ebb9b5-openshift-default-67b599f46-g2j6g\" (UID: \"ea9c266c-bd2e-450d-9eb2-7c5d145e9ce3\") " pod="openshift-ingress/e2e-ipp-cleanup-ebb9b5-openshift-default-67b599f46-g2j6g"
Sep 11 04:06:36.927147 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:06:36.926788    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea9c266c-bd2e-450d-9eb2-7c5d145e9ce3-istiod-ca-cert\") pod \"e2e-ipp-cleanup-ebb9b5-openshift-default-67b599f46-g2j6g\" (UID: \"ea9c266c-bd2e-450d-9eb2-7c5d145e9ce3\") " pod="openshift-ingress/e2e-ipp-cleanup-ebb9b5-openshift-default-67b599f46-g2j6g"
Sep 11 04:06:36.927147 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:06:36.926819    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dq22d\" (UniqueName: \"kubernetes.io/projected/ea9c266c-bd2e-450d-9eb2-7c5d145e9ce3-kube-api-access-dq22d\") pod \"e2e-ipp-cleanup-ebb9b5-openshift-default-67b599f46-g2j6g\" (UID: \"ea9c266c-bd2e-450d-9eb2-7c5d145e9ce3\") " pod="openshift-ingress/e2e-ipp-cleanup-ebb9b5-openshift-default-67b599f46-g2j6g"
Sep 11 04:06:36.927147 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:06:36.926848    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea9c266c-bd2e-450d-9eb2-7c5d145e9ce3-istio-envoy\") pod \"e2e-ipp-cleanup-ebb9b5-openshift-default-67b599f46-g2j6g\" (UID: \"ea9c266c-bd2e-450d-9eb2-7c5d145e9ce3\") " pod="openshift-ingress/e2e-ipp-cleanup-ebb9b5-openshift-default-67b599f46-g2j6g"
Sep 11 04:06:36.927147 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:06:36.926993    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea9c266c-bd2e-450d-9eb2-7c5d145e9ce3-credential-socket\") pod \"e2e-ipp-cleanup-ebb9b5-openshift-default-67b599f46-g2j6g\" (UID: \"ea9c266c-bd2e-450d-9eb2-7c5d145e9ce3\") " pod="openshift-ingress/e2e-ipp-cleanup-ebb9b5-openshift-default-67b599f46-g2j6g"
Sep 11 04:06:36.927147 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:06:36.927049    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea9c266c-bd2e-450d-9eb2-7c5d145e9ce3-workload-certs\") pod \"e2e-ipp-cleanup-ebb9b5-openshift-default-67b599f46-g2j6g\" (UID: \"ea9c266c-bd2e-450d-9eb2-7c5d145e9ce3\") " pod="openshift-ingress/e2e-ipp-cleanup-ebb9b5-openshift-default-67b599f46-g2j6g"
Sep 11 04:06:36.927147 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:06:36.927089    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea9c266c-bd2e-450d-9eb2-7c5d145e9ce3-istio-data\") pod \"e2e-ipp-cleanup-ebb9b5-openshift-default-67b599f46-g2j6g\" (UID: \"ea9c266c-bd2e-450d-9eb2-7c5d145e9ce3\") " pod="openshift-ingress/e2e-ipp-cleanup-ebb9b5-openshift-default-67b599f46-g2j6g"
Sep 11 04:06:36.927147 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:06:36.927135    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea9c266c-bd2e-450d-9eb2-7c5d145e9ce3-workload-socket\") pod \"e2e-ipp-cleanup-ebb9b5-openshift-default-67b599f46-g2j6g\" (UID: \"ea9c266c-bd2e-450d-9eb2-7c5d145e9ce3\") " pod="openshift-ingress/e2e-ipp-cleanup-ebb9b5-openshift-default-67b599f46-g2j6g"
Sep 11 04:06:36.927505 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:06:36.927190    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea9c266c-bd2e-450d-9eb2-7c5d145e9ce3-istio-podinfo\") pod \"e2e-ipp-cleanup-ebb9b5-openshift-default-67b599f46-g2j6g\" (UID: \"ea9c266c-bd2e-450d-9eb2-7c5d145e9ce3\") " pod="openshift-ingress/e2e-ipp-cleanup-ebb9b5-openshift-default-67b599f46-g2j6g"
Sep 11 04:06:36.927505 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:06:36.927332    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea9c266c-bd2e-450d-9eb2-7c5d145e9ce3-credential-socket\") pod \"e2e-ipp-cleanup-ebb9b5-openshift-default-67b599f46-g2j6g\" (UID: \"ea9c266c-bd2e-450d-9eb2-7c5d145e9ce3\") " pod="openshift-ingress/e2e-ipp-cleanup-ebb9b5-openshift-default-67b599f46-g2j6g"
Sep 11 04:06:36.927505 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:06:36.927460    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea9c266c-bd2e-450d-9eb2-7c5d145e9ce3-workload-socket\") pod \"e2e-ipp-cleanup-ebb9b5-openshift-default-67b599f46-g2j6g\" (UID: \"ea9c266c-bd2e-450d-9eb2-7c5d145e9ce3\") " pod="openshift-ingress/e2e-ipp-cleanup-ebb9b5-openshift-default-67b599f46-g2j6g"
Sep 11 04:06:36.927505 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:06:36.927469    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea9c266c-bd2e-450d-9eb2-7c5d145e9ce3-workload-certs\") pod \"e2e-ipp-cleanup-ebb9b5-openshift-default-67b599f46-g2j6g\" (UID: \"ea9c266c-bd2e-450d-9eb2-7c5d145e9ce3\") " pod="openshift-ingress/e2e-ipp-cleanup-ebb9b5-openshift-default-67b599f46-g2j6g"
Sep 11 04:06:36.927759 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:06:36.927515    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea9c266c-bd2e-450d-9eb2-7c5d145e9ce3-istio-data\") pod \"e2e-ipp-cleanup-ebb9b5-openshift-default-67b599f46-g2j6g\" (UID: \"ea9c266c-bd2e-450d-9eb2-7c5d145e9ce3\") " pod="openshift-ingress/e2e-ipp-cleanup-ebb9b5-openshift-default-67b599f46-g2j6g"
Sep 11 04:06:36.927759 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:06:36.927569    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea9c266c-bd2e-450d-9eb2-7c5d145e9ce3-istiod-ca-cert\") pod \"e2e-ipp-cleanup-ebb9b5-openshift-default-67b599f46-g2j6g\" (UID: \"ea9c266c-bd2e-450d-9eb2-7c5d145e9ce3\") " pod="openshift-ingress/e2e-ipp-cleanup-ebb9b5-openshift-default-67b599f46-g2j6g"
Sep 11 04:06:36.929281 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:06:36.929254    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea9c266c-bd2e-450d-9eb2-7c5d145e9ce3-istio-envoy\") pod \"e2e-ipp-cleanup-ebb9b5-openshift-default-67b599f46-g2j6g\" (UID: \"ea9c266c-bd2e-450d-9eb2-7c5d145e9ce3\") " pod="openshift-ingress/e2e-ipp-cleanup-ebb9b5-openshift-default-67b599f46-g2j6g"
Sep 11 04:06:36.929548 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:06:36.929526    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea9c266c-bd2e-450d-9eb2-7c5d145e9ce3-istio-podinfo\") pod \"e2e-ipp-cleanup-ebb9b5-openshift-default-67b599f46-g2j6g\" (UID: \"ea9c266c-bd2e-450d-9eb2-7c5d145e9ce3\") " pod="openshift-ingress/e2e-ipp-cleanup-ebb9b5-openshift-default-67b599f46-g2j6g"
Sep 11 04:06:36.936827 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:06:36.936797    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dq22d\" (UniqueName: \"kubernetes.io/projected/ea9c266c-bd2e-450d-9eb2-7c5d145e9ce3-kube-api-access-dq22d\") pod \"e2e-ipp-cleanup-ebb9b5-openshift-default-67b599f46-g2j6g\" (UID: \"ea9c266c-bd2e-450d-9eb2-7c5d145e9ce3\") " pod="openshift-ingress/e2e-ipp-cleanup-ebb9b5-openshift-default-67b599f46-g2j6g"
Sep 11 04:06:36.937166 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:06:36.937146    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea9c266c-bd2e-450d-9eb2-7c5d145e9ce3-istio-token\") pod \"e2e-ipp-cleanup-ebb9b5-openshift-default-67b599f46-g2j6g\" (UID: \"ea9c266c-bd2e-450d-9eb2-7c5d145e9ce3\") " pod="openshift-ingress/e2e-ipp-cleanup-ebb9b5-openshift-default-67b599f46-g2j6g"
Sep 11 04:06:37.025051 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:06:37.024960    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-ebb9b5-openshift-default-67b599f46-g2j6g"
Sep 11 04:06:37.300603 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:06:37.300528    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-ebb9b5-openshift-default-67b599f46-g2j6g"]
Sep 11 04:06:37.303879 ip-10-0-138-121 kubenswrapper[2580]: W0911 04:06:37.303851    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea9c266c_bd2e_450d_9eb2_7c5d145e9ce3.slice/crio-1435615d36b611c0147b13d118c364d150b818d2e666fc9ea4d64c0baf58839a WatchSource:0}: Error finding container 1435615d36b611c0147b13d118c364d150b818d2e666fc9ea4d64c0baf58839a: Status 404 returned error can't find the container with id 1435615d36b611c0147b13d118c364d150b818d2e666fc9ea4d64c0baf58839a
Sep 11 04:06:37.306036 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:06:37.306005    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 04:06:37.306099 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:06:37.306083    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 04:06:37.306143 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:06:37.306109    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 04:06:37.551229 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:06:37.551135    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-ebb9b5-openshift-default-67b599f46-g2j6g" event={"ID":"ea9c266c-bd2e-450d-9eb2-7c5d145e9ce3","Type":"ContainerStarted","Data":"69ca7b72dc1c0834dc68bf7d284f4b594b9bfbe53fb4cc36313c3f1602f54813"}
Sep 11 04:06:37.551229 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:06:37.551210    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-ebb9b5-openshift-default-67b599f46-g2j6g" event={"ID":"ea9c266c-bd2e-450d-9eb2-7c5d145e9ce3","Type":"ContainerStarted","Data":"1435615d36b611c0147b13d118c364d150b818d2e666fc9ea4d64c0baf58839a"}
Sep 11 04:06:37.570907 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:06:37.570853    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-cleanup-ebb9b5-openshift-default-67b599f46-g2j6g" podStartSLOduration=1.570837856 podStartE2EDuration="1.570837856s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:06:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:06:37.567917537 +0000 UTC m=+1585.726403870" watchObservedRunningTime="2026-09-11 04:06:37.570837856 +0000 UTC m=+1585.729324176"
Sep 11 04:06:38.025330 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:06:38.025291    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-ebb9b5-openshift-default-67b599f46-g2j6g"
Sep 11 04:06:38.030380 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:06:38.030354    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-ebb9b5-openshift-default-67b599f46-g2j6g"
Sep 11 04:06:38.556064 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:06:38.556034    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-ebb9b5-openshift-default-67b599f46-g2j6g"
Sep 11 04:06:38.556925 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:06:38.556898    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-ebb9b5-openshift-default-67b599f46-g2j6g"
Sep 11 04:06:43.116703 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:06:43.116653    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ebb9b5-67768c486b-mg5zc"]
Sep 11 04:06:43.122890 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:06:43.122856    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ebb9b5-67768c486b-mg5zc"
Sep 11 04:06:43.125430 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:06:43.125405    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-ebb9b5\""
Sep 11 04:06:43.125566 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:06:43.125452    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-ebb9b5-dockercfg-r62h5\""
Sep 11 04:06:43.129329 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:06:43.129303    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ebb9b5-67768c486b-mg5zc"]
Sep 11 04:06:43.192760 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:06:43.192722    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8jr6\" (UniqueName: \"kubernetes.io/projected/3e5ae4b4-303a-4500-9aef-dd5bd4ef39fb-kube-api-access-r8jr6\") pod \"payload-pre-processing-e2e-ipp-cleanup-ebb9b5-67768c486b-mg5zc\" (UID: \"3e5ae4b4-303a-4500-9aef-dd5bd4ef39fb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ebb9b5-67768c486b-mg5zc"
Sep 11 04:06:43.192930 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:06:43.192774    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e5ae4b4-303a-4500-9aef-dd5bd4ef39fb-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-ebb9b5-67768c486b-mg5zc\" (UID: \"3e5ae4b4-303a-4500-9aef-dd5bd4ef39fb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ebb9b5-67768c486b-mg5zc"
Sep 11 04:06:43.294295 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:06:43.294266    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8jr6\" (UniqueName: \"kubernetes.io/projected/3e5ae4b4-303a-4500-9aef-dd5bd4ef39fb-kube-api-access-r8jr6\") pod \"payload-pre-processing-e2e-ipp-cleanup-ebb9b5-67768c486b-mg5zc\" (UID: \"3e5ae4b4-303a-4500-9aef-dd5bd4ef39fb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ebb9b5-67768c486b-mg5zc"
Sep 11 04:06:43.294501 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:06:43.294312    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e5ae4b4-303a-4500-9aef-dd5bd4ef39fb-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-ebb9b5-67768c486b-mg5zc\" (UID: \"3e5ae4b4-303a-4500-9aef-dd5bd4ef39fb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ebb9b5-67768c486b-mg5zc"
Sep 11 04:06:43.295074 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:06:43.295048    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e5ae4b4-303a-4500-9aef-dd5bd4ef39fb-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-ebb9b5-67768c486b-mg5zc\" (UID: \"3e5ae4b4-303a-4500-9aef-dd5bd4ef39fb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ebb9b5-67768c486b-mg5zc"
Sep 11 04:06:43.302507 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:06:43.302479    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8jr6\" (UniqueName: \"kubernetes.io/projected/3e5ae4b4-303a-4500-9aef-dd5bd4ef39fb-kube-api-access-r8jr6\") pod \"payload-pre-processing-e2e-ipp-cleanup-ebb9b5-67768c486b-mg5zc\" (UID: \"3e5ae4b4-303a-4500-9aef-dd5bd4ef39fb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ebb9b5-67768c486b-mg5zc"
Sep 11 04:06:43.436261 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:06:43.436231    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ebb9b5-67768c486b-mg5zc"
Sep 11 04:06:43.708559 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:06:43.708535    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ebb9b5-67768c486b-mg5zc"]
Sep 11 04:06:43.710488 ip-10-0-138-121 kubenswrapper[2580]: W0911 04:06:43.710461    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e5ae4b4_303a_4500_9aef_dd5bd4ef39fb.slice/crio-6d7d08897d14db30e465b317700dcf7fef1d8a452600e7f00fd299957a455c62 WatchSource:0}: Error finding container 6d7d08897d14db30e465b317700dcf7fef1d8a452600e7f00fd299957a455c62: Status 404 returned error can't find the container with id 6d7d08897d14db30e465b317700dcf7fef1d8a452600e7f00fd299957a455c62
Sep 11 04:06:44.588358 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:06:44.588320    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ebb9b5-67768c486b-mg5zc" event={"ID":"3e5ae4b4-303a-4500-9aef-dd5bd4ef39fb","Type":"ContainerStarted","Data":"54a0bba0e5ee8a335c7c6806ab7d62a3cebe75a49ca49663b0837d257b3a289b"}
Sep 11 04:06:44.588861 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:06:44.588377    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ebb9b5-67768c486b-mg5zc" event={"ID":"3e5ae4b4-303a-4500-9aef-dd5bd4ef39fb","Type":"ContainerStarted","Data":"6d7d08897d14db30e465b317700dcf7fef1d8a452600e7f00fd299957a455c62"}
Sep 11 04:06:44.588861 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:06:44.588459    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ebb9b5-67768c486b-mg5zc"
Sep 11 04:06:44.605839 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:06:44.605793    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ebb9b5-67768c486b-mg5zc" podStartSLOduration=1.605779217 podStartE2EDuration="1.605779217s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:06:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:06:44.603981684 +0000 UTC m=+1592.762468006" watchObservedRunningTime="2026-09-11 04:06:44.605779217 +0000 UTC m=+1592.764265577"
Sep 11 04:06:55.595553 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:06:55.595522    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ebb9b5-67768c486b-mg5zc"
Sep 11 04:06:59.673746 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:06:59.673712    2580 generic.go:356] "Generic (PLEG): container finished" podID="ea9c266c-bd2e-450d-9eb2-7c5d145e9ce3" containerID="69ca7b72dc1c0834dc68bf7d284f4b594b9bfbe53fb4cc36313c3f1602f54813" exitCode=0
Sep 11 04:06:59.674152 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:06:59.673779    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-ebb9b5-openshift-default-67b599f46-g2j6g" event={"ID":"ea9c266c-bd2e-450d-9eb2-7c5d145e9ce3","Type":"ContainerDied","Data":"69ca7b72dc1c0834dc68bf7d284f4b594b9bfbe53fb4cc36313c3f1602f54813"}
Sep 11 04:06:59.674211 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:06:59.674168    2580 scope.go:117] "RemoveContainer" containerID="69ca7b72dc1c0834dc68bf7d284f4b594b9bfbe53fb4cc36313c3f1602f54813"
Sep 11 04:06:59.675275 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:06:59.675250    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 04:06:59.675350 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:06:59.675313    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 04:06:59.675350 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:06:59.675345    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 04:07:00.682026 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:07:00.681991    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-ebb9b5-openshift-default-67b599f46-g2j6g" event={"ID":"ea9c266c-bd2e-450d-9eb2-7c5d145e9ce3","Type":"ContainerStarted","Data":"452779d3eaee57d1994d803fd4df5f2a7caf87a013f14ceac08af3b43b13e271"}
Sep 11 04:07:01.026170 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:07:01.026077    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-ebb9b5-openshift-default-67b599f46-g2j6g"
Sep 11 04:07:01.026367 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:07:01.026342    2580 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-ebb9b5-openshift-default-67b599f46-g2j6g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Sep 11 04:07:01.026417 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:07:01.026394    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-ebb9b5-openshift-default-67b599f46-g2j6g" podUID="ea9c266c-bd2e-450d-9eb2-7c5d145e9ce3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Sep 11 04:07:02.025406 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:07:02.025366    2580 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-ebb9b5-openshift-default-67b599f46-g2j6g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Sep 11 04:07:02.025919 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:07:02.025438    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-ebb9b5-openshift-default-67b599f46-g2j6g" podUID="ea9c266c-bd2e-450d-9eb2-7c5d145e9ce3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Sep 11 04:07:03.026331 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:07:03.026294    2580 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-ebb9b5-openshift-default-67b599f46-g2j6g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Sep 11 04:07:03.026766 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:07:03.026359    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-ebb9b5-openshift-default-67b599f46-g2j6g" podUID="ea9c266c-bd2e-450d-9eb2-7c5d145e9ce3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Sep 11 04:07:04.026024 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:07:04.025990    2580 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-ebb9b5-openshift-default-67b599f46-g2j6g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Sep 11 04:07:04.026205 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:07:04.026042    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-ebb9b5-openshift-default-67b599f46-g2j6g" podUID="ea9c266c-bd2e-450d-9eb2-7c5d145e9ce3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Sep 11 04:07:05.025924 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:07:05.025839    2580 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-ebb9b5-openshift-default-67b599f46-g2j6g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Sep 11 04:07:05.026300 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:07:05.025916    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-ebb9b5-openshift-default-67b599f46-g2j6g" podUID="ea9c266c-bd2e-450d-9eb2-7c5d145e9ce3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Sep 11 04:07:06.029775 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:07:06.029746    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-ebb9b5-openshift-default-67b599f46-g2j6g"
Sep 11 04:07:06.030289 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:07:06.030079    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-ebb9b5-openshift-default-67b599f46-g2j6g"
Sep 11 04:07:06.030912 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:07:06.030895    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-ebb9b5-openshift-default-67b599f46-g2j6g"
Sep 11 04:07:21.334579 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:07:21.334542    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ebb9b5-67768c486b-mg5zc"]
Sep 11 04:07:21.335056 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:07:21.334814    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ebb9b5-67768c486b-mg5zc" podUID="3e5ae4b4-303a-4500-9aef-dd5bd4ef39fb" containerName="payload-pre-processing" containerID="cri-o://54a0bba0e5ee8a335c7c6806ab7d62a3cebe75a49ca49663b0837d257b3a289b" gracePeriod=30
Sep 11 04:07:21.587550 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:07:21.587489    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ebb9b5-67768c486b-mg5zc"
Sep 11 04:07:21.683355 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:07:21.683305    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e5ae4b4-303a-4500-9aef-dd5bd4ef39fb-plugins-config-volume\") pod \"3e5ae4b4-303a-4500-9aef-dd5bd4ef39fb\" (UID: \"3e5ae4b4-303a-4500-9aef-dd5bd4ef39fb\") "
Sep 11 04:07:21.683526 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:07:21.683414    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r8jr6\" (UniqueName: \"kubernetes.io/projected/3e5ae4b4-303a-4500-9aef-dd5bd4ef39fb-kube-api-access-r8jr6\") pod \"3e5ae4b4-303a-4500-9aef-dd5bd4ef39fb\" (UID: \"3e5ae4b4-303a-4500-9aef-dd5bd4ef39fb\") "
Sep 11 04:07:21.683714 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:07:21.683688    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e5ae4b4-303a-4500-9aef-dd5bd4ef39fb-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3e5ae4b4-303a-4500-9aef-dd5bd4ef39fb" (UID: "3e5ae4b4-303a-4500-9aef-dd5bd4ef39fb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 04:07:21.685742 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:07:21.685716    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e5ae4b4-303a-4500-9aef-dd5bd4ef39fb-kube-api-access-r8jr6" (OuterVolumeSpecName: "kube-api-access-r8jr6") pod "3e5ae4b4-303a-4500-9aef-dd5bd4ef39fb" (UID: "3e5ae4b4-303a-4500-9aef-dd5bd4ef39fb"). InnerVolumeSpecName "kube-api-access-r8jr6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 04:07:21.784708 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:07:21.784648    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e5ae4b4-303a-4500-9aef-dd5bd4ef39fb-plugins-config-volume\") on node \"ip-10-0-138-121.ec2.internal\" DevicePath \"\""
Sep 11 04:07:21.784708 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:07:21.784701    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r8jr6\" (UniqueName: \"kubernetes.io/projected/3e5ae4b4-303a-4500-9aef-dd5bd4ef39fb-kube-api-access-r8jr6\") on node \"ip-10-0-138-121.ec2.internal\" DevicePath \"\""
Sep 11 04:07:21.800689 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:07:21.796562    2580 generic.go:356] "Generic (PLEG): container finished" podID="3e5ae4b4-303a-4500-9aef-dd5bd4ef39fb" containerID="54a0bba0e5ee8a335c7c6806ab7d62a3cebe75a49ca49663b0837d257b3a289b" exitCode=0
Sep 11 04:07:21.800689 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:07:21.796810    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ebb9b5-67768c486b-mg5zc" event={"ID":"3e5ae4b4-303a-4500-9aef-dd5bd4ef39fb","Type":"ContainerDied","Data":"54a0bba0e5ee8a335c7c6806ab7d62a3cebe75a49ca49663b0837d257b3a289b"}
Sep 11 04:07:21.800689 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:07:21.796912    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ebb9b5-67768c486b-mg5zc" event={"ID":"3e5ae4b4-303a-4500-9aef-dd5bd4ef39fb","Type":"ContainerDied","Data":"6d7d08897d14db30e465b317700dcf7fef1d8a452600e7f00fd299957a455c62"}
Sep 11 04:07:21.800689 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:07:21.796955    2580 scope.go:117] "RemoveContainer" containerID="54a0bba0e5ee8a335c7c6806ab7d62a3cebe75a49ca49663b0837d257b3a289b"
Sep 11 04:07:21.800689 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:07:21.797162    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ebb9b5-67768c486b-mg5zc"
Sep 11 04:07:21.812902 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:07:21.812877    2580 scope.go:117] "RemoveContainer" containerID="54a0bba0e5ee8a335c7c6806ab7d62a3cebe75a49ca49663b0837d257b3a289b"
Sep 11 04:07:21.813195 ip-10-0-138-121 kubenswrapper[2580]: E0911 04:07:21.813174    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54a0bba0e5ee8a335c7c6806ab7d62a3cebe75a49ca49663b0837d257b3a289b\": container with ID starting with 54a0bba0e5ee8a335c7c6806ab7d62a3cebe75a49ca49663b0837d257b3a289b not found: ID does not exist" containerID="54a0bba0e5ee8a335c7c6806ab7d62a3cebe75a49ca49663b0837d257b3a289b"
Sep 11 04:07:21.813267 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:07:21.813208    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54a0bba0e5ee8a335c7c6806ab7d62a3cebe75a49ca49663b0837d257b3a289b"} err="failed to get container status \"54a0bba0e5ee8a335c7c6806ab7d62a3cebe75a49ca49663b0837d257b3a289b\": rpc error: code = NotFound desc = could not find container \"54a0bba0e5ee8a335c7c6806ab7d62a3cebe75a49ca49663b0837d257b3a289b\": container with ID starting with 54a0bba0e5ee8a335c7c6806ab7d62a3cebe75a49ca49663b0837d257b3a289b not found: ID does not exist"
Sep 11 04:07:21.823757 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:07:21.823730    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ebb9b5-67768c486b-mg5zc"]
Sep 11 04:07:21.827960 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:07:21.827935    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ebb9b5-67768c486b-mg5zc"]
Sep 11 04:07:22.465531 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:07:22.465500    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3e5ae4b4-303a-4500-9aef-dd5bd4ef39fb" path="/var/lib/kubelet/pods/3e5ae4b4-303a-4500-9aef-dd5bd4ef39fb/volumes"
Sep 11 04:07:27.970521 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:07:27.970484    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-ebb9b5-openshift-default-67b599f46-g2j6g"]
Sep 11 04:07:27.971060 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:07:27.970761    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-cleanup-ebb9b5-openshift-default-67b599f46-g2j6g" podUID="ea9c266c-bd2e-450d-9eb2-7c5d145e9ce3" containerName="istio-proxy" containerID="cri-o://452779d3eaee57d1994d803fd4df5f2a7caf87a013f14ceac08af3b43b13e271" gracePeriod=30
Sep 11 04:07:33.220984 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:07:33.220959    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-ebb9b5-openshift-default-67b599f46-g2j6g"
Sep 11 04:07:33.396977 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:07:33.396893    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea9c266c-bd2e-450d-9eb2-7c5d145e9ce3-istio-envoy\") pod \"ea9c266c-bd2e-450d-9eb2-7c5d145e9ce3\" (UID: \"ea9c266c-bd2e-450d-9eb2-7c5d145e9ce3\") "
Sep 11 04:07:33.396977 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:07:33.396941    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea9c266c-bd2e-450d-9eb2-7c5d145e9ce3-istio-podinfo\") pod \"ea9c266c-bd2e-450d-9eb2-7c5d145e9ce3\" (UID: \"ea9c266c-bd2e-450d-9eb2-7c5d145e9ce3\") "
Sep 11 04:07:33.396977 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:07:33.396960    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea9c266c-bd2e-450d-9eb2-7c5d145e9ce3-istiod-ca-cert\") pod \"ea9c266c-bd2e-450d-9eb2-7c5d145e9ce3\" (UID: \"ea9c266c-bd2e-450d-9eb2-7c5d145e9ce3\") "
Sep 11 04:07:33.397268 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:07:33.397025    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea9c266c-bd2e-450d-9eb2-7c5d145e9ce3-istio-token\") pod \"ea9c266c-bd2e-450d-9eb2-7c5d145e9ce3\" (UID: \"ea9c266c-bd2e-450d-9eb2-7c5d145e9ce3\") "
Sep 11 04:07:33.397268 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:07:33.397080    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea9c266c-bd2e-450d-9eb2-7c5d145e9ce3-workload-certs\") pod \"ea9c266c-bd2e-450d-9eb2-7c5d145e9ce3\" (UID: \"ea9c266c-bd2e-450d-9eb2-7c5d145e9ce3\") "
Sep 11 04:07:33.397268 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:07:33.397110    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea9c266c-bd2e-450d-9eb2-7c5d145e9ce3-credential-socket\") pod \"ea9c266c-bd2e-450d-9eb2-7c5d145e9ce3\" (UID: \"ea9c266c-bd2e-450d-9eb2-7c5d145e9ce3\") "
Sep 11 04:07:33.397268 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:07:33.397137    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea9c266c-bd2e-450d-9eb2-7c5d145e9ce3-istio-data\") pod \"ea9c266c-bd2e-450d-9eb2-7c5d145e9ce3\" (UID: \"ea9c266c-bd2e-450d-9eb2-7c5d145e9ce3\") "
Sep 11 04:07:33.397268 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:07:33.397166    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dq22d\" (UniqueName: \"kubernetes.io/projected/ea9c266c-bd2e-450d-9eb2-7c5d145e9ce3-kube-api-access-dq22d\") pod \"ea9c266c-bd2e-450d-9eb2-7c5d145e9ce3\" (UID: \"ea9c266c-bd2e-450d-9eb2-7c5d145e9ce3\") "
Sep 11 04:07:33.397268 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:07:33.397195    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea9c266c-bd2e-450d-9eb2-7c5d145e9ce3-workload-socket\") pod \"ea9c266c-bd2e-450d-9eb2-7c5d145e9ce3\" (UID: \"ea9c266c-bd2e-450d-9eb2-7c5d145e9ce3\") "
Sep 11 04:07:33.397574 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:07:33.397355    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea9c266c-bd2e-450d-9eb2-7c5d145e9ce3-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "ea9c266c-bd2e-450d-9eb2-7c5d145e9ce3" (UID: "ea9c266c-bd2e-450d-9eb2-7c5d145e9ce3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 04:07:33.397574 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:07:33.397389    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea9c266c-bd2e-450d-9eb2-7c5d145e9ce3-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "ea9c266c-bd2e-450d-9eb2-7c5d145e9ce3" (UID: "ea9c266c-bd2e-450d-9eb2-7c5d145e9ce3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 04:07:33.397574 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:07:33.397497    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea9c266c-bd2e-450d-9eb2-7c5d145e9ce3-istio-data" (OuterVolumeSpecName: "istio-data") pod "ea9c266c-bd2e-450d-9eb2-7c5d145e9ce3" (UID: "ea9c266c-bd2e-450d-9eb2-7c5d145e9ce3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 04:07:33.397574 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:07:33.397569    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea9c266c-bd2e-450d-9eb2-7c5d145e9ce3-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "ea9c266c-bd2e-450d-9eb2-7c5d145e9ce3" (UID: "ea9c266c-bd2e-450d-9eb2-7c5d145e9ce3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 04:07:33.397834 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:07:33.397617    2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea9c266c-bd2e-450d-9eb2-7c5d145e9ce3-workload-certs\") on node \"ip-10-0-138-121.ec2.internal\" DevicePath \"\""
Sep 11 04:07:33.397834 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:07:33.397638    2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea9c266c-bd2e-450d-9eb2-7c5d145e9ce3-credential-socket\") on node \"ip-10-0-138-121.ec2.internal\" DevicePath \"\""
Sep 11 04:07:33.397834 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:07:33.397653    2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea9c266c-bd2e-450d-9eb2-7c5d145e9ce3-istio-data\") on node \"ip-10-0-138-121.ec2.internal\" DevicePath \"\""
Sep 11 04:07:33.397968 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:07:33.397838    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea9c266c-bd2e-450d-9eb2-7c5d145e9ce3-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "ea9c266c-bd2e-450d-9eb2-7c5d145e9ce3" (UID: "ea9c266c-bd2e-450d-9eb2-7c5d145e9ce3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 04:07:33.399285 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:07:33.399262    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ea9c266c-bd2e-450d-9eb2-7c5d145e9ce3-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "ea9c266c-bd2e-450d-9eb2-7c5d145e9ce3" (UID: "ea9c266c-bd2e-450d-9eb2-7c5d145e9ce3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 04:07:33.399414 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:07:33.399349    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea9c266c-bd2e-450d-9eb2-7c5d145e9ce3-kube-api-access-dq22d" (OuterVolumeSpecName: "kube-api-access-dq22d") pod "ea9c266c-bd2e-450d-9eb2-7c5d145e9ce3" (UID: "ea9c266c-bd2e-450d-9eb2-7c5d145e9ce3"). InnerVolumeSpecName "kube-api-access-dq22d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 04:07:33.399561 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:07:33.399539    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea9c266c-bd2e-450d-9eb2-7c5d145e9ce3-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "ea9c266c-bd2e-450d-9eb2-7c5d145e9ce3" (UID: "ea9c266c-bd2e-450d-9eb2-7c5d145e9ce3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 04:07:33.399606 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:07:33.399556    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea9c266c-bd2e-450d-9eb2-7c5d145e9ce3-istio-token" (OuterVolumeSpecName: "istio-token") pod "ea9c266c-bd2e-450d-9eb2-7c5d145e9ce3" (UID: "ea9c266c-bd2e-450d-9eb2-7c5d145e9ce3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 04:07:33.499108 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:07:33.499073    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dq22d\" (UniqueName: \"kubernetes.io/projected/ea9c266c-bd2e-450d-9eb2-7c5d145e9ce3-kube-api-access-dq22d\") on node \"ip-10-0-138-121.ec2.internal\" DevicePath \"\""
Sep 11 04:07:33.499108 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:07:33.499102    2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea9c266c-bd2e-450d-9eb2-7c5d145e9ce3-workload-socket\") on node \"ip-10-0-138-121.ec2.internal\" DevicePath \"\""
Sep 11 04:07:33.499108 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:07:33.499113    2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea9c266c-bd2e-450d-9eb2-7c5d145e9ce3-istio-envoy\") on node \"ip-10-0-138-121.ec2.internal\" DevicePath \"\""
Sep 11 04:07:33.499325 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:07:33.499121    2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea9c266c-bd2e-450d-9eb2-7c5d145e9ce3-istio-podinfo\") on node \"ip-10-0-138-121.ec2.internal\" DevicePath \"\""
Sep 11 04:07:33.499325 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:07:33.499129    2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea9c266c-bd2e-450d-9eb2-7c5d145e9ce3-istiod-ca-cert\") on node \"ip-10-0-138-121.ec2.internal\" DevicePath \"\""
Sep 11 04:07:33.499325 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:07:33.499137    2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea9c266c-bd2e-450d-9eb2-7c5d145e9ce3-istio-token\") on node \"ip-10-0-138-121.ec2.internal\" DevicePath \"\""
Sep 11 04:07:33.862531 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:07:33.862494    2580 generic.go:356] "Generic (PLEG): container finished" podID="ea9c266c-bd2e-450d-9eb2-7c5d145e9ce3" containerID="452779d3eaee57d1994d803fd4df5f2a7caf87a013f14ceac08af3b43b13e271" exitCode=0
Sep 11 04:07:33.862725 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:07:33.862543    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-ebb9b5-openshift-default-67b599f46-g2j6g"
Sep 11 04:07:33.862725 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:07:33.862580    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-ebb9b5-openshift-default-67b599f46-g2j6g" event={"ID":"ea9c266c-bd2e-450d-9eb2-7c5d145e9ce3","Type":"ContainerDied","Data":"452779d3eaee57d1994d803fd4df5f2a7caf87a013f14ceac08af3b43b13e271"}
Sep 11 04:07:33.862725 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:07:33.862634    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-ebb9b5-openshift-default-67b599f46-g2j6g" event={"ID":"ea9c266c-bd2e-450d-9eb2-7c5d145e9ce3","Type":"ContainerDied","Data":"1435615d36b611c0147b13d118c364d150b818d2e666fc9ea4d64c0baf58839a"}
Sep 11 04:07:33.862725 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:07:33.862658    2580 scope.go:117] "RemoveContainer" containerID="452779d3eaee57d1994d803fd4df5f2a7caf87a013f14ceac08af3b43b13e271"
Sep 11 04:07:33.872436 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:07:33.872417    2580 scope.go:117] "RemoveContainer" containerID="69ca7b72dc1c0834dc68bf7d284f4b594b9bfbe53fb4cc36313c3f1602f54813"
Sep 11 04:07:33.880360 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:07:33.880343    2580 scope.go:117] "RemoveContainer" containerID="452779d3eaee57d1994d803fd4df5f2a7caf87a013f14ceac08af3b43b13e271"
Sep 11 04:07:33.880626 ip-10-0-138-121 kubenswrapper[2580]: E0911 04:07:33.880607    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"452779d3eaee57d1994d803fd4df5f2a7caf87a013f14ceac08af3b43b13e271\": container with ID starting with 452779d3eaee57d1994d803fd4df5f2a7caf87a013f14ceac08af3b43b13e271 not found: ID does not exist" containerID="452779d3eaee57d1994d803fd4df5f2a7caf87a013f14ceac08af3b43b13e271"
Sep 11 04:07:33.880767 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:07:33.880635    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"452779d3eaee57d1994d803fd4df5f2a7caf87a013f14ceac08af3b43b13e271"} err="failed to get container status \"452779d3eaee57d1994d803fd4df5f2a7caf87a013f14ceac08af3b43b13e271\": rpc error: code = NotFound desc = could not find container \"452779d3eaee57d1994d803fd4df5f2a7caf87a013f14ceac08af3b43b13e271\": container with ID starting with 452779d3eaee57d1994d803fd4df5f2a7caf87a013f14ceac08af3b43b13e271 not found: ID does not exist"
Sep 11 04:07:33.880767 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:07:33.880652    2580 scope.go:117] "RemoveContainer" containerID="69ca7b72dc1c0834dc68bf7d284f4b594b9bfbe53fb4cc36313c3f1602f54813"
Sep 11 04:07:33.880919 ip-10-0-138-121 kubenswrapper[2580]: E0911 04:07:33.880904    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69ca7b72dc1c0834dc68bf7d284f4b594b9bfbe53fb4cc36313c3f1602f54813\": container with ID starting with 69ca7b72dc1c0834dc68bf7d284f4b594b9bfbe53fb4cc36313c3f1602f54813 not found: ID does not exist" containerID="69ca7b72dc1c0834dc68bf7d284f4b594b9bfbe53fb4cc36313c3f1602f54813"
Sep 11 04:07:33.880962 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:07:33.880920    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69ca7b72dc1c0834dc68bf7d284f4b594b9bfbe53fb4cc36313c3f1602f54813"} err="failed to get container status \"69ca7b72dc1c0834dc68bf7d284f4b594b9bfbe53fb4cc36313c3f1602f54813\": rpc error: code = NotFound desc = could not find container \"69ca7b72dc1c0834dc68bf7d284f4b594b9bfbe53fb4cc36313c3f1602f54813\": container with ID starting with 69ca7b72dc1c0834dc68bf7d284f4b594b9bfbe53fb4cc36313c3f1602f54813 not found: ID does not exist"
Sep 11 04:07:33.883708 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:07:33.883687    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-ebb9b5-openshift-default-67b599f46-g2j6g"]
Sep 11 04:07:33.893769 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:07:33.888474    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-ebb9b5-openshift-default-67b599f46-g2j6g"]
Sep 11 04:07:34.464887 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:07:34.464850    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ea9c266c-bd2e-450d-9eb2-7c5d145e9ce3" path="/var/lib/kubelet/pods/ea9c266c-bd2e-450d-9eb2-7c5d145e9ce3/volumes"
Sep 11 04:07:47.555024 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:07:47.554986    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-4cb558-84665c6b56-wqdpk"]
Sep 11 04:07:47.555517 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:07:47.555239    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4cb558-84665c6b56-wqdpk" podUID="71f813c3-4021-4dbc-a261-c86af71f8478" containerName="maas-api" containerID="cri-o://b3ee8c44990de819c4a9d6f867e365f371f16e622529451c1e8ec36bf7c24bd4" gracePeriod=30
Sep 11 04:07:47.611609 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:07:47.611573    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-4cb558-6d89c46799-s6l4v"]
Sep 11 04:07:47.611890 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:07:47.611857    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-4cb558-6d89c46799-s6l4v" podUID="6e580a41-dba0-4a29-8476-54ca64cd8af4" containerName="payload-processing" containerID="cri-o://2f82a0803ea98b3cfbd21c19a62f06d21db56b71cd679ec9361b1901f96ee6be" gracePeriod=30
Sep 11 04:07:47.619142 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:07:47.619114    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-4cb558-6fcbd7474b-crdl2"]
Sep 11 04:07:47.619400 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:07:47.619355    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4cb558-6fcbd7474b-crdl2" podUID="4aca6dec-dc7a-4afc-b894-5393423f6313" containerName="payload-pre-processing" containerID="cri-o://65a7793b8a2bd50013849f7c987121a6e0d1f31d2bbb97caf4dbf7e4317c11cb" gracePeriod=30
Sep 11 04:07:47.904328 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:07:47.904296    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-4cb558-6d89c46799-s6l4v"
Sep 11 04:07:47.907255 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:07:47.907237    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4cb558-84665c6b56-wqdpk"
Sep 11 04:07:47.910619 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:07:47.910601    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4cb558-6fcbd7474b-crdl2"
Sep 11 04:07:47.934971 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:07:47.934949    2580 generic.go:356] "Generic (PLEG): container finished" podID="6e580a41-dba0-4a29-8476-54ca64cd8af4" containerID="2f82a0803ea98b3cfbd21c19a62f06d21db56b71cd679ec9361b1901f96ee6be" exitCode=0
Sep 11 04:07:47.935110 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:07:47.934991    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-4cb558-6d89c46799-s6l4v"
Sep 11 04:07:47.935110 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:07:47.935024    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-4cb558-6d89c46799-s6l4v" event={"ID":"6e580a41-dba0-4a29-8476-54ca64cd8af4","Type":"ContainerDied","Data":"2f82a0803ea98b3cfbd21c19a62f06d21db56b71cd679ec9361b1901f96ee6be"}
Sep 11 04:07:47.935110 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:07:47.935074    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-4cb558-6d89c46799-s6l4v" event={"ID":"6e580a41-dba0-4a29-8476-54ca64cd8af4","Type":"ContainerDied","Data":"df40233cbe360a67569089ef1308daf7b022b6eac213f20bb6cf554e3ba000cd"}
Sep 11 04:07:47.935110 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:07:47.935102    2580 scope.go:117] "RemoveContainer" containerID="2f82a0803ea98b3cfbd21c19a62f06d21db56b71cd679ec9361b1901f96ee6be"
Sep 11 04:07:47.936290 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:07:47.936267    2580 generic.go:356] "Generic (PLEG): container finished" podID="4aca6dec-dc7a-4afc-b894-5393423f6313" containerID="65a7793b8a2bd50013849f7c987121a6e0d1f31d2bbb97caf4dbf7e4317c11cb" exitCode=0
Sep 11 04:07:47.936395 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:07:47.936297    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4cb558-6fcbd7474b-crdl2"
Sep 11 04:07:47.936395 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:07:47.936353    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4cb558-6fcbd7474b-crdl2" event={"ID":"4aca6dec-dc7a-4afc-b894-5393423f6313","Type":"ContainerDied","Data":"65a7793b8a2bd50013849f7c987121a6e0d1f31d2bbb97caf4dbf7e4317c11cb"}
Sep 11 04:07:47.936491 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:07:47.936401    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4cb558-6fcbd7474b-crdl2" event={"ID":"4aca6dec-dc7a-4afc-b894-5393423f6313","Type":"ContainerDied","Data":"e0bd75e08ebc9480f8908be9851e17285cb74077c125b6e93267dc6ed5d957f1"}
Sep 11 04:07:47.938652 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:07:47.938620    2580 generic.go:356] "Generic (PLEG): container finished" podID="71f813c3-4021-4dbc-a261-c86af71f8478" containerID="b3ee8c44990de819c4a9d6f867e365f371f16e622529451c1e8ec36bf7c24bd4" exitCode=0
Sep 11 04:07:47.938844 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:07:47.938761    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4cb558-84665c6b56-wqdpk" event={"ID":"71f813c3-4021-4dbc-a261-c86af71f8478","Type":"ContainerDied","Data":"b3ee8c44990de819c4a9d6f867e365f371f16e622529451c1e8ec36bf7c24bd4"}
Sep 11 04:07:47.938844 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:07:47.938801    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4cb558-84665c6b56-wqdpk" event={"ID":"71f813c3-4021-4dbc-a261-c86af71f8478","Type":"ContainerDied","Data":"47c488f4f2b74a30a564e831dd1ab54d2604a7ff2af55a2808053cfda130b312"}
Sep 11 04:07:47.939081 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:07:47.939060    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4cb558-84665c6b56-wqdpk"
Sep 11 04:07:47.946448 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:07:47.946430    2580 scope.go:117] "RemoveContainer" containerID="2f82a0803ea98b3cfbd21c19a62f06d21db56b71cd679ec9361b1901f96ee6be"
Sep 11 04:07:47.946733 ip-10-0-138-121 kubenswrapper[2580]: E0911 04:07:47.946713    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f82a0803ea98b3cfbd21c19a62f06d21db56b71cd679ec9361b1901f96ee6be\": container with ID starting with 2f82a0803ea98b3cfbd21c19a62f06d21db56b71cd679ec9361b1901f96ee6be not found: ID does not exist" containerID="2f82a0803ea98b3cfbd21c19a62f06d21db56b71cd679ec9361b1901f96ee6be"
Sep 11 04:07:47.946824 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:07:47.946744    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f82a0803ea98b3cfbd21c19a62f06d21db56b71cd679ec9361b1901f96ee6be"} err="failed to get container status \"2f82a0803ea98b3cfbd21c19a62f06d21db56b71cd679ec9361b1901f96ee6be\": rpc error: code = NotFound desc = could not find container \"2f82a0803ea98b3cfbd21c19a62f06d21db56b71cd679ec9361b1901f96ee6be\": container with ID starting with 2f82a0803ea98b3cfbd21c19a62f06d21db56b71cd679ec9361b1901f96ee6be not found: ID does not exist"
Sep 11 04:07:47.946824 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:07:47.946767    2580 scope.go:117] "RemoveContainer" containerID="65a7793b8a2bd50013849f7c987121a6e0d1f31d2bbb97caf4dbf7e4317c11cb"
Sep 11 04:07:47.956987 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:07:47.956970    2580 scope.go:117] "RemoveContainer" containerID="65a7793b8a2bd50013849f7c987121a6e0d1f31d2bbb97caf4dbf7e4317c11cb"
Sep 11 04:07:47.957313 ip-10-0-138-121 kubenswrapper[2580]: E0911 04:07:47.957283    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65a7793b8a2bd50013849f7c987121a6e0d1f31d2bbb97caf4dbf7e4317c11cb\": container with ID starting with 65a7793b8a2bd50013849f7c987121a6e0d1f31d2bbb97caf4dbf7e4317c11cb not found: ID does not exist" containerID="65a7793b8a2bd50013849f7c987121a6e0d1f31d2bbb97caf4dbf7e4317c11cb"
Sep 11 04:07:47.957427 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:07:47.957322    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65a7793b8a2bd50013849f7c987121a6e0d1f31d2bbb97caf4dbf7e4317c11cb"} err="failed to get container status \"65a7793b8a2bd50013849f7c987121a6e0d1f31d2bbb97caf4dbf7e4317c11cb\": rpc error: code = NotFound desc = could not find container \"65a7793b8a2bd50013849f7c987121a6e0d1f31d2bbb97caf4dbf7e4317c11cb\": container with ID starting with 65a7793b8a2bd50013849f7c987121a6e0d1f31d2bbb97caf4dbf7e4317c11cb not found: ID does not exist"
Sep 11 04:07:47.957427 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:07:47.957345    2580 scope.go:117] "RemoveContainer" containerID="b3ee8c44990de819c4a9d6f867e365f371f16e622529451c1e8ec36bf7c24bd4"
Sep 11 04:07:47.968303 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:07:47.968284    2580 scope.go:117] "RemoveContainer" containerID="b3ee8c44990de819c4a9d6f867e365f371f16e622529451c1e8ec36bf7c24bd4"
Sep 11 04:07:47.968584 ip-10-0-138-121 kubenswrapper[2580]: E0911 04:07:47.968561    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3ee8c44990de819c4a9d6f867e365f371f16e622529451c1e8ec36bf7c24bd4\": container with ID starting with b3ee8c44990de819c4a9d6f867e365f371f16e622529451c1e8ec36bf7c24bd4 not found: ID does not exist" containerID="b3ee8c44990de819c4a9d6f867e365f371f16e622529451c1e8ec36bf7c24bd4"
Sep 11 04:07:47.968739 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:07:47.968586    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3ee8c44990de819c4a9d6f867e365f371f16e622529451c1e8ec36bf7c24bd4"} err="failed to get container status \"b3ee8c44990de819c4a9d6f867e365f371f16e622529451c1e8ec36bf7c24bd4\": rpc error: code = NotFound desc = could not find container \"b3ee8c44990de819c4a9d6f867e365f371f16e622529451c1e8ec36bf7c24bd4\": container with ID starting with b3ee8c44990de819c4a9d6f867e365f371f16e622529451c1e8ec36bf7c24bd4 not found: ID does not exist"
Sep 11 04:07:48.030182 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:07:48.030154    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4aca6dec-dc7a-4afc-b894-5393423f6313-plugins-config-volume\") pod \"4aca6dec-dc7a-4afc-b894-5393423f6313\" (UID: \"4aca6dec-dc7a-4afc-b894-5393423f6313\") "
Sep 11 04:07:48.030316 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:07:48.030212    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/71f813c3-4021-4dbc-a261-c86af71f8478-openshift-service-ca\") pod \"71f813c3-4021-4dbc-a261-c86af71f8478\" (UID: \"71f813c3-4021-4dbc-a261-c86af71f8478\") "
Sep 11 04:07:48.030316 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:07:48.030244    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hltlr\" (UniqueName: \"kubernetes.io/projected/4aca6dec-dc7a-4afc-b894-5393423f6313-kube-api-access-hltlr\") pod \"4aca6dec-dc7a-4afc-b894-5393423f6313\" (UID: \"4aca6dec-dc7a-4afc-b894-5393423f6313\") "
Sep 11 04:07:48.030316 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:07:48.030284    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e580a41-dba0-4a29-8476-54ca64cd8af4-plugins-config-volume\") pod \"6e580a41-dba0-4a29-8476-54ca64cd8af4\" (UID: \"6e580a41-dba0-4a29-8476-54ca64cd8af4\") "
Sep 11 04:07:48.030488 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:07:48.030325    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71f813c3-4021-4dbc-a261-c86af71f8478-maas-api-tls\") pod \"71f813c3-4021-4dbc-a261-c86af71f8478\" (UID: \"71f813c3-4021-4dbc-a261-c86af71f8478\") "
Sep 11 04:07:48.030488 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:07:48.030445    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71f813c3-4021-4dbc-a261-c86af71f8478-metrics-certs\") pod \"71f813c3-4021-4dbc-a261-c86af71f8478\" (UID: \"71f813c3-4021-4dbc-a261-c86af71f8478\") "
Sep 11 04:07:48.030488 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:07:48.030479    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jfkbc\" (UniqueName: \"kubernetes.io/projected/71f813c3-4021-4dbc-a261-c86af71f8478-kube-api-access-jfkbc\") pod \"71f813c3-4021-4dbc-a261-c86af71f8478\" (UID: \"71f813c3-4021-4dbc-a261-c86af71f8478\") "
Sep 11 04:07:48.030648 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:07:48.030551    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xwp98\" (UniqueName: \"kubernetes.io/projected/6e580a41-dba0-4a29-8476-54ca64cd8af4-kube-api-access-xwp98\") pod \"6e580a41-dba0-4a29-8476-54ca64cd8af4\" (UID: \"6e580a41-dba0-4a29-8476-54ca64cd8af4\") "
Sep 11 04:07:48.030648 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:07:48.030611    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4aca6dec-dc7a-4afc-b894-5393423f6313-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4aca6dec-dc7a-4afc-b894-5393423f6313" (UID: "4aca6dec-dc7a-4afc-b894-5393423f6313"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 04:07:48.030795 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:07:48.030651    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e580a41-dba0-4a29-8476-54ca64cd8af4-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6e580a41-dba0-4a29-8476-54ca64cd8af4" (UID: "6e580a41-dba0-4a29-8476-54ca64cd8af4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 04:07:48.030855 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:07:48.030836    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71f813c3-4021-4dbc-a261-c86af71f8478-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "71f813c3-4021-4dbc-a261-c86af71f8478" (UID: "71f813c3-4021-4dbc-a261-c86af71f8478"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 04:07:48.031027 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:07:48.030940    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4aca6dec-dc7a-4afc-b894-5393423f6313-plugins-config-volume\") on node \"ip-10-0-138-121.ec2.internal\" DevicePath \"\""
Sep 11 04:07:48.031027 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:07:48.030958    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/71f813c3-4021-4dbc-a261-c86af71f8478-openshift-service-ca\") on node \"ip-10-0-138-121.ec2.internal\" DevicePath \"\""
Sep 11 04:07:48.031027 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:07:48.030973    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e580a41-dba0-4a29-8476-54ca64cd8af4-plugins-config-volume\") on node \"ip-10-0-138-121.ec2.internal\" DevicePath \"\""
Sep 11 04:07:48.032521 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:07:48.032493    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4aca6dec-dc7a-4afc-b894-5393423f6313-kube-api-access-hltlr" (OuterVolumeSpecName: "kube-api-access-hltlr") pod "4aca6dec-dc7a-4afc-b894-5393423f6313" (UID: "4aca6dec-dc7a-4afc-b894-5393423f6313"). InnerVolumeSpecName "kube-api-access-hltlr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 04:07:48.032626 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:07:48.032546    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71f813c3-4021-4dbc-a261-c86af71f8478-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "71f813c3-4021-4dbc-a261-c86af71f8478" (UID: "71f813c3-4021-4dbc-a261-c86af71f8478"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 04:07:48.032626 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:07:48.032594    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e580a41-dba0-4a29-8476-54ca64cd8af4-kube-api-access-xwp98" (OuterVolumeSpecName: "kube-api-access-xwp98") pod "6e580a41-dba0-4a29-8476-54ca64cd8af4" (UID: "6e580a41-dba0-4a29-8476-54ca64cd8af4"). InnerVolumeSpecName "kube-api-access-xwp98". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 04:07:48.032830 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:07:48.032815    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71f813c3-4021-4dbc-a261-c86af71f8478-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "71f813c3-4021-4dbc-a261-c86af71f8478" (UID: "71f813c3-4021-4dbc-a261-c86af71f8478"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 04:07:48.032937 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:07:48.032918    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71f813c3-4021-4dbc-a261-c86af71f8478-kube-api-access-jfkbc" (OuterVolumeSpecName: "kube-api-access-jfkbc") pod "71f813c3-4021-4dbc-a261-c86af71f8478" (UID: "71f813c3-4021-4dbc-a261-c86af71f8478"). InnerVolumeSpecName "kube-api-access-jfkbc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 04:07:48.131634 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:07:48.131541    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71f813c3-4021-4dbc-a261-c86af71f8478-maas-api-tls\") on node \"ip-10-0-138-121.ec2.internal\" DevicePath \"\""
Sep 11 04:07:48.131634 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:07:48.131573    2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71f813c3-4021-4dbc-a261-c86af71f8478-metrics-certs\") on node \"ip-10-0-138-121.ec2.internal\" DevicePath \"\""
Sep 11 04:07:48.131634 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:07:48.131586    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jfkbc\" (UniqueName: \"kubernetes.io/projected/71f813c3-4021-4dbc-a261-c86af71f8478-kube-api-access-jfkbc\") on node \"ip-10-0-138-121.ec2.internal\" DevicePath \"\""
Sep 11 04:07:48.131634 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:07:48.131600    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xwp98\" (UniqueName: \"kubernetes.io/projected/6e580a41-dba0-4a29-8476-54ca64cd8af4-kube-api-access-xwp98\") on node \"ip-10-0-138-121.ec2.internal\" DevicePath \"\""
Sep 11 04:07:48.131634 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:07:48.131615    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hltlr\" (UniqueName: \"kubernetes.io/projected/4aca6dec-dc7a-4afc-b894-5393423f6313-kube-api-access-hltlr\") on node \"ip-10-0-138-121.ec2.internal\" DevicePath \"\""
Sep 11 04:07:48.260098 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:07:48.260071    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-4cb558-6d89c46799-s6l4v"]
Sep 11 04:07:48.270144 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:07:48.270118    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-4cb558-6d89c46799-s6l4v"]
Sep 11 04:07:48.282209 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:07:48.282180    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-4cb558-84665c6b56-wqdpk"]
Sep 11 04:07:48.284107 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:07:48.284083    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-4cb558-84665c6b56-wqdpk"]
Sep 11 04:07:48.292033 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:07:48.292007    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-4cb558-6fcbd7474b-crdl2"]
Sep 11 04:07:48.294959 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:07:48.294939    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-4cb558-6fcbd7474b-crdl2"]
Sep 11 04:07:48.465753 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:07:48.465719    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4aca6dec-dc7a-4afc-b894-5393423f6313" path="/var/lib/kubelet/pods/4aca6dec-dc7a-4afc-b894-5393423f6313/volumes"
Sep 11 04:07:48.466097 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:07:48.466084    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6e580a41-dba0-4a29-8476-54ca64cd8af4" path="/var/lib/kubelet/pods/6e580a41-dba0-4a29-8476-54ca64cd8af4/volumes"
Sep 11 04:07:48.466425 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:07:48.466414    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="71f813c3-4021-4dbc-a261-c86af71f8478" path="/var/lib/kubelet/pods/71f813c3-4021-4dbc-a261-c86af71f8478/volumes"
Sep 11 04:09:04.932123 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:09:04.932089    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-b4d3cf-29818320-n2k46"]
Sep 11 04:09:04.934523 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:09:04.934498    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-b4d3cf-29818320-n2k46"]
Sep 11 04:09:04.958736 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:09:04.958704    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-b4d3cf-5bf969dcf-ggj5v"]
Sep 11 04:09:04.958960 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:09:04.958909    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-b4d3cf-5bf969dcf-ggj5v" podUID="390e944a-c647-4015-9aa9-10e3a374795b" containerName="payload-processing" containerID="cri-o://59ab76ed2e5c096f2a89a41b6bbaea125b3cf03c03673f9f2ff2273df413b6e3" gracePeriod=30
Sep 11 04:09:04.966413 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:09:04.966381    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-b4d3cf-559484c998-cq9hg"]
Sep 11 04:09:04.966851 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:09:04.966627    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-b4d3cf-559484c998-cq9hg" podUID="2870913c-68b6-4753-9352-ee0547316428" containerName="payload-pre-processing" containerID="cri-o://de571050b673cffa0248dea75d6c1dfcb7a7e3cf042ac4769d5654ddb7ba571b" gracePeriod=30
Sep 11 04:09:05.226675 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:09:05.226639    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-b4d3cf-5bf969dcf-ggj5v"
Sep 11 04:09:05.243415 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:09:05.243390    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/390e944a-c647-4015-9aa9-10e3a374795b-plugins-config-volume\") pod \"390e944a-c647-4015-9aa9-10e3a374795b\" (UID: \"390e944a-c647-4015-9aa9-10e3a374795b\") "
Sep 11 04:09:05.243557 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:09:05.243433    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h7brk\" (UniqueName: \"kubernetes.io/projected/390e944a-c647-4015-9aa9-10e3a374795b-kube-api-access-h7brk\") pod \"390e944a-c647-4015-9aa9-10e3a374795b\" (UID: \"390e944a-c647-4015-9aa9-10e3a374795b\") "
Sep 11 04:09:05.243737 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:09:05.243714    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/390e944a-c647-4015-9aa9-10e3a374795b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "390e944a-c647-4015-9aa9-10e3a374795b" (UID: "390e944a-c647-4015-9aa9-10e3a374795b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 04:09:05.243809 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:09:05.243793    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/390e944a-c647-4015-9aa9-10e3a374795b-plugins-config-volume\") on node \"ip-10-0-138-121.ec2.internal\" DevicePath \"\""
Sep 11 04:09:05.246250 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:09:05.246218    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/390e944a-c647-4015-9aa9-10e3a374795b-kube-api-access-h7brk" (OuterVolumeSpecName: "kube-api-access-h7brk") pod "390e944a-c647-4015-9aa9-10e3a374795b" (UID: "390e944a-c647-4015-9aa9-10e3a374795b"). InnerVolumeSpecName "kube-api-access-h7brk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 04:09:05.253120 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:09:05.253095    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-b4d3cf-559484c998-cq9hg"
Sep 11 04:09:05.333389 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:09:05.333350    2580 generic.go:356] "Generic (PLEG): container finished" podID="2870913c-68b6-4753-9352-ee0547316428" containerID="de571050b673cffa0248dea75d6c1dfcb7a7e3cf042ac4769d5654ddb7ba571b" exitCode=0
Sep 11 04:09:05.333580 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:09:05.333419    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-b4d3cf-559484c998-cq9hg"
Sep 11 04:09:05.333580 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:09:05.333427    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-b4d3cf-559484c998-cq9hg" event={"ID":"2870913c-68b6-4753-9352-ee0547316428","Type":"ContainerDied","Data":"de571050b673cffa0248dea75d6c1dfcb7a7e3cf042ac4769d5654ddb7ba571b"}
Sep 11 04:09:05.333580 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:09:05.333514    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-b4d3cf-559484c998-cq9hg" event={"ID":"2870913c-68b6-4753-9352-ee0547316428","Type":"ContainerDied","Data":"125d96c8ced7ab6122d168f34be14d79600df5be4a55143501d8083334b2146c"}
Sep 11 04:09:05.333580 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:09:05.333547    2580 scope.go:117] "RemoveContainer" containerID="de571050b673cffa0248dea75d6c1dfcb7a7e3cf042ac4769d5654ddb7ba571b"
Sep 11 04:09:05.334898 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:09:05.334871    2580 generic.go:356] "Generic (PLEG): container finished" podID="390e944a-c647-4015-9aa9-10e3a374795b" containerID="59ab76ed2e5c096f2a89a41b6bbaea125b3cf03c03673f9f2ff2273df413b6e3" exitCode=0
Sep 11 04:09:05.335017 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:09:05.334921    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-b4d3cf-5bf969dcf-ggj5v"
Sep 11 04:09:05.335017 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:09:05.334920    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-b4d3cf-5bf969dcf-ggj5v" event={"ID":"390e944a-c647-4015-9aa9-10e3a374795b","Type":"ContainerDied","Data":"59ab76ed2e5c096f2a89a41b6bbaea125b3cf03c03673f9f2ff2273df413b6e3"}
Sep 11 04:09:05.335131 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:09:05.335036    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-b4d3cf-5bf969dcf-ggj5v" event={"ID":"390e944a-c647-4015-9aa9-10e3a374795b","Type":"ContainerDied","Data":"b7c842c8a4dae77b34fadba684bfe83d1a8a028d76deb65576078a7ed247f45a"}
Sep 11 04:09:05.342694 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:09:05.342606    2580 scope.go:117] "RemoveContainer" containerID="de571050b673cffa0248dea75d6c1dfcb7a7e3cf042ac4769d5654ddb7ba571b"
Sep 11 04:09:05.343034 ip-10-0-138-121 kubenswrapper[2580]: E0911 04:09:05.343010    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de571050b673cffa0248dea75d6c1dfcb7a7e3cf042ac4769d5654ddb7ba571b\": container with ID starting with de571050b673cffa0248dea75d6c1dfcb7a7e3cf042ac4769d5654ddb7ba571b not found: ID does not exist" containerID="de571050b673cffa0248dea75d6c1dfcb7a7e3cf042ac4769d5654ddb7ba571b"
Sep 11 04:09:05.343104 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:09:05.343043    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de571050b673cffa0248dea75d6c1dfcb7a7e3cf042ac4769d5654ddb7ba571b"} err="failed to get container status \"de571050b673cffa0248dea75d6c1dfcb7a7e3cf042ac4769d5654ddb7ba571b\": rpc error: code = NotFound desc = could not find container \"de571050b673cffa0248dea75d6c1dfcb7a7e3cf042ac4769d5654ddb7ba571b\": container with ID starting with de571050b673cffa0248dea75d6c1dfcb7a7e3cf042ac4769d5654ddb7ba571b not found: ID does not exist"
Sep 11 04:09:05.343104 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:09:05.343067    2580 scope.go:117] "RemoveContainer" containerID="59ab76ed2e5c096f2a89a41b6bbaea125b3cf03c03673f9f2ff2273df413b6e3"
Sep 11 04:09:05.344110 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:09:05.344083    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2870913c-68b6-4753-9352-ee0547316428-plugins-config-volume\") pod \"2870913c-68b6-4753-9352-ee0547316428\" (UID: \"2870913c-68b6-4753-9352-ee0547316428\") "
Sep 11 04:09:05.344180 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:09:05.344170    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2ck85\" (UniqueName: \"kubernetes.io/projected/2870913c-68b6-4753-9352-ee0547316428-kube-api-access-2ck85\") pod \"2870913c-68b6-4753-9352-ee0547316428\" (UID: \"2870913c-68b6-4753-9352-ee0547316428\") "
Sep 11 04:09:05.344476 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:09:05.344459    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h7brk\" (UniqueName: \"kubernetes.io/projected/390e944a-c647-4015-9aa9-10e3a374795b-kube-api-access-h7brk\") on node \"ip-10-0-138-121.ec2.internal\" DevicePath \"\""
Sep 11 04:09:05.344544 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:09:05.344476    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2870913c-68b6-4753-9352-ee0547316428-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2870913c-68b6-4753-9352-ee0547316428" (UID: "2870913c-68b6-4753-9352-ee0547316428"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 04:09:05.346198 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:09:05.346177    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2870913c-68b6-4753-9352-ee0547316428-kube-api-access-2ck85" (OuterVolumeSpecName: "kube-api-access-2ck85") pod "2870913c-68b6-4753-9352-ee0547316428" (UID: "2870913c-68b6-4753-9352-ee0547316428"). InnerVolumeSpecName "kube-api-access-2ck85". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 04:09:05.351259 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:09:05.351240    2580 scope.go:117] "RemoveContainer" containerID="59ab76ed2e5c096f2a89a41b6bbaea125b3cf03c03673f9f2ff2273df413b6e3"
Sep 11 04:09:05.351603 ip-10-0-138-121 kubenswrapper[2580]: E0911 04:09:05.351573    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59ab76ed2e5c096f2a89a41b6bbaea125b3cf03c03673f9f2ff2273df413b6e3\": container with ID starting with 59ab76ed2e5c096f2a89a41b6bbaea125b3cf03c03673f9f2ff2273df413b6e3 not found: ID does not exist" containerID="59ab76ed2e5c096f2a89a41b6bbaea125b3cf03c03673f9f2ff2273df413b6e3"
Sep 11 04:09:05.351710 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:09:05.351609    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59ab76ed2e5c096f2a89a41b6bbaea125b3cf03c03673f9f2ff2273df413b6e3"} err="failed to get container status \"59ab76ed2e5c096f2a89a41b6bbaea125b3cf03c03673f9f2ff2273df413b6e3\": rpc error: code = NotFound desc = could not find container \"59ab76ed2e5c096f2a89a41b6bbaea125b3cf03c03673f9f2ff2273df413b6e3\": container with ID starting with 59ab76ed2e5c096f2a89a41b6bbaea125b3cf03c03673f9f2ff2273df413b6e3 not found: ID does not exist"
Sep 11 04:09:05.358747 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:09:05.358723    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-b4d3cf-5bf969dcf-ggj5v"]
Sep 11 04:09:05.362087 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:09:05.362063    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-b4d3cf-5bf969dcf-ggj5v"]
Sep 11 04:09:05.444968 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:09:05.444938    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2ck85\" (UniqueName: \"kubernetes.io/projected/2870913c-68b6-4753-9352-ee0547316428-kube-api-access-2ck85\") on node \"ip-10-0-138-121.ec2.internal\" DevicePath \"\""
Sep 11 04:09:05.444968 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:09:05.444962    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2870913c-68b6-4753-9352-ee0547316428-plugins-config-volume\") on node \"ip-10-0-138-121.ec2.internal\" DevicePath \"\""
Sep 11 04:09:05.657771 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:09:05.657733    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-b4d3cf-559484c998-cq9hg"]
Sep 11 04:09:05.662032 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:09:05.662003    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-b4d3cf-559484c998-cq9hg"]
Sep 11 04:09:06.465366 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:09:06.465334    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2870913c-68b6-4753-9352-ee0547316428" path="/var/lib/kubelet/pods/2870913c-68b6-4753-9352-ee0547316428/volumes"
Sep 11 04:09:06.465769 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:09:06.465728    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="390e944a-c647-4015-9aa9-10e3a374795b" path="/var/lib/kubelet/pods/390e944a-c647-4015-9aa9-10e3a374795b/volumes"
Sep 11 04:09:06.465985 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:09:06.465973    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fb7e4860-ed01-4069-b9b4-070549d3a531" path="/var/lib/kubelet/pods/fb7e4860-ed01-4069-b9b4-070549d3a531/volumes"
Sep 11 04:09:19.135526 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:09:19.135495    2580 scope.go:117] "RemoveContainer" containerID="8590ed7d7578ddc517feceb2a3eec0e4138d47d63a9e22650a2b9bdd2387d892"
Sep 11 04:09:22.963392 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:09:22.963353    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-dd5b30-cbb7b64f-dx4z8"]
Sep 11 04:09:22.964009 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:09:22.963674    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-dd5b30-cbb7b64f-dx4z8" podUID="cc7227b2-19bf-4d54-b0b3-ce4de2c8a35b" containerName="maas-api" containerID="cri-o://2d1154dcbc9db4f5369bda9471dffcc96340426af7b7bc5365a77aed11ad9bbb" gracePeriod=30
Sep 11 04:09:23.031675 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:09:23.031627    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-dd5b30-59cd644494-hqhn2"]
Sep 11 04:09:23.031924 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:09:23.031894    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-dd5b30-59cd644494-hqhn2" podUID="0c012db4-89ca-40c3-a7e6-42681955e252" containerName="payload-pre-processing" containerID="cri-o://1738ccd7e17c9b55cdb29910ee41bebe0165d1b86a12e34c469e4d3985fb262a" gracePeriod=30
Sep 11 04:09:23.277541 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:09:23.277514    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-dd5b30-cbb7b64f-dx4z8"
Sep 11 04:09:23.306934 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:09:23.306901    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-dd5b30-59cd644494-hqhn2"
Sep 11 04:09:23.402675 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:09:23.402638    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc7227b2-19bf-4d54-b0b3-ce4de2c8a35b-openshift-service-ca\") pod \"cc7227b2-19bf-4d54-b0b3-ce4de2c8a35b\" (UID: \"cc7227b2-19bf-4d54-b0b3-ce4de2c8a35b\") "
Sep 11 04:09:23.402881 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:09:23.402708    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c012db4-89ca-40c3-a7e6-42681955e252-plugins-config-volume\") pod \"0c012db4-89ca-40c3-a7e6-42681955e252\" (UID: \"0c012db4-89ca-40c3-a7e6-42681955e252\") "
Sep 11 04:09:23.402881 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:09:23.402773    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h6pl4\" (UniqueName: \"kubernetes.io/projected/cc7227b2-19bf-4d54-b0b3-ce4de2c8a35b-kube-api-access-h6pl4\") pod \"cc7227b2-19bf-4d54-b0b3-ce4de2c8a35b\" (UID: \"cc7227b2-19bf-4d54-b0b3-ce4de2c8a35b\") "
Sep 11 04:09:23.402881 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:09:23.402832    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc7227b2-19bf-4d54-b0b3-ce4de2c8a35b-metrics-certs\") pod \"cc7227b2-19bf-4d54-b0b3-ce4de2c8a35b\" (UID: \"cc7227b2-19bf-4d54-b0b3-ce4de2c8a35b\") "
Sep 11 04:09:23.402881 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:09:23.402865    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lwglw\" (UniqueName: \"kubernetes.io/projected/0c012db4-89ca-40c3-a7e6-42681955e252-kube-api-access-lwglw\") pod \"0c012db4-89ca-40c3-a7e6-42681955e252\" (UID: \"0c012db4-89ca-40c3-a7e6-42681955e252\") "
Sep 11 04:09:23.403091 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:09:23.402889    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc7227b2-19bf-4d54-b0b3-ce4de2c8a35b-maas-api-tls\") pod \"cc7227b2-19bf-4d54-b0b3-ce4de2c8a35b\" (UID: \"cc7227b2-19bf-4d54-b0b3-ce4de2c8a35b\") "
Sep 11 04:09:23.403091 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:09:23.403039    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc7227b2-19bf-4d54-b0b3-ce4de2c8a35b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "cc7227b2-19bf-4d54-b0b3-ce4de2c8a35b" (UID: "cc7227b2-19bf-4d54-b0b3-ce4de2c8a35b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 04:09:23.403200 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:09:23.403144    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c012db4-89ca-40c3-a7e6-42681955e252-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0c012db4-89ca-40c3-a7e6-42681955e252" (UID: "0c012db4-89ca-40c3-a7e6-42681955e252"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 04:09:23.403255 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:09:23.403238    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc7227b2-19bf-4d54-b0b3-ce4de2c8a35b-openshift-service-ca\") on node \"ip-10-0-138-121.ec2.internal\" DevicePath \"\""
Sep 11 04:09:23.403309 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:09:23.403254    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c012db4-89ca-40c3-a7e6-42681955e252-plugins-config-volume\") on node \"ip-10-0-138-121.ec2.internal\" DevicePath \"\""
Sep 11 04:09:23.405422 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:09:23.405392    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc7227b2-19bf-4d54-b0b3-ce4de2c8a35b-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "cc7227b2-19bf-4d54-b0b3-ce4de2c8a35b" (UID: "cc7227b2-19bf-4d54-b0b3-ce4de2c8a35b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 04:09:23.405552 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:09:23.405436    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c012db4-89ca-40c3-a7e6-42681955e252-kube-api-access-lwglw" (OuterVolumeSpecName: "kube-api-access-lwglw") pod "0c012db4-89ca-40c3-a7e6-42681955e252" (UID: "0c012db4-89ca-40c3-a7e6-42681955e252"). InnerVolumeSpecName "kube-api-access-lwglw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 04:09:23.405552 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:09:23.405504    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc7227b2-19bf-4d54-b0b3-ce4de2c8a35b-kube-api-access-h6pl4" (OuterVolumeSpecName: "kube-api-access-h6pl4") pod "cc7227b2-19bf-4d54-b0b3-ce4de2c8a35b" (UID: "cc7227b2-19bf-4d54-b0b3-ce4de2c8a35b"). InnerVolumeSpecName "kube-api-access-h6pl4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 04:09:23.405552 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:09:23.405504    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc7227b2-19bf-4d54-b0b3-ce4de2c8a35b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "cc7227b2-19bf-4d54-b0b3-ce4de2c8a35b" (UID: "cc7227b2-19bf-4d54-b0b3-ce4de2c8a35b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 04:09:23.425436 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:09:23.425413    2580 generic.go:356] "Generic (PLEG): container finished" podID="cc7227b2-19bf-4d54-b0b3-ce4de2c8a35b" containerID="2d1154dcbc9db4f5369bda9471dffcc96340426af7b7bc5365a77aed11ad9bbb" exitCode=0
Sep 11 04:09:23.425554 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:09:23.425465    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-dd5b30-cbb7b64f-dx4z8"
Sep 11 04:09:23.425554 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:09:23.425494    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-dd5b30-cbb7b64f-dx4z8" event={"ID":"cc7227b2-19bf-4d54-b0b3-ce4de2c8a35b","Type":"ContainerDied","Data":"2d1154dcbc9db4f5369bda9471dffcc96340426af7b7bc5365a77aed11ad9bbb"}
Sep 11 04:09:23.425657 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:09:23.425557    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-dd5b30-cbb7b64f-dx4z8" event={"ID":"cc7227b2-19bf-4d54-b0b3-ce4de2c8a35b","Type":"ContainerDied","Data":"a480973ec5afe95a3a348a7e65dd5988cb9fc06fa6b2f4aa6b2b4e6c1bca9994"}
Sep 11 04:09:23.425657 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:09:23.425592    2580 scope.go:117] "RemoveContainer" containerID="2d1154dcbc9db4f5369bda9471dffcc96340426af7b7bc5365a77aed11ad9bbb"
Sep 11 04:09:23.427347 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:09:23.427320    2580 generic.go:356] "Generic (PLEG): container finished" podID="0c012db4-89ca-40c3-a7e6-42681955e252" containerID="1738ccd7e17c9b55cdb29910ee41bebe0165d1b86a12e34c469e4d3985fb262a" exitCode=0
Sep 11 04:09:23.427449 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:09:23.427358    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-dd5b30-59cd644494-hqhn2" event={"ID":"0c012db4-89ca-40c3-a7e6-42681955e252","Type":"ContainerDied","Data":"1738ccd7e17c9b55cdb29910ee41bebe0165d1b86a12e34c469e4d3985fb262a"}
Sep 11 04:09:23.427449 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:09:23.427400    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-dd5b30-59cd644494-hqhn2" event={"ID":"0c012db4-89ca-40c3-a7e6-42681955e252","Type":"ContainerDied","Data":"f188ba559075c0f640953a374e6c514e21645068e32b61b6db4210e01c2cf6f4"}
Sep 11 04:09:23.427449 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:09:23.427362    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-dd5b30-59cd644494-hqhn2"
Sep 11 04:09:23.435048 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:09:23.435023    2580 scope.go:117] "RemoveContainer" containerID="2d1154dcbc9db4f5369bda9471dffcc96340426af7b7bc5365a77aed11ad9bbb"
Sep 11 04:09:23.435392 ip-10-0-138-121 kubenswrapper[2580]: E0911 04:09:23.435370    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d1154dcbc9db4f5369bda9471dffcc96340426af7b7bc5365a77aed11ad9bbb\": container with ID starting with 2d1154dcbc9db4f5369bda9471dffcc96340426af7b7bc5365a77aed11ad9bbb not found: ID does not exist" containerID="2d1154dcbc9db4f5369bda9471dffcc96340426af7b7bc5365a77aed11ad9bbb"
Sep 11 04:09:23.435485 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:09:23.435402    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d1154dcbc9db4f5369bda9471dffcc96340426af7b7bc5365a77aed11ad9bbb"} err="failed to get container status \"2d1154dcbc9db4f5369bda9471dffcc96340426af7b7bc5365a77aed11ad9bbb\": rpc error: code = NotFound desc = could not find container \"2d1154dcbc9db4f5369bda9471dffcc96340426af7b7bc5365a77aed11ad9bbb\": container with ID starting with 2d1154dcbc9db4f5369bda9471dffcc96340426af7b7bc5365a77aed11ad9bbb not found: ID does not exist"
Sep 11 04:09:23.435485 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:09:23.435425    2580 scope.go:117] "RemoveContainer" containerID="1738ccd7e17c9b55cdb29910ee41bebe0165d1b86a12e34c469e4d3985fb262a"
Sep 11 04:09:23.443625 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:09:23.443610    2580 scope.go:117] "RemoveContainer" containerID="1738ccd7e17c9b55cdb29910ee41bebe0165d1b86a12e34c469e4d3985fb262a"
Sep 11 04:09:23.443903 ip-10-0-138-121 kubenswrapper[2580]: E0911 04:09:23.443885    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1738ccd7e17c9b55cdb29910ee41bebe0165d1b86a12e34c469e4d3985fb262a\": container with ID starting with 1738ccd7e17c9b55cdb29910ee41bebe0165d1b86a12e34c469e4d3985fb262a not found: ID does not exist" containerID="1738ccd7e17c9b55cdb29910ee41bebe0165d1b86a12e34c469e4d3985fb262a"
Sep 11 04:09:23.443951 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:09:23.443910    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1738ccd7e17c9b55cdb29910ee41bebe0165d1b86a12e34c469e4d3985fb262a"} err="failed to get container status \"1738ccd7e17c9b55cdb29910ee41bebe0165d1b86a12e34c469e4d3985fb262a\": rpc error: code = NotFound desc = could not find container \"1738ccd7e17c9b55cdb29910ee41bebe0165d1b86a12e34c469e4d3985fb262a\": container with ID starting with 1738ccd7e17c9b55cdb29910ee41bebe0165d1b86a12e34c469e4d3985fb262a not found: ID does not exist"
Sep 11 04:09:23.448573 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:09:23.448552    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-dd5b30-cbb7b64f-dx4z8"]
Sep 11 04:09:23.452710 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:09:23.452692    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-dd5b30-cbb7b64f-dx4z8"]
Sep 11 04:09:23.464534 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:09:23.464508    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-dd5b30-59cd644494-hqhn2"]
Sep 11 04:09:23.467159 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:09:23.467137    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-dd5b30-59cd644494-hqhn2"]
Sep 11 04:09:23.503815 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:09:23.503750    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h6pl4\" (UniqueName: \"kubernetes.io/projected/cc7227b2-19bf-4d54-b0b3-ce4de2c8a35b-kube-api-access-h6pl4\") on node \"ip-10-0-138-121.ec2.internal\" DevicePath \"\""
Sep 11 04:09:23.503815 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:09:23.503773    2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc7227b2-19bf-4d54-b0b3-ce4de2c8a35b-metrics-certs\") on node \"ip-10-0-138-121.ec2.internal\" DevicePath \"\""
Sep 11 04:09:23.503815 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:09:23.503783    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lwglw\" (UniqueName: \"kubernetes.io/projected/0c012db4-89ca-40c3-a7e6-42681955e252-kube-api-access-lwglw\") on node \"ip-10-0-138-121.ec2.internal\" DevicePath \"\""
Sep 11 04:09:23.503815 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:09:23.503792    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc7227b2-19bf-4d54-b0b3-ce4de2c8a35b-maas-api-tls\") on node \"ip-10-0-138-121.ec2.internal\" DevicePath \"\""
Sep 11 04:09:24.464996 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:09:24.464963    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0c012db4-89ca-40c3-a7e6-42681955e252" path="/var/lib/kubelet/pods/0c012db4-89ca-40c3-a7e6-42681955e252/volumes"
Sep 11 04:09:24.465376 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:09:24.465359    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cc7227b2-19bf-4d54-b0b3-ce4de2c8a35b" path="/var/lib/kubelet/pods/cc7227b2-19bf-4d54-b0b3-ce4de2c8a35b/volumes"
Sep 11 04:10:12.466985 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:10:12.466953    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j4pgw_28b30dae-a2d3-4251-99e8-04bf5153015b/ovn-acl-logging/0.log"
Sep 11 04:10:12.470957 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:10:12.470936    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j4pgw_28b30dae-a2d3-4251-99e8-04bf5153015b/ovn-acl-logging/0.log"
Sep 11 04:12:23.859813 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:12:23.859778    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-c6p8p"]
Sep 11 04:12:23.862299 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:12:23.860017    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-c6p8p" podUID="1066c072-19b7-4e56-92dd-c9e46a945e68" containerName="manager" containerID="cri-o://77c694e42788fed83af8bfd0466d8aee750dfdbc46621adbf2e380cd7f66c67e" gracePeriod=10
Sep 11 04:12:25.206017 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:12:25.205991    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-c6p8p"
Sep 11 04:12:25.242860 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:12:25.242831    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zglv5\" (UniqueName: \"kubernetes.io/projected/1066c072-19b7-4e56-92dd-c9e46a945e68-kube-api-access-zglv5\") pod \"1066c072-19b7-4e56-92dd-c9e46a945e68\" (UID: \"1066c072-19b7-4e56-92dd-c9e46a945e68\") "
Sep 11 04:12:25.243002 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:12:25.242983    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1066c072-19b7-4e56-92dd-c9e46a945e68-extensions-socket-volume\") pod \"1066c072-19b7-4e56-92dd-c9e46a945e68\" (UID: \"1066c072-19b7-4e56-92dd-c9e46a945e68\") "
Sep 11 04:12:25.243401 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:12:25.243374    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1066c072-19b7-4e56-92dd-c9e46a945e68-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "1066c072-19b7-4e56-92dd-c9e46a945e68" (UID: "1066c072-19b7-4e56-92dd-c9e46a945e68"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 04:12:25.245396 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:12:25.245375    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1066c072-19b7-4e56-92dd-c9e46a945e68-kube-api-access-zglv5" (OuterVolumeSpecName: "kube-api-access-zglv5") pod "1066c072-19b7-4e56-92dd-c9e46a945e68" (UID: "1066c072-19b7-4e56-92dd-c9e46a945e68"). InnerVolumeSpecName "kube-api-access-zglv5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 04:12:25.267618 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:12:25.267585    2580 generic.go:356] "Generic (PLEG): container finished" podID="1066c072-19b7-4e56-92dd-c9e46a945e68" containerID="77c694e42788fed83af8bfd0466d8aee750dfdbc46621adbf2e380cd7f66c67e" exitCode=0
Sep 11 04:12:25.267785 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:12:25.267626    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-c6p8p"
Sep 11 04:12:25.267785 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:12:25.267677    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-c6p8p" event={"ID":"1066c072-19b7-4e56-92dd-c9e46a945e68","Type":"ContainerDied","Data":"77c694e42788fed83af8bfd0466d8aee750dfdbc46621adbf2e380cd7f66c67e"}
Sep 11 04:12:25.267785 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:12:25.267728    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-c6p8p" event={"ID":"1066c072-19b7-4e56-92dd-c9e46a945e68","Type":"ContainerDied","Data":"f70a84b6d7b3643eda5e753d8f46878e8daaffb5b2a9676f09f4b63f97aa7ebb"}
Sep 11 04:12:25.267785 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:12:25.267750    2580 scope.go:117] "RemoveContainer" containerID="77c694e42788fed83af8bfd0466d8aee750dfdbc46621adbf2e380cd7f66c67e"
Sep 11 04:12:25.278464 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:12:25.278440    2580 scope.go:117] "RemoveContainer" containerID="77c694e42788fed83af8bfd0466d8aee750dfdbc46621adbf2e380cd7f66c67e"
Sep 11 04:12:25.278838 ip-10-0-138-121 kubenswrapper[2580]: E0911 04:12:25.278808    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77c694e42788fed83af8bfd0466d8aee750dfdbc46621adbf2e380cd7f66c67e\": container with ID starting with 77c694e42788fed83af8bfd0466d8aee750dfdbc46621adbf2e380cd7f66c67e not found: ID does not exist" containerID="77c694e42788fed83af8bfd0466d8aee750dfdbc46621adbf2e380cd7f66c67e"
Sep 11 04:12:25.278936 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:12:25.278843    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77c694e42788fed83af8bfd0466d8aee750dfdbc46621adbf2e380cd7f66c67e"} err="failed to get container status \"77c694e42788fed83af8bfd0466d8aee750dfdbc46621adbf2e380cd7f66c67e\": rpc error: code = NotFound desc = could not find container \"77c694e42788fed83af8bfd0466d8aee750dfdbc46621adbf2e380cd7f66c67e\": container with ID starting with 77c694e42788fed83af8bfd0466d8aee750dfdbc46621adbf2e380cd7f66c67e not found: ID does not exist"
Sep 11 04:12:25.296971 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:12:25.296948    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-c6p8p"]
Sep 11 04:12:25.302907 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:12:25.302882    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-c6p8p"]
Sep 11 04:12:25.344461 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:12:25.344383    2580 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1066c072-19b7-4e56-92dd-c9e46a945e68-extensions-socket-volume\") on node \"ip-10-0-138-121.ec2.internal\" DevicePath \"\""
Sep 11 04:12:25.344461 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:12:25.344412    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zglv5\" (UniqueName: \"kubernetes.io/projected/1066c072-19b7-4e56-92dd-c9e46a945e68-kube-api-access-zglv5\") on node \"ip-10-0-138-121.ec2.internal\" DevicePath \"\""
Sep 11 04:12:26.463309 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:12:26.463266    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1066c072-19b7-4e56-92dd-c9e46a945e68" path="/var/lib/kubelet/pods/1066c072-19b7-4e56-92dd-c9e46a945e68/volumes"
Sep 11 04:13:24.580112 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:13:24.580074    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-46xjp"]
Sep 11 04:13:24.580511 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:13:24.580407    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e580a41-dba0-4a29-8476-54ca64cd8af4" containerName="payload-processing"
Sep 11 04:13:24.580511 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:13:24.580418    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e580a41-dba0-4a29-8476-54ca64cd8af4" containerName="payload-processing"
Sep 11 04:13:24.580511 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:13:24.580427    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71f813c3-4021-4dbc-a261-c86af71f8478" containerName="maas-api"
Sep 11 04:13:24.580511 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:13:24.580432    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="71f813c3-4021-4dbc-a261-c86af71f8478" containerName="maas-api"
Sep 11 04:13:24.580511 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:13:24.580443    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc7227b2-19bf-4d54-b0b3-ce4de2c8a35b" containerName="maas-api"
Sep 11 04:13:24.580511 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:13:24.580449    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc7227b2-19bf-4d54-b0b3-ce4de2c8a35b" containerName="maas-api"
Sep 11 04:13:24.580511 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:13:24.580457    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1066c072-19b7-4e56-92dd-c9e46a945e68" containerName="manager"
Sep 11 04:13:24.580511 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:13:24.580463    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="1066c072-19b7-4e56-92dd-c9e46a945e68" containerName="manager"
Sep 11 04:13:24.580511 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:13:24.580469    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0c012db4-89ca-40c3-a7e6-42681955e252" containerName="payload-pre-processing"
Sep 11 04:13:24.580511 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:13:24.580474    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="0c012db4-89ca-40c3-a7e6-42681955e252" containerName="payload-pre-processing"
Sep 11 04:13:24.580511 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:13:24.580482    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4aca6dec-dc7a-4afc-b894-5393423f6313" containerName="payload-pre-processing"
Sep 11 04:13:24.580511 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:13:24.580486    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="4aca6dec-dc7a-4afc-b894-5393423f6313" containerName="payload-pre-processing"
Sep 11 04:13:24.580511 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:13:24.580493    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea9c266c-bd2e-450d-9eb2-7c5d145e9ce3" containerName="istio-proxy"
Sep 11 04:13:24.580511 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:13:24.580497    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea9c266c-bd2e-450d-9eb2-7c5d145e9ce3" containerName="istio-proxy"
Sep 11 04:13:24.580511 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:13:24.580508    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3e5ae4b4-303a-4500-9aef-dd5bd4ef39fb" containerName="payload-pre-processing"
Sep 11 04:13:24.580511 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:13:24.580514    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="3e5ae4b4-303a-4500-9aef-dd5bd4ef39fb" containerName="payload-pre-processing"
Sep 11 04:13:24.581042 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:13:24.580522    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="390e944a-c647-4015-9aa9-10e3a374795b" containerName="payload-processing"
Sep 11 04:13:24.581042 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:13:24.580527    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="390e944a-c647-4015-9aa9-10e3a374795b" containerName="payload-processing"
Sep 11 04:13:24.581042 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:13:24.580533    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea9c266c-bd2e-450d-9eb2-7c5d145e9ce3" containerName="istio-proxy"
Sep 11 04:13:24.581042 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:13:24.580538    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea9c266c-bd2e-450d-9eb2-7c5d145e9ce3" containerName="istio-proxy"
Sep 11 04:13:24.581042 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:13:24.580550    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2870913c-68b6-4753-9352-ee0547316428" containerName="payload-pre-processing"
Sep 11 04:13:24.581042 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:13:24.580555    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="2870913c-68b6-4753-9352-ee0547316428" containerName="payload-pre-processing"
Sep 11 04:13:24.581042 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:13:24.580606    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="71f813c3-4021-4dbc-a261-c86af71f8478" containerName="maas-api"
Sep 11 04:13:24.581042 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:13:24.580612    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="1066c072-19b7-4e56-92dd-c9e46a945e68" containerName="manager"
Sep 11 04:13:24.581042 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:13:24.580619    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="390e944a-c647-4015-9aa9-10e3a374795b" containerName="payload-processing"
Sep 11 04:13:24.581042 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:13:24.580626    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="cc7227b2-19bf-4d54-b0b3-ce4de2c8a35b" containerName="maas-api"
Sep 11 04:13:24.581042 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:13:24.580632    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e580a41-dba0-4a29-8476-54ca64cd8af4" containerName="payload-processing"
Sep 11 04:13:24.581042 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:13:24.580637    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="ea9c266c-bd2e-450d-9eb2-7c5d145e9ce3" containerName="istio-proxy"
Sep 11 04:13:24.581042 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:13:24.580643    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="0c012db4-89ca-40c3-a7e6-42681955e252" containerName="payload-pre-processing"
Sep 11 04:13:24.581042 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:13:24.580650    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="3e5ae4b4-303a-4500-9aef-dd5bd4ef39fb" containerName="payload-pre-processing"
Sep 11 04:13:24.581042 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:13:24.580656    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="2870913c-68b6-4753-9352-ee0547316428" containerName="payload-pre-processing"
Sep 11 04:13:24.581042 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:13:24.580674    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="4aca6dec-dc7a-4afc-b894-5393423f6313" containerName="payload-pre-processing"
Sep 11 04:13:24.581042 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:13:24.580679    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="ea9c266c-bd2e-450d-9eb2-7c5d145e9ce3" containerName="istio-proxy"
Sep 11 04:13:24.584025 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:13:24.584010    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-46xjp"
Sep 11 04:13:24.587203 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:13:24.587183    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 11 04:13:24.587321 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:13:24.587183    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-f82vk\""
Sep 11 04:13:24.587761 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:13:24.587745    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 11 04:13:24.599394 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:13:24.599367    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-46xjp"]
Sep 11 04:13:24.663846 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:13:24.663800    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/01f64146-896b-419e-a404-587dd2ef58ce-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-46xjp\" (UID: \"01f64146-896b-419e-a404-587dd2ef58ce\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-46xjp"
Sep 11 04:13:24.664021 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:13:24.663867    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtcc2\" (UniqueName: \"kubernetes.io/projected/01f64146-896b-419e-a404-587dd2ef58ce-kube-api-access-wtcc2\") pod \"kuadrant-operator-controller-manager-5769f99d9f-46xjp\" (UID: \"01f64146-896b-419e-a404-587dd2ef58ce\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-46xjp"
Sep 11 04:13:24.765131 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:13:24.765092    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/01f64146-896b-419e-a404-587dd2ef58ce-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-46xjp\" (UID: \"01f64146-896b-419e-a404-587dd2ef58ce\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-46xjp"
Sep 11 04:13:24.765372 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:13:24.765155    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtcc2\" (UniqueName: \"kubernetes.io/projected/01f64146-896b-419e-a404-587dd2ef58ce-kube-api-access-wtcc2\") pod \"kuadrant-operator-controller-manager-5769f99d9f-46xjp\" (UID: \"01f64146-896b-419e-a404-587dd2ef58ce\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-46xjp"
Sep 11 04:13:24.765478 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:13:24.765458    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/01f64146-896b-419e-a404-587dd2ef58ce-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-46xjp\" (UID: \"01f64146-896b-419e-a404-587dd2ef58ce\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-46xjp"
Sep 11 04:13:24.776437 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:13:24.776412    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtcc2\" (UniqueName: \"kubernetes.io/projected/01f64146-896b-419e-a404-587dd2ef58ce-kube-api-access-wtcc2\") pod \"kuadrant-operator-controller-manager-5769f99d9f-46xjp\" (UID: \"01f64146-896b-419e-a404-587dd2ef58ce\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-46xjp"
Sep 11 04:13:24.893854 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:13:24.893766    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-46xjp"
Sep 11 04:13:25.178964 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:13:25.178941    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-46xjp"]
Sep 11 04:13:25.180759 ip-10-0-138-121 kubenswrapper[2580]: W0911 04:13:25.180730    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01f64146_896b_419e_a404_587dd2ef58ce.slice/crio-21a311a80c00344c6c35bb577d0b9e5625b3ab1438e058370804f36a307f093d WatchSource:0}: Error finding container 21a311a80c00344c6c35bb577d0b9e5625b3ab1438e058370804f36a307f093d: Status 404 returned error can't find the container with id 21a311a80c00344c6c35bb577d0b9e5625b3ab1438e058370804f36a307f093d
Sep 11 04:13:25.545738 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:13:25.545628    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-46xjp" event={"ID":"01f64146-896b-419e-a404-587dd2ef58ce","Type":"ContainerStarted","Data":"201cbfdea1836637ae5516115f0c941fad52098e0f80aa375f5a9b924c868b44"}
Sep 11 04:13:25.545738 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:13:25.545708    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-46xjp" event={"ID":"01f64146-896b-419e-a404-587dd2ef58ce","Type":"ContainerStarted","Data":"21a311a80c00344c6c35bb577d0b9e5625b3ab1438e058370804f36a307f093d"}
Sep 11 04:13:25.545738 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:13:25.545731    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-46xjp"
Sep 11 04:13:25.570477 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:13:25.570418    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-46xjp" podStartSLOduration=1.570400198 podStartE2EDuration="1.570400198s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:13:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:13:25.568750118 +0000 UTC m=+1993.727236445" watchObservedRunningTime="2026-09-11 04:13:25.570400198 +0000 UTC m=+1993.728886499"
Sep 11 04:13:36.552227 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:13:36.552197    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-46xjp"
Sep 11 04:14:26.431716 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:14:26.431684    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-fa15b45f-kserve-74df5cf7ff-l8s9p"]
Sep 11 04:14:26.435116 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:14:26.435094    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-fa15b45f-kserve-74df5cf7ff-l8s9p"
Sep 11 04:14:26.437977 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:14:26.437953    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-fa15b45f-kserve-self-signed-certs\""
Sep 11 04:14:26.451278 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:14:26.451255    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-fa15b45f-kserve-74df5cf7ff-l8s9p"]
Sep 11 04:14:26.526843 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:14:26.526811    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d66a5c52-d8a9-4293-b125-4e182c9f3223-tmp-dir\") pod \"e2e-same-id-b-fa15b45f-kserve-74df5cf7ff-l8s9p\" (UID: \"d66a5c52-d8a9-4293-b125-4e182c9f3223\") " pod="llm/e2e-same-id-b-fa15b45f-kserve-74df5cf7ff-l8s9p"
Sep 11 04:14:26.527000 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:14:26.526865    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hfltf\" (UniqueName: \"kubernetes.io/projected/d66a5c52-d8a9-4293-b125-4e182c9f3223-kube-api-access-hfltf\") pod \"e2e-same-id-b-fa15b45f-kserve-74df5cf7ff-l8s9p\" (UID: \"d66a5c52-d8a9-4293-b125-4e182c9f3223\") " pod="llm/e2e-same-id-b-fa15b45f-kserve-74df5cf7ff-l8s9p"
Sep 11 04:14:26.527000 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:14:26.526927    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d66a5c52-d8a9-4293-b125-4e182c9f3223-dshm\") pod \"e2e-same-id-b-fa15b45f-kserve-74df5cf7ff-l8s9p\" (UID: \"d66a5c52-d8a9-4293-b125-4e182c9f3223\") " pod="llm/e2e-same-id-b-fa15b45f-kserve-74df5cf7ff-l8s9p"
Sep 11 04:14:26.527000 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:14:26.526980    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d66a5c52-d8a9-4293-b125-4e182c9f3223-tls-certs\") pod \"e2e-same-id-b-fa15b45f-kserve-74df5cf7ff-l8s9p\" (UID: \"d66a5c52-d8a9-4293-b125-4e182c9f3223\") " pod="llm/e2e-same-id-b-fa15b45f-kserve-74df5cf7ff-l8s9p"
Sep 11 04:14:26.527000 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:14:26.526997    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d66a5c52-d8a9-4293-b125-4e182c9f3223-model-cache\") pod \"e2e-same-id-b-fa15b45f-kserve-74df5cf7ff-l8s9p\" (UID: \"d66a5c52-d8a9-4293-b125-4e182c9f3223\") " pod="llm/e2e-same-id-b-fa15b45f-kserve-74df5cf7ff-l8s9p"
Sep 11 04:14:26.527151 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:14:26.527014    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d66a5c52-d8a9-4293-b125-4e182c9f3223-home\") pod \"e2e-same-id-b-fa15b45f-kserve-74df5cf7ff-l8s9p\" (UID: \"d66a5c52-d8a9-4293-b125-4e182c9f3223\") " pod="llm/e2e-same-id-b-fa15b45f-kserve-74df5cf7ff-l8s9p"
Sep 11 04:14:26.627977 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:14:26.627937    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d66a5c52-d8a9-4293-b125-4e182c9f3223-tls-certs\") pod \"e2e-same-id-b-fa15b45f-kserve-74df5cf7ff-l8s9p\" (UID: \"d66a5c52-d8a9-4293-b125-4e182c9f3223\") " pod="llm/e2e-same-id-b-fa15b45f-kserve-74df5cf7ff-l8s9p"
Sep 11 04:14:26.627977 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:14:26.627977    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d66a5c52-d8a9-4293-b125-4e182c9f3223-model-cache\") pod \"e2e-same-id-b-fa15b45f-kserve-74df5cf7ff-l8s9p\" (UID: \"d66a5c52-d8a9-4293-b125-4e182c9f3223\") " pod="llm/e2e-same-id-b-fa15b45f-kserve-74df5cf7ff-l8s9p"
Sep 11 04:14:26.628232 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:14:26.627997    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d66a5c52-d8a9-4293-b125-4e182c9f3223-home\") pod \"e2e-same-id-b-fa15b45f-kserve-74df5cf7ff-l8s9p\" (UID: \"d66a5c52-d8a9-4293-b125-4e182c9f3223\") " pod="llm/e2e-same-id-b-fa15b45f-kserve-74df5cf7ff-l8s9p"
Sep 11 04:14:26.628232 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:14:26.628017    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d66a5c52-d8a9-4293-b125-4e182c9f3223-tmp-dir\") pod \"e2e-same-id-b-fa15b45f-kserve-74df5cf7ff-l8s9p\" (UID: \"d66a5c52-d8a9-4293-b125-4e182c9f3223\") " pod="llm/e2e-same-id-b-fa15b45f-kserve-74df5cf7ff-l8s9p"
Sep 11 04:14:26.628232 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:14:26.628049    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfltf\" (UniqueName: \"kubernetes.io/projected/d66a5c52-d8a9-4293-b125-4e182c9f3223-kube-api-access-hfltf\") pod \"e2e-same-id-b-fa15b45f-kserve-74df5cf7ff-l8s9p\" (UID: \"d66a5c52-d8a9-4293-b125-4e182c9f3223\") " pod="llm/e2e-same-id-b-fa15b45f-kserve-74df5cf7ff-l8s9p"
Sep 11 04:14:26.628232 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:14:26.628097    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d66a5c52-d8a9-4293-b125-4e182c9f3223-dshm\") pod \"e2e-same-id-b-fa15b45f-kserve-74df5cf7ff-l8s9p\" (UID: \"d66a5c52-d8a9-4293-b125-4e182c9f3223\") " pod="llm/e2e-same-id-b-fa15b45f-kserve-74df5cf7ff-l8s9p"
Sep 11 04:14:26.628445 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:14:26.628371    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d66a5c52-d8a9-4293-b125-4e182c9f3223-model-cache\") pod \"e2e-same-id-b-fa15b45f-kserve-74df5cf7ff-l8s9p\" (UID: \"d66a5c52-d8a9-4293-b125-4e182c9f3223\") " pod="llm/e2e-same-id-b-fa15b45f-kserve-74df5cf7ff-l8s9p"
Sep 11 04:14:26.628508 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:14:26.628433    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d66a5c52-d8a9-4293-b125-4e182c9f3223-home\") pod \"e2e-same-id-b-fa15b45f-kserve-74df5cf7ff-l8s9p\" (UID: \"d66a5c52-d8a9-4293-b125-4e182c9f3223\") " pod="llm/e2e-same-id-b-fa15b45f-kserve-74df5cf7ff-l8s9p"
Sep 11 04:14:26.628508 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:14:26.628464    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d66a5c52-d8a9-4293-b125-4e182c9f3223-tmp-dir\") pod \"e2e-same-id-b-fa15b45f-kserve-74df5cf7ff-l8s9p\" (UID: \"d66a5c52-d8a9-4293-b125-4e182c9f3223\") " pod="llm/e2e-same-id-b-fa15b45f-kserve-74df5cf7ff-l8s9p"
Sep 11 04:14:26.630361 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:14:26.630341    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d66a5c52-d8a9-4293-b125-4e182c9f3223-dshm\") pod \"e2e-same-id-b-fa15b45f-kserve-74df5cf7ff-l8s9p\" (UID: \"d66a5c52-d8a9-4293-b125-4e182c9f3223\") " pod="llm/e2e-same-id-b-fa15b45f-kserve-74df5cf7ff-l8s9p"
Sep 11 04:14:26.630529 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:14:26.630513    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d66a5c52-d8a9-4293-b125-4e182c9f3223-tls-certs\") pod \"e2e-same-id-b-fa15b45f-kserve-74df5cf7ff-l8s9p\" (UID: \"d66a5c52-d8a9-4293-b125-4e182c9f3223\") " pod="llm/e2e-same-id-b-fa15b45f-kserve-74df5cf7ff-l8s9p"
Sep 11 04:14:26.637835 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:14:26.637809    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hfltf\" (UniqueName: \"kubernetes.io/projected/d66a5c52-d8a9-4293-b125-4e182c9f3223-kube-api-access-hfltf\") pod \"e2e-same-id-b-fa15b45f-kserve-74df5cf7ff-l8s9p\" (UID: \"d66a5c52-d8a9-4293-b125-4e182c9f3223\") " pod="llm/e2e-same-id-b-fa15b45f-kserve-74df5cf7ff-l8s9p"
Sep 11 04:14:26.744357 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:14:26.744258    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-fa15b45f-kserve-74df5cf7ff-l8s9p"
Sep 11 04:14:27.035674 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:14:27.035633    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-fa15b45f-kserve-74df5cf7ff-l8s9p"]
Sep 11 04:14:27.037627 ip-10-0-138-121 kubenswrapper[2580]: W0911 04:14:27.037596    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd66a5c52_d8a9_4293_b125_4e182c9f3223.slice/crio-54aaa7d19b870f0a1fe5cf9ade0b34f1b5cf59ed498ab1e23452e6dd37bd6314 WatchSource:0}: Error finding container 54aaa7d19b870f0a1fe5cf9ade0b34f1b5cf59ed498ab1e23452e6dd37bd6314: Status 404 returned error can't find the container with id 54aaa7d19b870f0a1fe5cf9ade0b34f1b5cf59ed498ab1e23452e6dd37bd6314
Sep 11 04:14:27.844792 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:14:27.844750    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-fa15b45f-kserve-74df5cf7ff-l8s9p" event={"ID":"d66a5c52-d8a9-4293-b125-4e182c9f3223","Type":"ContainerStarted","Data":"acf7900cd1e6a3e4bcf55899cade4ae810e71e057baa9cb442878fab18fa2cd4"}
Sep 11 04:14:27.845177 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:14:27.844800    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-fa15b45f-kserve-74df5cf7ff-l8s9p" event={"ID":"d66a5c52-d8a9-4293-b125-4e182c9f3223","Type":"ContainerStarted","Data":"54aaa7d19b870f0a1fe5cf9ade0b34f1b5cf59ed498ab1e23452e6dd37bd6314"}
Sep 11 04:14:27.866041 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:14:27.865902    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-fa15b45f-kserve-74df5cf7ff-l8s9p" podStartSLOduration=1.865881684 podStartE2EDuration="1.865881684s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:14:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:14:27.865393893 +0000 UTC m=+2056.023880215" watchObservedRunningTime="2026-09-11 04:14:27.865881684 +0000 UTC m=+2056.024368007"
Sep 11 04:14:36.744749 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:14:36.744646    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-fa15b45f-kserve-74df5cf7ff-l8s9p"
Sep 11 04:14:36.744749 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:14:36.744714    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-fa15b45f-kserve-74df5cf7ff-l8s9p"
Sep 11 04:14:36.757187 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:14:36.757160    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-fa15b45f-kserve-74df5cf7ff-l8s9p"
Sep 11 04:14:36.898810 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:14:36.898783    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-fa15b45f-kserve-74df5cf7ff-l8s9p"
Sep 11 04:15:00.150818 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:15:00.150784    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29818335-xgjjx"]
Sep 11 04:15:00.154241 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:15:00.154224    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29818335-xgjjx"
Sep 11 04:15:00.157779 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:15:00.157760    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-95ckr\""
Sep 11 04:15:00.163986 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:15:00.163969    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29818335-xgjjx"
Sep 11 04:15:00.170028 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:15:00.170004    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29818335-xgjjx"]
Sep 11 04:15:00.440351 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:15:00.440322    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29818335-xgjjx"]
Sep 11 04:15:01.000701 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:15:01.000649    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29818335-xgjjx" event={"ID":"76a0217e-2ce2-4597-a0b1-fc43825054dc","Type":"ContainerStarted","Data":"d87a8d64f8867985f4ef5b7895db9fa1f68af8450fabf429a4c28ae6ceb00973"}
Sep 11 04:15:01.000920 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:15:01.000715    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29818335-xgjjx" event={"ID":"76a0217e-2ce2-4597-a0b1-fc43825054dc","Type":"ContainerStarted","Data":"ea0264bc23bff86c6305922718234d66411eec9b1be4dceb7b4fdf0b568bb942"}
Sep 11 04:15:01.019983 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:15:01.019931    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29818335-xgjjx" podStartSLOduration=1.019914185 podStartE2EDuration="1.019914185s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:15:01.018392577 +0000 UTC m=+2089.176878911" watchObservedRunningTime="2026-09-11 04:15:01.019914185 +0000 UTC m=+2089.178400537"
Sep 11 04:15:02.006851 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:15:02.006811    2580 generic.go:356] "Generic (PLEG): container finished" podID="76a0217e-2ce2-4597-a0b1-fc43825054dc" containerID="d87a8d64f8867985f4ef5b7895db9fa1f68af8450fabf429a4c28ae6ceb00973" exitCode=0
Sep 11 04:15:02.007242 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:15:02.006892    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29818335-xgjjx" event={"ID":"76a0217e-2ce2-4597-a0b1-fc43825054dc","Type":"ContainerDied","Data":"d87a8d64f8867985f4ef5b7895db9fa1f68af8450fabf429a4c28ae6ceb00973"}
Sep 11 04:15:03.139335 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:15:03.139305    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29818335-xgjjx"
Sep 11 04:15:04.017582 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:15:04.017556    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29818335-xgjjx"
Sep 11 04:15:04.017582 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:15:04.017571    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29818335-xgjjx" event={"ID":"76a0217e-2ce2-4597-a0b1-fc43825054dc","Type":"ContainerDied","Data":"ea0264bc23bff86c6305922718234d66411eec9b1be4dceb7b4fdf0b568bb942"}
Sep 11 04:15:04.017830 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:15:04.017617    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ea0264bc23bff86c6305922718234d66411eec9b1be4dceb7b4fdf0b568bb942"
Sep 11 04:15:12.496243 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:15:12.496210    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j4pgw_28b30dae-a2d3-4251-99e8-04bf5153015b/ovn-acl-logging/0.log"
Sep 11 04:15:12.501689 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:15:12.501654    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j4pgw_28b30dae-a2d3-4251-99e8-04bf5153015b/ovn-acl-logging/0.log"
Sep 11 04:15:13.950241 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:15:13.950209    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-fa15b45f-kserve-74df5cf7ff-l8s9p"]
Sep 11 04:15:28.968980 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:15:28.968916    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-fa15b45f-kserve-74df5cf7ff-l8s9p" podUID="d66a5c52-d8a9-4293-b125-4e182c9f3223" containerName="main" containerID="cri-o://acf7900cd1e6a3e4bcf55899cade4ae810e71e057baa9cb442878fab18fa2cd4" gracePeriod=45
Sep 11 04:15:29.139050 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:15:29.139012    2580 generic.go:356] "Generic (PLEG): container finished" podID="d66a5c52-d8a9-4293-b125-4e182c9f3223" containerID="acf7900cd1e6a3e4bcf55899cade4ae810e71e057baa9cb442878fab18fa2cd4" exitCode=0
Sep 11 04:15:29.139182 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:15:29.139075    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-fa15b45f-kserve-74df5cf7ff-l8s9p" event={"ID":"d66a5c52-d8a9-4293-b125-4e182c9f3223","Type":"ContainerDied","Data":"acf7900cd1e6a3e4bcf55899cade4ae810e71e057baa9cb442878fab18fa2cd4"}
Sep 11 04:15:29.214999 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:15:29.214974    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-fa15b45f-kserve-74df5cf7ff-l8s9p"
Sep 11 04:15:29.303202 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:15:29.303116    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d66a5c52-d8a9-4293-b125-4e182c9f3223-home\") pod \"d66a5c52-d8a9-4293-b125-4e182c9f3223\" (UID: \"d66a5c52-d8a9-4293-b125-4e182c9f3223\") "
Sep 11 04:15:29.303202 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:15:29.303152    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d66a5c52-d8a9-4293-b125-4e182c9f3223-tls-certs\") pod \"d66a5c52-d8a9-4293-b125-4e182c9f3223\" (UID: \"d66a5c52-d8a9-4293-b125-4e182c9f3223\") "
Sep 11 04:15:29.303202 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:15:29.303186    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d66a5c52-d8a9-4293-b125-4e182c9f3223-model-cache\") pod \"d66a5c52-d8a9-4293-b125-4e182c9f3223\" (UID: \"d66a5c52-d8a9-4293-b125-4e182c9f3223\") "
Sep 11 04:15:29.303202 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:15:29.303203    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d66a5c52-d8a9-4293-b125-4e182c9f3223-tmp-dir\") pod \"d66a5c52-d8a9-4293-b125-4e182c9f3223\" (UID: \"d66a5c52-d8a9-4293-b125-4e182c9f3223\") "
Sep 11 04:15:29.303525 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:15:29.303317    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d66a5c52-d8a9-4293-b125-4e182c9f3223-dshm\") pod \"d66a5c52-d8a9-4293-b125-4e182c9f3223\" (UID: \"d66a5c52-d8a9-4293-b125-4e182c9f3223\") "
Sep 11 04:15:29.303525 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:15:29.303389    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hfltf\" (UniqueName: \"kubernetes.io/projected/d66a5c52-d8a9-4293-b125-4e182c9f3223-kube-api-access-hfltf\") pod \"d66a5c52-d8a9-4293-b125-4e182c9f3223\" (UID: \"d66a5c52-d8a9-4293-b125-4e182c9f3223\") "
Sep 11 04:15:29.303525 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:15:29.303458    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d66a5c52-d8a9-4293-b125-4e182c9f3223-home" (OuterVolumeSpecName: "home") pod "d66a5c52-d8a9-4293-b125-4e182c9f3223" (UID: "d66a5c52-d8a9-4293-b125-4e182c9f3223"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 04:15:29.303525 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:15:29.303453    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d66a5c52-d8a9-4293-b125-4e182c9f3223-model-cache" (OuterVolumeSpecName: "model-cache") pod "d66a5c52-d8a9-4293-b125-4e182c9f3223" (UID: "d66a5c52-d8a9-4293-b125-4e182c9f3223"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 04:15:29.303525 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:15:29.303474    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d66a5c52-d8a9-4293-b125-4e182c9f3223-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d66a5c52-d8a9-4293-b125-4e182c9f3223" (UID: "d66a5c52-d8a9-4293-b125-4e182c9f3223"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 04:15:29.303831 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:15:29.303650    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d66a5c52-d8a9-4293-b125-4e182c9f3223-home\") on node \"ip-10-0-138-121.ec2.internal\" DevicePath \"\""
Sep 11 04:15:29.303831 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:15:29.303692    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d66a5c52-d8a9-4293-b125-4e182c9f3223-model-cache\") on node \"ip-10-0-138-121.ec2.internal\" DevicePath \"\""
Sep 11 04:15:29.303831 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:15:29.303707    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d66a5c52-d8a9-4293-b125-4e182c9f3223-tmp-dir\") on node \"ip-10-0-138-121.ec2.internal\" DevicePath \"\""
Sep 11 04:15:29.305270 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:15:29.305241    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d66a5c52-d8a9-4293-b125-4e182c9f3223-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d66a5c52-d8a9-4293-b125-4e182c9f3223" (UID: "d66a5c52-d8a9-4293-b125-4e182c9f3223"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 04:15:29.305443 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:15:29.305418    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d66a5c52-d8a9-4293-b125-4e182c9f3223-kube-api-access-hfltf" (OuterVolumeSpecName: "kube-api-access-hfltf") pod "d66a5c52-d8a9-4293-b125-4e182c9f3223" (UID: "d66a5c52-d8a9-4293-b125-4e182c9f3223"). InnerVolumeSpecName "kube-api-access-hfltf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 04:15:29.305503 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:15:29.305447    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d66a5c52-d8a9-4293-b125-4e182c9f3223-dshm" (OuterVolumeSpecName: "dshm") pod "d66a5c52-d8a9-4293-b125-4e182c9f3223" (UID: "d66a5c52-d8a9-4293-b125-4e182c9f3223"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 04:15:29.404438 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:15:29.404396    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d66a5c52-d8a9-4293-b125-4e182c9f3223-dshm\") on node \"ip-10-0-138-121.ec2.internal\" DevicePath \"\""
Sep 11 04:15:29.404438 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:15:29.404431    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hfltf\" (UniqueName: \"kubernetes.io/projected/d66a5c52-d8a9-4293-b125-4e182c9f3223-kube-api-access-hfltf\") on node \"ip-10-0-138-121.ec2.internal\" DevicePath \"\""
Sep 11 04:15:29.404438 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:15:29.404442    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d66a5c52-d8a9-4293-b125-4e182c9f3223-tls-certs\") on node \"ip-10-0-138-121.ec2.internal\" DevicePath \"\""
Sep 11 04:15:30.145387 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:15:30.145360    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-fa15b45f-kserve-74df5cf7ff-l8s9p"
Sep 11 04:15:30.145810 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:15:30.145360    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-fa15b45f-kserve-74df5cf7ff-l8s9p" event={"ID":"d66a5c52-d8a9-4293-b125-4e182c9f3223","Type":"ContainerDied","Data":"54aaa7d19b870f0a1fe5cf9ade0b34f1b5cf59ed498ab1e23452e6dd37bd6314"}
Sep 11 04:15:30.145810 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:15:30.145489    2580 scope.go:117] "RemoveContainer" containerID="acf7900cd1e6a3e4bcf55899cade4ae810e71e057baa9cb442878fab18fa2cd4"
Sep 11 04:15:30.177861 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:15:30.177825    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-fa15b45f-kserve-74df5cf7ff-l8s9p"]
Sep 11 04:15:30.188303 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:15:30.188274    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-fa15b45f-kserve-74df5cf7ff-l8s9p"]
Sep 11 04:15:30.464904 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:15:30.464873    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d66a5c52-d8a9-4293-b125-4e182c9f3223" path="/var/lib/kubelet/pods/d66a5c52-d8a9-4293-b125-4e182c9f3223/volumes"
Sep 11 04:15:37.519770 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:15:37.519735    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-577f88484f-jrfgt"]
Sep 11 04:15:37.520332 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:15:37.520264    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="76a0217e-2ce2-4597-a0b1-fc43825054dc" containerName="cleanup"
Sep 11 04:15:37.520332 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:15:37.520285    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="76a0217e-2ce2-4597-a0b1-fc43825054dc" containerName="cleanup"
Sep 11 04:15:37.520332 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:15:37.520303    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d66a5c52-d8a9-4293-b125-4e182c9f3223" containerName="main"
Sep 11 04:15:37.520332 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:15:37.520313    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="d66a5c52-d8a9-4293-b125-4e182c9f3223" containerName="main"
Sep 11 04:15:37.520572 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:15:37.520419    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="d66a5c52-d8a9-4293-b125-4e182c9f3223" containerName="main"
Sep 11 04:15:37.520572 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:15:37.520431    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="76a0217e-2ce2-4597-a0b1-fc43825054dc" containerName="cleanup"
Sep 11 04:15:37.524745 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:15:37.524723    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-577f88484f-jrfgt"
Sep 11 04:15:37.528071 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:15:37.528048    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-fcpfs\""
Sep 11 04:15:37.528215 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:15:37.528193    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 04:15:37.528335 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:15:37.528055    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 11 04:15:37.528335 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:15:37.528065    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 11 04:15:37.528335 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:15:37.528125    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Sep 11 04:15:37.532968 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:15:37.532942    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-577f88484f-jrfgt"]
Sep 11 04:15:37.537235 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:15:37.537195    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-778c967957-xjm9j"]
Sep 11 04:15:37.540778 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:15:37.540760    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-778c967957-xjm9j"
Sep 11 04:15:37.550100 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:15:37.550046    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-778c967957-xjm9j"]
Sep 11 04:15:37.579746 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:15:37.579709    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b6c247a1-7182-4bdc-b493-fa6c1740f074-metrics-certs\") pod \"maas-api-577f88484f-jrfgt\" (UID: \"b6c247a1-7182-4bdc-b493-fa6c1740f074\") " pod="odh-ai-gateway-infra/maas-api-577f88484f-jrfgt"
Sep 11 04:15:37.579916 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:15:37.579766    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6c247a1-7182-4bdc-b493-fa6c1740f074-openshift-service-ca\") pod \"maas-api-577f88484f-jrfgt\" (UID: \"b6c247a1-7182-4bdc-b493-fa6c1740f074\") " pod="odh-ai-gateway-infra/maas-api-577f88484f-jrfgt"
Sep 11 04:15:37.579916 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:15:37.579829    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b6c247a1-7182-4bdc-b493-fa6c1740f074-maas-api-tls\") pod \"maas-api-577f88484f-jrfgt\" (UID: \"b6c247a1-7182-4bdc-b493-fa6c1740f074\") " pod="odh-ai-gateway-infra/maas-api-577f88484f-jrfgt"
Sep 11 04:15:37.579997 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:15:37.579919    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lckf7\" (UniqueName: \"kubernetes.io/projected/b6c247a1-7182-4bdc-b493-fa6c1740f074-kube-api-access-lckf7\") pod \"maas-api-577f88484f-jrfgt\" (UID: \"b6c247a1-7182-4bdc-b493-fa6c1740f074\") " pod="odh-ai-gateway-infra/maas-api-577f88484f-jrfgt"
Sep 11 04:15:37.680886 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:15:37.680851    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b6c247a1-7182-4bdc-b493-fa6c1740f074-maas-api-tls\") pod \"maas-api-577f88484f-jrfgt\" (UID: \"b6c247a1-7182-4bdc-b493-fa6c1740f074\") " pod="odh-ai-gateway-infra/maas-api-577f88484f-jrfgt"
Sep 11 04:15:37.681086 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:15:37.680904    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/283d1b87-ffdc-4e07-83ef-a5b931691cf9-metrics-certs\") pod \"maas-api-778c967957-xjm9j\" (UID: \"283d1b87-ffdc-4e07-83ef-a5b931691cf9\") " pod="odh-ai-gateway-infra/maas-api-778c967957-xjm9j"
Sep 11 04:15:37.681086 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:15:37.680945    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lckf7\" (UniqueName: \"kubernetes.io/projected/b6c247a1-7182-4bdc-b493-fa6c1740f074-kube-api-access-lckf7\") pod \"maas-api-577f88484f-jrfgt\" (UID: \"b6c247a1-7182-4bdc-b493-fa6c1740f074\") " pod="odh-ai-gateway-infra/maas-api-577f88484f-jrfgt"
Sep 11 04:15:37.681086 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:15:37.680977    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/283d1b87-ffdc-4e07-83ef-a5b931691cf9-maas-api-tls\") pod \"maas-api-778c967957-xjm9j\" (UID: \"283d1b87-ffdc-4e07-83ef-a5b931691cf9\") " pod="odh-ai-gateway-infra/maas-api-778c967957-xjm9j"
Sep 11 04:15:37.681086 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:15:37.680998    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b6c247a1-7182-4bdc-b493-fa6c1740f074-metrics-certs\") pod \"maas-api-577f88484f-jrfgt\" (UID: \"b6c247a1-7182-4bdc-b493-fa6c1740f074\") " pod="odh-ai-gateway-infra/maas-api-577f88484f-jrfgt"
Sep 11 04:15:37.681086 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:15:37.681036    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6c247a1-7182-4bdc-b493-fa6c1740f074-openshift-service-ca\") pod \"maas-api-577f88484f-jrfgt\" (UID: \"b6c247a1-7182-4bdc-b493-fa6c1740f074\") " pod="odh-ai-gateway-infra/maas-api-577f88484f-jrfgt"
Sep 11 04:15:37.681366 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:15:37.681122    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/283d1b87-ffdc-4e07-83ef-a5b931691cf9-openshift-service-ca\") pod \"maas-api-778c967957-xjm9j\" (UID: \"283d1b87-ffdc-4e07-83ef-a5b931691cf9\") " pod="odh-ai-gateway-infra/maas-api-778c967957-xjm9j"
Sep 11 04:15:37.681366 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:15:37.681161    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znvq2\" (UniqueName: \"kubernetes.io/projected/283d1b87-ffdc-4e07-83ef-a5b931691cf9-kube-api-access-znvq2\") pod \"maas-api-778c967957-xjm9j\" (UID: \"283d1b87-ffdc-4e07-83ef-a5b931691cf9\") " pod="odh-ai-gateway-infra/maas-api-778c967957-xjm9j"
Sep 11 04:15:37.681717 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:15:37.681691    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6c247a1-7182-4bdc-b493-fa6c1740f074-openshift-service-ca\") pod \"maas-api-577f88484f-jrfgt\" (UID: \"b6c247a1-7182-4bdc-b493-fa6c1740f074\") " pod="odh-ai-gateway-infra/maas-api-577f88484f-jrfgt"
Sep 11 04:15:37.683422 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:15:37.683397    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b6c247a1-7182-4bdc-b493-fa6c1740f074-maas-api-tls\") pod \"maas-api-577f88484f-jrfgt\" (UID: \"b6c247a1-7182-4bdc-b493-fa6c1740f074\") " pod="odh-ai-gateway-infra/maas-api-577f88484f-jrfgt"
Sep 11 04:15:37.683884 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:15:37.683863    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b6c247a1-7182-4bdc-b493-fa6c1740f074-metrics-certs\") pod \"maas-api-577f88484f-jrfgt\" (UID: \"b6c247a1-7182-4bdc-b493-fa6c1740f074\") " pod="odh-ai-gateway-infra/maas-api-577f88484f-jrfgt"
Sep 11 04:15:37.688478 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:15:37.688451    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lckf7\" (UniqueName: \"kubernetes.io/projected/b6c247a1-7182-4bdc-b493-fa6c1740f074-kube-api-access-lckf7\") pod \"maas-api-577f88484f-jrfgt\" (UID: \"b6c247a1-7182-4bdc-b493-fa6c1740f074\") " pod="odh-ai-gateway-infra/maas-api-577f88484f-jrfgt"
Sep 11 04:15:37.781979 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:15:37.781898    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/283d1b87-ffdc-4e07-83ef-a5b931691cf9-openshift-service-ca\") pod \"maas-api-778c967957-xjm9j\" (UID: \"283d1b87-ffdc-4e07-83ef-a5b931691cf9\") " pod="odh-ai-gateway-infra/maas-api-778c967957-xjm9j"
Sep 11 04:15:37.781979 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:15:37.781931    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znvq2\" (UniqueName: \"kubernetes.io/projected/283d1b87-ffdc-4e07-83ef-a5b931691cf9-kube-api-access-znvq2\") pod \"maas-api-778c967957-xjm9j\" (UID: \"283d1b87-ffdc-4e07-83ef-a5b931691cf9\") " pod="odh-ai-gateway-infra/maas-api-778c967957-xjm9j"
Sep 11 04:15:37.781979 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:15:37.781981    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/283d1b87-ffdc-4e07-83ef-a5b931691cf9-metrics-certs\") pod \"maas-api-778c967957-xjm9j\" (UID: \"283d1b87-ffdc-4e07-83ef-a5b931691cf9\") " pod="odh-ai-gateway-infra/maas-api-778c967957-xjm9j"
Sep 11 04:15:37.782208 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:15:37.782016    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/283d1b87-ffdc-4e07-83ef-a5b931691cf9-maas-api-tls\") pod \"maas-api-778c967957-xjm9j\" (UID: \"283d1b87-ffdc-4e07-83ef-a5b931691cf9\") " pod="odh-ai-gateway-infra/maas-api-778c967957-xjm9j"
Sep 11 04:15:37.782645 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:15:37.782619    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/283d1b87-ffdc-4e07-83ef-a5b931691cf9-openshift-service-ca\") pod \"maas-api-778c967957-xjm9j\" (UID: \"283d1b87-ffdc-4e07-83ef-a5b931691cf9\") " pod="odh-ai-gateway-infra/maas-api-778c967957-xjm9j"
Sep 11 04:15:37.784358 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:15:37.784316    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/283d1b87-ffdc-4e07-83ef-a5b931691cf9-metrics-certs\") pod \"maas-api-778c967957-xjm9j\" (UID: \"283d1b87-ffdc-4e07-83ef-a5b931691cf9\") " pod="odh-ai-gateway-infra/maas-api-778c967957-xjm9j"
Sep 11 04:15:37.784485 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:15:37.784406    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/283d1b87-ffdc-4e07-83ef-a5b931691cf9-maas-api-tls\") pod \"maas-api-778c967957-xjm9j\" (UID: \"283d1b87-ffdc-4e07-83ef-a5b931691cf9\") " pod="odh-ai-gateway-infra/maas-api-778c967957-xjm9j"
Sep 11 04:15:37.790325 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:15:37.790303    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-znvq2\" (UniqueName: \"kubernetes.io/projected/283d1b87-ffdc-4e07-83ef-a5b931691cf9-kube-api-access-znvq2\") pod \"maas-api-778c967957-xjm9j\" (UID: \"283d1b87-ffdc-4e07-83ef-a5b931691cf9\") " pod="odh-ai-gateway-infra/maas-api-778c967957-xjm9j"
Sep 11 04:15:37.838337 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:15:37.838303    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-577f88484f-jrfgt"
Sep 11 04:15:37.854720 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:15:37.854692    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-778c967957-xjm9j"
Sep 11 04:15:38.115383 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:15:38.115355    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-577f88484f-jrfgt"]
Sep 11 04:15:38.118166 ip-10-0-138-121 kubenswrapper[2580]: W0911 04:15:38.118115    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6c247a1_7182_4bdc_b493_fa6c1740f074.slice/crio-93b94a0d8cb6e0eb6e0d1f1af731a998ad807f75efff630d98089f18f98d763e WatchSource:0}: Error finding container 93b94a0d8cb6e0eb6e0d1f1af731a998ad807f75efff630d98089f18f98d763e: Status 404 returned error can't find the container with id 93b94a0d8cb6e0eb6e0d1f1af731a998ad807f75efff630d98089f18f98d763e
Sep 11 04:15:38.119529 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:15:38.119509    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 04:15:38.137797 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:15:38.137689    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-778c967957-xjm9j"]
Sep 11 04:15:38.139850 ip-10-0-138-121 kubenswrapper[2580]: W0911 04:15:38.139817    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod283d1b87_ffdc_4e07_83ef_a5b931691cf9.slice/crio-beb63ecf487bd1d21cd86a2368c6dca27108cf8a9ed5a892497e0c10b5bc4d58 WatchSource:0}: Error finding container beb63ecf487bd1d21cd86a2368c6dca27108cf8a9ed5a892497e0c10b5bc4d58: Status 404 returned error can't find the container with id beb63ecf487bd1d21cd86a2368c6dca27108cf8a9ed5a892497e0c10b5bc4d58
Sep 11 04:15:38.185242 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:15:38.185201    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-577f88484f-jrfgt" event={"ID":"b6c247a1-7182-4bdc-b493-fa6c1740f074","Type":"ContainerStarted","Data":"93b94a0d8cb6e0eb6e0d1f1af731a998ad807f75efff630d98089f18f98d763e"}
Sep 11 04:15:38.186413 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:15:38.186379    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-778c967957-xjm9j" event={"ID":"283d1b87-ffdc-4e07-83ef-a5b931691cf9","Type":"ContainerStarted","Data":"beb63ecf487bd1d21cd86a2368c6dca27108cf8a9ed5a892497e0c10b5bc4d58"}
Sep 11 04:15:39.192451 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:15:39.192406    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-577f88484f-jrfgt" event={"ID":"b6c247a1-7182-4bdc-b493-fa6c1740f074","Type":"ContainerStarted","Data":"b5969a1abefe6b32476d4909d46cb6f74ac9b2bb2f9c55ef01e5140e821241d4"}
Sep 11 04:15:39.192928 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:15:39.192512    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-577f88484f-jrfgt"
Sep 11 04:15:39.193976 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:15:39.193955    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-778c967957-xjm9j" event={"ID":"283d1b87-ffdc-4e07-83ef-a5b931691cf9","Type":"ContainerStarted","Data":"5d673563bd5bab4b200c01c235f931f1a1fb683c5e7a244549faa89531170e46"}
Sep 11 04:15:39.194098 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:15:39.194006    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-778c967957-xjm9j"
Sep 11 04:15:39.210438 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:15:39.210394    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-577f88484f-jrfgt" podStartSLOduration=1.707239792 podStartE2EDuration="2.210380475s" totalImagesPullingTime="503.140683ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:15:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:15:39.208933821 +0000 UTC m=+2127.367420147" watchObservedRunningTime="2026-09-11 04:15:39.210380475 +0000 UTC m=+2127.368866798"
Sep 11 04:15:39.226481 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:15:39.226436    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-778c967957-xjm9j" podStartSLOduration=1.7749776659999998 podStartE2EDuration="2.226418831s" totalImagesPullingTime="451.441165ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:15:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:15:39.225019548 +0000 UTC m=+2127.383505868" watchObservedRunningTime="2026-09-11 04:15:39.226418831 +0000 UTC m=+2127.384905306"
Sep 11 04:15:45.205020 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:15:45.204987    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-778c967957-xjm9j"
Sep 11 04:15:45.205462 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:15:45.205308    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-577f88484f-jrfgt"
Sep 11 04:15:45.285994 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:15:45.285959    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-577f88484f-jrfgt"]
Sep 11 04:15:45.286218 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:15:45.286189    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-577f88484f-jrfgt" podUID="b6c247a1-7182-4bdc-b493-fa6c1740f074" containerName="maas-api" containerID="cri-o://b5969a1abefe6b32476d4909d46cb6f74ac9b2bb2f9c55ef01e5140e821241d4" gracePeriod=30
Sep 11 04:15:45.529345 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:15:45.529319    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-577f88484f-jrfgt"
Sep 11 04:15:45.658391 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:15:45.658359    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lckf7\" (UniqueName: \"kubernetes.io/projected/b6c247a1-7182-4bdc-b493-fa6c1740f074-kube-api-access-lckf7\") pod \"b6c247a1-7182-4bdc-b493-fa6c1740f074\" (UID: \"b6c247a1-7182-4bdc-b493-fa6c1740f074\") "
Sep 11 04:15:45.658572 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:15:45.658411    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b6c247a1-7182-4bdc-b493-fa6c1740f074-metrics-certs\") pod \"b6c247a1-7182-4bdc-b493-fa6c1740f074\" (UID: \"b6c247a1-7182-4bdc-b493-fa6c1740f074\") "
Sep 11 04:15:45.658572 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:15:45.658451    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6c247a1-7182-4bdc-b493-fa6c1740f074-openshift-service-ca\") pod \"b6c247a1-7182-4bdc-b493-fa6c1740f074\" (UID: \"b6c247a1-7182-4bdc-b493-fa6c1740f074\") "
Sep 11 04:15:45.658572 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:15:45.658533    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b6c247a1-7182-4bdc-b493-fa6c1740f074-maas-api-tls\") pod \"b6c247a1-7182-4bdc-b493-fa6c1740f074\" (UID: \"b6c247a1-7182-4bdc-b493-fa6c1740f074\") "
Sep 11 04:15:45.658894 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:15:45.658861    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6c247a1-7182-4bdc-b493-fa6c1740f074-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b6c247a1-7182-4bdc-b493-fa6c1740f074" (UID: "b6c247a1-7182-4bdc-b493-fa6c1740f074"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 04:15:45.660725 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:15:45.660632    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6c247a1-7182-4bdc-b493-fa6c1740f074-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "b6c247a1-7182-4bdc-b493-fa6c1740f074" (UID: "b6c247a1-7182-4bdc-b493-fa6c1740f074"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 04:15:45.660830 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:15:45.660734    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6c247a1-7182-4bdc-b493-fa6c1740f074-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b6c247a1-7182-4bdc-b493-fa6c1740f074" (UID: "b6c247a1-7182-4bdc-b493-fa6c1740f074"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 04:15:45.660830 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:15:45.660779    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6c247a1-7182-4bdc-b493-fa6c1740f074-kube-api-access-lckf7" (OuterVolumeSpecName: "kube-api-access-lckf7") pod "b6c247a1-7182-4bdc-b493-fa6c1740f074" (UID: "b6c247a1-7182-4bdc-b493-fa6c1740f074"). InnerVolumeSpecName "kube-api-access-lckf7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 04:15:45.760156 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:15:45.760063    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b6c247a1-7182-4bdc-b493-fa6c1740f074-maas-api-tls\") on node \"ip-10-0-138-121.ec2.internal\" DevicePath \"\""
Sep 11 04:15:45.760156 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:15:45.760105    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lckf7\" (UniqueName: \"kubernetes.io/projected/b6c247a1-7182-4bdc-b493-fa6c1740f074-kube-api-access-lckf7\") on node \"ip-10-0-138-121.ec2.internal\" DevicePath \"\""
Sep 11 04:15:45.760156 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:15:45.760120    2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b6c247a1-7182-4bdc-b493-fa6c1740f074-metrics-certs\") on node \"ip-10-0-138-121.ec2.internal\" DevicePath \"\""
Sep 11 04:15:45.760156 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:15:45.760135    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6c247a1-7182-4bdc-b493-fa6c1740f074-openshift-service-ca\") on node \"ip-10-0-138-121.ec2.internal\" DevicePath \"\""
Sep 11 04:15:46.232493 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:15:46.232460    2580 generic.go:356] "Generic (PLEG): container finished" podID="b6c247a1-7182-4bdc-b493-fa6c1740f074" containerID="b5969a1abefe6b32476d4909d46cb6f74ac9b2bb2f9c55ef01e5140e821241d4" exitCode=0
Sep 11 04:15:46.232939 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:15:46.232506    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-577f88484f-jrfgt"
Sep 11 04:15:46.232939 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:15:46.232536    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-577f88484f-jrfgt" event={"ID":"b6c247a1-7182-4bdc-b493-fa6c1740f074","Type":"ContainerDied","Data":"b5969a1abefe6b32476d4909d46cb6f74ac9b2bb2f9c55ef01e5140e821241d4"}
Sep 11 04:15:46.232939 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:15:46.232578    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-577f88484f-jrfgt" event={"ID":"b6c247a1-7182-4bdc-b493-fa6c1740f074","Type":"ContainerDied","Data":"93b94a0d8cb6e0eb6e0d1f1af731a998ad807f75efff630d98089f18f98d763e"}
Sep 11 04:15:46.232939 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:15:46.232603    2580 scope.go:117] "RemoveContainer" containerID="b5969a1abefe6b32476d4909d46cb6f74ac9b2bb2f9c55ef01e5140e821241d4"
Sep 11 04:15:46.241409 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:15:46.241392    2580 scope.go:117] "RemoveContainer" containerID="b5969a1abefe6b32476d4909d46cb6f74ac9b2bb2f9c55ef01e5140e821241d4"
Sep 11 04:15:46.241696 ip-10-0-138-121 kubenswrapper[2580]: E0911 04:15:46.241652    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5969a1abefe6b32476d4909d46cb6f74ac9b2bb2f9c55ef01e5140e821241d4\": container with ID starting with b5969a1abefe6b32476d4909d46cb6f74ac9b2bb2f9c55ef01e5140e821241d4 not found: ID does not exist" containerID="b5969a1abefe6b32476d4909d46cb6f74ac9b2bb2f9c55ef01e5140e821241d4"
Sep 11 04:15:46.241780 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:15:46.241696    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5969a1abefe6b32476d4909d46cb6f74ac9b2bb2f9c55ef01e5140e821241d4"} err="failed to get container status \"b5969a1abefe6b32476d4909d46cb6f74ac9b2bb2f9c55ef01e5140e821241d4\": rpc error: code = NotFound desc = could not find container \"b5969a1abefe6b32476d4909d46cb6f74ac9b2bb2f9c55ef01e5140e821241d4\": container with ID starting with b5969a1abefe6b32476d4909d46cb6f74ac9b2bb2f9c55ef01e5140e821241d4 not found: ID does not exist"
Sep 11 04:15:46.253813 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:15:46.253785    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-577f88484f-jrfgt"]
Sep 11 04:15:46.257107 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:15:46.257084    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-577f88484f-jrfgt"]
Sep 11 04:15:46.464836 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:15:46.464803    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b6c247a1-7182-4bdc-b493-fa6c1740f074" path="/var/lib/kubelet/pods/b6c247a1-7182-4bdc-b493-fa6c1740f074/volumes"
Sep 11 04:16:14.940990 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:16:14.940915    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6cf4d46bbf-sbztx_86bf55e1-90b9-436a-98a4-7997e90e0f84/copy-manifests/0.log"
Sep 11 04:16:14.947365 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:16:14.947343    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6cf4d46bbf-sbztx_86bf55e1-90b9-436a-98a4-7997e90e0f84/manager/0.log"
Sep 11 04:16:15.543781 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:16:15.543753    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-778c967957-xjm9j_283d1b87-ffdc-4e07-83ef-a5b931691cf9/maas-api/0.log"
Sep 11 04:16:15.772233 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:16:15.772201    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29818335-xgjjx_76a0217e-2ce2-4597-a0b1-fc43825054dc/cleanup/0.log"
Sep 11 04:16:17.233069 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:16:17.233025    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-46xjp_01f64146-896b-419e-a404-587dd2ef58ce/manager/0.log"
Sep 11 04:16:17.991781 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:16:17.991748    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-8kfvp_bf009cc0-6d19-4ecb-adbc-81b9751265d5/discovery/0.log"
Sep 11 04:16:19.100097 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:16:19.100073    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-2rvkj_3cd86dce-6008-4b5d-9455-069889bc79cd/main/0.log"
Sep 11 04:16:19.334310 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:16:19.334282    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-bcc9ff8d8-nf79k_a2800068-b767-412b-ba7a-60a7f8ab4dc6/main/0.log"
Sep 11 04:16:28.887677 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:16:28.887641    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-8knjw_f6fa8b71-4ef9-4c91-b7ee-7ef235532ea0/global-pull-secret-syncer/0.log"
Sep 11 04:16:29.118535 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:16:29.118506    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-tm9tp_a7954230-378f-4cf1-b574-3c3b9f0ec4fb/konnectivity-agent/0.log"
Sep 11 04:16:29.147739 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:16:29.147678    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-138-121.ec2.internal_355aa2f2ac5d4d26d63779c83ede8936/haproxy/0.log"
Sep 11 04:16:34.434707 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:16:34.434654    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-46xjp_01f64146-896b-419e-a404-587dd2ef58ce/manager/0.log"
Sep 11 04:16:35.991871 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:16:35.991841    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_e5988370-558f-43e2-af52-5369936a43ae/alertmanager/0.log"
Sep 11 04:16:36.016922 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:16:36.016898    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_e5988370-558f-43e2-af52-5369936a43ae/config-reloader/0.log"
Sep 11 04:16:36.049920 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:16:36.049892    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_e5988370-558f-43e2-af52-5369936a43ae/kube-rbac-proxy-web/0.log"
Sep 11 04:16:36.076479 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:16:36.076457    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_e5988370-558f-43e2-af52-5369936a43ae/kube-rbac-proxy/0.log"
Sep 11 04:16:36.102212 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:16:36.102190    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_e5988370-558f-43e2-af52-5369936a43ae/kube-rbac-proxy-metric/0.log"
Sep 11 04:16:36.131005 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:16:36.130976    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_e5988370-558f-43e2-af52-5369936a43ae/prom-label-proxy/0.log"
Sep 11 04:16:36.156517 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:16:36.156490    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_e5988370-558f-43e2-af52-5369936a43ae/init-config-reloader/0.log"
Sep 11 04:16:36.400585 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:16:36.400508    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5d85778b9d-g4xkc_065b6bbb-a5f5-4602-9f4b-fc191f92a4bd/monitoring-plugin/0.log"
Sep 11 04:16:36.448841 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:16:36.448815    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-gppcl_c8257ffa-bddf-478e-8b16-e246ad50e29a/node-exporter/0.log"
Sep 11 04:16:36.493715 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:16:36.493691    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-gppcl_c8257ffa-bddf-478e-8b16-e246ad50e29a/kube-rbac-proxy/0.log"
Sep 11 04:16:36.541285 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:16:36.541260    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-gppcl_c8257ffa-bddf-478e-8b16-e246ad50e29a/init-textfile/0.log"
Sep 11 04:16:36.980856 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:16:36.980827    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-lq5nl_4859b737-e190-44e9-ae1b-0812bbd1c927/kube-rbac-proxy-main/0.log"
Sep 11 04:16:37.026431 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:16:37.026400    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-lq5nl_4859b737-e190-44e9-ae1b-0812bbd1c927/kube-rbac-proxy-self/0.log"
Sep 11 04:16:37.070503 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:16:37.070469    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-lq5nl_4859b737-e190-44e9-ae1b-0812bbd1c927/openshift-state-metrics/0.log"
Sep 11 04:16:37.534351 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:16:37.534326    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-68bcd7f987-c6zjw_a5f45b3b-f5ae-427c-b253-399d0dcd896d/prometheus-operator/0.log"
Sep 11 04:16:37.560507 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:16:37.560480    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-68bcd7f987-c6zjw_a5f45b3b-f5ae-427c-b253-399d0dcd896d/kube-rbac-proxy/0.log"
Sep 11 04:16:37.588593 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:16:37.588563    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-5864cd99d7-xd7gn_f30e3e6e-ad79-489c-bffa-99b81296ac25/prometheus-operator-admission-webhook/0.log"
Sep 11 04:16:37.635038 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:16:37.635012    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-67577cf4bb-9pb7v_0f67bb82-1dc5-418b-8778-b43028862742/telemeter-client/0.log"
Sep 11 04:16:37.657886 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:16:37.657854    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-67577cf4bb-9pb7v_0f67bb82-1dc5-418b-8778-b43028862742/reload/0.log"
Sep 11 04:16:37.693423 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:16:37.693391    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-67577cf4bb-9pb7v_0f67bb82-1dc5-418b-8778-b43028862742/kube-rbac-proxy/0.log"
Sep 11 04:16:39.709983 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:16:39.709950    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-85rjf/perf-node-gather-daemonset-rkjwd"]
Sep 11 04:16:39.710406 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:16:39.710393    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b6c247a1-7182-4bdc-b493-fa6c1740f074" containerName="maas-api"
Sep 11 04:16:39.710460 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:16:39.710409    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="b6c247a1-7182-4bdc-b493-fa6c1740f074" containerName="maas-api"
Sep 11 04:16:39.710493 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:16:39.710482    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="b6c247a1-7182-4bdc-b493-fa6c1740f074" containerName="maas-api"
Sep 11 04:16:39.714928 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:16:39.714910    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-85rjf/perf-node-gather-daemonset-rkjwd"
Sep 11 04:16:39.718306 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:16:39.718284    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-85rjf\"/\"default-dockercfg-j767r\""
Sep 11 04:16:39.718394 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:16:39.718284    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-85rjf\"/\"openshift-service-ca.crt\""
Sep 11 04:16:39.723345 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:16:39.723327    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-85rjf\"/\"kube-root-ca.crt\""
Sep 11 04:16:39.726867 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:16:39.726843    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-85rjf/perf-node-gather-daemonset-rkjwd"]
Sep 11 04:16:39.870545 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:16:39.870504    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e3dd4949-1077-4e19-8db8-2a0ef91d63bd-proc\") pod \"perf-node-gather-daemonset-rkjwd\" (UID: \"e3dd4949-1077-4e19-8db8-2a0ef91d63bd\") " pod="openshift-must-gather-85rjf/perf-node-gather-daemonset-rkjwd"
Sep 11 04:16:39.870745 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:16:39.870606    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e3dd4949-1077-4e19-8db8-2a0ef91d63bd-sys\") pod \"perf-node-gather-daemonset-rkjwd\" (UID: \"e3dd4949-1077-4e19-8db8-2a0ef91d63bd\") " pod="openshift-must-gather-85rjf/perf-node-gather-daemonset-rkjwd"
Sep 11 04:16:39.870745 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:16:39.870652    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hd2jx\" (UniqueName: \"kubernetes.io/projected/e3dd4949-1077-4e19-8db8-2a0ef91d63bd-kube-api-access-hd2jx\") pod \"perf-node-gather-daemonset-rkjwd\" (UID: \"e3dd4949-1077-4e19-8db8-2a0ef91d63bd\") " pod="openshift-must-gather-85rjf/perf-node-gather-daemonset-rkjwd"
Sep 11 04:16:39.870745 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:16:39.870722    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e3dd4949-1077-4e19-8db8-2a0ef91d63bd-podres\") pod \"perf-node-gather-daemonset-rkjwd\" (UID: \"e3dd4949-1077-4e19-8db8-2a0ef91d63bd\") " pod="openshift-must-gather-85rjf/perf-node-gather-daemonset-rkjwd"
Sep 11 04:16:39.870860 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:16:39.870752    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e3dd4949-1077-4e19-8db8-2a0ef91d63bd-lib-modules\") pod \"perf-node-gather-daemonset-rkjwd\" (UID: \"e3dd4949-1077-4e19-8db8-2a0ef91d63bd\") " pod="openshift-must-gather-85rjf/perf-node-gather-daemonset-rkjwd"
Sep 11 04:16:39.972157 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:16:39.972066    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e3dd4949-1077-4e19-8db8-2a0ef91d63bd-sys\") pod \"perf-node-gather-daemonset-rkjwd\" (UID: \"e3dd4949-1077-4e19-8db8-2a0ef91d63bd\") " pod="openshift-must-gather-85rjf/perf-node-gather-daemonset-rkjwd"
Sep 11 04:16:39.972157 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:16:39.972123    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hd2jx\" (UniqueName: \"kubernetes.io/projected/e3dd4949-1077-4e19-8db8-2a0ef91d63bd-kube-api-access-hd2jx\") pod \"perf-node-gather-daemonset-rkjwd\" (UID: \"e3dd4949-1077-4e19-8db8-2a0ef91d63bd\") " pod="openshift-must-gather-85rjf/perf-node-gather-daemonset-rkjwd"
Sep 11 04:16:39.972157 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:16:39.972153    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e3dd4949-1077-4e19-8db8-2a0ef91d63bd-podres\") pod \"perf-node-gather-daemonset-rkjwd\" (UID: \"e3dd4949-1077-4e19-8db8-2a0ef91d63bd\") " pod="openshift-must-gather-85rjf/perf-node-gather-daemonset-rkjwd"
Sep 11 04:16:39.972371 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:16:39.972177    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e3dd4949-1077-4e19-8db8-2a0ef91d63bd-lib-modules\") pod \"perf-node-gather-daemonset-rkjwd\" (UID: \"e3dd4949-1077-4e19-8db8-2a0ef91d63bd\") " pod="openshift-must-gather-85rjf/perf-node-gather-daemonset-rkjwd"
Sep 11 04:16:39.972371 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:16:39.972180    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e3dd4949-1077-4e19-8db8-2a0ef91d63bd-sys\") pod \"perf-node-gather-daemonset-rkjwd\" (UID: \"e3dd4949-1077-4e19-8db8-2a0ef91d63bd\") " pod="openshift-must-gather-85rjf/perf-node-gather-daemonset-rkjwd"
Sep 11 04:16:39.972371 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:16:39.972211    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e3dd4949-1077-4e19-8db8-2a0ef91d63bd-proc\") pod \"perf-node-gather-daemonset-rkjwd\" (UID: \"e3dd4949-1077-4e19-8db8-2a0ef91d63bd\") " pod="openshift-must-gather-85rjf/perf-node-gather-daemonset-rkjwd"
Sep 11 04:16:39.972371 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:16:39.972294    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e3dd4949-1077-4e19-8db8-2a0ef91d63bd-lib-modules\") pod \"perf-node-gather-daemonset-rkjwd\" (UID: \"e3dd4949-1077-4e19-8db8-2a0ef91d63bd\") " pod="openshift-must-gather-85rjf/perf-node-gather-daemonset-rkjwd"
Sep 11 04:16:39.972371 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:16:39.972305    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e3dd4949-1077-4e19-8db8-2a0ef91d63bd-proc\") pod \"perf-node-gather-daemonset-rkjwd\" (UID: \"e3dd4949-1077-4e19-8db8-2a0ef91d63bd\") " pod="openshift-must-gather-85rjf/perf-node-gather-daemonset-rkjwd"
Sep 11 04:16:39.972551 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:16:39.972372    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e3dd4949-1077-4e19-8db8-2a0ef91d63bd-podres\") pod \"perf-node-gather-daemonset-rkjwd\" (UID: \"e3dd4949-1077-4e19-8db8-2a0ef91d63bd\") " pod="openshift-must-gather-85rjf/perf-node-gather-daemonset-rkjwd"
Sep 11 04:16:39.980758 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:16:39.980727    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hd2jx\" (UniqueName: \"kubernetes.io/projected/e3dd4949-1077-4e19-8db8-2a0ef91d63bd-kube-api-access-hd2jx\") pod \"perf-node-gather-daemonset-rkjwd\" (UID: \"e3dd4949-1077-4e19-8db8-2a0ef91d63bd\") " pod="openshift-must-gather-85rjf/perf-node-gather-daemonset-rkjwd"
Sep 11 04:16:40.024822 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:16:40.024793    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-85rjf/perf-node-gather-daemonset-rkjwd"
Sep 11 04:16:40.297130 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:16:40.297100    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-85rjf/perf-node-gather-daemonset-rkjwd"]
Sep 11 04:16:40.299230 ip-10-0-138-121 kubenswrapper[2580]: W0911 04:16:40.299204    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pode3dd4949_1077_4e19_8db8_2a0ef91d63bd.slice/crio-dc63bafd18a8a6ce6d43ee471a35d51bed689870a92ad829288f9a07c0ed0824 WatchSource:0}: Error finding container dc63bafd18a8a6ce6d43ee471a35d51bed689870a92ad829288f9a07c0ed0824: Status 404 returned error can't find the container with id dc63bafd18a8a6ce6d43ee471a35d51bed689870a92ad829288f9a07c0ed0824
Sep 11 04:16:40.430473 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:16:40.430448    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-7886644f6d-qpjhb_9db50db7-59ba-4828-b3c4-2ef68b00d22c/console/0.log"
Sep 11 04:16:40.491789 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:16:40.491762    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-76f99584f4-xt54w_15478e4f-2c48-4b48-b6cc-64da0510c1fa/download-server/0.log"
Sep 11 04:16:40.502000 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:16:40.501977    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-85rjf/perf-node-gather-daemonset-rkjwd" event={"ID":"e3dd4949-1077-4e19-8db8-2a0ef91d63bd","Type":"ContainerStarted","Data":"9f5da78b092722bcce0b5ffe7752691f339bf57b8b0e1e90dab64689c64b3f88"}
Sep 11 04:16:40.502117 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:16:40.502017    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-85rjf/perf-node-gather-daemonset-rkjwd" event={"ID":"e3dd4949-1077-4e19-8db8-2a0ef91d63bd","Type":"ContainerStarted","Data":"dc63bafd18a8a6ce6d43ee471a35d51bed689870a92ad829288f9a07c0ed0824"}
Sep 11 04:16:40.502159 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:16:40.502117    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-85rjf/perf-node-gather-daemonset-rkjwd"
Sep 11 04:16:40.519574 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:16:40.519531    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-85rjf/perf-node-gather-daemonset-rkjwd" podStartSLOduration=1.519517883 podStartE2EDuration="1.519517883s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 04:16:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:16:40.518940703 +0000 UTC m=+2188.677427028" watchObservedRunningTime="2026-09-11 04:16:40.519517883 +0000 UTC m=+2188.678004205"
Sep 11 04:16:42.266837 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:16:42.266806    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-pwk5f_61f124d5-c24b-4150-b19a-ae500bce32fe/dns/0.log"
Sep 11 04:16:42.292618 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:16:42.292588    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-pwk5f_61f124d5-c24b-4150-b19a-ae500bce32fe/kube-rbac-proxy/0.log"
Sep 11 04:16:42.424321 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:16:42.424292    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-wkqmr_896af27b-fde3-43f0-bdbb-5bdf0815873a/dns-node-resolver/0.log"
Sep 11 04:16:42.962125 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:16:42.962094    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-6dc5b6c9b6-rt5ml_dc23fd71-e4f6-4e97-b9c6-b5d200a69802/registry/0.log"
Sep 11 04:16:43.008955 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:16:43.008923    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-mlcbg_84aade61-b056-43d4-bbcd-fd68eba2cb48/node-ca/0.log"
Sep 11 04:16:44.310824 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:16:44.310789    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-8kfvp_bf009cc0-6d19-4ecb-adbc-81b9751265d5/discovery/0.log"
Sep 11 04:16:45.480378 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:16:45.480348    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-f5d7k_46e75c33-310a-420f-b9c9-f92bbe261dd7/serve-healthcheck-canary/0.log"
Sep 11 04:16:46.170481 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:16:46.170453    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5844b856db-l7n7s_2a8f992e-1768-401c-afd6-b87e5a289d61/insights-operator/0.log"
Sep 11 04:16:46.170747 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:16:46.170550    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5844b856db-l7n7s_2a8f992e-1768-401c-afd6-b87e5a289d61/insights-operator/1.log"
Sep 11 04:16:46.456867 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:16:46.456793    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-qgz5l_40db11ed-2328-4a59-9db8-6efeaf55b54b/kube-rbac-proxy/0.log"
Sep 11 04:16:46.495226 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:16:46.495197    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-qgz5l_40db11ed-2328-4a59-9db8-6efeaf55b54b/exporter/0.log"
Sep 11 04:16:46.515759 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:16:46.515736    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-85rjf/perf-node-gather-daemonset-rkjwd"
Sep 11 04:16:46.547651 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:16:46.547627    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-qgz5l_40db11ed-2328-4a59-9db8-6efeaf55b54b/extractor/0.log"
Sep 11 04:16:49.938879 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:16:49.938847    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6cf4d46bbf-sbztx_86bf55e1-90b9-436a-98a4-7997e90e0f84/manager/0.log"
Sep 11 04:16:50.004190 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:16:50.004160    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6cf4d46bbf-sbztx_86bf55e1-90b9-436a-98a4-7997e90e0f84/copy-manifests/0.log"
Sep 11 04:16:50.468713 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:16:50.468681    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-8kfvp_bf009cc0-6d19-4ecb-adbc-81b9751265d5/discovery/0.log"
Sep 11 04:16:51.543804 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:16:51.543778    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-67c8664749-8wbt4_cd30e29b-6de1-44ea-9ba1-f67605725a2a/manager/0.log"
Sep 11 04:16:52.583675 ip-10-0-138-121 kubenswrapper[2580]: I0911 04:16:52.583641    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-8kfvp_bf009cc0-6d19-4ecb-adbc-81b9751265d5/discovery/0.log"