Sep 10 17:35:55.887840 ip-10-0-129-122 systemd[1]: Starting Kubernetes Kubelet...
Sep 10 17:35:56.283087 ip-10-0-129-122 kubenswrapper[2579]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 10 17:35:56.283087 ip-10-0-129-122 kubenswrapper[2579]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 10 17:35:56.283087 ip-10-0-129-122 kubenswrapper[2579]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 10 17:35:56.283087 ip-10-0-129-122 kubenswrapper[2579]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 10 17:35:56.283087 ip-10-0-129-122 kubenswrapper[2579]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 10 17:35:56.284071 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.283994    2579 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 10 17:35:56.289554 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.289532    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 17:35:56.289554 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.289551    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 17:35:56.289554 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.289555    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 17:35:56.289554 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.289558    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 17:35:56.289735 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.289579    2579 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 17:35:56.289735 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.289583    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 17:35:56.289735 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.289586    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 17:35:56.289735 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.289588    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 17:35:56.289735 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.289591    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 17:35:56.289735 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.289593    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 17:35:56.289735 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.289598    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 17:35:56.289735 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.289602    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 17:35:56.289735 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.289605    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 17:35:56.289735 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.289609    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 17:35:56.289735 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.289612    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 17:35:56.289735 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.289615    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 17:35:56.289735 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.289618    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 17:35:56.289735 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.289621    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 17:35:56.289735 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.289623    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 17:35:56.289735 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.289626    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 17:35:56.289735 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.289629    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 17:35:56.289735 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.289631    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 17:35:56.289735 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.289634    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 17:35:56.289735 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.289637    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 17:35:56.290205 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.289640    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 17:35:56.290205 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.289642    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 17:35:56.290205 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.289645    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 17:35:56.290205 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.289647    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 17:35:56.290205 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.289650    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 17:35:56.290205 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.289652    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 17:35:56.290205 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.289662    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 17:35:56.290205 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.289665    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 17:35:56.290205 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.289669    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 17:35:56.290205 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.289673    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 17:35:56.290205 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.289678    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 17:35:56.290205 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.289680    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 17:35:56.290205 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.289683    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 17:35:56.290205 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.289686    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 17:35:56.290205 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.289689    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 17:35:56.290205 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.289692    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 17:35:56.290205 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.289694    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 17:35:56.290205 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.289697    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 17:35:56.290205 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.289699    2579 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 17:35:56.290718 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.289702    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 17:35:56.290718 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.289705    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 17:35:56.290718 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.289707    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 17:35:56.290718 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.289710    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 17:35:56.290718 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.289712    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 17:35:56.290718 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.289715    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 17:35:56.290718 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.289717    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 17:35:56.290718 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.289720    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 17:35:56.290718 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.289723    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 17:35:56.290718 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.289725    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 17:35:56.290718 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.289728    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 17:35:56.290718 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.289730    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 17:35:56.290718 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.289734    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 17:35:56.290718 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.289738    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 17:35:56.290718 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.289741    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 17:35:56.290718 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.289745    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 17:35:56.290718 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.289748    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 17:35:56.290718 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.289751    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 17:35:56.290718 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.289754    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 17:35:56.291169 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.289756    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 17:35:56.291169 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.289759    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 17:35:56.291169 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.289761    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 17:35:56.291169 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.289764    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 17:35:56.291169 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.289767    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 17:35:56.291169 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.289769    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 17:35:56.291169 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.289772    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 17:35:56.291169 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.289775    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 17:35:56.291169 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.289778    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 17:35:56.291169 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.289781    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 17:35:56.291169 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.289783    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 17:35:56.291169 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.289786    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 17:35:56.291169 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.289789    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 17:35:56.291169 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.289792    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 17:35:56.291169 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.289794    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 17:35:56.291169 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.289797    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 17:35:56.291169 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.289799    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 17:35:56.291169 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.289801    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 17:35:56.291169 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.289804    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 17:35:56.291169 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.289807    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 17:35:56.291676 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.289810    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 17:35:56.291676 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.289812    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 17:35:56.291676 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.289815    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 17:35:56.291676 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.289818    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 17:35:56.291676 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.289820    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 17:35:56.291676 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.289823    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 17:35:56.291676 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.289825    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 17:35:56.291676 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.289828    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 17:35:56.291676 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.289830    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 17:35:56.291676 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.289833    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 17:35:56.291676 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.289835    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 17:35:56.291676 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.289838    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 17:35:56.291676 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.289840    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 17:35:56.291676 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.289843    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 17:35:56.291676 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.289845    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 17:35:56.291676 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.289848    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 17:35:56.291676 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.289851    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 17:35:56.291676 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.290557    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 17:35:56.291676 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.290588    2579 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 17:35:56.291676 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.290593    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 17:35:56.292158 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.290596    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 17:35:56.292158 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.290599    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 17:35:56.292158 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.290602    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 17:35:56.292158 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.290605    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 17:35:56.292158 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.290607    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 17:35:56.292158 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.290610    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 17:35:56.292158 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.290613    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 17:35:56.292158 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.290615    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 17:35:56.292158 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.290618    2579 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 17:35:56.292158 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.290621    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 17:35:56.292158 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.290623    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 17:35:56.292158 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.290626    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 17:35:56.292158 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.290629    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 17:35:56.292158 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.290631    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 17:35:56.292158 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.290635    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 17:35:56.292158 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.290637    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 17:35:56.292158 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.290640    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 17:35:56.292158 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.290643    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 17:35:56.292158 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.290645    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 17:35:56.292158 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.290648    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 17:35:56.292671 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.290650    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 17:35:56.292671 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.290653    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 17:35:56.292671 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.290655    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 17:35:56.292671 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.290657    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 17:35:56.292671 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.290660    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 17:35:56.292671 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.290664    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 17:35:56.292671 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.290668    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 17:35:56.292671 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.290671    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 17:35:56.292671 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.290674    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 17:35:56.292671 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.290677    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 17:35:56.292671 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.290680    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 17:35:56.292671 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.290683    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 17:35:56.292671 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.290685    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 17:35:56.292671 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.290688    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 17:35:56.292671 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.290690    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 17:35:56.292671 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.290693    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 17:35:56.292671 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.290696    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 17:35:56.292671 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.290699    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 17:35:56.292671 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.290701    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 17:35:56.293134 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.290704    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 17:35:56.293134 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.290706    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 17:35:56.293134 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.290709    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 17:35:56.293134 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.290711    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 17:35:56.293134 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.290714    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 17:35:56.293134 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.290716    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 17:35:56.293134 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.290719    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 17:35:56.293134 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.290723    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 17:35:56.293134 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.290726    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 17:35:56.293134 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.290729    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 17:35:56.293134 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.290732    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 17:35:56.293134 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.290734    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 17:35:56.293134 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.290737    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 17:35:56.293134 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.290739    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 17:35:56.293134 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.290742    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 17:35:56.293134 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.290746    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 17:35:56.293134 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.290749    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 17:35:56.293134 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.290753    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 17:35:56.293134 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.290755    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 17:35:56.293646 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.290758    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 17:35:56.293646 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.290760    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 17:35:56.293646 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.290764    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 17:35:56.293646 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.290768    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 17:35:56.293646 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.290771    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 17:35:56.293646 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.290774    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 17:35:56.293646 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.290777    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 17:35:56.293646 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.290780    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 17:35:56.293646 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.290784    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 17:35:56.293646 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.290786    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 17:35:56.293646 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.290789    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 17:35:56.293646 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.290791    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 17:35:56.293646 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.290794    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 17:35:56.293646 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.290796    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 17:35:56.293646 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.290799    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 17:35:56.293646 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.290801    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 17:35:56.293646 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.290803    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 17:35:56.293646 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.290808    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 17:35:56.293646 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.290811    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 17:35:56.293646 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.290813    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 17:35:56.294114 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.290816    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 17:35:56.294114 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.290818    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 17:35:56.294114 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.290822    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 17:35:56.294114 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.290825    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 17:35:56.294114 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.290827    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 17:35:56.294114 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.290830    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 17:35:56.294114 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.290832    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 17:35:56.294114 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.290835    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 17:35:56.294114 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.290837    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 17:35:56.294114 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.290840    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 17:35:56.294114 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.290842    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 17:35:56.294114 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.290845    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 17:35:56.294114 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.290847    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 17:35:56.294114 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.290850    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 17:35:56.294114 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.290853    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 17:35:56.294114 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.290856    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 17:35:56.294114 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.290859    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 17:35:56.294114 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.290861    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 17:35:56.294114 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.291244    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 17:35:56.294114 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.291250    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 17:35:56.294620 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.291253    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 17:35:56.294620 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.291256    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 17:35:56.294620 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.291259    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 17:35:56.294620 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.291263    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 17:35:56.294620 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.291267    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 17:35:56.294620 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.291270    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 17:35:56.294620 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.291273    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 17:35:56.294620 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.291275    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 17:35:56.294620 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.291278    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 17:35:56.294620 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.291283    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 17:35:56.294620 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.291285    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 17:35:56.294620 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.291288    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 17:35:56.294620 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.291290    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 17:35:56.294620 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.291293    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 17:35:56.294620 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.291296    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 17:35:56.294620 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.291298    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 17:35:56.294620 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.291301    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 17:35:56.294620 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.291303    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 17:35:56.294620 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.291306    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 17:35:56.294620 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.291308    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 17:35:56.295185 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.291311    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 17:35:56.295185 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.291314    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 17:35:56.295185 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.291317    2579 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 17:35:56.295185 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.291319    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 17:35:56.295185 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.291322    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 17:35:56.295185 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.291327    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 17:35:56.295185 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.291330    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 17:35:56.295185 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.291333    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 17:35:56.295185 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.291336    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 17:35:56.295185 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.291339    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 17:35:56.295185 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.291341    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 17:35:56.295185 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.291344    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 17:35:56.295185 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.291347    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 17:35:56.295185 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.291350    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 17:35:56.295185 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.291352    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 17:35:56.295185 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.291355    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 17:35:56.295185 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.291357    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 17:35:56.295185 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.291360    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 17:35:56.295185 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.291362    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 17:35:56.295185 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.291365    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 17:35:56.295700 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.291367    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 17:35:56.295700 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.291371    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 17:35:56.295700 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.291374    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 17:35:56.295700 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.291376    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 17:35:56.295700 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.291379    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 17:35:56.295700 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.291381    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 17:35:56.295700 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.291384    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 17:35:56.295700 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.291387    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 17:35:56.295700 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.291390    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 17:35:56.295700 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.291392    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 17:35:56.295700 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.291395    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 17:35:56.295700 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.291397    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 17:35:56.295700 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.291399    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 17:35:56.295700 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.291402    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 17:35:56.295700 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.291405    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 17:35:56.295700 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.291407    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 17:35:56.295700 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.291409    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 17:35:56.295700 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.291413    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 17:35:56.295700 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.291417    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 17:35:56.296151 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.291420    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 17:35:56.296151 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.291423    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 17:35:56.296151 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.291426    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 17:35:56.296151 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.291429    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 17:35:56.296151 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.291432    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 17:35:56.296151 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.291435    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 17:35:56.296151 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.291438    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 17:35:56.296151 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.291441    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 17:35:56.296151 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.291444    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 17:35:56.296151 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.291446    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 17:35:56.296151 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.291449    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 17:35:56.296151 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.291451    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 17:35:56.296151 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.291454    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 17:35:56.296151 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.291456    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 17:35:56.296151 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.291460    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 17:35:56.296151 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.291462    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 17:35:56.296151 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.291465    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 17:35:56.296151 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.291467    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 17:35:56.296151 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.291470    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 17:35:56.296151 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.291472    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 17:35:56.296648 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.291475    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 17:35:56.296648 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.291478    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 17:35:56.296648 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.291480    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 17:35:56.296648 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.291483    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 17:35:56.296648 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.291485    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 17:35:56.296648 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.291488    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 17:35:56.296648 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.291491    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 17:35:56.296648 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.291493    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 17:35:56.296648 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.291495    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 17:35:56.296648 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.291507    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 17:35:56.296648 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.291511    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 17:35:56.296648 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.291514    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 17:35:56.296648 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.291517    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 17:35:56.296648 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.291519    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 17:35:56.296648 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.291522    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 17:35:56.296648 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.291524    2579 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 17:35:56.296648 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.291527    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 17:35:56.296648 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.291529    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 17:35:56.296648 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.291653    2579 flags.go:64] FLAG: --address="0.0.0.0"
Sep 10 17:35:56.296648 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.291690    2579 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 10 17:35:56.296648 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.291696    2579 flags.go:64] FLAG: --anonymous-auth="true"
Sep 10 17:35:56.297168 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.291700    2579 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 10 17:35:56.297168 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.291704    2579 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 10 17:35:56.297168 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.291707    2579 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 10 17:35:56.297168 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.291711    2579 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 10 17:35:56.297168 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.291715    2579 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 10 17:35:56.297168 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.291719    2579 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 10 17:35:56.297168 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.291721    2579 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 10 17:35:56.297168 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.291725    2579 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 10 17:35:56.297168 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.291728    2579 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 10 17:35:56.297168 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.291731    2579 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 10 17:35:56.297168 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.291734    2579 flags.go:64] FLAG: --cgroup-root=""
Sep 10 17:35:56.297168 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.291737    2579 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 10 17:35:56.297168 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.291739    2579 flags.go:64] FLAG: --client-ca-file=""
Sep 10 17:35:56.297168 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.291743    2579 flags.go:64] FLAG: --cloud-provider="external"
Sep 10 17:35:56.297168 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.291746    2579 flags.go:64] FLAG: --cluster-dns="[]"
Sep 10 17:35:56.297168 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.291751    2579 flags.go:64] FLAG: --cluster-domain=""
Sep 10 17:35:56.297168 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.291754    2579 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 10 17:35:56.297168 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.291757    2579 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 10 17:35:56.297168 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.291760    2579 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 10 17:35:56.297168 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.291763    2579 flags.go:64] FLAG: --container-log-max-files="5"
Sep 10 17:35:56.297168 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.291767    2579 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 10 17:35:56.297168 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.291770    2579 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 10 17:35:56.297168 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.291773    2579 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 10 17:35:56.297727 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.291776    2579 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 10 17:35:56.297727 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.291779    2579 flags.go:64] FLAG: --contention-profiling="false"
Sep 10 17:35:56.297727 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.291782    2579 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 10 17:35:56.297727 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.291785    2579 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 10 17:35:56.297727 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.291788    2579 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 10 17:35:56.297727 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.291791    2579 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 10 17:35:56.297727 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.291795    2579 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 10 17:35:56.297727 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.291798    2579 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 10 17:35:56.297727 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.291801    2579 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 10 17:35:56.297727 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.291804    2579 flags.go:64] FLAG: --enable-load-reader="false"
Sep 10 17:35:56.297727 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.291807    2579 flags.go:64] FLAG: --enable-server="true"
Sep 10 17:35:56.297727 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.291809    2579 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 10 17:35:56.297727 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.291813    2579 flags.go:64] FLAG: --event-burst="100"
Sep 10 17:35:56.297727 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.291816    2579 flags.go:64] FLAG: --event-qps="50"
Sep 10 17:35:56.297727 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.291819    2579 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 10 17:35:56.297727 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.291822    2579 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 10 17:35:56.297727 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.291825    2579 flags.go:64] FLAG: --eviction-hard=""
Sep 10 17:35:56.297727 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.291828    2579 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 10 17:35:56.297727 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.291831    2579 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 10 17:35:56.297727 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.291834    2579 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 10 17:35:56.297727 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.291837    2579 flags.go:64] FLAG: --eviction-soft=""
Sep 10 17:35:56.297727 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.291840    2579 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 10 17:35:56.297727 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.291843    2579 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 10 17:35:56.297727 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.291846    2579 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 10 17:35:56.297727 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.291849    2579 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 10 17:35:56.298346 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.291852    2579 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 10 17:35:56.298346 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.291854    2579 flags.go:64] FLAG: --fail-swap-on="true"
Sep 10 17:35:56.298346 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.291857    2579 flags.go:64] FLAG: --feature-gates=""
Sep 10 17:35:56.298346 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.291861    2579 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 10 17:35:56.298346 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.291863    2579 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 10 17:35:56.298346 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.291881    2579 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 10 17:35:56.298346 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.291885    2579 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 10 17:35:56.298346 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.291888    2579 flags.go:64] FLAG: --healthz-port="10248"
Sep 10 17:35:56.298346 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.291892    2579 flags.go:64] FLAG: --help="false"
Sep 10 17:35:56.298346 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.291894    2579 flags.go:64] FLAG: --hostname-override="ip-10-0-129-122.ec2.internal"
Sep 10 17:35:56.298346 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.291898    2579 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 10 17:35:56.298346 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.291901    2579 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 10 17:35:56.298346 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.291904    2579 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 10 17:35:56.298346 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.291907    2579 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 10 17:35:56.298346 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.291910    2579 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 10 17:35:56.298346 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.291913    2579 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 10 17:35:56.298346 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.291916    2579 flags.go:64] FLAG: --image-service-endpoint=""
Sep 10 17:35:56.298346 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.291919    2579 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 10 17:35:56.298346 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.291921    2579 flags.go:64] FLAG: --kube-api-burst="100"
Sep 10 17:35:56.298346 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.291924    2579 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 10 17:35:56.298346 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.291927    2579 flags.go:64] FLAG: --kube-api-qps="50"
Sep 10 17:35:56.298346 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.291930    2579 flags.go:64] FLAG: --kube-reserved=""
Sep 10 17:35:56.298346 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.291933    2579 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 10 17:35:56.298346 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.291935    2579 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 10 17:35:56.298926 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.291938    2579 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 10 17:35:56.298926 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.291941    2579 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 10 17:35:56.298926 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.291943    2579 flags.go:64] FLAG: --lock-file=""
Sep 10 17:35:56.298926 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.291946    2579 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 10 17:35:56.298926 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.291949    2579 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 10 17:35:56.298926 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.291952    2579 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 10 17:35:56.298926 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.291957    2579 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 10 17:35:56.298926 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.291960    2579 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 10 17:35:56.298926 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.291963    2579 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 10 17:35:56.298926 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.291966    2579 flags.go:64] FLAG: --logging-format="text"
Sep 10 17:35:56.298926 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.291969    2579 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 10 17:35:56.298926 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.291972    2579 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 10 17:35:56.298926 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.291975    2579 flags.go:64] FLAG: --manifest-url=""
Sep 10 17:35:56.298926 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.291977    2579 flags.go:64] FLAG: --manifest-url-header=""
Sep 10 17:35:56.298926 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.291981    2579 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 10 17:35:56.298926 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.291984    2579 flags.go:64] FLAG: --max-open-files="1000000"
Sep 10 17:35:56.298926 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.291988    2579 flags.go:64] FLAG: --max-pods="110"
Sep 10 17:35:56.298926 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.291991    2579 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 10 17:35:56.298926 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.291994    2579 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 10 17:35:56.298926 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.291997    2579 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 10 17:35:56.298926 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.292000    2579 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 10 17:35:56.298926 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.292003    2579 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 10 17:35:56.298926 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.292005    2579 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 10 17:35:56.298926 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.292008    2579 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 10 17:35:56.298926 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.292016    2579 flags.go:64] FLAG: --node-status-max-images="50"
Sep 10 17:35:56.299548 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.292019    2579 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 10 17:35:56.299548 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.292022    2579 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 10 17:35:56.299548 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.292025    2579 flags.go:64] FLAG: --pod-cidr=""
Sep 10 17:35:56.299548 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.292028    2579 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ccbacc999dbf9abad8f4009240188312b683f36ced4763eed166c1500c164b1"
Sep 10 17:35:56.299548 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.292034    2579 flags.go:64] FLAG: --pod-manifest-path=""
Sep 10 17:35:56.299548 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.292036    2579 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 10 17:35:56.299548 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.292039    2579 flags.go:64] FLAG: --pods-per-core="0"
Sep 10 17:35:56.299548 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.292042    2579 flags.go:64] FLAG: --port="10250"
Sep 10 17:35:56.299548 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.292045    2579 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 10 17:35:56.299548 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.292048    2579 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-060f857167635542b"
Sep 10 17:35:56.299548 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.292051    2579 flags.go:64] FLAG: --qos-reserved=""
Sep 10 17:35:56.299548 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.292053    2579 flags.go:64] FLAG: --read-only-port="10255"
Sep 10 17:35:56.299548 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.292058    2579 flags.go:64] FLAG: --register-node="true"
Sep 10 17:35:56.299548 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.292061    2579 flags.go:64] FLAG: --register-with-taints=""
Sep 10 17:35:56.299548 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.292064    2579 flags.go:64] FLAG: --registry-burst="10"
Sep 10 17:35:56.299548 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.292068    2579 flags.go:64] FLAG: --registry-qps="5"
Sep 10 17:35:56.299548 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.292071    2579 flags.go:64] FLAG: --reserved-cpus=""
Sep 10 17:35:56.299548 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.292074    2579 flags.go:64] FLAG: --reserved-memory=""
Sep 10 17:35:56.299548 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.292077    2579 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 10 17:35:56.299548 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.292080    2579 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 10 17:35:56.299548 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.292083    2579 flags.go:64] FLAG: --rotate-certificates="false"
Sep 10 17:35:56.299548 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.292086    2579 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 10 17:35:56.299548 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.292089    2579 flags.go:64] FLAG: --runonce="false"
Sep 10 17:35:56.299548 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.292091    2579 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 10 17:35:56.299548 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.292094    2579 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 10 17:35:56.300214 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.292097    2579 flags.go:64] FLAG: --seccomp-default="false"
Sep 10 17:35:56.300214 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.292100    2579 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 10 17:35:56.300214 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.292102    2579 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 10 17:35:56.300214 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.292105    2579 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 10 17:35:56.300214 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.292108    2579 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 10 17:35:56.300214 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.292111    2579 flags.go:64] FLAG: --storage-driver-password="root"
Sep 10 17:35:56.300214 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.292113    2579 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 10 17:35:56.300214 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.292116    2579 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 10 17:35:56.300214 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.292119    2579 flags.go:64] FLAG: --storage-driver-user="root"
Sep 10 17:35:56.300214 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.292122    2579 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 10 17:35:56.300214 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.292124    2579 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 10 17:35:56.300214 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.292127    2579 flags.go:64] FLAG: --system-cgroups=""
Sep 10 17:35:56.300214 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.292130    2579 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 10 17:35:56.300214 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.292135    2579 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 10 17:35:56.300214 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.292138    2579 flags.go:64] FLAG: --tls-cert-file=""
Sep 10 17:35:56.300214 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.292140    2579 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 10 17:35:56.300214 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.292147    2579 flags.go:64] FLAG: --tls-min-version=""
Sep 10 17:35:56.300214 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.292149    2579 flags.go:64] FLAG: --tls-private-key-file=""
Sep 10 17:35:56.300214 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.292152    2579 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 10 17:35:56.300214 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.292156    2579 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 10 17:35:56.300214 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.292159    2579 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 10 17:35:56.300214 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.292162    2579 flags.go:64] FLAG: --v="2"
Sep 10 17:35:56.300214 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.292166    2579 flags.go:64] FLAG: --version="false"
Sep 10 17:35:56.300214 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.292171    2579 flags.go:64] FLAG: --vmodule=""
Sep 10 17:35:56.300214 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.292175    2579 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 10 17:35:56.300856 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.292178    2579 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 10 17:35:56.300856 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.292295    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 17:35:56.300856 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.292299    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 17:35:56.300856 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.292302    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 17:35:56.300856 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.292305    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 17:35:56.300856 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.292308    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 17:35:56.300856 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.292311    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 17:35:56.300856 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.292314    2579 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 17:35:56.300856 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.292317    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 17:35:56.300856 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.292320    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 17:35:56.300856 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.292322    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 17:35:56.300856 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.292325    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 17:35:56.300856 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.292327    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 17:35:56.300856 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.292330    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 17:35:56.300856 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.292333    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 17:35:56.300856 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.292335    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 17:35:56.300856 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.292338    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 17:35:56.300856 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.292342    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 17:35:56.300856 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.292345    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 17:35:56.300856 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.292348    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 17:35:56.301334 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.292351    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 17:35:56.301334 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.292354    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 17:35:56.301334 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.292356    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 17:35:56.301334 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.292359    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 17:35:56.301334 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.292361    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 17:35:56.301334 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.292364    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 17:35:56.301334 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.292367    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 17:35:56.301334 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.292370    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 17:35:56.301334 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.292373    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 17:35:56.301334 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.292375    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 17:35:56.301334 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.292378    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 17:35:56.301334 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.292382    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 17:35:56.301334 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.292385    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 17:35:56.301334 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.292387    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 17:35:56.301334 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.292390    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 17:35:56.301334 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.292392    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 17:35:56.301334 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.292395    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 17:35:56.301334 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.292397    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 17:35:56.301334 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.292400    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 17:35:56.301334 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.292402    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 17:35:56.301845 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.292405    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 17:35:56.301845 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.292408    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 17:35:56.301845 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.292410    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 17:35:56.301845 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.292413    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 17:35:56.301845 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.292415    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 17:35:56.301845 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.292417    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 17:35:56.301845 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.292420    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 17:35:56.301845 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.292422    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 17:35:56.301845 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.292425    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 17:35:56.301845 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.292427    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 17:35:56.301845 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.292430    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 17:35:56.301845 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.292432    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 17:35:56.301845 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.292435    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 17:35:56.301845 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.292437    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 17:35:56.301845 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.292440    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 17:35:56.301845 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.292442    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 17:35:56.301845 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.292445    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 17:35:56.301845 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.292447    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 17:35:56.301845 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.292451    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 17:35:56.301845 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.292453    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 17:35:56.302329 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.292456    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 17:35:56.302329 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.292458    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 17:35:56.302329 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.292462    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 17:35:56.302329 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.292467    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 17:35:56.302329 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.292471    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 17:35:56.302329 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.292474    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 17:35:56.302329 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.292476    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 17:35:56.302329 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.292479    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 17:35:56.302329 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.292482    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 17:35:56.302329 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.292485    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 17:35:56.302329 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.292487    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 17:35:56.302329 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.292490    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 17:35:56.302329 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.292493    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 17:35:56.302329 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.292496    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 17:35:56.302329 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.292498    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 17:35:56.302329 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.292501    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 17:35:56.302329 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.292503    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 17:35:56.302329 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.292506    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 17:35:56.302329 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.292508    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 17:35:56.302329 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.292511    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 17:35:56.302827 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.292513    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 17:35:56.302827 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.292516    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 17:35:56.302827 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.292519    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 17:35:56.302827 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.292521    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 17:35:56.302827 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.292524    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 17:35:56.302827 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.292526    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 17:35:56.302827 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.292528    2579 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 17:35:56.302827 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.292531    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 17:35:56.302827 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.292534    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 17:35:56.302827 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.292536    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 17:35:56.302827 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.292540    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 17:35:56.302827 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.292542    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 17:35:56.302827 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.292545    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 17:35:56.302827 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.292547    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 17:35:56.302827 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.292549    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 17:35:56.302827 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.292553    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 17:35:56.302827 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.292556    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 17:35:56.302827 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.292579    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 17:35:56.302827 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.292585    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 17:35:56.303285 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.292588    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 17:35:56.303285 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.293290    2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 10 17:35:56.303285 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.301514    2579 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 10 17:35:56.303285 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.301528    2579 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 10 17:35:56.303285 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.301541    2579 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 10 17:35:56.303285 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.301547    2579 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 10 17:35:56.303285 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.301627    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 17:35:56.303285 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.301634    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 17:35:56.303285 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.301637    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 17:35:56.303285 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.301641    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 17:35:56.303285 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.301644    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 17:35:56.303285 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.301647    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 17:35:56.303285 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.301649    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 17:35:56.303285 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.301652    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 17:35:56.303714 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.301655    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 17:35:56.303714 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.301658    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 17:35:56.303714 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.301661    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 17:35:56.303714 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.301663    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 17:35:56.303714 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.301665    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 17:35:56.303714 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.301668    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 17:35:56.303714 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.301670    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 17:35:56.303714 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.301673    2579 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 17:35:56.303714 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.301675    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 17:35:56.303714 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.301678    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 17:35:56.303714 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.301681    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 17:35:56.303714 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.301683    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 17:35:56.303714 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.301686    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 17:35:56.303714 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.301688    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 17:35:56.303714 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.301691    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 17:35:56.303714 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.301694    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 17:35:56.303714 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.301696    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 17:35:56.303714 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.301699    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 17:35:56.303714 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.301701    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 17:35:56.303714 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.301704    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 17:35:56.304199 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.301707    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 17:35:56.304199 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.301710    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 17:35:56.304199 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.301712    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 17:35:56.304199 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.301715    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 17:35:56.304199 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.301717    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 17:35:56.304199 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.301720    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 17:35:56.304199 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.301722    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 17:35:56.304199 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.301725    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 17:35:56.304199 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.301729    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 17:35:56.304199 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.301732    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 17:35:56.304199 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.301735    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 17:35:56.304199 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.301737    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 17:35:56.304199 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.301740    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 17:35:56.304199 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.301742    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 17:35:56.304199 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.301745    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 17:35:56.304199 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.301747    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 17:35:56.304199 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.301750    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 17:35:56.304199 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.301752    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 17:35:56.304199 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.301755    2579 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 17:35:56.304199 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.301758    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 17:35:56.304700 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.301760    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 17:35:56.304700 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.301763    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 17:35:56.304700 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.301765    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 17:35:56.304700 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.301768    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 17:35:56.304700 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.301771    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 17:35:56.304700 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.301774    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 17:35:56.304700 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.301777    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 17:35:56.304700 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.301780    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 17:35:56.304700 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.301782    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 17:35:56.304700 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.301785    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 17:35:56.304700 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.301787    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 17:35:56.304700 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.301790    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 17:35:56.304700 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.301792    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 17:35:56.304700 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.301795    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 17:35:56.304700 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.301798    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 17:35:56.304700 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.301801    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 17:35:56.304700 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.301803    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 17:35:56.304700 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.301806    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 17:35:56.304700 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.301808    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 17:35:56.304700 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.301811    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 17:35:56.305170 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.301814    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 17:35:56.305170 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.301816    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 17:35:56.305170 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.301819    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 17:35:56.305170 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.301821    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 17:35:56.305170 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.301824    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 17:35:56.305170 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.301826    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 17:35:56.305170 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.301829    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 17:35:56.305170 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.301831    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 17:35:56.305170 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.301835    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 17:35:56.305170 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.301837    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 17:35:56.305170 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.301840    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 17:35:56.305170 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.301842    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 17:35:56.305170 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.301845    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 17:35:56.305170 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.301847    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 17:35:56.305170 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.301851    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 17:35:56.305170 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.301855    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 17:35:56.305170 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.301858    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 17:35:56.305170 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.301861    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 17:35:56.305170 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.301864    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 17:35:56.305641 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.301867    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 17:35:56.305641 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.301870    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 17:35:56.305641 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.301873    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 17:35:56.305641 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.301875    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 17:35:56.305641 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.301878    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 17:35:56.305641 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.301880    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 17:35:56.305641 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.301883    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 17:35:56.305641 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.301885    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 17:35:56.305641 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.301888    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 17:35:56.305641 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.301891    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 17:35:56.305641 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.301893    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 17:35:56.305641 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.301896    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 17:35:56.305641 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.301901    2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 10 17:35:56.305641 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.302045    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 17:35:56.305641 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.302050    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 17:35:56.306024 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.302054    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 17:35:56.306024 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.302057    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 17:35:56.306024 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.302059    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 17:35:56.306024 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.302062    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 17:35:56.306024 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.302064    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 17:35:56.306024 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.302067    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 17:35:56.306024 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.302070    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 17:35:56.306024 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.302072    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 17:35:56.306024 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.302075    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 17:35:56.306024 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.302077    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 17:35:56.306024 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.302080    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 17:35:56.306024 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.302082    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 17:35:56.306024 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.302086    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 17:35:56.306024 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.302090    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 17:35:56.306024 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.302093    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 17:35:56.306024 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.302096    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 17:35:56.306024 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.302099    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 17:35:56.306024 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.302101    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 17:35:56.306024 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.302104    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 17:35:56.306024 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.302107    2579 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 17:35:56.306494 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.302110    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 17:35:56.306494 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.302113    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 17:35:56.306494 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.302115    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 17:35:56.306494 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.302118    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 17:35:56.306494 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.302121    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 17:35:56.306494 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.302123    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 17:35:56.306494 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.302132    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 17:35:56.306494 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.302135    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 17:35:56.306494 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.302137    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 17:35:56.306494 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.302140    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 17:35:56.306494 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.302142    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 17:35:56.306494 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.302144    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 17:35:56.306494 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.302147    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 17:35:56.306494 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.302149    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 17:35:56.306494 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.302152    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 17:35:56.306494 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.302154    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 17:35:56.306494 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.302157    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 17:35:56.306494 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.302159    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 17:35:56.306494 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.302161    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 17:35:56.306494 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.302164    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 17:35:56.306972 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.302166    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 17:35:56.306972 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.302169    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 17:35:56.306972 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.302171    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 17:35:56.306972 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.302173    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 17:35:56.306972 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.302176    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 17:35:56.306972 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.302178    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 17:35:56.306972 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.302180    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 17:35:56.306972 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.302183    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 17:35:56.306972 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.302186    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 17:35:56.306972 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.302188    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 17:35:56.306972 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.302191    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 17:35:56.306972 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.302193    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 17:35:56.306972 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.302196    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 17:35:56.306972 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.302198    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 17:35:56.306972 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.302201    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 17:35:56.306972 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.302203    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 17:35:56.306972 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.302205    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 17:35:56.306972 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.302208    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 17:35:56.306972 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.302210    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 17:35:56.307451 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.302222    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 17:35:56.307451 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.302225    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 17:35:56.307451 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.302227    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 17:35:56.307451 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.302229    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 17:35:56.307451 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.302233    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 17:35:56.307451 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.302236    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 17:35:56.307451 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.302238    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 17:35:56.307451 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.302241    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 17:35:56.307451 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.302243    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 17:35:56.307451 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.302246    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 17:35:56.307451 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.302248    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 17:35:56.307451 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.302251    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 17:35:56.307451 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.302253    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 17:35:56.307451 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.302255    2579 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 17:35:56.307451 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.302258    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 17:35:56.307451 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.302260    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 17:35:56.307451 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.302263    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 17:35:56.307451 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.302265    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 17:35:56.307451 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.302268    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 17:35:56.307451 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.302270    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 17:35:56.307944 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.302273    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 17:35:56.307944 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.302275    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 17:35:56.307944 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.302277    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 17:35:56.307944 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.302280    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 17:35:56.307944 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.302282    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 17:35:56.307944 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.302285    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 17:35:56.307944 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.302287    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 17:35:56.307944 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.302290    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 17:35:56.307944 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.302293    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 17:35:56.307944 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.302297    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 17:35:56.307944 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.302300    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 17:35:56.307944 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.302302    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 17:35:56.307944 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.302305    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 17:35:56.307944 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.302313    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 17:35:56.307944 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.302316    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 17:35:56.307944 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.302318    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 17:35:56.307944 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.302320    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 17:35:56.307944 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:56.302323    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 17:35:56.308394 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.302328    2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 10 17:35:56.308394 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.302435    2579 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 10 17:35:56.308394 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.305197    2579 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 10 17:35:56.308394 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.306303    2579 server.go:1019] "Starting client certificate rotation"
Sep 10 17:35:56.308394 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.306401    2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 10 17:35:56.308394 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.306439    2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 10 17:35:56.309273 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.309261    2579 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 10 17:35:56.310597 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.310560    2579 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 10 17:35:56.319805 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.319787    2579 log.go:25] "Validated CRI v1 runtime API"
Sep 10 17:35:56.325445 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.325423    2579 log.go:25] "Validated CRI v1 image API"
Sep 10 17:35:56.328492 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.328472    2579 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 10 17:35:56.332827 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.332811    2579 fs.go:135] Filesystem UUIDs: map[1d833e8a-e46c-4a52-9ebd-0efa157f4fd6:/dev/nvme0n1p3 5466f4f3-3e62-4756-84ea-214e643fce38:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2]
Sep 10 17:35:56.332891 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.332828    2579 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 10 17:35:56.337631 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.337502    2579 manager.go:216] Machine: {Timestamp:2026-09-10 17:35:56.336286534 +0000 UTC m=+0.336116591 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099890 MemoryCapacity:32812163072 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2fe8c660626064cb045dceabebeb2b SystemUUID:ec2fe8c6-6062-6064-cb04-5dceabebeb2b BootID:3cbaf042-e3bf-491f-8dcb-b9e106394bf6 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406081536 Type:vfs Inodes:4005391 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406081536 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:29:a6:dc:f2:e7 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:29:a6:dc:f2:e7 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:1a:66:55:8f:3d:24 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812163072 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 10 17:35:56.337631 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.337629    2579 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 10 17:35:56.337728 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.337701    2579 manager.go:232] Version: {KernelVersion:5.14.0-570.138.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260901-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 10 17:35:56.340024 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.340002    2579 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 10 17:35:56.340169 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.340026    2579 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-129-122.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 10 17:35:56.341284 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.341268    2579 topology_manager.go:138] "Creating topology manager with none policy"
Sep 10 17:35:56.341322 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.341294    2579 container_manager_linux.go:309] "Creating device plugin manager"
Sep 10 17:35:56.341322 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.341307    2579 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 10 17:35:56.341721 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.341706    2579 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 10 17:35:56.342113 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.342101    2579 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 10 17:35:56.342827 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.342817    2579 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 10 17:35:56.346493 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.346481    2579 state_mem.go:36] "Initialized new in-memory state store"
Sep 10 17:35:56.346625 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.346616    2579 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 10 17:35:56.346659 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.346654    2579 kubelet.go:486] "Attempting to sync node with API server"
Sep 10 17:35:56.346694 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.346664    2579 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 10 17:35:56.346694 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.346677    2579 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 10 17:35:56.346694 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.346686    2579 kubelet.go:398] "Adding apiserver pod source"
Sep 10 17:35:56.346694 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.346695    2579 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 10 17:35:56.347119 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.347107    2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 10 17:35:56.347159 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.347130    2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 10 17:35:56.349087 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.349070    2579 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 10 17:35:56.351347 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.351334    2579 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 10 17:35:56.351402 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.351365    2579 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 10 17:35:56.353239 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.353225    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 10 17:35:56.353304 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.353243    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 10 17:35:56.353304 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.353249    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 10 17:35:56.353304 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.353254    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 10 17:35:56.353304 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.353263    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 10 17:35:56.353304 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.353268    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 10 17:35:56.353304 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.353282    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 10 17:35:56.353304 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.353289    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 10 17:35:56.353304 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.353295    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 10 17:35:56.353304 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.353300    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 10 17:35:56.353304 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.353309    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 10 17:35:56.353604 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.353314    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 10 17:35:56.354098 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.354085    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 10 17:35:56.354098 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.354099    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 10 17:35:56.357435 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.357422    2579 server.go:1268] "Started kubelet"
Sep 10 17:35:56.357607 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.357535    2579 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 10 17:35:56.357695 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.357670    2579 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 10 17:35:56.357747 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.357713    2579 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 10 17:35:56.358332 ip-10-0-129-122 systemd[1]: Started Kubernetes Kubelet.
Sep 10 17:35:56.358886 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.358863    2579 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 10 17:35:56.358975 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:35:56.358870    2579 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 10 17:35:56.359127 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.358861    2579 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-129-122.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 10 17:35:56.359127 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:35:56.358999    2579 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-129-122.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 10 17:35:56.360182 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.360053    2579 server.go:310] "Adding debug handlers to kubelet server"
Sep 10 17:35:56.362216 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:35:56.362197    2579 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 10 17:35:56.364931 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.364915    2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 10 17:35:56.365668 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.365652    2579 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 10 17:35:56.365806 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:35:56.365791    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-122.ec2.internal\" not found"
Sep 10 17:35:56.366703 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.366607    2579 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 10 17:35:56.366703 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.366704    2579 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 10 17:35:56.366844 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:35:56.364858    2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-122.ec2.internal.18d4063044eb4978  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-122.ec2.internal,UID:ip-10-0-129-122.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-129-122.ec2.internal,},FirstTimestamp:2026-09-10 17:35:56.357396856 +0000 UTC m=+0.357226915,LastTimestamp:2026-09-10 17:35:56.357396856 +0000 UTC m=+0.357226915,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-122.ec2.internal,}"
Sep 10 17:35:56.366943 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.366602    2579 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 10 17:35:56.367030 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.366960    2579 reconstruct.go:97] "Volume reconstruction finished"
Sep 10 17:35:56.367132 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.367019    2579 reconciler.go:29] "Reconciler: start to sync state"
Sep 10 17:35:56.367667 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.367648    2579 factory.go:55] Registering systemd factory
Sep 10 17:35:56.367667 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.367668    2579 factory.go:223] Registration of the systemd container factory successfully
Sep 10 17:35:56.367933 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.367919    2579 factory.go:153] Registering CRI-O factory
Sep 10 17:35:56.367933 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.367931    2579 factory.go:223] Registration of the crio container factory successfully
Sep 10 17:35:56.368096 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.368019    2579 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 10 17:35:56.368096 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.368039    2579 factory.go:103] Registering Raw factory
Sep 10 17:35:56.368096 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.368090    2579 manager.go:1196] Started watching for new ooms in manager
Sep 10 17:35:56.368694 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.368676    2579 manager.go:349] Starting recovery of all containers
Sep 10 17:35:56.373945 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:35:56.373912    2579 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-129-122.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 10 17:35:56.374046 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:35:56.374023    2579 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 10 17:35:56.378309 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.378172    2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 10 17:35:56.379303 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.379282    2579 manager.go:354] Recovery completed
Sep 10 17:35:56.384774 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.384759    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 17:35:56.386486 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.386462    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-122.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 17:35:56.386546 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.386501    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-122.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 17:35:56.386546 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.386511    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-122.ec2.internal" event="NodeHasSufficientPID"
Sep 10 17:35:56.387044 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.387031    2579 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 10 17:35:56.387093 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.387045    2579 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 10 17:35:56.387093 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.387045    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 17:35:56.387093 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.387060    2579 state_mem.go:36] "Initialized new in-memory state store"
Sep 10 17:35:56.387750 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.387733    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-122.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 17:35:56.387811 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.387762    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-122.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 17:35:56.387811 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.387777    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-122.ec2.internal" event="NodeHasSufficientPID"
Sep 10 17:35:56.388468 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:35:56.388401    2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-122.ec2.internal.18d4063046a72836  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-122.ec2.internal,UID:ip-10-0-129-122.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-129-122.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-129-122.ec2.internal,},FirstTimestamp:2026-09-10 17:35:56.386486326 +0000 UTC m=+0.386316391,LastTimestamp:2026-09-10 17:35:56.386486326 +0000 UTC m=+0.386316391,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-122.ec2.internal,}"
Sep 10 17:35:56.389252 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.389241    2579 policy_none.go:49] "None policy: Start"
Sep 10 17:35:56.389286 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.389257    2579 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 10 17:35:56.389286 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.389266    2579 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 10 17:35:56.391178 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.391167    2579 policy_none.go:47] "Start"
Sep 10 17:35:56.393644 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.393627    2579 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-5pjx5"
Sep 10 17:35:56.397914 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:35:56.397847    2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-122.ec2.internal.18d4063046a776d6  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-122.ec2.internal,UID:ip-10-0-129-122.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-129-122.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-129-122.ec2.internal,},FirstTimestamp:2026-09-10 17:35:56.386506454 +0000 UTC m=+0.386336512,LastTimestamp:2026-09-10 17:35:56.386506454 +0000 UTC m=+0.386336512,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-122.ec2.internal,}"
Sep 10 17:35:56.400342 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:35:56.400255    2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-122.ec2.internal.18d4063046a79a9e  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-122.ec2.internal,UID:ip-10-0-129-122.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-129-122.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-129-122.ec2.internal,},FirstTimestamp:2026-09-10 17:35:56.386515614 +0000 UTC m=+0.386345672,LastTimestamp:2026-09-10 17:35:56.386515614 +0000 UTC m=+0.386345672,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-122.ec2.internal,}"
Sep 10 17:35:56.401114 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.401094    2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-5pjx5"
Sep 10 17:35:56.428112 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.428098    2579 manager.go:337] "Starting Device Plugin manager"
Sep 10 17:35:56.448109 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:35:56.428130    2579 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 10 17:35:56.448109 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.428139    2579 server.go:87] "Starting device plugin registration server"
Sep 10 17:35:56.448109 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.428388    2579 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 10 17:35:56.448109 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.428397    2579 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 10 17:35:56.448109 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.428507    2579 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 10 17:35:56.448109 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.428609    2579 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 10 17:35:56.448109 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.428618    2579 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 10 17:35:56.448109 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:35:56.429160    2579 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 10 17:35:56.448109 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:35:56.429208    2579 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-129-122.ec2.internal\" not found"
Sep 10 17:35:56.468251 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.468234    2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 10 17:35:56.468317 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.468277    2579 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 10 17:35:56.468317 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.468295    2579 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 10 17:35:56.468398 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:35:56.468326    2579 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 10 17:35:56.470444 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.470430    2579 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 17:35:56.529334 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.529311    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 17:35:56.530517 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.530502    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-122.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 17:35:56.530581 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.530532    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-122.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 17:35:56.530581 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.530547    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-122.ec2.internal" event="NodeHasSufficientPID"
Sep 10 17:35:56.530649 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.530590    2579 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-129-122.ec2.internal"
Sep 10 17:35:56.538834 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.538783    2579 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-129-122.ec2.internal"
Sep 10 17:35:56.538834 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:35:56.538810    2579 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-129-122.ec2.internal\": node \"ip-10-0-129-122.ec2.internal\" not found"
Sep 10 17:35:56.558917 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:35:56.558891    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-122.ec2.internal\" not found"
Sep 10 17:35:56.569137 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.569117    2579 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-122.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-129-122.ec2.internal"]
Sep 10 17:35:56.569189 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.569184    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 17:35:56.571050 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.570638    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-122.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 17:35:56.571050 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.570681    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-122.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 17:35:56.571050 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.570695    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-122.ec2.internal" event="NodeHasSufficientPID"
Sep 10 17:35:56.572974 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.572960    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 17:35:56.573102 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.573087    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-122.ec2.internal"
Sep 10 17:35:56.573164 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.573123    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 17:35:56.573692 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.573675    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-122.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 17:35:56.573761 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.573707    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-122.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 17:35:56.573761 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.573718    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-122.ec2.internal" event="NodeHasSufficientPID"
Sep 10 17:35:56.573823 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.573681    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-122.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 17:35:56.573823 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.573787    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-122.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 17:35:56.573823 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.573801    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-122.ec2.internal" event="NodeHasSufficientPID"
Sep 10 17:35:56.575942 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.575925    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-122.ec2.internal"
Sep 10 17:35:56.576028 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.575950    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 17:35:56.576632 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.576614    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-122.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 17:35:56.576715 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.576650    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-122.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 17:35:56.576715 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.576666    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-122.ec2.internal" event="NodeHasSufficientPID"
Sep 10 17:35:56.589186 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:35:56.589167    2579 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-122.ec2.internal\" not found" node="ip-10-0-129-122.ec2.internal"
Sep 10 17:35:56.593359 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:35:56.593343    2579 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-122.ec2.internal\" not found" node="ip-10-0-129-122.ec2.internal"
Sep 10 17:35:56.659611 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:35:56.659593    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-122.ec2.internal\" not found"
Sep 10 17:35:56.668962 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.668942    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/39d5711007096867f7ca5e1bd92438d9-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-122.ec2.internal\" (UID: \"39d5711007096867f7ca5e1bd92438d9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-122.ec2.internal"
Sep 10 17:35:56.669024 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.668967    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/39d5711007096867f7ca5e1bd92438d9-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-122.ec2.internal\" (UID: \"39d5711007096867f7ca5e1bd92438d9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-122.ec2.internal"
Sep 10 17:35:56.669024 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.668983    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bd750ec1329c03eb8935ed3e9791d542-config\") pod \"kube-apiserver-proxy-ip-10-0-129-122.ec2.internal\" (UID: \"bd750ec1329c03eb8935ed3e9791d542\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-122.ec2.internal"
Sep 10 17:35:56.760117 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:35:56.760092    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-122.ec2.internal\" not found"
Sep 10 17:35:56.769484 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.769461    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/39d5711007096867f7ca5e1bd92438d9-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-122.ec2.internal\" (UID: \"39d5711007096867f7ca5e1bd92438d9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-122.ec2.internal"
Sep 10 17:35:56.769545 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.769491    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/39d5711007096867f7ca5e1bd92438d9-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-122.ec2.internal\" (UID: \"39d5711007096867f7ca5e1bd92438d9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-122.ec2.internal"
Sep 10 17:35:56.769545 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.769507    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bd750ec1329c03eb8935ed3e9791d542-config\") pod \"kube-apiserver-proxy-ip-10-0-129-122.ec2.internal\" (UID: \"bd750ec1329c03eb8935ed3e9791d542\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-122.ec2.internal"
Sep 10 17:35:56.769545 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.769541    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bd750ec1329c03eb8935ed3e9791d542-config\") pod \"kube-apiserver-proxy-ip-10-0-129-122.ec2.internal\" (UID: \"bd750ec1329c03eb8935ed3e9791d542\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-122.ec2.internal"
Sep 10 17:35:56.769661 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.769559    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/39d5711007096867f7ca5e1bd92438d9-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-122.ec2.internal\" (UID: \"39d5711007096867f7ca5e1bd92438d9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-122.ec2.internal"
Sep 10 17:35:56.769661 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.769593    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/39d5711007096867f7ca5e1bd92438d9-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-122.ec2.internal\" (UID: \"39d5711007096867f7ca5e1bd92438d9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-122.ec2.internal"
Sep 10 17:35:56.860939 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:35:56.860889    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-122.ec2.internal\" not found"
Sep 10 17:35:56.891423 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.891390    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-122.ec2.internal"
Sep 10 17:35:56.896297 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:56.896237    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-122.ec2.internal"
Sep 10 17:35:56.961668 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:35:56.961638    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-122.ec2.internal\" not found"
Sep 10 17:35:57.062182 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:35:57.062157    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-122.ec2.internal\" not found"
Sep 10 17:35:57.162741 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:35:57.162681    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-122.ec2.internal\" not found"
Sep 10 17:35:57.263276 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:35:57.263248    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-122.ec2.internal\" not found"
Sep 10 17:35:57.305787 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.305762    2579 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 10 17:35:57.306186 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.305898    2579 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 10 17:35:57.341522 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.341501    2579 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 17:35:57.347613 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.347595    2579 apiserver.go:52] "Watching apiserver"
Sep 10 17:35:57.358412 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.358387    2579 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 10 17:35:57.360222 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.360202    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-lpm4k","openshift-cluster-node-tuning-operator/tuned-n27c5","openshift-dns/node-resolver-t4l6s","openshift-image-registry/node-ca-rkzwg","openshift-multus/multus-x2sqz","openshift-multus/network-metrics-daemon-ncmn7","openshift-network-diagnostics/network-check-target-8k8z5","openshift-network-operator/iptables-alerter-pdstz","kube-system/global-pull-secret-syncer-m4npt","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-plzx8","openshift-multus/multus-additional-cni-plugins-drhxj","openshift-ovn-kubernetes/ovnkube-node-4fsjz"]
Sep 10 17:35:57.362546 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.362531    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-n27c5"
Sep 10 17:35:57.364624 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.364608    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-t4l6s"
Sep 10 17:35:57.364968 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.364953    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 10 17:35:57.364968 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.364962    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 10 17:35:57.365141 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.364992    2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 10 17:35:57.365141 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.365025    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-fhwhr\""
Sep 10 17:35:57.366189 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.366176    2579 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-122.ec2.internal"
Sep 10 17:35:57.366786 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.366771    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-rkzwg"
Sep 10 17:35:57.367085 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.367068    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 10 17:35:57.367220 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.367201    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-j2qvw\""
Sep 10 17:35:57.367295 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.367211    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 10 17:35:57.368925 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.368907    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 10 17:35:57.369036 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.368952    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-x2sqz"
Sep 10 17:35:57.369200 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.369183    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 10 17:35:57.369275 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.369220    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-r6cnt\""
Sep 10 17:35:57.369327 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.369274    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 10 17:35:57.370963 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.370949    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ncmn7"
Sep 10 17:35:57.371041 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:35:57.371021    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ncmn7" podUID="3fcacac9-c23e-4ba5-8b1d-dfeede6b57e3"
Sep 10 17:35:57.371941 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.371927    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 10 17:35:57.372056 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.372040    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 10 17:35:57.372250 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.372238    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 10 17:35:57.372250 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.372245    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 10 17:35:57.372345 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.372332    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-fh95l\""
Sep 10 17:35:57.373461 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.373447    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ppvg\" (UniqueName: \"kubernetes.io/projected/3dc43868-1f6c-48fa-85fc-59607a50c103-kube-api-access-4ppvg\") pod \"node-resolver-t4l6s\" (UID: \"3dc43868-1f6c-48fa-85fc-59607a50c103\") " pod="openshift-dns/node-resolver-t4l6s"
Sep 10 17:35:57.373515 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.373478    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/85fae606-5227-404b-95a8-d7de60dc1041-host-run-netns\") pod \"multus-x2sqz\" (UID: \"85fae606-5227-404b-95a8-d7de60dc1041\") " pod="openshift-multus/multus-x2sqz"
Sep 10 17:35:57.373515 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.373493    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/85fae606-5227-404b-95a8-d7de60dc1041-hostroot\") pod \"multus-x2sqz\" (UID: \"85fae606-5227-404b-95a8-d7de60dc1041\") " pod="openshift-multus/multus-x2sqz"
Sep 10 17:35:57.373515 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.373506    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/85fae606-5227-404b-95a8-d7de60dc1041-multus-cni-dir\") pod \"multus-x2sqz\" (UID: \"85fae606-5227-404b-95a8-d7de60dc1041\") " pod="openshift-multus/multus-x2sqz"
Sep 10 17:35:57.373655 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.373527    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/88321231-7aaf-4278-be6f-32bee3777450-etc-modprobe-d\") pod \"tuned-n27c5\" (UID: \"88321231-7aaf-4278-be6f-32bee3777450\") " pod="openshift-cluster-node-tuning-operator/tuned-n27c5"
Sep 10 17:35:57.373655 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.373543    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/88321231-7aaf-4278-be6f-32bee3777450-lib-modules\") pod \"tuned-n27c5\" (UID: \"88321231-7aaf-4278-be6f-32bee3777450\") " pod="openshift-cluster-node-tuning-operator/tuned-n27c5"
Sep 10 17:35:57.373655 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.373645    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/88321231-7aaf-4278-be6f-32bee3777450-etc-tuned\") pod \"tuned-n27c5\" (UID: \"88321231-7aaf-4278-be6f-32bee3777450\") " pod="openshift-cluster-node-tuning-operator/tuned-n27c5"
Sep 10 17:35:57.373801 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.373705    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/eca4abd0-e29b-4c77-9a3b-c9c32e5b5101-host\") pod \"node-ca-rkzwg\" (UID: \"eca4abd0-e29b-4c77-9a3b-c9c32e5b5101\") " pod="openshift-image-registry/node-ca-rkzwg"
Sep 10 17:35:57.373801 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.373732    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/85fae606-5227-404b-95a8-d7de60dc1041-cni-binary-copy\") pod \"multus-x2sqz\" (UID: \"85fae606-5227-404b-95a8-d7de60dc1041\") " pod="openshift-multus/multus-x2sqz"
Sep 10 17:35:57.373801 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.373783    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/85fae606-5227-404b-95a8-d7de60dc1041-host-var-lib-cni-multus\") pod \"multus-x2sqz\" (UID: \"85fae606-5227-404b-95a8-d7de60dc1041\") " pod="openshift-multus/multus-x2sqz"
Sep 10 17:35:57.373925 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.373810    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/88321231-7aaf-4278-be6f-32bee3777450-run\") pod \"tuned-n27c5\" (UID: \"88321231-7aaf-4278-be6f-32bee3777450\") " pod="openshift-cluster-node-tuning-operator/tuned-n27c5"
Sep 10 17:35:57.373925 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.373826    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/88321231-7aaf-4278-be6f-32bee3777450-sys\") pod \"tuned-n27c5\" (UID: \"88321231-7aaf-4278-be6f-32bee3777450\") " pod="openshift-cluster-node-tuning-operator/tuned-n27c5"
Sep 10 17:35:57.373925 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.373859    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hpjgl\" (UniqueName: \"kubernetes.io/projected/eca4abd0-e29b-4c77-9a3b-c9c32e5b5101-kube-api-access-hpjgl\") pod \"node-ca-rkzwg\" (UID: \"eca4abd0-e29b-4c77-9a3b-c9c32e5b5101\") " pod="openshift-image-registry/node-ca-rkzwg"
Sep 10 17:35:57.373925 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.373885    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/85fae606-5227-404b-95a8-d7de60dc1041-host-var-lib-cni-bin\") pod \"multus-x2sqz\" (UID: \"85fae606-5227-404b-95a8-d7de60dc1041\") " pod="openshift-multus/multus-x2sqz"
Sep 10 17:35:57.373925 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.373897    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8k8z5"
Sep 10 17:35:57.373925 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.373911    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dfj4\" (UniqueName: \"kubernetes.io/projected/85fae606-5227-404b-95a8-d7de60dc1041-kube-api-access-2dfj4\") pod \"multus-x2sqz\" (UID: \"85fae606-5227-404b-95a8-d7de60dc1041\") " pod="openshift-multus/multus-x2sqz"
Sep 10 17:35:57.374188 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.373933    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/eca4abd0-e29b-4c77-9a3b-c9c32e5b5101-serviceca\") pod \"node-ca-rkzwg\" (UID: \"eca4abd0-e29b-4c77-9a3b-c9c32e5b5101\") " pod="openshift-image-registry/node-ca-rkzwg"
Sep 10 17:35:57.374188 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.373954    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/85fae606-5227-404b-95a8-d7de60dc1041-cnibin\") pod \"multus-x2sqz\" (UID: \"85fae606-5227-404b-95a8-d7de60dc1041\") " pod="openshift-multus/multus-x2sqz"
Sep 10 17:35:57.374188 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:35:57.373952    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8k8z5" podUID="8436b9d2-f47a-411e-a695-cb52b8277b62"
Sep 10 17:35:57.374188 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.373975    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/85fae606-5227-404b-95a8-d7de60dc1041-host-run-multus-certs\") pod \"multus-x2sqz\" (UID: \"85fae606-5227-404b-95a8-d7de60dc1041\") " pod="openshift-multus/multus-x2sqz"
Sep 10 17:35:57.374188 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.374007    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/88321231-7aaf-4278-be6f-32bee3777450-etc-sysconfig\") pod \"tuned-n27c5\" (UID: \"88321231-7aaf-4278-be6f-32bee3777450\") " pod="openshift-cluster-node-tuning-operator/tuned-n27c5"
Sep 10 17:35:57.374188 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.374043    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/88321231-7aaf-4278-be6f-32bee3777450-etc-sysctl-conf\") pod \"tuned-n27c5\" (UID: \"88321231-7aaf-4278-be6f-32bee3777450\") " pod="openshift-cluster-node-tuning-operator/tuned-n27c5"
Sep 10 17:35:57.374188 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.374057    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zsqvf\" (UniqueName: \"kubernetes.io/projected/88321231-7aaf-4278-be6f-32bee3777450-kube-api-access-zsqvf\") pod \"tuned-n27c5\" (UID: \"88321231-7aaf-4278-be6f-32bee3777450\") " pod="openshift-cluster-node-tuning-operator/tuned-n27c5"
Sep 10 17:35:57.374188 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.374071    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3dc43868-1f6c-48fa-85fc-59607a50c103-tmp-dir\") pod \"node-resolver-t4l6s\" (UID: \"3dc43868-1f6c-48fa-85fc-59607a50c103\") " pod="openshift-dns/node-resolver-t4l6s"
Sep 10 17:35:57.374188 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.374088    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/85fae606-5227-404b-95a8-d7de60dc1041-multus-conf-dir\") pod \"multus-x2sqz\" (UID: \"85fae606-5227-404b-95a8-d7de60dc1041\") " pod="openshift-multus/multus-x2sqz"
Sep 10 17:35:57.374188 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.374115    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/85fae606-5227-404b-95a8-d7de60dc1041-multus-daemon-config\") pod \"multus-x2sqz\" (UID: \"85fae606-5227-404b-95a8-d7de60dc1041\") " pod="openshift-multus/multus-x2sqz"
Sep 10 17:35:57.374188 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.374145    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/88321231-7aaf-4278-be6f-32bee3777450-etc-kubernetes\") pod \"tuned-n27c5\" (UID: \"88321231-7aaf-4278-be6f-32bee3777450\") " pod="openshift-cluster-node-tuning-operator/tuned-n27c5"
Sep 10 17:35:57.374188 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.374165    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/88321231-7aaf-4278-be6f-32bee3777450-etc-sysctl-d\") pod \"tuned-n27c5\" (UID: \"88321231-7aaf-4278-be6f-32bee3777450\") " pod="openshift-cluster-node-tuning-operator/tuned-n27c5"
Sep 10 17:35:57.374188 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.374178    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/88321231-7aaf-4278-be6f-32bee3777450-host\") pod \"tuned-n27c5\" (UID: \"88321231-7aaf-4278-be6f-32bee3777450\") " pod="openshift-cluster-node-tuning-operator/tuned-n27c5"
Sep 10 17:35:57.374188 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.374192    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3dc43868-1f6c-48fa-85fc-59607a50c103-hosts-file\") pod \"node-resolver-t4l6s\" (UID: \"3dc43868-1f6c-48fa-85fc-59607a50c103\") " pod="openshift-dns/node-resolver-t4l6s"
Sep 10 17:35:57.374580 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.374212    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/85fae606-5227-404b-95a8-d7de60dc1041-system-cni-dir\") pod \"multus-x2sqz\" (UID: \"85fae606-5227-404b-95a8-d7de60dc1041\") " pod="openshift-multus/multus-x2sqz"
Sep 10 17:35:57.374580 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.374225    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/85fae606-5227-404b-95a8-d7de60dc1041-multus-socket-dir-parent\") pod \"multus-x2sqz\" (UID: \"85fae606-5227-404b-95a8-d7de60dc1041\") " pod="openshift-multus/multus-x2sqz"
Sep 10 17:35:57.374580 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.374238    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/85fae606-5227-404b-95a8-d7de60dc1041-etc-kubernetes\") pod \"multus-x2sqz\" (UID: \"85fae606-5227-404b-95a8-d7de60dc1041\") " pod="openshift-multus/multus-x2sqz"
Sep 10 17:35:57.374580 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.374250    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/88321231-7aaf-4278-be6f-32bee3777450-etc-systemd\") pod \"tuned-n27c5\" (UID: \"88321231-7aaf-4278-be6f-32bee3777450\") " pod="openshift-cluster-node-tuning-operator/tuned-n27c5"
Sep 10 17:35:57.374580 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.374263    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/88321231-7aaf-4278-be6f-32bee3777450-tmp\") pod \"tuned-n27c5\" (UID: \"88321231-7aaf-4278-be6f-32bee3777450\") " pod="openshift-cluster-node-tuning-operator/tuned-n27c5"
Sep 10 17:35:57.374580 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.374276    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/85fae606-5227-404b-95a8-d7de60dc1041-host-var-lib-kubelet\") pod \"multus-x2sqz\" (UID: \"85fae606-5227-404b-95a8-d7de60dc1041\") " pod="openshift-multus/multus-x2sqz"
Sep 10 17:35:57.374580 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.374304    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/85fae606-5227-404b-95a8-d7de60dc1041-host-run-k8s-cni-cncf-io\") pod \"multus-x2sqz\" (UID: \"85fae606-5227-404b-95a8-d7de60dc1041\") " pod="openshift-multus/multus-x2sqz"
Sep 10 17:35:57.374580 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.374329    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/88321231-7aaf-4278-be6f-32bee3777450-var-lib-kubelet\") pod \"tuned-n27c5\" (UID: \"88321231-7aaf-4278-be6f-32bee3777450\") " pod="openshift-cluster-node-tuning-operator/tuned-n27c5"
Sep 10 17:35:57.375931 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.375917    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-pdstz"
Sep 10 17:35:57.378473 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.376424    2579 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 10 17:35:57.378967 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.378740    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 10 17:35:57.378967 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.378783    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 10 17:35:57.378967 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.378953    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 10 17:35:57.379078 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.378964    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-m9rq7\""
Sep 10 17:35:57.381070 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.381049    2579 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 10 17:35:57.381204 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.381072    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-plzx8"
Sep 10 17:35:57.381204 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.381131    2579 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-122.ec2.internal"
Sep 10 17:35:57.383120 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.383100    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-drhxj"
Sep 10 17:35:57.383370 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.383348    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-mcxr9\""
Sep 10 17:35:57.384078 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.383982    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 10 17:35:57.384078 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.384048    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 10 17:35:57.384078 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.384053    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 10 17:35:57.385521 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.385506    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-4fsjz"
Sep 10 17:35:57.387623 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.387608    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-lpm4k"
Sep 10 17:35:57.388730 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.388714    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 10 17:35:57.388823 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.388809    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 10 17:35:57.389236 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.389224    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-vhqr9\""
Sep 10 17:35:57.389318 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.389305    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 10 17:35:57.389370 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.389317    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-2t96x\""
Sep 10 17:35:57.389414 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.389401    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 10 17:35:57.389458 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.389446    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 10 17:35:57.389805 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.389786    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 10 17:35:57.389919 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.389786    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 10 17:35:57.389919 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.389851    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 10 17:35:57.389919 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.389860    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 10 17:35:57.389919 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.389904    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-glkpd\""
Sep 10 17:35:57.390117 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.390087    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-122.ec2.internal"]
Sep 10 17:35:57.390117 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.390089    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 10 17:35:57.390182 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.390165    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m4npt"
Sep 10 17:35:57.390247 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:35:57.390226    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m4npt" podUID="e0fe9300-104f-489d-9962-908000359645"
Sep 10 17:35:57.393799 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.393783    2579 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 10 17:35:57.393970 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.393955    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-129-122.ec2.internal"]
Sep 10 17:35:57.405149 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.405108    2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-09 17:30:56 +0000 UTC" deadline="2028-04-22 15:49:21.904701062 +0000 UTC"
Sep 10 17:35:57.405149 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.405147    2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14158h13m24.499556141s"
Sep 10 17:35:57.407153 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.407135    2579 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-sjksb"
Sep 10 17:35:57.415012 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.414966    2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-sjksb"
Sep 10 17:35:57.468136 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.468119    2579 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 10 17:35:57.474492 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.474460    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d4fad94e-7939-4949-bdfd-1ef38e016c19-run-openvswitch\") pod \"ovnkube-node-4fsjz\" (UID: \"d4fad94e-7939-4949-bdfd-1ef38e016c19\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fsjz"
Sep 10 17:35:57.474606 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.474491    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d4fad94e-7939-4949-bdfd-1ef38e016c19-ovnkube-script-lib\") pod \"ovnkube-node-4fsjz\" (UID: \"d4fad94e-7939-4949-bdfd-1ef38e016c19\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fsjz"
Sep 10 17:35:57.474606 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.474516    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1cd8f80e-f5c5-4e80-9abb-b43af15ed1d1-agent-certs\") pod \"konnectivity-agent-lpm4k\" (UID: \"1cd8f80e-f5c5-4e80-9abb-b43af15ed1d1\") " pod="kube-system/konnectivity-agent-lpm4k"
Sep 10 17:35:57.474606 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.474576    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/88321231-7aaf-4278-be6f-32bee3777450-etc-sysctl-conf\") pod \"tuned-n27c5\" (UID: \"88321231-7aaf-4278-be6f-32bee3777450\") " pod="openshift-cluster-node-tuning-operator/tuned-n27c5"
Sep 10 17:35:57.474773 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.474608    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zsqvf\" (UniqueName: \"kubernetes.io/projected/88321231-7aaf-4278-be6f-32bee3777450-kube-api-access-zsqvf\") pod \"tuned-n27c5\" (UID: \"88321231-7aaf-4278-be6f-32bee3777450\") " pod="openshift-cluster-node-tuning-operator/tuned-n27c5"
Sep 10 17:35:57.474773 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.474630    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3dc43868-1f6c-48fa-85fc-59607a50c103-tmp-dir\") pod \"node-resolver-t4l6s\" (UID: \"3dc43868-1f6c-48fa-85fc-59607a50c103\") " pod="openshift-dns/node-resolver-t4l6s"
Sep 10 17:35:57.474773 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.474651    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/56f29fc7-7f69-42cb-b80e-1a2bfc298f1a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-plzx8\" (UID: \"56f29fc7-7f69-42cb-b80e-1a2bfc298f1a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-plzx8"
Sep 10 17:35:57.474773 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.474676    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/48d6ee6d-6025-4ab7-a095-6a31180ac9f7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-drhxj\" (UID: \"48d6ee6d-6025-4ab7-a095-6a31180ac9f7\") " pod="openshift-multus/multus-additional-cni-plugins-drhxj"
Sep 10 17:35:57.474773 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.474699    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e0fe9300-104f-489d-9962-908000359645-dbus\") pod \"global-pull-secret-syncer-m4npt\" (UID: \"e0fe9300-104f-489d-9962-908000359645\") " pod="kube-system/global-pull-secret-syncer-m4npt"
Sep 10 17:35:57.475001 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.474776    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/88321231-7aaf-4278-be6f-32bee3777450-etc-kubernetes\") pod \"tuned-n27c5\" (UID: \"88321231-7aaf-4278-be6f-32bee3777450\") " pod="openshift-cluster-node-tuning-operator/tuned-n27c5"
Sep 10 17:35:57.475001 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.474800    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/88321231-7aaf-4278-be6f-32bee3777450-host\") pod \"tuned-n27c5\" (UID: \"88321231-7aaf-4278-be6f-32bee3777450\") " pod="openshift-cluster-node-tuning-operator/tuned-n27c5"
Sep 10 17:35:57.475001 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.474815    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/88321231-7aaf-4278-be6f-32bee3777450-etc-sysctl-conf\") pod \"tuned-n27c5\" (UID: \"88321231-7aaf-4278-be6f-32bee3777450\") " pod="openshift-cluster-node-tuning-operator/tuned-n27c5"
Sep 10 17:35:57.475001 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.474849    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/88321231-7aaf-4278-be6f-32bee3777450-host\") pod \"tuned-n27c5\" (UID: \"88321231-7aaf-4278-be6f-32bee3777450\") " pod="openshift-cluster-node-tuning-operator/tuned-n27c5"
Sep 10 17:35:57.475001 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.474869    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/48d6ee6d-6025-4ab7-a095-6a31180ac9f7-system-cni-dir\") pod \"multus-additional-cni-plugins-drhxj\" (UID: \"48d6ee6d-6025-4ab7-a095-6a31180ac9f7\") " pod="openshift-multus/multus-additional-cni-plugins-drhxj"
Sep 10 17:35:57.475001 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.474883    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/88321231-7aaf-4278-be6f-32bee3777450-tmp\") pod \"tuned-n27c5\" (UID: \"88321231-7aaf-4278-be6f-32bee3777450\") " pod="openshift-cluster-node-tuning-operator/tuned-n27c5"
Sep 10 17:35:57.475001 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.474902    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/85fae606-5227-404b-95a8-d7de60dc1041-host-var-lib-kubelet\") pod \"multus-x2sqz\" (UID: \"85fae606-5227-404b-95a8-d7de60dc1041\") " pod="openshift-multus/multus-x2sqz"
Sep 10 17:35:57.475001 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.474907    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/88321231-7aaf-4278-be6f-32bee3777450-etc-kubernetes\") pod \"tuned-n27c5\" (UID: \"88321231-7aaf-4278-be6f-32bee3777450\") " pod="openshift-cluster-node-tuning-operator/tuned-n27c5"
Sep 10 17:35:57.475001 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.474927    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/48d6ee6d-6025-4ab7-a095-6a31180ac9f7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-drhxj\" (UID: \"48d6ee6d-6025-4ab7-a095-6a31180ac9f7\") " pod="openshift-multus/multus-additional-cni-plugins-drhxj"
Sep 10 17:35:57.475001 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.474942    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/85fae606-5227-404b-95a8-d7de60dc1041-host-var-lib-kubelet\") pod \"multus-x2sqz\" (UID: \"85fae606-5227-404b-95a8-d7de60dc1041\") " pod="openshift-multus/multus-x2sqz"
Sep 10 17:35:57.475001 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.474960    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/32a43910-3aa0-41e0-ab5b-ad08c1ab6b41-host-slash\") pod \"iptables-alerter-pdstz\" (UID: \"32a43910-3aa0-41e0-ab5b-ad08c1ab6b41\") " pod="openshift-network-operator/iptables-alerter-pdstz"
Sep 10 17:35:57.475001 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.474979    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1cd8f80e-f5c5-4e80-9abb-b43af15ed1d1-konnectivity-ca\") pod \"konnectivity-agent-lpm4k\" (UID: \"1cd8f80e-f5c5-4e80-9abb-b43af15ed1d1\") " pod="kube-system/konnectivity-agent-lpm4k"
Sep 10 17:35:57.475483 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.475005    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d4fad94e-7939-4949-bdfd-1ef38e016c19-run-systemd\") pod \"ovnkube-node-4fsjz\" (UID: \"d4fad94e-7939-4949-bdfd-1ef38e016c19\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fsjz"
Sep 10 17:35:57.475483 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.475034    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d4fad94e-7939-4949-bdfd-1ef38e016c19-run-ovn\") pod \"ovnkube-node-4fsjz\" (UID: \"d4fad94e-7939-4949-bdfd-1ef38e016c19\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fsjz"
Sep 10 17:35:57.475483 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.475073    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d4fad94e-7939-4949-bdfd-1ef38e016c19-ovn-node-metrics-cert\") pod \"ovnkube-node-4fsjz\" (UID: \"d4fad94e-7939-4949-bdfd-1ef38e016c19\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fsjz"
Sep 10 17:35:57.475483 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.475092    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/88321231-7aaf-4278-be6f-32bee3777450-var-lib-kubelet\") pod \"tuned-n27c5\" (UID: \"88321231-7aaf-4278-be6f-32bee3777450\") " pod="openshift-cluster-node-tuning-operator/tuned-n27c5"
Sep 10 17:35:57.475483 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.475108    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/85fae606-5227-404b-95a8-d7de60dc1041-host-run-netns\") pod \"multus-x2sqz\" (UID: \"85fae606-5227-404b-95a8-d7de60dc1041\") " pod="openshift-multus/multus-x2sqz"
Sep 10 17:35:57.475483 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.475143    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/85fae606-5227-404b-95a8-d7de60dc1041-host-run-netns\") pod \"multus-x2sqz\" (UID: \"85fae606-5227-404b-95a8-d7de60dc1041\") " pod="openshift-multus/multus-x2sqz"
Sep 10 17:35:57.475483 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.475153    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/48d6ee6d-6025-4ab7-a095-6a31180ac9f7-cnibin\") pod \"multus-additional-cni-plugins-drhxj\" (UID: \"48d6ee6d-6025-4ab7-a095-6a31180ac9f7\") " pod="openshift-multus/multus-additional-cni-plugins-drhxj"
Sep 10 17:35:57.475483 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.475179    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/88321231-7aaf-4278-be6f-32bee3777450-var-lib-kubelet\") pod \"tuned-n27c5\" (UID: \"88321231-7aaf-4278-be6f-32bee3777450\") " pod="openshift-cluster-node-tuning-operator/tuned-n27c5"
Sep 10 17:35:57.475483 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.475236    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/85fae606-5227-404b-95a8-d7de60dc1041-multus-cni-dir\") pod \"multus-x2sqz\" (UID: \"85fae606-5227-404b-95a8-d7de60dc1041\") " pod="openshift-multus/multus-x2sqz"
Sep 10 17:35:57.475483 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.475266    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/88321231-7aaf-4278-be6f-32bee3777450-lib-modules\") pod \"tuned-n27c5\" (UID: \"88321231-7aaf-4278-be6f-32bee3777450\") " pod="openshift-cluster-node-tuning-operator/tuned-n27c5"
Sep 10 17:35:57.475483 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.475291    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/85fae606-5227-404b-95a8-d7de60dc1041-host-var-lib-cni-multus\") pod \"multus-x2sqz\" (UID: \"85fae606-5227-404b-95a8-d7de60dc1041\") " pod="openshift-multus/multus-x2sqz"
Sep 10 17:35:57.475483 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.475304    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/85fae606-5227-404b-95a8-d7de60dc1041-multus-cni-dir\") pod \"multus-x2sqz\" (UID: \"85fae606-5227-404b-95a8-d7de60dc1041\") " pod="openshift-multus/multus-x2sqz"
Sep 10 17:35:57.475483 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.475318    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/56f29fc7-7f69-42cb-b80e-1a2bfc298f1a-etc-selinux\") pod \"aws-ebs-csi-driver-node-plzx8\" (UID: \"56f29fc7-7f69-42cb-b80e-1a2bfc298f1a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-plzx8"
Sep 10 17:35:57.475483 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.475357    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/85fae606-5227-404b-95a8-d7de60dc1041-host-var-lib-cni-multus\") pod \"multus-x2sqz\" (UID: \"85fae606-5227-404b-95a8-d7de60dc1041\") " pod="openshift-multus/multus-x2sqz"
Sep 10 17:35:57.475483 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.475376    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d4fad94e-7939-4949-bdfd-1ef38e016c19-host-kubelet\") pod \"ovnkube-node-4fsjz\" (UID: \"d4fad94e-7939-4949-bdfd-1ef38e016c19\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fsjz"
Sep 10 17:35:57.475483 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.475376    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/88321231-7aaf-4278-be6f-32bee3777450-lib-modules\") pod \"tuned-n27c5\" (UID: \"88321231-7aaf-4278-be6f-32bee3777450\") " pod="openshift-cluster-node-tuning-operator/tuned-n27c5"
Sep 10 17:35:57.475483 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.475384    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3dc43868-1f6c-48fa-85fc-59607a50c103-tmp-dir\") pod \"node-resolver-t4l6s\" (UID: \"3dc43868-1f6c-48fa-85fc-59607a50c103\") " pod="openshift-dns/node-resolver-t4l6s"
Sep 10 17:35:57.476343 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.475419    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d4fad94e-7939-4949-bdfd-1ef38e016c19-log-socket\") pod \"ovnkube-node-4fsjz\" (UID: \"d4fad94e-7939-4949-bdfd-1ef38e016c19\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fsjz"
Sep 10 17:35:57.476343 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.475456    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d4fad94e-7939-4949-bdfd-1ef38e016c19-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-4fsjz\" (UID: \"d4fad94e-7939-4949-bdfd-1ef38e016c19\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fsjz"
Sep 10 17:35:57.476343 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.475466    2579 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 10 17:35:57.476343 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.475483    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/56f29fc7-7f69-42cb-b80e-1a2bfc298f1a-registration-dir\") pod \"aws-ebs-csi-driver-node-plzx8\" (UID: \"56f29fc7-7f69-42cb-b80e-1a2bfc298f1a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-plzx8"
Sep 10 17:35:57.476343 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.475516    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-csj86\" (UniqueName: \"kubernetes.io/projected/56f29fc7-7f69-42cb-b80e-1a2bfc298f1a-kube-api-access-csj86\") pod \"aws-ebs-csi-driver-node-plzx8\" (UID: \"56f29fc7-7f69-42cb-b80e-1a2bfc298f1a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-plzx8"
Sep 10 17:35:57.476343 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.475542    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d4fad94e-7939-4949-bdfd-1ef38e016c19-host-cni-netd\") pod \"ovnkube-node-4fsjz\" (UID: \"d4fad94e-7939-4949-bdfd-1ef38e016c19\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fsjz"
Sep 10 17:35:57.476343 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.475579    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/32a43910-3aa0-41e0-ab5b-ad08c1ab6b41-iptables-alerter-script\") pod \"iptables-alerter-pdstz\" (UID: \"32a43910-3aa0-41e0-ab5b-ad08c1ab6b41\") " pod="openshift-network-operator/iptables-alerter-pdstz"
Sep 10 17:35:57.476343 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.475602    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/85fae606-5227-404b-95a8-d7de60dc1041-cnibin\") pod \"multus-x2sqz\" (UID: \"85fae606-5227-404b-95a8-d7de60dc1041\") " pod="openshift-multus/multus-x2sqz"
Sep 10 17:35:57.476343 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.475616    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/48d6ee6d-6025-4ab7-a095-6a31180ac9f7-cni-binary-copy\") pod \"multus-additional-cni-plugins-drhxj\" (UID: \"48d6ee6d-6025-4ab7-a095-6a31180ac9f7\") " pod="openshift-multus/multus-additional-cni-plugins-drhxj"
Sep 10 17:35:57.476343 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.475630    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d4fad94e-7939-4949-bdfd-1ef38e016c19-var-lib-openvswitch\") pod \"ovnkube-node-4fsjz\" (UID: \"d4fad94e-7939-4949-bdfd-1ef38e016c19\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fsjz"
Sep 10 17:35:57.476343 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.475649    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d4fad94e-7939-4949-bdfd-1ef38e016c19-etc-openvswitch\") pod \"ovnkube-node-4fsjz\" (UID: \"d4fad94e-7939-4949-bdfd-1ef38e016c19\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fsjz"
Sep 10 17:35:57.476343 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.475672    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d4fad94e-7939-4949-bdfd-1ef38e016c19-env-overrides\") pod \"ovnkube-node-4fsjz\" (UID: \"d4fad94e-7939-4949-bdfd-1ef38e016c19\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fsjz"
Sep 10 17:35:57.476343 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.475679    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/85fae606-5227-404b-95a8-d7de60dc1041-cnibin\") pod \"multus-x2sqz\" (UID: \"85fae606-5227-404b-95a8-d7de60dc1041\") " pod="openshift-multus/multus-x2sqz"
Sep 10 17:35:57.476343 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.475701    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e0fe9300-104f-489d-9962-908000359645-original-pull-secret\") pod \"global-pull-secret-syncer-m4npt\" (UID: \"e0fe9300-104f-489d-9962-908000359645\") " pod="kube-system/global-pull-secret-syncer-m4npt"
Sep 10 17:35:57.476343 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.475742    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/88321231-7aaf-4278-be6f-32bee3777450-etc-sysconfig\") pod \"tuned-n27c5\" (UID: \"88321231-7aaf-4278-be6f-32bee3777450\") " pod="openshift-cluster-node-tuning-operator/tuned-n27c5"
Sep 10 17:35:57.476343 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.475758    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/85fae606-5227-404b-95a8-d7de60dc1041-multus-conf-dir\") pod \"multus-x2sqz\" (UID: \"85fae606-5227-404b-95a8-d7de60dc1041\") " pod="openshift-multus/multus-x2sqz"
Sep 10 17:35:57.476343 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.475774    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/85fae606-5227-404b-95a8-d7de60dc1041-multus-daemon-config\") pod \"multus-x2sqz\" (UID: \"85fae606-5227-404b-95a8-d7de60dc1041\") " pod="openshift-multus/multus-x2sqz"
Sep 10 17:35:57.477214 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.475798    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g56qb\" (UniqueName: \"kubernetes.io/projected/3fcacac9-c23e-4ba5-8b1d-dfeede6b57e3-kube-api-access-g56qb\") pod \"network-metrics-daemon-ncmn7\" (UID: \"3fcacac9-c23e-4ba5-8b1d-dfeede6b57e3\") " pod="openshift-multus/network-metrics-daemon-ncmn7"
Sep 10 17:35:57.477214 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.475823    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/56f29fc7-7f69-42cb-b80e-1a2bfc298f1a-device-dir\") pod \"aws-ebs-csi-driver-node-plzx8\" (UID: \"56f29fc7-7f69-42cb-b80e-1a2bfc298f1a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-plzx8"
Sep 10 17:35:57.477214 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.475835    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/85fae606-5227-404b-95a8-d7de60dc1041-multus-conf-dir\") pod \"multus-x2sqz\" (UID: \"85fae606-5227-404b-95a8-d7de60dc1041\") " pod="openshift-multus/multus-x2sqz"
Sep 10 17:35:57.477214 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.475845    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d4fad94e-7939-4949-bdfd-1ef38e016c19-host-run-ovn-kubernetes\") pod \"ovnkube-node-4fsjz\" (UID: \"d4fad94e-7939-4949-bdfd-1ef38e016c19\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fsjz"
Sep 10 17:35:57.477214 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.475824    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/88321231-7aaf-4278-be6f-32bee3777450-etc-sysconfig\") pod \"tuned-n27c5\" (UID: \"88321231-7aaf-4278-be6f-32bee3777450\") " pod="openshift-cluster-node-tuning-operator/tuned-n27c5"
Sep 10 17:35:57.477214 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.475868    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98dkx\" (UniqueName: \"kubernetes.io/projected/d4fad94e-7939-4949-bdfd-1ef38e016c19-kube-api-access-98dkx\") pod \"ovnkube-node-4fsjz\" (UID: \"d4fad94e-7939-4949-bdfd-1ef38e016c19\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fsjz"
Sep 10 17:35:57.477214 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.475890    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e0fe9300-104f-489d-9962-908000359645-kubelet-config\") pod \"global-pull-secret-syncer-m4npt\" (UID: \"e0fe9300-104f-489d-9962-908000359645\") " pod="kube-system/global-pull-secret-syncer-m4npt"
Sep 10 17:35:57.477214 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.475913    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/88321231-7aaf-4278-be6f-32bee3777450-etc-sysctl-d\") pod \"tuned-n27c5\" (UID: \"88321231-7aaf-4278-be6f-32bee3777450\") " pod="openshift-cluster-node-tuning-operator/tuned-n27c5"
Sep 10 17:35:57.477214 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.475951    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3dc43868-1f6c-48fa-85fc-59607a50c103-hosts-file\") pod \"node-resolver-t4l6s\" (UID: \"3dc43868-1f6c-48fa-85fc-59607a50c103\") " pod="openshift-dns/node-resolver-t4l6s"
Sep 10 17:35:57.477214 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.475974    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/85fae606-5227-404b-95a8-d7de60dc1041-system-cni-dir\") pod \"multus-x2sqz\" (UID: \"85fae606-5227-404b-95a8-d7de60dc1041\") " pod="openshift-multus/multus-x2sqz"
Sep 10 17:35:57.477214 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.476002    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/85fae606-5227-404b-95a8-d7de60dc1041-multus-socket-dir-parent\") pod \"multus-x2sqz\" (UID: \"85fae606-5227-404b-95a8-d7de60dc1041\") " pod="openshift-multus/multus-x2sqz"
Sep 10 17:35:57.477214 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.476034    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/85fae606-5227-404b-95a8-d7de60dc1041-etc-kubernetes\") pod \"multus-x2sqz\" (UID: \"85fae606-5227-404b-95a8-d7de60dc1041\") " pod="openshift-multus/multus-x2sqz"
Sep 10 17:35:57.477214 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.476057    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/48d6ee6d-6025-4ab7-a095-6a31180ac9f7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-drhxj\" (UID: \"48d6ee6d-6025-4ab7-a095-6a31180ac9f7\") " pod="openshift-multus/multus-additional-cni-plugins-drhxj"
Sep 10 17:35:57.477214 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.476080    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9zqd\" (UniqueName: \"kubernetes.io/projected/8436b9d2-f47a-411e-a695-cb52b8277b62-kube-api-access-p9zqd\") pod \"network-check-target-8k8z5\" (UID: \"8436b9d2-f47a-411e-a695-cb52b8277b62\") " pod="openshift-network-diagnostics/network-check-target-8k8z5"
Sep 10 17:35:57.477214 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.476107    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wsc6v\" (UniqueName: \"kubernetes.io/projected/32a43910-3aa0-41e0-ab5b-ad08c1ab6b41-kube-api-access-wsc6v\") pod \"iptables-alerter-pdstz\" (UID: \"32a43910-3aa0-41e0-ab5b-ad08c1ab6b41\") " pod="openshift-network-operator/iptables-alerter-pdstz"
Sep 10 17:35:57.477214 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.476137    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/88321231-7aaf-4278-be6f-32bee3777450-etc-systemd\") pod \"tuned-n27c5\" (UID: \"88321231-7aaf-4278-be6f-32bee3777450\") " pod="openshift-cluster-node-tuning-operator/tuned-n27c5"
Sep 10 17:35:57.477214 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.476162    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d4fad94e-7939-4949-bdfd-1ef38e016c19-ovnkube-config\") pod \"ovnkube-node-4fsjz\" (UID: \"d4fad94e-7939-4949-bdfd-1ef38e016c19\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fsjz"
Sep 10 17:35:57.477993 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.476191    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/85fae606-5227-404b-95a8-d7de60dc1041-host-run-k8s-cni-cncf-io\") pod \"multus-x2sqz\" (UID: \"85fae606-5227-404b-95a8-d7de60dc1041\") " pod="openshift-multus/multus-x2sqz"
Sep 10 17:35:57.477993 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.476264    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/85fae606-5227-404b-95a8-d7de60dc1041-host-run-k8s-cni-cncf-io\") pod \"multus-x2sqz\" (UID: \"85fae606-5227-404b-95a8-d7de60dc1041\") " pod="openshift-multus/multus-x2sqz"
Sep 10 17:35:57.477993 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.476268    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/85fae606-5227-404b-95a8-d7de60dc1041-multus-socket-dir-parent\") pod \"multus-x2sqz\" (UID: \"85fae606-5227-404b-95a8-d7de60dc1041\") " pod="openshift-multus/multus-x2sqz"
Sep 10 17:35:57.477993 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.476293    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/85fae606-5227-404b-95a8-d7de60dc1041-etc-kubernetes\") pod \"multus-x2sqz\" (UID: \"85fae606-5227-404b-95a8-d7de60dc1041\") " pod="openshift-multus/multus-x2sqz"
Sep 10 17:35:57.477993 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.476302    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/56f29fc7-7f69-42cb-b80e-1a2bfc298f1a-socket-dir\") pod \"aws-ebs-csi-driver-node-plzx8\" (UID: \"56f29fc7-7f69-42cb-b80e-1a2bfc298f1a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-plzx8"
Sep 10 17:35:57.477993 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.476331    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d4fad94e-7939-4949-bdfd-1ef38e016c19-node-log\") pod \"ovnkube-node-4fsjz\" (UID: \"d4fad94e-7939-4949-bdfd-1ef38e016c19\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fsjz"
Sep 10 17:35:57.477993 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.476352    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/88321231-7aaf-4278-be6f-32bee3777450-etc-systemd\") pod \"tuned-n27c5\" (UID: \"88321231-7aaf-4278-be6f-32bee3777450\") " pod="openshift-cluster-node-tuning-operator/tuned-n27c5"
Sep 10 17:35:57.477993 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.476357    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ppvg\" (UniqueName: \"kubernetes.io/projected/3dc43868-1f6c-48fa-85fc-59607a50c103-kube-api-access-4ppvg\") pod \"node-resolver-t4l6s\" (UID: \"3dc43868-1f6c-48fa-85fc-59607a50c103\") " pod="openshift-dns/node-resolver-t4l6s"
Sep 10 17:35:57.477993 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.476390    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/85fae606-5227-404b-95a8-d7de60dc1041-hostroot\") pod \"multus-x2sqz\" (UID: \"85fae606-5227-404b-95a8-d7de60dc1041\") " pod="openshift-multus/multus-x2sqz"
Sep 10 17:35:57.477993 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.476417    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/88321231-7aaf-4278-be6f-32bee3777450-etc-modprobe-d\") pod \"tuned-n27c5\" (UID: \"88321231-7aaf-4278-be6f-32bee3777450\") " pod="openshift-cluster-node-tuning-operator/tuned-n27c5"
Sep 10 17:35:57.477993 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.476414    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/88321231-7aaf-4278-be6f-32bee3777450-etc-sysctl-d\") pod \"tuned-n27c5\" (UID: \"88321231-7aaf-4278-be6f-32bee3777450\") " pod="openshift-cluster-node-tuning-operator/tuned-n27c5"
Sep 10 17:35:57.477993 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.476445    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/88321231-7aaf-4278-be6f-32bee3777450-etc-tuned\") pod \"tuned-n27c5\" (UID: \"88321231-7aaf-4278-be6f-32bee3777450\") " pod="openshift-cluster-node-tuning-operator/tuned-n27c5"
Sep 10 17:35:57.477993 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.476469    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/eca4abd0-e29b-4c77-9a3b-c9c32e5b5101-host\") pod \"node-ca-rkzwg\" (UID: \"eca4abd0-e29b-4c77-9a3b-c9c32e5b5101\") " pod="openshift-image-registry/node-ca-rkzwg"
Sep 10 17:35:57.477993 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.476492    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/85fae606-5227-404b-95a8-d7de60dc1041-cni-binary-copy\") pod \"multus-x2sqz\" (UID: \"85fae606-5227-404b-95a8-d7de60dc1041\") " pod="openshift-multus/multus-x2sqz"
Sep 10 17:35:57.477993 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.476521    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d4fad94e-7939-4949-bdfd-1ef38e016c19-host-slash\") pod \"ovnkube-node-4fsjz\" (UID: \"d4fad94e-7939-4949-bdfd-1ef38e016c19\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fsjz"
Sep 10 17:35:57.477993 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.476502    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/85fae606-5227-404b-95a8-d7de60dc1041-hostroot\") pod \"multus-x2sqz\" (UID: \"85fae606-5227-404b-95a8-d7de60dc1041\") " pod="openshift-multus/multus-x2sqz"
Sep 10 17:35:57.477993 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.476547    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d4fad94e-7939-4949-bdfd-1ef38e016c19-host-cni-bin\") pod \"ovnkube-node-4fsjz\" (UID: \"d4fad94e-7939-4949-bdfd-1ef38e016c19\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fsjz"
Sep 10 17:35:57.477993 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.476588    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/eca4abd0-e29b-4c77-9a3b-c9c32e5b5101-host\") pod \"node-ca-rkzwg\" (UID: \"eca4abd0-e29b-4c77-9a3b-c9c32e5b5101\") " pod="openshift-image-registry/node-ca-rkzwg"
Sep 10 17:35:57.478768 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.476592    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/88321231-7aaf-4278-be6f-32bee3777450-run\") pod \"tuned-n27c5\" (UID: \"88321231-7aaf-4278-be6f-32bee3777450\") " pod="openshift-cluster-node-tuning-operator/tuned-n27c5"
Sep 10 17:35:57.478768 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.476620    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/88321231-7aaf-4278-be6f-32bee3777450-sys\") pod \"tuned-n27c5\" (UID: \"88321231-7aaf-4278-be6f-32bee3777450\") " pod="openshift-cluster-node-tuning-operator/tuned-n27c5"
Sep 10 17:35:57.478768 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.476637    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/88321231-7aaf-4278-be6f-32bee3777450-run\") pod \"tuned-n27c5\" (UID: \"88321231-7aaf-4278-be6f-32bee3777450\") " pod="openshift-cluster-node-tuning-operator/tuned-n27c5"
Sep 10 17:35:57.478768 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.476644    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpjgl\" (UniqueName: \"kubernetes.io/projected/eca4abd0-e29b-4c77-9a3b-c9c32e5b5101-kube-api-access-hpjgl\") pod \"node-ca-rkzwg\" (UID: \"eca4abd0-e29b-4c77-9a3b-c9c32e5b5101\") " pod="openshift-image-registry/node-ca-rkzwg"
Sep 10 17:35:57.478768 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.476671    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/85fae606-5227-404b-95a8-d7de60dc1041-host-var-lib-cni-bin\") pod \"multus-x2sqz\" (UID: \"85fae606-5227-404b-95a8-d7de60dc1041\") " pod="openshift-multus/multus-x2sqz"
Sep 10 17:35:57.478768 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.476694    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dfj4\" (UniqueName: \"kubernetes.io/projected/85fae606-5227-404b-95a8-d7de60dc1041-kube-api-access-2dfj4\") pod \"multus-x2sqz\" (UID: \"85fae606-5227-404b-95a8-d7de60dc1041\") " pod="openshift-multus/multus-x2sqz"
Sep 10 17:35:57.478768 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.476714    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/85fae606-5227-404b-95a8-d7de60dc1041-system-cni-dir\") pod \"multus-x2sqz\" (UID: \"85fae606-5227-404b-95a8-d7de60dc1041\") " pod="openshift-multus/multus-x2sqz"
Sep 10 17:35:57.478768 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.476721    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4jkd\" (UniqueName: \"kubernetes.io/projected/48d6ee6d-6025-4ab7-a095-6a31180ac9f7-kube-api-access-z4jkd\") pod \"multus-additional-cni-plugins-drhxj\" (UID: \"48d6ee6d-6025-4ab7-a095-6a31180ac9f7\") " pod="openshift-multus/multus-additional-cni-plugins-drhxj"
Sep 10 17:35:57.478768 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.476746    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d4fad94e-7939-4949-bdfd-1ef38e016c19-host-run-netns\") pod \"ovnkube-node-4fsjz\" (UID: \"d4fad94e-7939-4949-bdfd-1ef38e016c19\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fsjz"
Sep 10 17:35:57.478768 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.476767    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3dc43868-1f6c-48fa-85fc-59607a50c103-hosts-file\") pod \"node-resolver-t4l6s\" (UID: \"3dc43868-1f6c-48fa-85fc-59607a50c103\") " pod="openshift-dns/node-resolver-t4l6s"
Sep 10 17:35:57.478768 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.476770    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/eca4abd0-e29b-4c77-9a3b-c9c32e5b5101-serviceca\") pod \"node-ca-rkzwg\" (UID: \"eca4abd0-e29b-4c77-9a3b-c9c32e5b5101\") " pod="openshift-image-registry/node-ca-rkzwg"
Sep 10 17:35:57.478768 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.476794    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/85fae606-5227-404b-95a8-d7de60dc1041-host-run-multus-certs\") pod \"multus-x2sqz\" (UID: \"85fae606-5227-404b-95a8-d7de60dc1041\") " pod="openshift-multus/multus-x2sqz"
Sep 10 17:35:57.478768 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.476812    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/85fae606-5227-404b-95a8-d7de60dc1041-multus-daemon-config\") pod \"multus-x2sqz\" (UID: \"85fae606-5227-404b-95a8-d7de60dc1041\") " pod="openshift-multus/multus-x2sqz"
Sep 10 17:35:57.478768 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.476816    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3fcacac9-c23e-4ba5-8b1d-dfeede6b57e3-metrics-certs\") pod \"network-metrics-daemon-ncmn7\" (UID: \"3fcacac9-c23e-4ba5-8b1d-dfeede6b57e3\") " pod="openshift-multus/network-metrics-daemon-ncmn7"
Sep 10 17:35:57.478768 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.476840    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/56f29fc7-7f69-42cb-b80e-1a2bfc298f1a-sys-fs\") pod \"aws-ebs-csi-driver-node-plzx8\" (UID: \"56f29fc7-7f69-42cb-b80e-1a2bfc298f1a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-plzx8"
Sep 10 17:35:57.478768 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.476863    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d4fad94e-7939-4949-bdfd-1ef38e016c19-systemd-units\") pod \"ovnkube-node-4fsjz\" (UID: \"d4fad94e-7939-4949-bdfd-1ef38e016c19\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fsjz"
Sep 10 17:35:57.478768 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.476885    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/88321231-7aaf-4278-be6f-32bee3777450-sys\") pod \"tuned-n27c5\" (UID: \"88321231-7aaf-4278-be6f-32bee3777450\") " pod="openshift-cluster-node-tuning-operator/tuned-n27c5"
Sep 10 17:35:57.478768 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.476952    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/85fae606-5227-404b-95a8-d7de60dc1041-host-var-lib-cni-bin\") pod \"multus-x2sqz\" (UID: \"85fae606-5227-404b-95a8-d7de60dc1041\") " pod="openshift-multus/multus-x2sqz"
Sep 10 17:35:57.479600 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.477226    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/85fae606-5227-404b-95a8-d7de60dc1041-cni-binary-copy\") pod \"multus-x2sqz\" (UID: \"85fae606-5227-404b-95a8-d7de60dc1041\") " pod="openshift-multus/multus-x2sqz"
Sep 10 17:35:57.479600 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.477311    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/88321231-7aaf-4278-be6f-32bee3777450-etc-modprobe-d\") pod \"tuned-n27c5\" (UID: \"88321231-7aaf-4278-be6f-32bee3777450\") " pod="openshift-cluster-node-tuning-operator/tuned-n27c5"
Sep 10 17:35:57.479600 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.477368    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/85fae606-5227-404b-95a8-d7de60dc1041-host-run-multus-certs\") pod \"multus-x2sqz\" (UID: \"85fae606-5227-404b-95a8-d7de60dc1041\") " pod="openshift-multus/multus-x2sqz"
Sep 10 17:35:57.479600 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.477429    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/eca4abd0-e29b-4c77-9a3b-c9c32e5b5101-serviceca\") pod \"node-ca-rkzwg\" (UID: \"eca4abd0-e29b-4c77-9a3b-c9c32e5b5101\") " pod="openshift-image-registry/node-ca-rkzwg"
Sep 10 17:35:57.479600 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.479583    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/88321231-7aaf-4278-be6f-32bee3777450-tmp\") pod \"tuned-n27c5\" (UID: \"88321231-7aaf-4278-be6f-32bee3777450\") " pod="openshift-cluster-node-tuning-operator/tuned-n27c5"
Sep 10 17:35:57.482247 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.482216    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/88321231-7aaf-4278-be6f-32bee3777450-etc-tuned\") pod \"tuned-n27c5\" (UID: \"88321231-7aaf-4278-be6f-32bee3777450\") " pod="openshift-cluster-node-tuning-operator/tuned-n27c5"
Sep 10 17:35:57.483061 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.483040    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zsqvf\" (UniqueName: \"kubernetes.io/projected/88321231-7aaf-4278-be6f-32bee3777450-kube-api-access-zsqvf\") pod \"tuned-n27c5\" (UID: \"88321231-7aaf-4278-be6f-32bee3777450\") " pod="openshift-cluster-node-tuning-operator/tuned-n27c5"
Sep 10 17:35:57.484690 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.484663    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dfj4\" (UniqueName: \"kubernetes.io/projected/85fae606-5227-404b-95a8-d7de60dc1041-kube-api-access-2dfj4\") pod \"multus-x2sqz\" (UID: \"85fae606-5227-404b-95a8-d7de60dc1041\") " pod="openshift-multus/multus-x2sqz"
Sep 10 17:35:57.484790 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.484722    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpjgl\" (UniqueName: \"kubernetes.io/projected/eca4abd0-e29b-4c77-9a3b-c9c32e5b5101-kube-api-access-hpjgl\") pod \"node-ca-rkzwg\" (UID: \"eca4abd0-e29b-4c77-9a3b-c9c32e5b5101\") " pod="openshift-image-registry/node-ca-rkzwg"
Sep 10 17:35:57.485043 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.485021    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ppvg\" (UniqueName: \"kubernetes.io/projected/3dc43868-1f6c-48fa-85fc-59607a50c103-kube-api-access-4ppvg\") pod \"node-resolver-t4l6s\" (UID: \"3dc43868-1f6c-48fa-85fc-59607a50c103\") " pod="openshift-dns/node-resolver-t4l6s"
Sep 10 17:35:57.544380 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:57.544234    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd750ec1329c03eb8935ed3e9791d542.slice/crio-4e76463ed4e1d1c1a88c496d5e03da44d91f5bd9d4a917ba143deaa3085337ee WatchSource:0}: Error finding container 4e76463ed4e1d1c1a88c496d5e03da44d91f5bd9d4a917ba143deaa3085337ee: Status 404 returned error can't find the container with id 4e76463ed4e1d1c1a88c496d5e03da44d91f5bd9d4a917ba143deaa3085337ee
Sep 10 17:35:57.550945 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.550928    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 17:35:57.556237 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:57.556220    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39d5711007096867f7ca5e1bd92438d9.slice/crio-1ae37df971b93797f3816312cec363907fd8fe87375bbb4af3bb1127ce457da8 WatchSource:0}: Error finding container 1ae37df971b93797f3816312cec363907fd8fe87375bbb4af3bb1127ce457da8: Status 404 returned error can't find the container with id 1ae37df971b93797f3816312cec363907fd8fe87375bbb4af3bb1127ce457da8
Sep 10 17:35:57.577729 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.577709    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/48d6ee6d-6025-4ab7-a095-6a31180ac9f7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-drhxj\" (UID: \"48d6ee6d-6025-4ab7-a095-6a31180ac9f7\") " pod="openshift-multus/multus-additional-cni-plugins-drhxj"
Sep 10 17:35:57.577825 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.577736    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/32a43910-3aa0-41e0-ab5b-ad08c1ab6b41-host-slash\") pod \"iptables-alerter-pdstz\" (UID: \"32a43910-3aa0-41e0-ab5b-ad08c1ab6b41\") " pod="openshift-network-operator/iptables-alerter-pdstz"
Sep 10 17:35:57.577825 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.577751    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1cd8f80e-f5c5-4e80-9abb-b43af15ed1d1-konnectivity-ca\") pod \"konnectivity-agent-lpm4k\" (UID: \"1cd8f80e-f5c5-4e80-9abb-b43af15ed1d1\") " pod="kube-system/konnectivity-agent-lpm4k"
Sep 10 17:35:57.577825 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.577765    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d4fad94e-7939-4949-bdfd-1ef38e016c19-run-systemd\") pod \"ovnkube-node-4fsjz\" (UID: \"d4fad94e-7939-4949-bdfd-1ef38e016c19\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fsjz"
Sep 10 17:35:57.577825 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.577779    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d4fad94e-7939-4949-bdfd-1ef38e016c19-run-ovn\") pod \"ovnkube-node-4fsjz\" (UID: \"d4fad94e-7939-4949-bdfd-1ef38e016c19\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fsjz"
Sep 10 17:35:57.577825 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.577825    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d4fad94e-7939-4949-bdfd-1ef38e016c19-run-systemd\") pod \"ovnkube-node-4fsjz\" (UID: \"d4fad94e-7939-4949-bdfd-1ef38e016c19\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fsjz"
Sep 10 17:35:57.578008 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.577831    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/32a43910-3aa0-41e0-ab5b-ad08c1ab6b41-host-slash\") pod \"iptables-alerter-pdstz\" (UID: \"32a43910-3aa0-41e0-ab5b-ad08c1ab6b41\") " pod="openshift-network-operator/iptables-alerter-pdstz"
Sep 10 17:35:57.578008 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.577856    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d4fad94e-7939-4949-bdfd-1ef38e016c19-run-ovn\") pod \"ovnkube-node-4fsjz\" (UID: \"d4fad94e-7939-4949-bdfd-1ef38e016c19\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fsjz"
Sep 10 17:35:57.578008 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.577881    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d4fad94e-7939-4949-bdfd-1ef38e016c19-ovn-node-metrics-cert\") pod \"ovnkube-node-4fsjz\" (UID: \"d4fad94e-7939-4949-bdfd-1ef38e016c19\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fsjz"
Sep 10 17:35:57.578008 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.577911    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/48d6ee6d-6025-4ab7-a095-6a31180ac9f7-cnibin\") pod \"multus-additional-cni-plugins-drhxj\" (UID: \"48d6ee6d-6025-4ab7-a095-6a31180ac9f7\") " pod="openshift-multus/multus-additional-cni-plugins-drhxj"
Sep 10 17:35:57.578008 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.577941    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/56f29fc7-7f69-42cb-b80e-1a2bfc298f1a-etc-selinux\") pod \"aws-ebs-csi-driver-node-plzx8\" (UID: \"56f29fc7-7f69-42cb-b80e-1a2bfc298f1a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-plzx8"
Sep 10 17:35:57.578008 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.577966    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d4fad94e-7939-4949-bdfd-1ef38e016c19-host-kubelet\") pod \"ovnkube-node-4fsjz\" (UID: \"d4fad94e-7939-4949-bdfd-1ef38e016c19\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fsjz"
Sep 10 17:35:57.578008 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.577982    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/48d6ee6d-6025-4ab7-a095-6a31180ac9f7-cnibin\") pod \"multus-additional-cni-plugins-drhxj\" (UID: \"48d6ee6d-6025-4ab7-a095-6a31180ac9f7\") " pod="openshift-multus/multus-additional-cni-plugins-drhxj"
Sep 10 17:35:57.578008 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.577987    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d4fad94e-7939-4949-bdfd-1ef38e016c19-log-socket\") pod \"ovnkube-node-4fsjz\" (UID: \"d4fad94e-7939-4949-bdfd-1ef38e016c19\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fsjz"
Sep 10 17:35:57.578370 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.578018    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d4fad94e-7939-4949-bdfd-1ef38e016c19-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-4fsjz\" (UID: \"d4fad94e-7939-4949-bdfd-1ef38e016c19\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fsjz"
Sep 10 17:35:57.578370 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.578028    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d4fad94e-7939-4949-bdfd-1ef38e016c19-log-socket\") pod \"ovnkube-node-4fsjz\" (UID: \"d4fad94e-7939-4949-bdfd-1ef38e016c19\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fsjz"
Sep 10 17:35:57.578370 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.578043    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/56f29fc7-7f69-42cb-b80e-1a2bfc298f1a-registration-dir\") pod \"aws-ebs-csi-driver-node-plzx8\" (UID: \"56f29fc7-7f69-42cb-b80e-1a2bfc298f1a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-plzx8"
Sep 10 17:35:57.578370 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.578067    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-csj86\" (UniqueName: \"kubernetes.io/projected/56f29fc7-7f69-42cb-b80e-1a2bfc298f1a-kube-api-access-csj86\") pod \"aws-ebs-csi-driver-node-plzx8\" (UID: \"56f29fc7-7f69-42cb-b80e-1a2bfc298f1a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-plzx8"
Sep 10 17:35:57.578370 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.578062    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d4fad94e-7939-4949-bdfd-1ef38e016c19-host-kubelet\") pod \"ovnkube-node-4fsjz\" (UID: \"d4fad94e-7939-4949-bdfd-1ef38e016c19\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fsjz"
Sep 10 17:35:57.578370 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.578088    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d4fad94e-7939-4949-bdfd-1ef38e016c19-host-cni-netd\") pod \"ovnkube-node-4fsjz\" (UID: \"d4fad94e-7939-4949-bdfd-1ef38e016c19\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fsjz"
Sep 10 17:35:57.578370 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.578113    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/32a43910-3aa0-41e0-ab5b-ad08c1ab6b41-iptables-alerter-script\") pod \"iptables-alerter-pdstz\" (UID: \"32a43910-3aa0-41e0-ab5b-ad08c1ab6b41\") " pod="openshift-network-operator/iptables-alerter-pdstz"
Sep 10 17:35:57.578370 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.578118    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/56f29fc7-7f69-42cb-b80e-1a2bfc298f1a-registration-dir\") pod \"aws-ebs-csi-driver-node-plzx8\" (UID: \"56f29fc7-7f69-42cb-b80e-1a2bfc298f1a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-plzx8"
Sep 10 17:35:57.578370 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.578072    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/56f29fc7-7f69-42cb-b80e-1a2bfc298f1a-etc-selinux\") pod \"aws-ebs-csi-driver-node-plzx8\" (UID: \"56f29fc7-7f69-42cb-b80e-1a2bfc298f1a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-plzx8"
Sep 10 17:35:57.578370 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.578142    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/48d6ee6d-6025-4ab7-a095-6a31180ac9f7-cni-binary-copy\") pod \"multus-additional-cni-plugins-drhxj\" (UID: \"48d6ee6d-6025-4ab7-a095-6a31180ac9f7\") " pod="openshift-multus/multus-additional-cni-plugins-drhxj"
Sep 10 17:35:57.578370 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.578164    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d4fad94e-7939-4949-bdfd-1ef38e016c19-var-lib-openvswitch\") pod \"ovnkube-node-4fsjz\" (UID: \"d4fad94e-7939-4949-bdfd-1ef38e016c19\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fsjz"
Sep 10 17:35:57.578370 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.578188    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d4fad94e-7939-4949-bdfd-1ef38e016c19-etc-openvswitch\") pod \"ovnkube-node-4fsjz\" (UID: \"d4fad94e-7939-4949-bdfd-1ef38e016c19\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fsjz"
Sep 10 17:35:57.578370 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.578212    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d4fad94e-7939-4949-bdfd-1ef38e016c19-env-overrides\") pod \"ovnkube-node-4fsjz\" (UID: \"d4fad94e-7939-4949-bdfd-1ef38e016c19\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fsjz"
Sep 10 17:35:57.578370 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.578258    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e0fe9300-104f-489d-9962-908000359645-original-pull-secret\") pod \"global-pull-secret-syncer-m4npt\" (UID: \"e0fe9300-104f-489d-9962-908000359645\") " pod="kube-system/global-pull-secret-syncer-m4npt"
Sep 10 17:35:57.578370 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.578276    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1cd8f80e-f5c5-4e80-9abb-b43af15ed1d1-konnectivity-ca\") pod \"konnectivity-agent-lpm4k\" (UID: \"1cd8f80e-f5c5-4e80-9abb-b43af15ed1d1\") " pod="kube-system/konnectivity-agent-lpm4k"
Sep 10 17:35:57.578370 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.578285    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g56qb\" (UniqueName: \"kubernetes.io/projected/3fcacac9-c23e-4ba5-8b1d-dfeede6b57e3-kube-api-access-g56qb\") pod \"network-metrics-daemon-ncmn7\" (UID: \"3fcacac9-c23e-4ba5-8b1d-dfeede6b57e3\") " pod="openshift-multus/network-metrics-daemon-ncmn7"
Sep 10 17:35:57.579084 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.578317    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/48d6ee6d-6025-4ab7-a095-6a31180ac9f7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-drhxj\" (UID: \"48d6ee6d-6025-4ab7-a095-6a31180ac9f7\") " pod="openshift-multus/multus-additional-cni-plugins-drhxj"
Sep 10 17:35:57.579084 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.578323    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/56f29fc7-7f69-42cb-b80e-1a2bfc298f1a-device-dir\") pod \"aws-ebs-csi-driver-node-plzx8\" (UID: \"56f29fc7-7f69-42cb-b80e-1a2bfc298f1a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-plzx8"
Sep 10 17:35:57.579084 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.578347    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d4fad94e-7939-4949-bdfd-1ef38e016c19-etc-openvswitch\") pod \"ovnkube-node-4fsjz\" (UID: \"d4fad94e-7939-4949-bdfd-1ef38e016c19\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fsjz"
Sep 10 17:35:57.579084 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.578361    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/56f29fc7-7f69-42cb-b80e-1a2bfc298f1a-device-dir\") pod \"aws-ebs-csi-driver-node-plzx8\" (UID: \"56f29fc7-7f69-42cb-b80e-1a2bfc298f1a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-plzx8"
Sep 10 17:35:57.579084 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.578366    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d4fad94e-7939-4949-bdfd-1ef38e016c19-host-run-ovn-kubernetes\") pod \"ovnkube-node-4fsjz\" (UID: \"d4fad94e-7939-4949-bdfd-1ef38e016c19\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fsjz"
Sep 10 17:35:57.579084 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.578163    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d4fad94e-7939-4949-bdfd-1ef38e016c19-host-cni-netd\") pod \"ovnkube-node-4fsjz\" (UID: \"d4fad94e-7939-4949-bdfd-1ef38e016c19\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fsjz"
Sep 10 17:35:57.579084 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.578400    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d4fad94e-7939-4949-bdfd-1ef38e016c19-host-run-ovn-kubernetes\") pod \"ovnkube-node-4fsjz\" (UID: \"d4fad94e-7939-4949-bdfd-1ef38e016c19\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fsjz"
Sep 10 17:35:57.579084 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.578404    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d4fad94e-7939-4949-bdfd-1ef38e016c19-var-lib-openvswitch\") pod \"ovnkube-node-4fsjz\" (UID: \"d4fad94e-7939-4949-bdfd-1ef38e016c19\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fsjz"
Sep 10 17:35:57.579084 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.578422    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98dkx\" (UniqueName: \"kubernetes.io/projected/d4fad94e-7939-4949-bdfd-1ef38e016c19-kube-api-access-98dkx\") pod \"ovnkube-node-4fsjz\" (UID: \"d4fad94e-7939-4949-bdfd-1ef38e016c19\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fsjz"
Sep 10 17:35:57.579084 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.578451    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e0fe9300-104f-489d-9962-908000359645-kubelet-config\") pod \"global-pull-secret-syncer-m4npt\" (UID: \"e0fe9300-104f-489d-9962-908000359645\") " pod="kube-system/global-pull-secret-syncer-m4npt"
Sep 10 17:35:57.579084 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.578482    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/48d6ee6d-6025-4ab7-a095-6a31180ac9f7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-drhxj\" (UID: \"48d6ee6d-6025-4ab7-a095-6a31180ac9f7\") " pod="openshift-multus/multus-additional-cni-plugins-drhxj"
Sep 10 17:35:57.579084 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:35:57.578493    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 17:35:57.579084 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:35:57.578588    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0fe9300-104f-489d-9962-908000359645-original-pull-secret podName:e0fe9300-104f-489d-9962-908000359645 nodeName:}" failed. No retries permitted until 2026-09-10 17:35:58.078544126 +0000 UTC m=+2.078374174 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e0fe9300-104f-489d-9962-908000359645-original-pull-secret") pod "global-pull-secret-syncer-m4npt" (UID: "e0fe9300-104f-489d-9962-908000359645") : object "kube-system"/"original-pull-secret" not registered
Sep 10 17:35:57.579084 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.578652    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/48d6ee6d-6025-4ab7-a095-6a31180ac9f7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-drhxj\" (UID: \"48d6ee6d-6025-4ab7-a095-6a31180ac9f7\") " pod="openshift-multus/multus-additional-cni-plugins-drhxj"
Sep 10 17:35:57.579084 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.578659    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9zqd\" (UniqueName: \"kubernetes.io/projected/8436b9d2-f47a-411e-a695-cb52b8277b62-kube-api-access-p9zqd\") pod \"network-check-target-8k8z5\" (UID: \"8436b9d2-f47a-411e-a695-cb52b8277b62\") " pod="openshift-network-diagnostics/network-check-target-8k8z5"
Sep 10 17:35:57.579084 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.578687    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wsc6v\" (UniqueName: \"kubernetes.io/projected/32a43910-3aa0-41e0-ab5b-ad08c1ab6b41-kube-api-access-wsc6v\") pod \"iptables-alerter-pdstz\" (UID: \"32a43910-3aa0-41e0-ab5b-ad08c1ab6b41\") " pod="openshift-network-operator/iptables-alerter-pdstz"
Sep 10 17:35:57.579711 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.578709    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d4fad94e-7939-4949-bdfd-1ef38e016c19-ovnkube-config\") pod \"ovnkube-node-4fsjz\" (UID: \"d4fad94e-7939-4949-bdfd-1ef38e016c19\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fsjz"
Sep 10 17:35:57.579711 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.578071    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d4fad94e-7939-4949-bdfd-1ef38e016c19-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-4fsjz\" (UID: \"d4fad94e-7939-4949-bdfd-1ef38e016c19\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fsjz"
Sep 10 17:35:57.579711 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.578712    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/48d6ee6d-6025-4ab7-a095-6a31180ac9f7-cni-binary-copy\") pod \"multus-additional-cni-plugins-drhxj\" (UID: \"48d6ee6d-6025-4ab7-a095-6a31180ac9f7\") " pod="openshift-multus/multus-additional-cni-plugins-drhxj"
Sep 10 17:35:57.579711 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.578730    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/56f29fc7-7f69-42cb-b80e-1a2bfc298f1a-socket-dir\") pod \"aws-ebs-csi-driver-node-plzx8\" (UID: \"56f29fc7-7f69-42cb-b80e-1a2bfc298f1a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-plzx8"
Sep 10 17:35:57.579711 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.578734    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/32a43910-3aa0-41e0-ab5b-ad08c1ab6b41-iptables-alerter-script\") pod \"iptables-alerter-pdstz\" (UID: \"32a43910-3aa0-41e0-ab5b-ad08c1ab6b41\") " pod="openshift-network-operator/iptables-alerter-pdstz"
Sep 10 17:35:57.579711 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.578707    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e0fe9300-104f-489d-9962-908000359645-kubelet-config\") pod \"global-pull-secret-syncer-m4npt\" (UID: \"e0fe9300-104f-489d-9962-908000359645\") " pod="kube-system/global-pull-secret-syncer-m4npt"
Sep 10 17:35:57.579711 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.578768    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d4fad94e-7939-4949-bdfd-1ef38e016c19-node-log\") pod \"ovnkube-node-4fsjz\" (UID: \"d4fad94e-7939-4949-bdfd-1ef38e016c19\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fsjz"
Sep 10 17:35:57.579711 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.578796    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d4fad94e-7939-4949-bdfd-1ef38e016c19-host-slash\") pod \"ovnkube-node-4fsjz\" (UID: \"d4fad94e-7939-4949-bdfd-1ef38e016c19\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fsjz"
Sep 10 17:35:57.579711 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.578811    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d4fad94e-7939-4949-bdfd-1ef38e016c19-host-cni-bin\") pod \"ovnkube-node-4fsjz\" (UID: \"d4fad94e-7939-4949-bdfd-1ef38e016c19\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fsjz"
Sep 10 17:35:57.579711 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.578795    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d4fad94e-7939-4949-bdfd-1ef38e016c19-env-overrides\") pod \"ovnkube-node-4fsjz\" (UID: \"d4fad94e-7939-4949-bdfd-1ef38e016c19\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fsjz"
Sep 10 17:35:57.579711 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.578839    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4jkd\" (UniqueName: \"kubernetes.io/projected/48d6ee6d-6025-4ab7-a095-6a31180ac9f7-kube-api-access-z4jkd\") pod \"multus-additional-cni-plugins-drhxj\" (UID: \"48d6ee6d-6025-4ab7-a095-6a31180ac9f7\") " pod="openshift-multus/multus-additional-cni-plugins-drhxj"
Sep 10 17:35:57.579711 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.578858    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d4fad94e-7939-4949-bdfd-1ef38e016c19-host-slash\") pod \"ovnkube-node-4fsjz\" (UID: \"d4fad94e-7939-4949-bdfd-1ef38e016c19\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fsjz"
Sep 10 17:35:57.579711 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.578863    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d4fad94e-7939-4949-bdfd-1ef38e016c19-host-run-netns\") pod \"ovnkube-node-4fsjz\" (UID: \"d4fad94e-7939-4949-bdfd-1ef38e016c19\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fsjz"
Sep 10 17:35:57.579711 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.578877    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d4fad94e-7939-4949-bdfd-1ef38e016c19-host-cni-bin\") pod \"ovnkube-node-4fsjz\" (UID: \"d4fad94e-7939-4949-bdfd-1ef38e016c19\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fsjz"
Sep 10 17:35:57.579711 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.578896    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3fcacac9-c23e-4ba5-8b1d-dfeede6b57e3-metrics-certs\") pod \"network-metrics-daemon-ncmn7\" (UID: \"3fcacac9-c23e-4ba5-8b1d-dfeede6b57e3\") " pod="openshift-multus/network-metrics-daemon-ncmn7"
Sep 10 17:35:57.579711 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.578905    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/56f29fc7-7f69-42cb-b80e-1a2bfc298f1a-socket-dir\") pod \"aws-ebs-csi-driver-node-plzx8\" (UID: \"56f29fc7-7f69-42cb-b80e-1a2bfc298f1a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-plzx8"
Sep 10 17:35:57.579711 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.578920    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/56f29fc7-7f69-42cb-b80e-1a2bfc298f1a-sys-fs\") pod \"aws-ebs-csi-driver-node-plzx8\" (UID: \"56f29fc7-7f69-42cb-b80e-1a2bfc298f1a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-plzx8"
Sep 10 17:35:57.580240 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.578943    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d4fad94e-7939-4949-bdfd-1ef38e016c19-systemd-units\") pod \"ovnkube-node-4fsjz\" (UID: \"d4fad94e-7939-4949-bdfd-1ef38e016c19\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fsjz"
Sep 10 17:35:57.580240 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.578945    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d4fad94e-7939-4949-bdfd-1ef38e016c19-host-run-netns\") pod \"ovnkube-node-4fsjz\" (UID: \"d4fad94e-7939-4949-bdfd-1ef38e016c19\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fsjz"
Sep 10 17:35:57.580240 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.578965    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d4fad94e-7939-4949-bdfd-1ef38e016c19-run-openvswitch\") pod \"ovnkube-node-4fsjz\" (UID: \"d4fad94e-7939-4949-bdfd-1ef38e016c19\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fsjz"
Sep 10 17:35:57.580240 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.578946    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d4fad94e-7939-4949-bdfd-1ef38e016c19-node-log\") pod \"ovnkube-node-4fsjz\" (UID: \"d4fad94e-7939-4949-bdfd-1ef38e016c19\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fsjz"
Sep 10 17:35:57.580240 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.579016    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d4fad94e-7939-4949-bdfd-1ef38e016c19-systemd-units\") pod \"ovnkube-node-4fsjz\" (UID: \"d4fad94e-7939-4949-bdfd-1ef38e016c19\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fsjz"
Sep 10 17:35:57.580240 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.578988    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d4fad94e-7939-4949-bdfd-1ef38e016c19-ovnkube-script-lib\") pod \"ovnkube-node-4fsjz\" (UID: \"d4fad94e-7939-4949-bdfd-1ef38e016c19\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fsjz"
Sep 10 17:35:57.580240 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.579046    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1cd8f80e-f5c5-4e80-9abb-b43af15ed1d1-agent-certs\") pod \"konnectivity-agent-lpm4k\" (UID: \"1cd8f80e-f5c5-4e80-9abb-b43af15ed1d1\") " pod="kube-system/konnectivity-agent-lpm4k"
Sep 10 17:35:57.580240 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.579050    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d4fad94e-7939-4949-bdfd-1ef38e016c19-run-openvswitch\") pod \"ovnkube-node-4fsjz\" (UID: \"d4fad94e-7939-4949-bdfd-1ef38e016c19\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fsjz"
Sep 10 17:35:57.580240 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.579104    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/56f29fc7-7f69-42cb-b80e-1a2bfc298f1a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-plzx8\" (UID: \"56f29fc7-7f69-42cb-b80e-1a2bfc298f1a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-plzx8"
Sep 10 17:35:57.580240 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.579111    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/56f29fc7-7f69-42cb-b80e-1a2bfc298f1a-sys-fs\") pod \"aws-ebs-csi-driver-node-plzx8\" (UID: \"56f29fc7-7f69-42cb-b80e-1a2bfc298f1a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-plzx8"
Sep 10 17:35:57.580240 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.579128    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/48d6ee6d-6025-4ab7-a095-6a31180ac9f7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-drhxj\" (UID: \"48d6ee6d-6025-4ab7-a095-6a31180ac9f7\") " pod="openshift-multus/multus-additional-cni-plugins-drhxj"
Sep 10 17:35:57.580240 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:35:57.578982    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 17:35:57.580240 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.579173    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/56f29fc7-7f69-42cb-b80e-1a2bfc298f1a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-plzx8\" (UID: \"56f29fc7-7f69-42cb-b80e-1a2bfc298f1a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-plzx8"
Sep 10 17:35:57.580240 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:35:57.579187    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3fcacac9-c23e-4ba5-8b1d-dfeede6b57e3-metrics-certs podName:3fcacac9-c23e-4ba5-8b1d-dfeede6b57e3 nodeName:}" failed. No retries permitted until 2026-09-10 17:35:58.079170635 +0000 UTC m=+2.079000697 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3fcacac9-c23e-4ba5-8b1d-dfeede6b57e3-metrics-certs") pod "network-metrics-daemon-ncmn7" (UID: "3fcacac9-c23e-4ba5-8b1d-dfeede6b57e3") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 17:35:57.580240 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.579212    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e0fe9300-104f-489d-9962-908000359645-dbus\") pod \"global-pull-secret-syncer-m4npt\" (UID: \"e0fe9300-104f-489d-9962-908000359645\") " pod="kube-system/global-pull-secret-syncer-m4npt"
Sep 10 17:35:57.580240 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.579238    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/48d6ee6d-6025-4ab7-a095-6a31180ac9f7-system-cni-dir\") pod \"multus-additional-cni-plugins-drhxj\" (UID: \"48d6ee6d-6025-4ab7-a095-6a31180ac9f7\") " pod="openshift-multus/multus-additional-cni-plugins-drhxj"
Sep 10 17:35:57.580240 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.579305    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/48d6ee6d-6025-4ab7-a095-6a31180ac9f7-system-cni-dir\") pod \"multus-additional-cni-plugins-drhxj\" (UID: \"48d6ee6d-6025-4ab7-a095-6a31180ac9f7\") " pod="openshift-multus/multus-additional-cni-plugins-drhxj"
Sep 10 17:35:57.580800 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.579369    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e0fe9300-104f-489d-9962-908000359645-dbus\") pod \"global-pull-secret-syncer-m4npt\" (UID: \"e0fe9300-104f-489d-9962-908000359645\") " pod="kube-system/global-pull-secret-syncer-m4npt"
Sep 10 17:35:57.580800 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.579481    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d4fad94e-7939-4949-bdfd-1ef38e016c19-ovnkube-config\") pod \"ovnkube-node-4fsjz\" (UID: \"d4fad94e-7939-4949-bdfd-1ef38e016c19\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fsjz"
Sep 10 17:35:57.580800 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.579510    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d4fad94e-7939-4949-bdfd-1ef38e016c19-ovnkube-script-lib\") pod \"ovnkube-node-4fsjz\" (UID: \"d4fad94e-7939-4949-bdfd-1ef38e016c19\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fsjz"
Sep 10 17:35:57.580800 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.579549    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/48d6ee6d-6025-4ab7-a095-6a31180ac9f7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-drhxj\" (UID: \"48d6ee6d-6025-4ab7-a095-6a31180ac9f7\") " pod="openshift-multus/multus-additional-cni-plugins-drhxj"
Sep 10 17:35:57.581483 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.581466    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1cd8f80e-f5c5-4e80-9abb-b43af15ed1d1-agent-certs\") pod \"konnectivity-agent-lpm4k\" (UID: \"1cd8f80e-f5c5-4e80-9abb-b43af15ed1d1\") " pod="kube-system/konnectivity-agent-lpm4k"
Sep 10 17:35:57.581739 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.581722    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d4fad94e-7939-4949-bdfd-1ef38e016c19-ovn-node-metrics-cert\") pod \"ovnkube-node-4fsjz\" (UID: \"d4fad94e-7939-4949-bdfd-1ef38e016c19\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fsjz"
Sep 10 17:35:57.588630 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:35:57.588615    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 17:35:57.588693 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:35:57.588633    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 17:35:57.588693 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:35:57.588643    2579 projected.go:196] Error preparing data for projected volume kube-api-access-p9zqd for pod openshift-network-diagnostics/network-check-target-8k8z5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 17:35:57.588775 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:35:57.588695    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8436b9d2-f47a-411e-a695-cb52b8277b62-kube-api-access-p9zqd podName:8436b9d2-f47a-411e-a695-cb52b8277b62 nodeName:}" failed. No retries permitted until 2026-09-10 17:35:58.088680882 +0000 UTC m=+2.088510944 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-p9zqd" (UniqueName: "kubernetes.io/projected/8436b9d2-f47a-411e-a695-cb52b8277b62-kube-api-access-p9zqd") pod "network-check-target-8k8z5" (UID: "8436b9d2-f47a-411e-a695-cb52b8277b62") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 17:35:57.590702 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.590682    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-98dkx\" (UniqueName: \"kubernetes.io/projected/d4fad94e-7939-4949-bdfd-1ef38e016c19-kube-api-access-98dkx\") pod \"ovnkube-node-4fsjz\" (UID: \"d4fad94e-7939-4949-bdfd-1ef38e016c19\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fsjz"
Sep 10 17:35:57.591029 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.591006    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wsc6v\" (UniqueName: \"kubernetes.io/projected/32a43910-3aa0-41e0-ab5b-ad08c1ab6b41-kube-api-access-wsc6v\") pod \"iptables-alerter-pdstz\" (UID: \"32a43910-3aa0-41e0-ab5b-ad08c1ab6b41\") " pod="openshift-network-operator/iptables-alerter-pdstz"
Sep 10 17:35:57.591112 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.591089    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g56qb\" (UniqueName: \"kubernetes.io/projected/3fcacac9-c23e-4ba5-8b1d-dfeede6b57e3-kube-api-access-g56qb\") pod \"network-metrics-daemon-ncmn7\" (UID: \"3fcacac9-c23e-4ba5-8b1d-dfeede6b57e3\") " pod="openshift-multus/network-metrics-daemon-ncmn7"
Sep 10 17:35:57.591301 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.591282    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-csj86\" (UniqueName: \"kubernetes.io/projected/56f29fc7-7f69-42cb-b80e-1a2bfc298f1a-kube-api-access-csj86\") pod \"aws-ebs-csi-driver-node-plzx8\" (UID: \"56f29fc7-7f69-42cb-b80e-1a2bfc298f1a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-plzx8"
Sep 10 17:35:57.591356 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.591303    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4jkd\" (UniqueName: \"kubernetes.io/projected/48d6ee6d-6025-4ab7-a095-6a31180ac9f7-kube-api-access-z4jkd\") pod \"multus-additional-cni-plugins-drhxj\" (UID: \"48d6ee6d-6025-4ab7-a095-6a31180ac9f7\") " pod="openshift-multus/multus-additional-cni-plugins-drhxj"
Sep 10 17:35:57.701751 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.701682    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-n27c5"
Sep 10 17:35:57.705479 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.705447    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-t4l6s"
Sep 10 17:35:57.711695 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:57.711673    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88321231_7aaf_4278_be6f_32bee3777450.slice/crio-bb85f7f1c2fde669d4a403be265a575c772f0fe8dea917aae9c5c348c051aeff WatchSource:0}: Error finding container bb85f7f1c2fde669d4a403be265a575c772f0fe8dea917aae9c5c348c051aeff: Status 404 returned error can't find the container with id bb85f7f1c2fde669d4a403be265a575c772f0fe8dea917aae9c5c348c051aeff
Sep 10 17:35:57.713711 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:57.713690    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3dc43868_1f6c_48fa_85fc_59607a50c103.slice/crio-8c5506df69d3a5ee738d6c2fdb2e2fc6a991b1259625075e7789ecbec8cf81d2 WatchSource:0}: Error finding container 8c5506df69d3a5ee738d6c2fdb2e2fc6a991b1259625075e7789ecbec8cf81d2: Status 404 returned error can't find the container with id 8c5506df69d3a5ee738d6c2fdb2e2fc6a991b1259625075e7789ecbec8cf81d2
Sep 10 17:35:57.734212 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.734193    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-rkzwg"
Sep 10 17:35:57.738892 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.738877    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-x2sqz"
Sep 10 17:35:57.743145 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:57.743122    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeca4abd0_e29b_4c77_9a3b_c9c32e5b5101.slice/crio-7ce1ff21b09df006f00d166456144394701209ff63cc72d7ca67cccc0500043d WatchSource:0}: Error finding container 7ce1ff21b09df006f00d166456144394701209ff63cc72d7ca67cccc0500043d: Status 404 returned error can't find the container with id 7ce1ff21b09df006f00d166456144394701209ff63cc72d7ca67cccc0500043d
Sep 10 17:35:57.746462 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:57.746443    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85fae606_5227_404b_95a8_d7de60dc1041.slice/crio-58d677aaeaebfbef980ceefff6d72ed3758e14ee775a0b7f1778357c96fb2590 WatchSource:0}: Error finding container 58d677aaeaebfbef980ceefff6d72ed3758e14ee775a0b7f1778357c96fb2590: Status 404 returned error can't find the container with id 58d677aaeaebfbef980ceefff6d72ed3758e14ee775a0b7f1778357c96fb2590
Sep 10 17:35:57.767959 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.767936    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-pdstz"
Sep 10 17:35:57.777825 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:57.777807    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32a43910_3aa0_41e0_ab5b_ad08c1ab6b41.slice/crio-536e095f5761eda91e5330f8b1cfb2a2712061b6e54f1336396992aa4e590723 WatchSource:0}: Error finding container 536e095f5761eda91e5330f8b1cfb2a2712061b6e54f1336396992aa4e590723: Status 404 returned error can't find the container with id 536e095f5761eda91e5330f8b1cfb2a2712061b6e54f1336396992aa4e590723
Sep 10 17:35:57.791736 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.791719    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-plzx8"
Sep 10 17:35:57.799057 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:57.799038    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56f29fc7_7f69_42cb_b80e_1a2bfc298f1a.slice/crio-3b9a0c989b1603e7fe0ae53cbc18fc0493e57b93a408298d4f91bb62ae1dcf08 WatchSource:0}: Error finding container 3b9a0c989b1603e7fe0ae53cbc18fc0493e57b93a408298d4f91bb62ae1dcf08: Status 404 returned error can't find the container with id 3b9a0c989b1603e7fe0ae53cbc18fc0493e57b93a408298d4f91bb62ae1dcf08
Sep 10 17:35:57.807012 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.806998    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-drhxj"
Sep 10 17:35:57.811762 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.811744    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-4fsjz"
Sep 10 17:35:57.814679 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:57.814517    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48d6ee6d_6025_4ab7_a095_6a31180ac9f7.slice/crio-2476cab16fe091bd57f21977bedccd245b9160900eff27e1a4b29c871732eb14 WatchSource:0}: Error finding container 2476cab16fe091bd57f21977bedccd245b9160900eff27e1a4b29c871732eb14: Status 404 returned error can't find the container with id 2476cab16fe091bd57f21977bedccd245b9160900eff27e1a4b29c871732eb14
Sep 10 17:35:57.818882 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.818811    2579 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 17:35:57.821250 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:57.821231    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4fad94e_7939_4949_bdfd_1ef38e016c19.slice/crio-63160f37ea908664295db135f9e0caf09cd8fad2d32207ecc1aa087925ba4f72 WatchSource:0}: Error finding container 63160f37ea908664295db135f9e0caf09cd8fad2d32207ecc1aa087925ba4f72: Status 404 returned error can't find the container with id 63160f37ea908664295db135f9e0caf09cd8fad2d32207ecc1aa087925ba4f72
Sep 10 17:35:57.830763 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.830746    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-lpm4k"
Sep 10 17:35:57.839700 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:35:57.839681    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1cd8f80e_f5c5_4e80_9abb_b43af15ed1d1.slice/crio-4f35c43a0973f353dd00d7a2dfa869370a60da2d91710c900200b9f726a0049b WatchSource:0}: Error finding container 4f35c43a0973f353dd00d7a2dfa869370a60da2d91710c900200b9f726a0049b: Status 404 returned error can't find the container with id 4f35c43a0973f353dd00d7a2dfa869370a60da2d91710c900200b9f726a0049b
Sep 10 17:35:57.930070 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:57.930051    2579 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 17:35:58.083426 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:58.083397    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3fcacac9-c23e-4ba5-8b1d-dfeede6b57e3-metrics-certs\") pod \"network-metrics-daemon-ncmn7\" (UID: \"3fcacac9-c23e-4ba5-8b1d-dfeede6b57e3\") " pod="openshift-multus/network-metrics-daemon-ncmn7"
Sep 10 17:35:58.083592 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:58.083451    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e0fe9300-104f-489d-9962-908000359645-original-pull-secret\") pod \"global-pull-secret-syncer-m4npt\" (UID: \"e0fe9300-104f-489d-9962-908000359645\") " pod="kube-system/global-pull-secret-syncer-m4npt"
Sep 10 17:35:58.083592 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:35:58.083569    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 17:35:58.083702 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:35:58.083598    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 17:35:58.083702 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:35:58.083638    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0fe9300-104f-489d-9962-908000359645-original-pull-secret podName:e0fe9300-104f-489d-9962-908000359645 nodeName:}" failed. No retries permitted until 2026-09-10 17:35:59.083624853 +0000 UTC m=+3.083454900 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e0fe9300-104f-489d-9962-908000359645-original-pull-secret") pod "global-pull-secret-syncer-m4npt" (UID: "e0fe9300-104f-489d-9962-908000359645") : object "kube-system"/"original-pull-secret" not registered
Sep 10 17:35:58.083702 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:35:58.083663    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3fcacac9-c23e-4ba5-8b1d-dfeede6b57e3-metrics-certs podName:3fcacac9-c23e-4ba5-8b1d-dfeede6b57e3 nodeName:}" failed. No retries permitted until 2026-09-10 17:35:59.083644891 +0000 UTC m=+3.083474945 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3fcacac9-c23e-4ba5-8b1d-dfeede6b57e3-metrics-certs") pod "network-metrics-daemon-ncmn7" (UID: "3fcacac9-c23e-4ba5-8b1d-dfeede6b57e3") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 17:35:58.184242 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:58.184209    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9zqd\" (UniqueName: \"kubernetes.io/projected/8436b9d2-f47a-411e-a695-cb52b8277b62-kube-api-access-p9zqd\") pod \"network-check-target-8k8z5\" (UID: \"8436b9d2-f47a-411e-a695-cb52b8277b62\") " pod="openshift-network-diagnostics/network-check-target-8k8z5"
Sep 10 17:35:58.184404 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:35:58.184390    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 17:35:58.184404 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:35:58.184406    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 17:35:58.184533 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:35:58.184416    2579 projected.go:196] Error preparing data for projected volume kube-api-access-p9zqd for pod openshift-network-diagnostics/network-check-target-8k8z5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 17:35:58.184533 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:35:58.184471    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8436b9d2-f47a-411e-a695-cb52b8277b62-kube-api-access-p9zqd podName:8436b9d2-f47a-411e-a695-cb52b8277b62 nodeName:}" failed. No retries permitted until 2026-09-10 17:35:59.184454793 +0000 UTC m=+3.184284839 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-p9zqd" (UniqueName: "kubernetes.io/projected/8436b9d2-f47a-411e-a695-cb52b8277b62-kube-api-access-p9zqd") pod "network-check-target-8k8z5" (UID: "8436b9d2-f47a-411e-a695-cb52b8277b62") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 17:35:58.418797 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:58.418581    2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-09 17:30:57 +0000 UTC" deadline="2028-03-30 05:01:37.612011017 +0000 UTC"
Sep 10 17:35:58.418797 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:58.418644    2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13595h25m39.193371687s"
Sep 10 17:35:58.514962 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:58.514895    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-lpm4k" event={"ID":"1cd8f80e-f5c5-4e80-9abb-b43af15ed1d1","Type":"ContainerStarted","Data":"4f35c43a0973f353dd00d7a2dfa869370a60da2d91710c900200b9f726a0049b"}
Sep 10 17:35:58.537404 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:58.537252    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-plzx8" event={"ID":"56f29fc7-7f69-42cb-b80e-1a2bfc298f1a","Type":"ContainerStarted","Data":"3b9a0c989b1603e7fe0ae53cbc18fc0493e57b93a408298d4f91bb62ae1dcf08"}
Sep 10 17:35:58.549304 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:58.549268    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-x2sqz" event={"ID":"85fae606-5227-404b-95a8-d7de60dc1041","Type":"ContainerStarted","Data":"58d677aaeaebfbef980ceefff6d72ed3758e14ee775a0b7f1778357c96fb2590"}
Sep 10 17:35:58.565954 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:58.565879    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-n27c5" event={"ID":"88321231-7aaf-4278-be6f-32bee3777450","Type":"ContainerStarted","Data":"bb85f7f1c2fde669d4a403be265a575c772f0fe8dea917aae9c5c348c051aeff"}
Sep 10 17:35:58.574194 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:58.574132    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-122.ec2.internal" event={"ID":"bd750ec1329c03eb8935ed3e9791d542","Type":"ContainerStarted","Data":"4e76463ed4e1d1c1a88c496d5e03da44d91f5bd9d4a917ba143deaa3085337ee"}
Sep 10 17:35:58.593248 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:58.593223    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4fsjz" event={"ID":"d4fad94e-7939-4949-bdfd-1ef38e016c19","Type":"ContainerStarted","Data":"63160f37ea908664295db135f9e0caf09cd8fad2d32207ecc1aa087925ba4f72"}
Sep 10 17:35:58.598914 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:58.598886    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-drhxj" event={"ID":"48d6ee6d-6025-4ab7-a095-6a31180ac9f7","Type":"ContainerStarted","Data":"2476cab16fe091bd57f21977bedccd245b9160900eff27e1a4b29c871732eb14"}
Sep 10 17:35:58.600704 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:58.600681    2579 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 17:35:58.612709 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:58.612609    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-pdstz" event={"ID":"32a43910-3aa0-41e0-ab5b-ad08c1ab6b41","Type":"ContainerStarted","Data":"536e095f5761eda91e5330f8b1cfb2a2712061b6e54f1336396992aa4e590723"}
Sep 10 17:35:58.657744 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:58.657709    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-rkzwg" event={"ID":"eca4abd0-e29b-4c77-9a3b-c9c32e5b5101","Type":"ContainerStarted","Data":"7ce1ff21b09df006f00d166456144394701209ff63cc72d7ca67cccc0500043d"}
Sep 10 17:35:58.667727 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:58.666978    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-t4l6s" event={"ID":"3dc43868-1f6c-48fa-85fc-59607a50c103","Type":"ContainerStarted","Data":"8c5506df69d3a5ee738d6c2fdb2e2fc6a991b1259625075e7789ecbec8cf81d2"}
Sep 10 17:35:58.677036 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:58.676930    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-122.ec2.internal" event={"ID":"39d5711007096867f7ca5e1bd92438d9","Type":"ContainerStarted","Data":"1ae37df971b93797f3816312cec363907fd8fe87375bbb4af3bb1127ce457da8"}
Sep 10 17:35:59.091737 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:59.091140    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e0fe9300-104f-489d-9962-908000359645-original-pull-secret\") pod \"global-pull-secret-syncer-m4npt\" (UID: \"e0fe9300-104f-489d-9962-908000359645\") " pod="kube-system/global-pull-secret-syncer-m4npt"
Sep 10 17:35:59.091737 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:59.091210    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3fcacac9-c23e-4ba5-8b1d-dfeede6b57e3-metrics-certs\") pod \"network-metrics-daemon-ncmn7\" (UID: \"3fcacac9-c23e-4ba5-8b1d-dfeede6b57e3\") " pod="openshift-multus/network-metrics-daemon-ncmn7"
Sep 10 17:35:59.091737 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:35:59.091312    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 17:35:59.091737 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:35:59.091368    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3fcacac9-c23e-4ba5-8b1d-dfeede6b57e3-metrics-certs podName:3fcacac9-c23e-4ba5-8b1d-dfeede6b57e3 nodeName:}" failed. No retries permitted until 2026-09-10 17:36:01.09134709 +0000 UTC m=+5.091177138 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3fcacac9-c23e-4ba5-8b1d-dfeede6b57e3-metrics-certs") pod "network-metrics-daemon-ncmn7" (UID: "3fcacac9-c23e-4ba5-8b1d-dfeede6b57e3") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 17:35:59.092109 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:35:59.091755    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 17:35:59.092109 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:35:59.091810    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0fe9300-104f-489d-9962-908000359645-original-pull-secret podName:e0fe9300-104f-489d-9962-908000359645 nodeName:}" failed. No retries permitted until 2026-09-10 17:36:01.091794708 +0000 UTC m=+5.091624769 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e0fe9300-104f-489d-9962-908000359645-original-pull-secret") pod "global-pull-secret-syncer-m4npt" (UID: "e0fe9300-104f-489d-9962-908000359645") : object "kube-system"/"original-pull-secret" not registered
Sep 10 17:35:59.192272 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:59.192239    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9zqd\" (UniqueName: \"kubernetes.io/projected/8436b9d2-f47a-411e-a695-cb52b8277b62-kube-api-access-p9zqd\") pod \"network-check-target-8k8z5\" (UID: \"8436b9d2-f47a-411e-a695-cb52b8277b62\") " pod="openshift-network-diagnostics/network-check-target-8k8z5"
Sep 10 17:35:59.192448 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:35:59.192431    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 17:35:59.192503 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:35:59.192455    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 17:35:59.192503 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:35:59.192467    2579 projected.go:196] Error preparing data for projected volume kube-api-access-p9zqd for pod openshift-network-diagnostics/network-check-target-8k8z5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 17:35:59.192622 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:35:59.192517    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8436b9d2-f47a-411e-a695-cb52b8277b62-kube-api-access-p9zqd podName:8436b9d2-f47a-411e-a695-cb52b8277b62 nodeName:}" failed. No retries permitted until 2026-09-10 17:36:01.192501613 +0000 UTC m=+5.192331658 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-p9zqd" (UniqueName: "kubernetes.io/projected/8436b9d2-f47a-411e-a695-cb52b8277b62-kube-api-access-p9zqd") pod "network-check-target-8k8z5" (UID: "8436b9d2-f47a-411e-a695-cb52b8277b62") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 17:35:59.419791 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:59.419705    2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-09 17:30:57 +0000 UTC" deadline="2028-04-12 05:18:53.633947659 +0000 UTC"
Sep 10 17:35:59.419791 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:59.419740    2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13907h42m54.214212326s"
Sep 10 17:35:59.469269 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:59.469240    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ncmn7"
Sep 10 17:35:59.469429 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:35:59.469382    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ncmn7" podUID="3fcacac9-c23e-4ba5-8b1d-dfeede6b57e3"
Sep 10 17:35:59.469873 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:59.469841    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8k8z5"
Sep 10 17:35:59.469972 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:35:59.469946    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8k8z5" podUID="8436b9d2-f47a-411e-a695-cb52b8277b62"
Sep 10 17:35:59.470033 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:35:59.470002    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m4npt"
Sep 10 17:35:59.470091 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:35:59.470069    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m4npt" podUID="e0fe9300-104f-489d-9962-908000359645"
Sep 10 17:36:01.111027 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:01.110989    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3fcacac9-c23e-4ba5-8b1d-dfeede6b57e3-metrics-certs\") pod \"network-metrics-daemon-ncmn7\" (UID: \"3fcacac9-c23e-4ba5-8b1d-dfeede6b57e3\") " pod="openshift-multus/network-metrics-daemon-ncmn7"
Sep 10 17:36:01.111619 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:01.111060    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e0fe9300-104f-489d-9962-908000359645-original-pull-secret\") pod \"global-pull-secret-syncer-m4npt\" (UID: \"e0fe9300-104f-489d-9962-908000359645\") " pod="kube-system/global-pull-secret-syncer-m4npt"
Sep 10 17:36:01.111619 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:36:01.111202    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 17:36:01.111619 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:36:01.111264    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0fe9300-104f-489d-9962-908000359645-original-pull-secret podName:e0fe9300-104f-489d-9962-908000359645 nodeName:}" failed. No retries permitted until 2026-09-10 17:36:05.11124606 +0000 UTC m=+9.111076110 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e0fe9300-104f-489d-9962-908000359645-original-pull-secret") pod "global-pull-secret-syncer-m4npt" (UID: "e0fe9300-104f-489d-9962-908000359645") : object "kube-system"/"original-pull-secret" not registered
Sep 10 17:36:01.111795 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:36:01.111650    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 17:36:01.111795 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:36:01.111698    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3fcacac9-c23e-4ba5-8b1d-dfeede6b57e3-metrics-certs podName:3fcacac9-c23e-4ba5-8b1d-dfeede6b57e3 nodeName:}" failed. No retries permitted until 2026-09-10 17:36:05.111683431 +0000 UTC m=+9.111513478 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3fcacac9-c23e-4ba5-8b1d-dfeede6b57e3-metrics-certs") pod "network-metrics-daemon-ncmn7" (UID: "3fcacac9-c23e-4ba5-8b1d-dfeede6b57e3") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 17:36:01.211405 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:01.211366    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9zqd\" (UniqueName: \"kubernetes.io/projected/8436b9d2-f47a-411e-a695-cb52b8277b62-kube-api-access-p9zqd\") pod \"network-check-target-8k8z5\" (UID: \"8436b9d2-f47a-411e-a695-cb52b8277b62\") " pod="openshift-network-diagnostics/network-check-target-8k8z5"
Sep 10 17:36:01.211716 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:36:01.211583    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 17:36:01.211716 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:36:01.211612    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 17:36:01.211716 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:36:01.211624    2579 projected.go:196] Error preparing data for projected volume kube-api-access-p9zqd for pod openshift-network-diagnostics/network-check-target-8k8z5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 17:36:01.211716 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:36:01.211685    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8436b9d2-f47a-411e-a695-cb52b8277b62-kube-api-access-p9zqd podName:8436b9d2-f47a-411e-a695-cb52b8277b62 nodeName:}" failed. No retries permitted until 2026-09-10 17:36:05.21166686 +0000 UTC m=+9.211496923 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-p9zqd" (UniqueName: "kubernetes.io/projected/8436b9d2-f47a-411e-a695-cb52b8277b62-kube-api-access-p9zqd") pod "network-check-target-8k8z5" (UID: "8436b9d2-f47a-411e-a695-cb52b8277b62") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 17:36:01.469434 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:01.469360    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ncmn7"
Sep 10 17:36:01.469434 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:01.469396    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m4npt"
Sep 10 17:36:01.469434 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:01.469396    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8k8z5"
Sep 10 17:36:01.469731 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:36:01.469504    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ncmn7" podUID="3fcacac9-c23e-4ba5-8b1d-dfeede6b57e3"
Sep 10 17:36:01.469731 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:36:01.469605    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m4npt" podUID="e0fe9300-104f-489d-9962-908000359645"
Sep 10 17:36:01.469731 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:36:01.469686    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8k8z5" podUID="8436b9d2-f47a-411e-a695-cb52b8277b62"
Sep 10 17:36:03.468760 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:03.468721    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ncmn7"
Sep 10 17:36:03.468760 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:03.468746    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8k8z5"
Sep 10 17:36:03.469275 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:36:03.468873    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ncmn7" podUID="3fcacac9-c23e-4ba5-8b1d-dfeede6b57e3"
Sep 10 17:36:03.469275 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:36:03.468934    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8k8z5" podUID="8436b9d2-f47a-411e-a695-cb52b8277b62"
Sep 10 17:36:03.469275 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:03.468997    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m4npt"
Sep 10 17:36:03.469275 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:36:03.469069    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m4npt" podUID="e0fe9300-104f-489d-9962-908000359645"
Sep 10 17:36:05.144842 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:05.144806    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e0fe9300-104f-489d-9962-908000359645-original-pull-secret\") pod \"global-pull-secret-syncer-m4npt\" (UID: \"e0fe9300-104f-489d-9962-908000359645\") " pod="kube-system/global-pull-secret-syncer-m4npt"
Sep 10 17:36:05.145370 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:05.144887    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3fcacac9-c23e-4ba5-8b1d-dfeede6b57e3-metrics-certs\") pod \"network-metrics-daemon-ncmn7\" (UID: \"3fcacac9-c23e-4ba5-8b1d-dfeede6b57e3\") " pod="openshift-multus/network-metrics-daemon-ncmn7"
Sep 10 17:36:05.145370 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:36:05.144978    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 17:36:05.145370 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:36:05.144992    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 17:36:05.145370 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:36:05.145047    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0fe9300-104f-489d-9962-908000359645-original-pull-secret podName:e0fe9300-104f-489d-9962-908000359645 nodeName:}" failed. No retries permitted until 2026-09-10 17:36:13.145025472 +0000 UTC m=+17.144855539 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e0fe9300-104f-489d-9962-908000359645-original-pull-secret") pod "global-pull-secret-syncer-m4npt" (UID: "e0fe9300-104f-489d-9962-908000359645") : object "kube-system"/"original-pull-secret" not registered
Sep 10 17:36:05.145370 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:36:05.145068    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3fcacac9-c23e-4ba5-8b1d-dfeede6b57e3-metrics-certs podName:3fcacac9-c23e-4ba5-8b1d-dfeede6b57e3 nodeName:}" failed. No retries permitted until 2026-09-10 17:36:13.145057744 +0000 UTC m=+17.144887792 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3fcacac9-c23e-4ba5-8b1d-dfeede6b57e3-metrics-certs") pod "network-metrics-daemon-ncmn7" (UID: "3fcacac9-c23e-4ba5-8b1d-dfeede6b57e3") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 17:36:05.246305 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:05.246273    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9zqd\" (UniqueName: \"kubernetes.io/projected/8436b9d2-f47a-411e-a695-cb52b8277b62-kube-api-access-p9zqd\") pod \"network-check-target-8k8z5\" (UID: \"8436b9d2-f47a-411e-a695-cb52b8277b62\") " pod="openshift-network-diagnostics/network-check-target-8k8z5"
Sep 10 17:36:05.246473 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:36:05.246443    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 17:36:05.246473 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:36:05.246466    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 17:36:05.246607 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:36:05.246478    2579 projected.go:196] Error preparing data for projected volume kube-api-access-p9zqd for pod openshift-network-diagnostics/network-check-target-8k8z5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 17:36:05.246607 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:36:05.246530    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8436b9d2-f47a-411e-a695-cb52b8277b62-kube-api-access-p9zqd podName:8436b9d2-f47a-411e-a695-cb52b8277b62 nodeName:}" failed. No retries permitted until 2026-09-10 17:36:13.246516045 +0000 UTC m=+17.246346100 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-p9zqd" (UniqueName: "kubernetes.io/projected/8436b9d2-f47a-411e-a695-cb52b8277b62-kube-api-access-p9zqd") pod "network-check-target-8k8z5" (UID: "8436b9d2-f47a-411e-a695-cb52b8277b62") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 17:36:05.468750 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:05.468673    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8k8z5"
Sep 10 17:36:05.468899 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:05.468674    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ncmn7"
Sep 10 17:36:05.468899 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:36:05.468791    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8k8z5" podUID="8436b9d2-f47a-411e-a695-cb52b8277b62"
Sep 10 17:36:05.468989 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:36:05.468899    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ncmn7" podUID="3fcacac9-c23e-4ba5-8b1d-dfeede6b57e3"
Sep 10 17:36:05.468989 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:05.468680    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m4npt"
Sep 10 17:36:05.469064 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:36:05.468997    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m4npt" podUID="e0fe9300-104f-489d-9962-908000359645"
Sep 10 17:36:07.468801 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:07.468715    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8k8z5"
Sep 10 17:36:07.469275 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:07.468835    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m4npt"
Sep 10 17:36:07.469275 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:07.468845    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ncmn7"
Sep 10 17:36:07.469275 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:36:07.468937    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m4npt" podUID="e0fe9300-104f-489d-9962-908000359645"
Sep 10 17:36:07.469275 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:36:07.468835    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8k8z5" podUID="8436b9d2-f47a-411e-a695-cb52b8277b62"
Sep 10 17:36:07.469275 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:36:07.469022    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ncmn7" podUID="3fcacac9-c23e-4ba5-8b1d-dfeede6b57e3"
Sep 10 17:36:09.468759 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:09.468734    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8k8z5"
Sep 10 17:36:09.469101 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:09.468763    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ncmn7"
Sep 10 17:36:09.469101 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:36:09.468821    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8k8z5" podUID="8436b9d2-f47a-411e-a695-cb52b8277b62"
Sep 10 17:36:09.469101 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:09.468852    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m4npt"
Sep 10 17:36:09.469101 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:36:09.468928    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ncmn7" podUID="3fcacac9-c23e-4ba5-8b1d-dfeede6b57e3"
Sep 10 17:36:09.469101 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:36:09.468983    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m4npt" podUID="e0fe9300-104f-489d-9962-908000359645"
Sep 10 17:36:11.470006 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:11.469462    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ncmn7"
Sep 10 17:36:11.470006 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:11.469493    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m4npt"
Sep 10 17:36:11.470006 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:11.469492    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8k8z5"
Sep 10 17:36:11.470006 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:36:11.469628    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ncmn7" podUID="3fcacac9-c23e-4ba5-8b1d-dfeede6b57e3"
Sep 10 17:36:11.470006 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:36:11.469770    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m4npt" podUID="e0fe9300-104f-489d-9962-908000359645"
Sep 10 17:36:11.470006 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:36:11.469848    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8k8z5" podUID="8436b9d2-f47a-411e-a695-cb52b8277b62"
Sep 10 17:36:13.206686 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:13.206650    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e0fe9300-104f-489d-9962-908000359645-original-pull-secret\") pod \"global-pull-secret-syncer-m4npt\" (UID: \"e0fe9300-104f-489d-9962-908000359645\") " pod="kube-system/global-pull-secret-syncer-m4npt"
Sep 10 17:36:13.207162 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:13.206731    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3fcacac9-c23e-4ba5-8b1d-dfeede6b57e3-metrics-certs\") pod \"network-metrics-daemon-ncmn7\" (UID: \"3fcacac9-c23e-4ba5-8b1d-dfeede6b57e3\") " pod="openshift-multus/network-metrics-daemon-ncmn7"
Sep 10 17:36:13.207162 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:36:13.206799    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 17:36:13.207162 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:36:13.206831    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 17:36:13.207162 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:36:13.206870    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0fe9300-104f-489d-9962-908000359645-original-pull-secret podName:e0fe9300-104f-489d-9962-908000359645 nodeName:}" failed. No retries permitted until 2026-09-10 17:36:29.206852018 +0000 UTC m=+33.206682064 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e0fe9300-104f-489d-9962-908000359645-original-pull-secret") pod "global-pull-secret-syncer-m4npt" (UID: "e0fe9300-104f-489d-9962-908000359645") : object "kube-system"/"original-pull-secret" not registered
Sep 10 17:36:13.207162 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:36:13.206894    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3fcacac9-c23e-4ba5-8b1d-dfeede6b57e3-metrics-certs podName:3fcacac9-c23e-4ba5-8b1d-dfeede6b57e3 nodeName:}" failed. No retries permitted until 2026-09-10 17:36:29.206881867 +0000 UTC m=+33.206711914 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3fcacac9-c23e-4ba5-8b1d-dfeede6b57e3-metrics-certs") pod "network-metrics-daemon-ncmn7" (UID: "3fcacac9-c23e-4ba5-8b1d-dfeede6b57e3") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 17:36:13.307753 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:13.307714    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9zqd\" (UniqueName: \"kubernetes.io/projected/8436b9d2-f47a-411e-a695-cb52b8277b62-kube-api-access-p9zqd\") pod \"network-check-target-8k8z5\" (UID: \"8436b9d2-f47a-411e-a695-cb52b8277b62\") " pod="openshift-network-diagnostics/network-check-target-8k8z5"
Sep 10 17:36:13.307926 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:36:13.307863    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 17:36:13.307926 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:36:13.307882    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 17:36:13.307926 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:36:13.307894    2579 projected.go:196] Error preparing data for projected volume kube-api-access-p9zqd for pod openshift-network-diagnostics/network-check-target-8k8z5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 17:36:13.308075 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:36:13.307960    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8436b9d2-f47a-411e-a695-cb52b8277b62-kube-api-access-p9zqd podName:8436b9d2-f47a-411e-a695-cb52b8277b62 nodeName:}" failed. No retries permitted until 2026-09-10 17:36:29.307939945 +0000 UTC m=+33.307770012 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-p9zqd" (UniqueName: "kubernetes.io/projected/8436b9d2-f47a-411e-a695-cb52b8277b62-kube-api-access-p9zqd") pod "network-check-target-8k8z5" (UID: "8436b9d2-f47a-411e-a695-cb52b8277b62") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 17:36:13.468648 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:13.468582    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ncmn7"
Sep 10 17:36:13.468774 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:13.468599    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8k8z5"
Sep 10 17:36:13.468774 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:36:13.468699    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ncmn7" podUID="3fcacac9-c23e-4ba5-8b1d-dfeede6b57e3"
Sep 10 17:36:13.468774 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:13.468747    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m4npt"
Sep 10 17:36:13.468866 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:36:13.468815    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8k8z5" podUID="8436b9d2-f47a-411e-a695-cb52b8277b62"
Sep 10 17:36:13.468910 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:36:13.468887    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m4npt" podUID="e0fe9300-104f-489d-9962-908000359645"
Sep 10 17:36:15.469385 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:15.469352    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ncmn7"
Sep 10 17:36:15.469975 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:15.469385    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8k8z5"
Sep 10 17:36:15.469975 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:36:15.469494    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ncmn7" podUID="3fcacac9-c23e-4ba5-8b1d-dfeede6b57e3"
Sep 10 17:36:15.469975 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:36:15.469547    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8k8z5" podUID="8436b9d2-f47a-411e-a695-cb52b8277b62"
Sep 10 17:36:15.469975 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:15.469626    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m4npt"
Sep 10 17:36:15.469975 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:36:15.469715    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m4npt" podUID="e0fe9300-104f-489d-9962-908000359645"
Sep 10 17:36:17.469270 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:17.468754    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m4npt"
Sep 10 17:36:17.469270 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:36:17.469022    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m4npt" podUID="e0fe9300-104f-489d-9962-908000359645"
Sep 10 17:36:17.469762 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:17.469393    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ncmn7"
Sep 10 17:36:17.469762 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:36:17.469499    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ncmn7" podUID="3fcacac9-c23e-4ba5-8b1d-dfeede6b57e3"
Sep 10 17:36:17.469762 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:17.469603    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8k8z5"
Sep 10 17:36:17.469762 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:36:17.469678    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8k8z5" podUID="8436b9d2-f47a-411e-a695-cb52b8277b62"
Sep 10 17:36:17.730835 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:17.730798    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-x2sqz" event={"ID":"85fae606-5227-404b-95a8-d7de60dc1041","Type":"ContainerStarted","Data":"e66fc67119df41535d770cee2dff5b12892283aa0c35206fa337d82186ed5ab8"}
Sep 10 17:36:17.736000 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:17.735964    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-n27c5" event={"ID":"88321231-7aaf-4278-be6f-32bee3777450","Type":"ContainerStarted","Data":"73931554e167a97728d3a90d67da39a4c5ae73fb38f1c3164cb6be36a06cf0ed"}
Sep 10 17:36:17.740331 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:17.740145    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-122.ec2.internal" event={"ID":"bd750ec1329c03eb8935ed3e9791d542","Type":"ContainerStarted","Data":"e4feaa1581a2f57abce4765ca2b4e24ae3da047399e7d38876a1027bfa98a91d"}
Sep 10 17:36:17.746421 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:17.746255    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4fsjz" event={"ID":"d4fad94e-7939-4949-bdfd-1ef38e016c19","Type":"ContainerStarted","Data":"aa85af69ddbd941a75eb8ca778a78e77ffed06bed3d51f1d97c2c36cae5328b1"}
Sep 10 17:36:17.746522 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:17.746469    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4fsjz" event={"ID":"d4fad94e-7939-4949-bdfd-1ef38e016c19","Type":"ContainerStarted","Data":"319a23cd8e818e85467785f15bec9a422708cd06e89a4aa526553633bda96929"}
Sep 10 17:36:17.746522 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:17.746503    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4fsjz" event={"ID":"d4fad94e-7939-4949-bdfd-1ef38e016c19","Type":"ContainerStarted","Data":"454979ff75143a88b6c19a576b51eac09ddd0f9b0a53f998f0a2b85ff14ae4f6"}
Sep 10 17:36:17.749642 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:17.749099    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-x2sqz" podStartSLOduration=2.037358984 podStartE2EDuration="21.749081389s" totalImagesPullingTime="19.711722405s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 17:35:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:36:17.748024403 +0000 UTC m=+21.747854472" watchObservedRunningTime="2026-09-10 17:36:17.749081389 +0000 UTC m=+21.748911456"
Sep 10 17:36:17.762158 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:17.762117    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-122.ec2.internal" podStartSLOduration=20.762102675 podStartE2EDuration="20.762102675s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 17:35:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:36:17.761718043 +0000 UTC m=+21.761548109" watchObservedRunningTime="2026-09-10 17:36:17.762102675 +0000 UTC m=+21.761932741"
Sep 10 17:36:18.755095 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:18.755052    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4fsjz" event={"ID":"d4fad94e-7939-4949-bdfd-1ef38e016c19","Type":"ContainerStarted","Data":"513e8a7cc06868311c33bcb7ae373458b51522f1054c7569b483b66fa916224a"}
Sep 10 17:36:18.755704 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:18.755160    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4fsjz" event={"ID":"d4fad94e-7939-4949-bdfd-1ef38e016c19","Type":"ContainerStarted","Data":"b0a861bae83af173a8f2c389f47502a99321c549b04632111c5a27a51b415ccc"}
Sep 10 17:36:18.755704 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:18.755192    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4fsjz" event={"ID":"d4fad94e-7939-4949-bdfd-1ef38e016c19","Type":"ContainerStarted","Data":"c9639800a7037deee21f834763d0f575ad91c5466b87f4c6aed8cbae54b4cbe4"}
Sep 10 17:36:18.757360 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:18.757325    2579 generic.go:356] "Generic (PLEG): container finished" podID="48d6ee6d-6025-4ab7-a095-6a31180ac9f7" containerID="9ca9407822b05272141919e54e729562f7363978b7e1db6e1b838cd6248f12b3" exitCode=0
Sep 10 17:36:18.757475 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:18.757409    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-drhxj" event={"ID":"48d6ee6d-6025-4ab7-a095-6a31180ac9f7","Type":"ContainerDied","Data":"9ca9407822b05272141919e54e729562f7363978b7e1db6e1b838cd6248f12b3"}
Sep 10 17:36:18.759693 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:18.759671    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-pdstz" event={"ID":"32a43910-3aa0-41e0-ab5b-ad08c1ab6b41","Type":"ContainerStarted","Data":"5c480a7f53a3bfef4d499dbbf1214ec0f69a9ac52df2a98cee9be46b60c0dc64"}
Sep 10 17:36:18.761304 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:18.761277    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-rkzwg" event={"ID":"eca4abd0-e29b-4c77-9a3b-c9c32e5b5101","Type":"ContainerStarted","Data":"cf9d959058d0105a15edc648c2a5a0224364ce63d3d4eed6821cfb2095bd26b9"}
Sep 10 17:36:18.764847 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:18.764810    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-t4l6s" event={"ID":"3dc43868-1f6c-48fa-85fc-59607a50c103","Type":"ContainerStarted","Data":"a75ed4b0f481c6bd970b0879a51637826f8244b1aec2de9017b8d89b951a997d"}
Sep 10 17:36:18.767095 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:18.767074    2579 generic.go:356] "Generic (PLEG): container finished" podID="39d5711007096867f7ca5e1bd92438d9" containerID="ffb78ebecc4142e4b90c11f9dabd276be67b0efecff0898bce37338502d21224" exitCode=0
Sep 10 17:36:18.767175 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:18.767139    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-122.ec2.internal" event={"ID":"39d5711007096867f7ca5e1bd92438d9","Type":"ContainerDied","Data":"ffb78ebecc4142e4b90c11f9dabd276be67b0efecff0898bce37338502d21224"}
Sep 10 17:36:18.769055 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:18.769028    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-lpm4k" event={"ID":"1cd8f80e-f5c5-4e80-9abb-b43af15ed1d1","Type":"ContainerStarted","Data":"c00ac5341bfdfb3200faa20e2e62b6f3b9c0b0a50781a59144bf9f1c7271fda0"}
Sep 10 17:36:18.770542 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:18.770519    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-plzx8" event={"ID":"56f29fc7-7f69-42cb-b80e-1a2bfc298f1a","Type":"ContainerStarted","Data":"4937fb069ce63865e5cba6c04426e2356a7d0f902a1e3e79f8fc55b64bc41f04"}
Sep 10 17:36:18.778501 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:18.778464    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-n27c5" podStartSLOduration=3.1766214919999998 podStartE2EDuration="22.778452362s" totalImagesPullingTime="19.60183087s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 17:35:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:36:17.779605828 +0000 UTC m=+21.779435895" watchObservedRunningTime="2026-09-10 17:36:18.778452362 +0000 UTC m=+22.778282428"
Sep 10 17:36:18.790991 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:18.790938    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-rkzwg" podStartSLOduration=7.026897495 podStartE2EDuration="22.790922554s" totalImagesPullingTime="15.764025059s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 17:35:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:36:18.790160445 +0000 UTC m=+22.789990514" watchObservedRunningTime="2026-09-10 17:36:18.790922554 +0000 UTC m=+22.790752623"
Sep 10 17:36:18.802023 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:18.801981    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-pdstz" podStartSLOduration=3.2972225809999998 podStartE2EDuration="22.801965978s" totalImagesPullingTime="19.504743397s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 17:35:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:36:18.801449335 +0000 UTC m=+22.801279404" watchObservedRunningTime="2026-09-10 17:36:18.801965978 +0000 UTC m=+22.801796047"
Sep 10 17:36:18.815341 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:18.815303    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-lpm4k" podStartSLOduration=2.372377241 podStartE2EDuration="21.815294059s" totalImagesPullingTime="19.442916818s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:35:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:36:18.815041838 +0000 UTC m=+22.814871904" watchObservedRunningTime="2026-09-10 17:36:18.815294059 +0000 UTC m=+22.815124125"
Sep 10 17:36:18.840041 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:18.839998    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-t4l6s" podStartSLOduration=3.2715829 podStartE2EDuration="22.839984014s" totalImagesPullingTime="19.568401114s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 17:35:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:36:18.83988588 +0000 UTC m=+22.839715948" watchObservedRunningTime="2026-09-10 17:36:18.839984014 +0000 UTC m=+22.839814082"
Sep 10 17:36:19.335367 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:19.335341    2579 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 10 17:36:19.439528 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:19.439297    2579 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-10T17:36:19.335359076Z","UUID":"54cc7b5d-38db-4cdf-9590-469d50840599","Handler":null,"Name":"","Endpoint":""}
Sep 10 17:36:19.440610 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:19.440594    2579 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 10 17:36:19.440682 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:19.440621    2579 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 10 17:36:19.468752 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:19.468730    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ncmn7"
Sep 10 17:36:19.468859 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:19.468828    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8k8z5"
Sep 10 17:36:19.468925 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:36:19.468909    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8k8z5" podUID="8436b9d2-f47a-411e-a695-cb52b8277b62"
Sep 10 17:36:19.468961 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:36:19.468830    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ncmn7" podUID="3fcacac9-c23e-4ba5-8b1d-dfeede6b57e3"
Sep 10 17:36:19.468961 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:19.468833    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m4npt"
Sep 10 17:36:19.469050 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:36:19.468999    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m4npt" podUID="e0fe9300-104f-489d-9962-908000359645"
Sep 10 17:36:19.676197 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:19.676128    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-lpm4k"
Sep 10 17:36:19.676850 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:19.676827    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-lpm4k"
Sep 10 17:36:19.774783 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:19.774754    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-plzx8" event={"ID":"56f29fc7-7f69-42cb-b80e-1a2bfc298f1a","Type":"ContainerStarted","Data":"4ce17fd873a6f1dba3ae42eced298450942d0e670f6c3f281c8195a8f9630bad"}
Sep 10 17:36:19.776898 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:19.776870    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-122.ec2.internal" event={"ID":"39d5711007096867f7ca5e1bd92438d9","Type":"ContainerStarted","Data":"33f7d8475cba9a1a2b43655930468ce08bb860664ecc93236d47825ea979e661"}
Sep 10 17:36:19.777616 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:19.777597    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-lpm4k"
Sep 10 17:36:19.778090 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:19.778073    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-lpm4k"
Sep 10 17:36:19.794583 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:19.794510    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-122.ec2.internal" podStartSLOduration=22.794491589 podStartE2EDuration="22.794491589s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 17:35:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:36:19.793946772 +0000 UTC m=+23.793776841" watchObservedRunningTime="2026-09-10 17:36:19.794491589 +0000 UTC m=+23.794321657"
Sep 10 17:36:20.780907 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:20.780866    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-plzx8" event={"ID":"56f29fc7-7f69-42cb-b80e-1a2bfc298f1a","Type":"ContainerStarted","Data":"54a6927ee94e2c30097c41c2b2719dcf1aa9332615cb89b4fb8cb59b9a849d3c"}
Sep 10 17:36:20.784296 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:20.784271    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4fsjz" event={"ID":"d4fad94e-7939-4949-bdfd-1ef38e016c19","Type":"ContainerStarted","Data":"207df43afc9d2795eca8e29d7f1a3ec887dba91a28fc34367ad4ada7e14cc5f4"}
Sep 10 17:36:20.799891 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:20.799818    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-plzx8" podStartSLOduration=2.700270486 podStartE2EDuration="24.799804739s" totalImagesPullingTime="22.099534253s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 17:35:56 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:36:20.799655081 +0000 UTC m=+24.799485149" watchObservedRunningTime="2026-09-10 17:36:20.799804739 +0000 UTC m=+24.799634806"
Sep 10 17:36:21.469331 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:21.469296    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m4npt"
Sep 10 17:36:21.469510 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:21.469296    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ncmn7"
Sep 10 17:36:21.469510 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:36:21.469414    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m4npt" podUID="e0fe9300-104f-489d-9962-908000359645"
Sep 10 17:36:21.469510 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:21.469296    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8k8z5"
Sep 10 17:36:21.469510 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:36:21.469479    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ncmn7" podUID="3fcacac9-c23e-4ba5-8b1d-dfeede6b57e3"
Sep 10 17:36:21.469733 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:36:21.469526    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8k8z5" podUID="8436b9d2-f47a-411e-a695-cb52b8277b62"
Sep 10 17:36:22.792359 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:22.792320    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4fsjz" event={"ID":"d4fad94e-7939-4949-bdfd-1ef38e016c19","Type":"ContainerStarted","Data":"00b28cf1013c93356d5746edb87d0cb32cf18d0f5ae8cce5b2ba0c4df3312b4c"}
Sep 10 17:36:22.825695 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:22.825642    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-4fsjz" podStartSLOduration=7.298037494 podStartE2EDuration="26.825626539s" totalImagesPullingTime="19.527589045s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 17:35:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:36:22.824916437 +0000 UTC m=+26.824746504" watchObservedRunningTime="2026-09-10 17:36:22.825626539 +0000 UTC m=+26.825456607"
Sep 10 17:36:23.468946 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:23.468911    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ncmn7"
Sep 10 17:36:23.469124 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:23.468961    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8k8z5"
Sep 10 17:36:23.469124 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:23.468911    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m4npt"
Sep 10 17:36:23.469124 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:36:23.469039    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ncmn7" podUID="3fcacac9-c23e-4ba5-8b1d-dfeede6b57e3"
Sep 10 17:36:23.469124 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:36:23.469093    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m4npt" podUID="e0fe9300-104f-489d-9962-908000359645"
Sep 10 17:36:23.469327 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:36:23.469162    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8k8z5" podUID="8436b9d2-f47a-411e-a695-cb52b8277b62"
Sep 10 17:36:23.796029 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:23.795962    2579 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 10 17:36:23.796646 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:23.796489    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-4fsjz"
Sep 10 17:36:23.796646 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:23.796599    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-4fsjz"
Sep 10 17:36:23.815346 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:23.815318    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-4fsjz"
Sep 10 17:36:23.815614 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:23.815597    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-4fsjz"
Sep 10 17:36:24.203591 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:24.203539    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-m4npt"]
Sep 10 17:36:24.203742 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:24.203621    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-8k8z5"]
Sep 10 17:36:24.203742 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:24.203644    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-ncmn7"]
Sep 10 17:36:24.203742 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:24.203709    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ncmn7"
Sep 10 17:36:24.203834 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:36:24.203805    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ncmn7" podUID="3fcacac9-c23e-4ba5-8b1d-dfeede6b57e3"
Sep 10 17:36:24.204206 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:24.204183    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m4npt"
Sep 10 17:36:24.204331 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:36:24.204291    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m4npt" podUID="e0fe9300-104f-489d-9962-908000359645"
Sep 10 17:36:24.204380 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:24.204336    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8k8z5"
Sep 10 17:36:24.204411 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:36:24.204397    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8k8z5" podUID="8436b9d2-f47a-411e-a695-cb52b8277b62"
Sep 10 17:36:24.797833 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:24.797806    2579 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 10 17:36:25.469056 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:25.469030    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ncmn7"
Sep 10 17:36:25.469244 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:36:25.469129    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ncmn7" podUID="3fcacac9-c23e-4ba5-8b1d-dfeede6b57e3"
Sep 10 17:36:25.801554 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:25.801467    2579 generic.go:356] "Generic (PLEG): container finished" podID="48d6ee6d-6025-4ab7-a095-6a31180ac9f7" containerID="8e5e75e9f8841c795559be6ee2f3188557593956400e4c22ec2bbfda4ae8eb68" exitCode=0
Sep 10 17:36:25.802195 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:25.801582    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-drhxj" event={"ID":"48d6ee6d-6025-4ab7-a095-6a31180ac9f7","Type":"ContainerDied","Data":"8e5e75e9f8841c795559be6ee2f3188557593956400e4c22ec2bbfda4ae8eb68"}
Sep 10 17:36:25.802195 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:25.801759    2579 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 10 17:36:26.470106 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:26.470066    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8k8z5"
Sep 10 17:36:26.470268 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:36:26.470164    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8k8z5" podUID="8436b9d2-f47a-411e-a695-cb52b8277b62"
Sep 10 17:36:26.470268 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:26.470216    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m4npt"
Sep 10 17:36:26.470375 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:36:26.470324    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m4npt" podUID="e0fe9300-104f-489d-9962-908000359645"
Sep 10 17:36:27.128055 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:27.127882    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-4fsjz"
Sep 10 17:36:27.128494 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:27.128286    2579 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 10 17:36:27.138001 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:27.137964    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-4fsjz" podUID="d4fad94e-7939-4949-bdfd-1ef38e016c19" containerName="ovnkube-controller" probeResult="failure" output=""
Sep 10 17:36:27.146414 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:27.146387    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-4fsjz" podUID="d4fad94e-7939-4949-bdfd-1ef38e016c19" containerName="ovnkube-controller" probeResult="failure" output=""
Sep 10 17:36:27.469188 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:27.469122    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ncmn7"
Sep 10 17:36:27.469297 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:36:27.469221    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ncmn7" podUID="3fcacac9-c23e-4ba5-8b1d-dfeede6b57e3"
Sep 10 17:36:27.807440 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:27.807370    2579 generic.go:356] "Generic (PLEG): container finished" podID="48d6ee6d-6025-4ab7-a095-6a31180ac9f7" containerID="2d03591aeb6425814feaff660cd597c0effa76541dcc2ff07a5957fbcaeebd3d" exitCode=0
Sep 10 17:36:27.807440 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:27.807421    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-drhxj" event={"ID":"48d6ee6d-6025-4ab7-a095-6a31180ac9f7","Type":"ContainerDied","Data":"2d03591aeb6425814feaff660cd597c0effa76541dcc2ff07a5957fbcaeebd3d"}
Sep 10 17:36:28.469178 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:28.469147    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8k8z5"
Sep 10 17:36:28.469630 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:36:28.469274    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8k8z5" podUID="8436b9d2-f47a-411e-a695-cb52b8277b62"
Sep 10 17:36:28.469630 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:28.469320    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m4npt"
Sep 10 17:36:28.469630 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:36:28.469423    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m4npt" podUID="e0fe9300-104f-489d-9962-908000359645"
Sep 10 17:36:29.218657 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:29.218629    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e0fe9300-104f-489d-9962-908000359645-original-pull-secret\") pod \"global-pull-secret-syncer-m4npt\" (UID: \"e0fe9300-104f-489d-9962-908000359645\") " pod="kube-system/global-pull-secret-syncer-m4npt"
Sep 10 17:36:29.218745 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:29.218716    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3fcacac9-c23e-4ba5-8b1d-dfeede6b57e3-metrics-certs\") pod \"network-metrics-daemon-ncmn7\" (UID: \"3fcacac9-c23e-4ba5-8b1d-dfeede6b57e3\") " pod="openshift-multus/network-metrics-daemon-ncmn7"
Sep 10 17:36:29.218786 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:36:29.218769    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 17:36:29.218823 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:36:29.218796    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 17:36:29.218823 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:36:29.218822    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0fe9300-104f-489d-9962-908000359645-original-pull-secret podName:e0fe9300-104f-489d-9962-908000359645 nodeName:}" failed. No retries permitted until 2026-09-10 17:37:01.218807305 +0000 UTC m=+65.218637354 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e0fe9300-104f-489d-9962-908000359645-original-pull-secret") pod "global-pull-secret-syncer-m4npt" (UID: "e0fe9300-104f-489d-9962-908000359645") : object "kube-system"/"original-pull-secret" not registered
Sep 10 17:36:29.218906 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:36:29.218837    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3fcacac9-c23e-4ba5-8b1d-dfeede6b57e3-metrics-certs podName:3fcacac9-c23e-4ba5-8b1d-dfeede6b57e3 nodeName:}" failed. No retries permitted until 2026-09-10 17:37:01.218830018 +0000 UTC m=+65.218660062 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3fcacac9-c23e-4ba5-8b1d-dfeede6b57e3-metrics-certs") pod "network-metrics-daemon-ncmn7" (UID: "3fcacac9-c23e-4ba5-8b1d-dfeede6b57e3") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 17:36:29.319868 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:29.319838    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9zqd\" (UniqueName: \"kubernetes.io/projected/8436b9d2-f47a-411e-a695-cb52b8277b62-kube-api-access-p9zqd\") pod \"network-check-target-8k8z5\" (UID: \"8436b9d2-f47a-411e-a695-cb52b8277b62\") " pod="openshift-network-diagnostics/network-check-target-8k8z5"
Sep 10 17:36:29.320008 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:36:29.319992    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 17:36:29.320050 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:36:29.320013    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 17:36:29.320050 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:36:29.320024    2579 projected.go:196] Error preparing data for projected volume kube-api-access-p9zqd for pod openshift-network-diagnostics/network-check-target-8k8z5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 17:36:29.320108 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:36:29.320073    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8436b9d2-f47a-411e-a695-cb52b8277b62-kube-api-access-p9zqd podName:8436b9d2-f47a-411e-a695-cb52b8277b62 nodeName:}" failed. No retries permitted until 2026-09-10 17:37:01.320056011 +0000 UTC m=+65.319886057 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-p9zqd" (UniqueName: "kubernetes.io/projected/8436b9d2-f47a-411e-a695-cb52b8277b62-kube-api-access-p9zqd") pod "network-check-target-8k8z5" (UID: "8436b9d2-f47a-411e-a695-cb52b8277b62") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 17:36:29.468610 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:29.468526    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ncmn7"
Sep 10 17:36:29.468750 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:36:29.468647    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ncmn7" podUID="3fcacac9-c23e-4ba5-8b1d-dfeede6b57e3"
Sep 10 17:36:29.814626 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:29.814525    2579 generic.go:356] "Generic (PLEG): container finished" podID="48d6ee6d-6025-4ab7-a095-6a31180ac9f7" containerID="ac19b775d0ffce5f3616a5152256940a12b7e288cab62ed093aec6dc614c9153" exitCode=0
Sep 10 17:36:29.814626 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:29.814559    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-drhxj" event={"ID":"48d6ee6d-6025-4ab7-a095-6a31180ac9f7","Type":"ContainerDied","Data":"ac19b775d0ffce5f3616a5152256940a12b7e288cab62ed093aec6dc614c9153"}
Sep 10 17:36:30.468670 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:30.468635    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8k8z5"
Sep 10 17:36:30.468854 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:36:30.468759    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8k8z5" podUID="8436b9d2-f47a-411e-a695-cb52b8277b62"
Sep 10 17:36:30.468854 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:30.468810    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m4npt"
Sep 10 17:36:30.468968 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:36:30.468927    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m4npt" podUID="e0fe9300-104f-489d-9962-908000359645"
Sep 10 17:36:31.229576 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:31.229486    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-122.ec2.internal" event="NodeReady"
Sep 10 17:36:31.230025 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:31.229650    2579 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 10 17:36:31.261970 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:31.261946    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-8dmcm"]
Sep 10 17:36:31.274640 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:31.274620    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-8f44f7f46-d2blr"]
Sep 10 17:36:31.274770 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:31.274749    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-8dmcm"
Sep 10 17:36:31.278034 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:31.277904    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Sep 10 17:36:31.278034 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:31.277923    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-hzbdf\""
Sep 10 17:36:31.278034 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:31.277942    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Sep 10 17:36:31.289741 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:31.289719    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-85b454874b-rb854"]
Sep 10 17:36:31.292754 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:31.292731    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-8f44f7f46-d2blr"
Sep 10 17:36:31.295165 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:31.295135    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 10 17:36:31.295426 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:31.295400    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 10 17:36:31.295691 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:31.295670    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-hcd8f\""
Sep 10 17:36:31.295828 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:31.295809    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 10 17:36:31.299836 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:31.299813    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-wt6xs"]
Sep 10 17:36:31.301046 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:31.301026    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-85b454874b-rb854"
Sep 10 17:36:31.313636 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:31.311954    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-mvd6n"]
Sep 10 17:36:31.313636 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:31.312912    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-wt6xs"
Sep 10 17:36:31.313636 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:31.313053    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Sep 10 17:36:31.313636 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:31.313527    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-d4wkp\""
Sep 10 17:36:31.313918 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:31.313785    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Sep 10 17:36:31.314089 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:31.314068    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Sep 10 17:36:31.314512 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:31.314491    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Sep 10 17:36:31.316505 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:31.316483    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-9gg7w\""
Sep 10 17:36:31.316621 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:31.316543    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 10 17:36:31.316775 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:31.316755    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Sep 10 17:36:31.316956 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:31.316940    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Sep 10 17:36:31.318547 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:31.318464    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Sep 10 17:36:31.319885 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:31.319863    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Sep 10 17:36:31.320084 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:31.320063    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Sep 10 17:36:31.326461 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:31.326443    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-859f94c4b-cmwpc"]
Sep 10 17:36:31.326674 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:31.326657    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-mvd6n"
Sep 10 17:36:31.329391 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:31.329367    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Sep 10 17:36:31.329391 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:31.329379    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Sep 10 17:36:31.329541 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:31.329402    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Sep 10 17:36:31.329541 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:31.329368    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-v4bqr\""
Sep 10 17:36:31.340859 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:31.340839    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-d7f986b9d-xvnsc"]
Sep 10 17:36:31.340981 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:31.340964    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-cmwpc"
Sep 10 17:36:31.343806 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:31.343781    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Sep 10 17:36:31.343908 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:31.343804    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Sep 10 17:36:31.343908 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:31.343834    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Sep 10 17:36:31.343994 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:31.343781    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-2ssmp\""
Sep 10 17:36:31.344100 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:31.344085    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Sep 10 17:36:31.351182 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:31.351162    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-8dmcm"]
Sep 10 17:36:31.351288 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:31.351200    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-859f94c4b-cmwpc"]
Sep 10 17:36:31.351288 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:31.351219    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-wt6xs"]
Sep 10 17:36:31.351288 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:31.351240    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cdcc96965-5cbrn"]
Sep 10 17:36:31.351816 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:31.351784    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-d7f986b9d-xvnsc"
Sep 10 17:36:31.354330 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:31.354303    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Sep 10 17:36:31.354330 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:31.354314    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 10 17:36:31.354330 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:31.354303    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 10 17:36:31.354622 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:31.354559    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-rf44j\""
Sep 10 17:36:31.354622 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:31.354612    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Sep 10 17:36:31.354832 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:31.354817    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Sep 10 17:36:31.355004 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:31.354983    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Sep 10 17:36:31.365593 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:31.365555    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-dxqxz"]
Sep 10 17:36:31.365733 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:31.365648    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-5cbrn"
Sep 10 17:36:31.368222 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:31.368198    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Sep 10 17:36:31.368222 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:31.368218    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 10 17:36:31.368383 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:31.368218    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-wrf4l\""
Sep 10 17:36:31.368383 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:31.368250    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 10 17:36:31.368383 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:31.368338    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Sep 10 17:36:31.381919 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:31.381903    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-mvd6n"]
Sep 10 17:36:31.382012 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:31.381935    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-76bcd6ccf5-rh6vv"]
Sep 10 17:36:31.382012 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:31.382008    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-dxqxz"
Sep 10 17:36:31.384218 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:31.384197    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 10 17:36:31.384313 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:31.384219    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-wtnr4\""
Sep 10 17:36:31.384313 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:31.384203    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 10 17:36:31.392173 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:31.392158    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-xctgk"]
Sep 10 17:36:31.392271 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:31.392258    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-76bcd6ccf5-rh6vv"
Sep 10 17:36:31.395287 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:31.395268    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Sep 10 17:36:31.395287 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:31.395280    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Sep 10 17:36:31.395431 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:31.395344    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 10 17:36:31.395850 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:31.395831    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-8kx8k\""
Sep 10 17:36:31.395944 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:31.395861    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 10 17:36:31.401708 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:31.401686    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Sep 10 17:36:31.408425 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:31.408409    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-685456655f-7798s"]
Sep 10 17:36:31.408534 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:31.408521    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-xctgk"
Sep 10 17:36:31.411394 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:31.411379    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-5h2r4\""
Sep 10 17:36:31.411648 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:31.411630    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 10 17:36:31.411729 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:31.411653    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 10 17:36:31.411729 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:31.411636    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 10 17:36:31.425693 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:31.425675    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-d7f986b9d-xvnsc"]
Sep 10 17:36:31.425770 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:31.425708    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-8f44f7f46-d2blr"]
Sep 10 17:36:31.425770 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:31.425734    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cdcc96965-5cbrn"]
Sep 10 17:36:31.425770 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:31.425751    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-685456655f-7798s"]
Sep 10 17:36:31.425770 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:31.425768    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-xctgk"]
Sep 10 17:36:31.425954 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:31.425770    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-685456655f-7798s"
Sep 10 17:36:31.425954 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:31.425786    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-85b454874b-rb854"]
Sep 10 17:36:31.425954 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:31.425815    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-76bcd6ccf5-rh6vv"]
Sep 10 17:36:31.425954 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:31.425845    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-dxqxz"]
Sep 10 17:36:31.428166 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:31.428150    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 10 17:36:31.428226 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:31.428175    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-2tcxf\""
Sep 10 17:36:31.428360 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:31.428346    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 10 17:36:31.435017 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:31.434999    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ad03af6b-287a-4b2f-9ea3-ec8c43c17cba-registry-tls\") pod \"image-registry-8f44f7f46-d2blr\" (UID: \"ad03af6b-287a-4b2f-9ea3-ec8c43c17cba\") " pod="openshift-image-registry/image-registry-8f44f7f46-d2blr"
Sep 10 17:36:31.435107 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:31.435029    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vsbfw\" (UniqueName: \"kubernetes.io/projected/f1713e2d-701b-435b-8d76-18824228a81e-kube-api-access-vsbfw\") pod \"cluster-samples-operator-7cdcfcf76c-mvd6n\" (UID: \"f1713e2d-701b-435b-8d76-18824228a81e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-mvd6n"
Sep 10 17:36:31.435107 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:31.435057    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77f7r\" (UniqueName: \"kubernetes.io/projected/7f13ef98-e00e-4fe2-8f80-41eb99b0c7f7-kube-api-access-77f7r\") pod \"volume-data-source-validator-695c895bd6-8dmcm\" (UID: \"7f13ef98-e00e-4fe2-8f80-41eb99b0c7f7\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-8dmcm"
Sep 10 17:36:31.435107 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:31.435081    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qp685\" (UniqueName: \"kubernetes.io/projected/ad03af6b-287a-4b2f-9ea3-ec8c43c17cba-kube-api-access-qp685\") pod \"image-registry-8f44f7f46-d2blr\" (UID: \"ad03af6b-287a-4b2f-9ea3-ec8c43c17cba\") " pod="openshift-image-registry/image-registry-8f44f7f46-d2blr"
Sep 10 17:36:31.435206 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:31.435120    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/97172b14-b84a-476e-89f1-05d3e70df416-trusted-ca\") pod \"console-operator-85b454874b-rb854\" (UID: \"97172b14-b84a-476e-89f1-05d3e70df416\") " pod="openshift-console-operator/console-operator-85b454874b-rb854"
Sep 10 17:36:31.435206 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:31.435174    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6976caf5-0ff1-4604-9e30-aab464b54edf-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-5cbrn\" (UID: \"6976caf5-0ff1-4604-9e30-aab464b54edf\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-5cbrn"
Sep 10 17:36:31.435206 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:31.435202    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ad03af6b-287a-4b2f-9ea3-ec8c43c17cba-image-registry-private-configuration\") pod \"image-registry-8f44f7f46-d2blr\" (UID: \"ad03af6b-287a-4b2f-9ea3-ec8c43c17cba\") " pod="openshift-image-registry/image-registry-8f44f7f46-d2blr"
Sep 10 17:36:31.435303 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:31.435233    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5edf557d-b123-458c-aaf8-e9760b5b6cfe-config\") pod \"service-ca-operator-859f94c4b-cmwpc\" (UID: \"5edf557d-b123-458c-aaf8-e9760b5b6cfe\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-cmwpc"
Sep 10 17:36:31.435303 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:31.435249    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/6976caf5-0ff1-4604-9e30-aab464b54edf-telemetry-config\") pod \"cluster-monitoring-operator-6cdcc96965-5cbrn\" (UID: \"6976caf5-0ff1-4604-9e30-aab464b54edf\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-5cbrn"
Sep 10 17:36:31.435303 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:31.435272    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5edf557d-b123-458c-aaf8-e9760b5b6cfe-serving-cert\") pod \"service-ca-operator-859f94c4b-cmwpc\" (UID: \"5edf557d-b123-458c-aaf8-e9760b5b6cfe\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-cmwpc"
Sep 10 17:36:31.435303 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:31.435294    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmphc\" (UniqueName: \"kubernetes.io/projected/5edf557d-b123-458c-aaf8-e9760b5b6cfe-kube-api-access-dmphc\") pod \"service-ca-operator-859f94c4b-cmwpc\" (UID: \"5edf557d-b123-458c-aaf8-e9760b5b6cfe\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-cmwpc"
Sep 10 17:36:31.435449 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:31.435314    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/97172b14-b84a-476e-89f1-05d3e70df416-config\") pod \"console-operator-85b454874b-rb854\" (UID: \"97172b14-b84a-476e-89f1-05d3e70df416\") " pod="openshift-console-operator/console-operator-85b454874b-rb854"
Sep 10 17:36:31.435449 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:31.435331    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f1713e2d-701b-435b-8d76-18824228a81e-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-mvd6n\" (UID: \"f1713e2d-701b-435b-8d76-18824228a81e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-mvd6n"
Sep 10 17:36:31.435449 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:31.435346    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c8f186e0-4c0d-4422-be62-96685cca0258-config\") pod \"kube-storage-version-migrator-operator-8d7dd4675-wt6xs\" (UID: \"c8f186e0-4c0d-4422-be62-96685cca0258\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-wt6xs"
Sep 10 17:36:31.435449 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:31.435369    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ad03af6b-287a-4b2f-9ea3-ec8c43c17cba-bound-sa-token\") pod \"image-registry-8f44f7f46-d2blr\" (UID: \"ad03af6b-287a-4b2f-9ea3-ec8c43c17cba\") " pod="openshift-image-registry/image-registry-8f44f7f46-d2blr"
Sep 10 17:36:31.435449 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:31.435428    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ad03af6b-287a-4b2f-9ea3-ec8c43c17cba-registry-certificates\") pod \"image-registry-8f44f7f46-d2blr\" (UID: \"ad03af6b-287a-4b2f-9ea3-ec8c43c17cba\") " pod="openshift-image-registry/image-registry-8f44f7f46-d2blr"
Sep 10 17:36:31.435648 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:31.435455    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/97172b14-b84a-476e-89f1-05d3e70df416-serving-cert\") pod \"console-operator-85b454874b-rb854\" (UID: \"97172b14-b84a-476e-89f1-05d3e70df416\") " pod="openshift-console-operator/console-operator-85b454874b-rb854"
Sep 10 17:36:31.435648 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:31.435485    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cpn9k\" (UniqueName: \"kubernetes.io/projected/678c4830-c56e-4af8-85a2-6f26a4e03aa8-kube-api-access-cpn9k\") pod \"router-default-d7f986b9d-xvnsc\" (UID: \"678c4830-c56e-4af8-85a2-6f26a4e03aa8\") " pod="openshift-ingress/router-default-d7f986b9d-xvnsc"
Sep 10 17:36:31.435648 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:31.435530    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c8f186e0-4c0d-4422-be62-96685cca0258-serving-cert\") pod \"kube-storage-version-migrator-operator-8d7dd4675-wt6xs\" (UID: \"c8f186e0-4c0d-4422-be62-96685cca0258\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-wt6xs"
Sep 10 17:36:31.435648 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:31.435557    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/678c4830-c56e-4af8-85a2-6f26a4e03aa8-metrics-certs\") pod \"router-default-d7f986b9d-xvnsc\" (UID: \"678c4830-c56e-4af8-85a2-6f26a4e03aa8\") " pod="openshift-ingress/router-default-d7f986b9d-xvnsc"
Sep 10 17:36:31.435818 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:31.435642    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/678c4830-c56e-4af8-85a2-6f26a4e03aa8-default-certificate\") pod \"router-default-d7f986b9d-xvnsc\" (UID: \"678c4830-c56e-4af8-85a2-6f26a4e03aa8\") " pod="openshift-ingress/router-default-d7f986b9d-xvnsc"
Sep 10 17:36:31.435818 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:31.435683    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/678c4830-c56e-4af8-85a2-6f26a4e03aa8-service-ca-bundle\") pod \"router-default-d7f986b9d-xvnsc\" (UID: \"678c4830-c56e-4af8-85a2-6f26a4e03aa8\") " pod="openshift-ingress/router-default-d7f986b9d-xvnsc"
Sep 10 17:36:31.435818 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:31.435709    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qt6bw\" (UniqueName: \"kubernetes.io/projected/97172b14-b84a-476e-89f1-05d3e70df416-kube-api-access-qt6bw\") pod \"console-operator-85b454874b-rb854\" (UID: \"97172b14-b84a-476e-89f1-05d3e70df416\") " pod="openshift-console-operator/console-operator-85b454874b-rb854"
Sep 10 17:36:31.435818 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:31.435735    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/678c4830-c56e-4af8-85a2-6f26a4e03aa8-stats-auth\") pod \"router-default-d7f986b9d-xvnsc\" (UID: \"678c4830-c56e-4af8-85a2-6f26a4e03aa8\") " pod="openshift-ingress/router-default-d7f986b9d-xvnsc"
Sep 10 17:36:31.435818 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:31.435783    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2x5w\" (UniqueName: \"kubernetes.io/projected/6976caf5-0ff1-4604-9e30-aab464b54edf-kube-api-access-q2x5w\") pod \"cluster-monitoring-operator-6cdcc96965-5cbrn\" (UID: \"6976caf5-0ff1-4604-9e30-aab464b54edf\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-5cbrn"
Sep 10 17:36:31.435818 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:31.435807    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ad03af6b-287a-4b2f-9ea3-ec8c43c17cba-ca-trust-extracted\") pod \"image-registry-8f44f7f46-d2blr\" (UID: \"ad03af6b-287a-4b2f-9ea3-ec8c43c17cba\") " pod="openshift-image-registry/image-registry-8f44f7f46-d2blr"
Sep 10 17:36:31.435992 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:31.435828    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ad03af6b-287a-4b2f-9ea3-ec8c43c17cba-installation-pull-secrets\") pod \"image-registry-8f44f7f46-d2blr\" (UID: \"ad03af6b-287a-4b2f-9ea3-ec8c43c17cba\") " pod="openshift-image-registry/image-registry-8f44f7f46-d2blr"
Sep 10 17:36:31.435992 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:31.435853    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmfg7\" (UniqueName: \"kubernetes.io/projected/c8f186e0-4c0d-4422-be62-96685cca0258-kube-api-access-tmfg7\") pod \"kube-storage-version-migrator-operator-8d7dd4675-wt6xs\" (UID: \"c8f186e0-4c0d-4422-be62-96685cca0258\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-wt6xs"
Sep 10 17:36:31.435992 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:31.435878    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ad03af6b-287a-4b2f-9ea3-ec8c43c17cba-trusted-ca\") pod \"image-registry-8f44f7f46-d2blr\" (UID: \"ad03af6b-287a-4b2f-9ea3-ec8c43c17cba\") " pod="openshift-image-registry/image-registry-8f44f7f46-d2blr"
Sep 10 17:36:31.468747 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:31.468724    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ncmn7"
Sep 10 17:36:31.471504 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:31.471488    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-7lms2\""
Sep 10 17:36:31.471605 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:31.471506    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 10 17:36:31.536578 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:31.536493    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/678c4830-c56e-4af8-85a2-6f26a4e03aa8-stats-auth\") pod \"router-default-d7f986b9d-xvnsc\" (UID: \"678c4830-c56e-4af8-85a2-6f26a4e03aa8\") " pod="openshift-ingress/router-default-d7f986b9d-xvnsc"
Sep 10 17:36:31.536578 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:31.536528    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2x5w\" (UniqueName: \"kubernetes.io/projected/6976caf5-0ff1-4604-9e30-aab464b54edf-kube-api-access-q2x5w\") pod \"cluster-monitoring-operator-6cdcc96965-5cbrn\" (UID: \"6976caf5-0ff1-4604-9e30-aab464b54edf\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-5cbrn"
Sep 10 17:36:31.536578 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:31.536553    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06ecbee3-6c6f-467b-831a-d9545f4588bd-tmp-dir\") pod \"dns-default-dxqxz\" (UID: \"06ecbee3-6c6f-467b-831a-d9545f4588bd\") " pod="openshift-dns/dns-default-dxqxz"
Sep 10 17:36:31.536816 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:31.536603    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ad03af6b-287a-4b2f-9ea3-ec8c43c17cba-ca-trust-extracted\") pod \"image-registry-8f44f7f46-d2blr\" (UID: \"ad03af6b-287a-4b2f-9ea3-ec8c43c17cba\") " pod="openshift-image-registry/image-registry-8f44f7f46-d2blr"
Sep 10 17:36:31.536816 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:31.536627    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bfa6c6a2-4e67-4c3c-b445-e07834780f92-cert\") pod \"ingress-canary-xctgk\" (UID: \"bfa6c6a2-4e67-4c3c-b445-e07834780f92\") " pod="openshift-ingress-canary/ingress-canary-xctgk"
Sep 10 17:36:31.536816 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:31.536652    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ad03af6b-287a-4b2f-9ea3-ec8c43c17cba-installation-pull-secrets\") pod \"image-registry-8f44f7f46-d2blr\" (UID: \"ad03af6b-287a-4b2f-9ea3-ec8c43c17cba\") " pod="openshift-image-registry/image-registry-8f44f7f46-d2blr"
Sep 10 17:36:31.536816 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:31.536676    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmfg7\" (UniqueName: \"kubernetes.io/projected/c8f186e0-4c0d-4422-be62-96685cca0258-kube-api-access-tmfg7\") pod \"kube-storage-version-migrator-operator-8d7dd4675-wt6xs\" (UID: \"c8f186e0-4c0d-4422-be62-96685cca0258\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-wt6xs"
Sep 10 17:36:31.536816 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:31.536704    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kx8ft\" (UniqueName: \"kubernetes.io/projected/bfa6c6a2-4e67-4c3c-b445-e07834780f92-kube-api-access-kx8ft\") pod \"ingress-canary-xctgk\" (UID: \"bfa6c6a2-4e67-4c3c-b445-e07834780f92\") " pod="openshift-ingress-canary/ingress-canary-xctgk"
Sep 10 17:36:31.536816 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:31.536725    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/fd21a8ca-3f90-4aa5-b1de-78ddebecf928-serving-cert\") pod \"insights-operator-76bcd6ccf5-rh6vv\" (UID: \"fd21a8ca-3f90-4aa5-b1de-78ddebecf928\") " pod="openshift-insights/insights-operator-76bcd6ccf5-rh6vv"
Sep 10 17:36:31.536816 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:31.536752    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ad03af6b-287a-4b2f-9ea3-ec8c43c17cba-trusted-ca\") pod \"image-registry-8f44f7f46-d2blr\" (UID: \"ad03af6b-287a-4b2f-9ea3-ec8c43c17cba\") " pod="openshift-image-registry/image-registry-8f44f7f46-d2blr"
Sep 10 17:36:31.536816 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:31.536774    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fd21a8ca-3f90-4aa5-b1de-78ddebecf928-trusted-ca-bundle\") pod \"insights-operator-76bcd6ccf5-rh6vv\" (UID: \"fd21a8ca-3f90-4aa5-b1de-78ddebecf928\") " pod="openshift-insights/insights-operator-76bcd6ccf5-rh6vv"
Sep 10 17:36:31.536816 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:31.536815    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/fd21a8ca-3f90-4aa5-b1de-78ddebecf928-snapshots\") pod \"insights-operator-76bcd6ccf5-rh6vv\" (UID: \"fd21a8ca-3f90-4aa5-b1de-78ddebecf928\") " pod="openshift-insights/insights-operator-76bcd6ccf5-rh6vv"
Sep 10 17:36:31.537247 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:31.536841    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ad03af6b-287a-4b2f-9ea3-ec8c43c17cba-registry-tls\") pod \"image-registry-8f44f7f46-d2blr\" (UID: \"ad03af6b-287a-4b2f-9ea3-ec8c43c17cba\") " pod="openshift-image-registry/image-registry-8f44f7f46-d2blr"
Sep 10 17:36:31.537247 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:31.536864    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vsbfw\" (UniqueName: \"kubernetes.io/projected/f1713e2d-701b-435b-8d76-18824228a81e-kube-api-access-vsbfw\") pod \"cluster-samples-operator-7cdcfcf76c-mvd6n\" (UID: \"f1713e2d-701b-435b-8d76-18824228a81e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-mvd6n"
Sep 10 17:36:31.537247 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:31.536903    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77f7r\" (UniqueName: \"kubernetes.io/projected/7f13ef98-e00e-4fe2-8f80-41eb99b0c7f7-kube-api-access-77f7r\") pod \"volume-data-source-validator-695c895bd6-8dmcm\" (UID: \"7f13ef98-e00e-4fe2-8f80-41eb99b0c7f7\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-8dmcm"
Sep 10 17:36:31.537247 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:31.536926    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svcd8\" (UniqueName: \"kubernetes.io/projected/fd21a8ca-3f90-4aa5-b1de-78ddebecf928-kube-api-access-svcd8\") pod \"insights-operator-76bcd6ccf5-rh6vv\" (UID: \"fd21a8ca-3f90-4aa5-b1de-78ddebecf928\") " pod="openshift-insights/insights-operator-76bcd6ccf5-rh6vv"
Sep 10 17:36:31.537247 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:31.536946    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qp685\" (UniqueName: \"kubernetes.io/projected/ad03af6b-287a-4b2f-9ea3-ec8c43c17cba-kube-api-access-qp685\") pod \"image-registry-8f44f7f46-d2blr\" (UID: \"ad03af6b-287a-4b2f-9ea3-ec8c43c17cba\") " pod="openshift-image-registry/image-registry-8f44f7f46-d2blr"
Sep 10 17:36:31.537247 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:31.536962    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/97172b14-b84a-476e-89f1-05d3e70df416-trusted-ca\") pod \"console-operator-85b454874b-rb854\" (UID: \"97172b14-b84a-476e-89f1-05d3e70df416\") " pod="openshift-console-operator/console-operator-85b454874b-rb854"
Sep 10 17:36:31.537247 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:31.536983    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6976caf5-0ff1-4604-9e30-aab464b54edf-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-5cbrn\" (UID: \"6976caf5-0ff1-4604-9e30-aab464b54edf\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-5cbrn"
Sep 10 17:36:31.537247 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:31.537003    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ad03af6b-287a-4b2f-9ea3-ec8c43c17cba-image-registry-private-configuration\") pod \"image-registry-8f44f7f46-d2blr\" (UID: \"ad03af6b-287a-4b2f-9ea3-ec8c43c17cba\") " pod="openshift-image-registry/image-registry-8f44f7f46-d2blr"
Sep 10 17:36:31.537247 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:31.537029    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5edf557d-b123-458c-aaf8-e9760b5b6cfe-config\") pod \"service-ca-operator-859f94c4b-cmwpc\" (UID: \"5edf557d-b123-458c-aaf8-e9760b5b6cfe\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-cmwpc"
Sep 10 17:36:31.537247 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:31.537044    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/6976caf5-0ff1-4604-9e30-aab464b54edf-telemetry-config\") pod \"cluster-monitoring-operator-6cdcc96965-5cbrn\" (UID: \"6976caf5-0ff1-4604-9e30-aab464b54edf\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-5cbrn"
Sep 10 17:36:31.537247 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:31.537059    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/06ecbee3-6c6f-467b-831a-d9545f4588bd-metrics-tls\") pod \"dns-default-dxqxz\" (UID: \"06ecbee3-6c6f-467b-831a-d9545f4588bd\") " pod="openshift-dns/dns-default-dxqxz"
Sep 10 17:36:31.537247 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:31.537058    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ad03af6b-287a-4b2f-9ea3-ec8c43c17cba-ca-trust-extracted\") pod \"image-registry-8f44f7f46-d2blr\" (UID: \"ad03af6b-287a-4b2f-9ea3-ec8c43c17cba\") " pod="openshift-image-registry/image-registry-8f44f7f46-d2blr"
Sep 10 17:36:31.537247 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:31.537081    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5edf557d-b123-458c-aaf8-e9760b5b6cfe-serving-cert\") pod \"service-ca-operator-859f94c4b-cmwpc\" (UID: \"5edf557d-b123-458c-aaf8-e9760b5b6cfe\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-cmwpc"
Sep 10 17:36:31.537247 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:31.537097    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmphc\" (UniqueName: \"kubernetes.io/projected/5edf557d-b123-458c-aaf8-e9760b5b6cfe-kube-api-access-dmphc\") pod \"service-ca-operator-859f94c4b-cmwpc\" (UID: \"5edf557d-b123-458c-aaf8-e9760b5b6cfe\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-cmwpc"
Sep 10 17:36:31.537247 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:31.537116    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/97172b14-b84a-476e-89f1-05d3e70df416-config\") pod \"console-operator-85b454874b-rb854\" (UID: \"97172b14-b84a-476e-89f1-05d3e70df416\") " pod="openshift-console-operator/console-operator-85b454874b-rb854"
Sep 10 17:36:31.537986 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:31.537150    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f1713e2d-701b-435b-8d76-18824228a81e-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-mvd6n\" (UID: \"f1713e2d-701b-435b-8d76-18824228a81e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-mvd6n"
Sep 10 17:36:31.537986 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:31.537181    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c8f186e0-4c0d-4422-be62-96685cca0258-config\") pod \"kube-storage-version-migrator-operator-8d7dd4675-wt6xs\" (UID: \"c8f186e0-4c0d-4422-be62-96685cca0258\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-wt6xs"
Sep 10 17:36:31.537986 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:31.537209    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/06ecbee3-6c6f-467b-831a-d9545f4588bd-config-volume\") pod \"dns-default-dxqxz\" (UID: \"06ecbee3-6c6f-467b-831a-d9545f4588bd\") " pod="openshift-dns/dns-default-dxqxz"
Sep 10 17:36:31.537986 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:31.537239    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ad03af6b-287a-4b2f-9ea3-ec8c43c17cba-bound-sa-token\") pod \"image-registry-8f44f7f46-d2blr\" (UID: \"ad03af6b-287a-4b2f-9ea3-ec8c43c17cba\") " pod="openshift-image-registry/image-registry-8f44f7f46-d2blr"
Sep 10 17:36:31.537986 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:31.537270    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fd21a8ca-3f90-4aa5-b1de-78ddebecf928-tmp\") pod \"insights-operator-76bcd6ccf5-rh6vv\" (UID: \"fd21a8ca-3f90-4aa5-b1de-78ddebecf928\") " pod="openshift-insights/insights-operator-76bcd6ccf5-rh6vv"
Sep 10 17:36:31.537986 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:31.537305    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ad03af6b-287a-4b2f-9ea3-ec8c43c17cba-registry-certificates\") pod \"image-registry-8f44f7f46-d2blr\" (UID: \"ad03af6b-287a-4b2f-9ea3-ec8c43c17cba\") " pod="openshift-image-registry/image-registry-8f44f7f46-d2blr"
Sep 10 17:36:31.537986 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:31.537329    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/97172b14-b84a-476e-89f1-05d3e70df416-serving-cert\") pod \"console-operator-85b454874b-rb854\" (UID: \"97172b14-b84a-476e-89f1-05d3e70df416\") " pod="openshift-console-operator/console-operator-85b454874b-rb854"
Sep 10 17:36:31.537986 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:36:31.537336    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 10 17:36:31.537986 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:36:31.537352    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-8f44f7f46-d2blr: secret "image-registry-tls" not found
Sep 10 17:36:31.537986 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:31.537353    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2nd4\" (UniqueName: \"kubernetes.io/projected/06ecbee3-6c6f-467b-831a-d9545f4588bd-kube-api-access-w2nd4\") pod \"dns-default-dxqxz\" (UID: \"06ecbee3-6c6f-467b-831a-d9545f4588bd\") " pod="openshift-dns/dns-default-dxqxz"
Sep 10 17:36:31.537986 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:31.537384    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4m67q\" (UniqueName: \"kubernetes.io/projected/1694e06a-5285-4434-b725-3384cc295951-kube-api-access-4m67q\") pod \"network-check-source-685456655f-7798s\" (UID: \"1694e06a-5285-4434-b725-3384cc295951\") " pod="openshift-network-diagnostics/network-check-source-685456655f-7798s"
Sep 10 17:36:31.537986 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:36:31.537412    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ad03af6b-287a-4b2f-9ea3-ec8c43c17cba-registry-tls podName:ad03af6b-287a-4b2f-9ea3-ec8c43c17cba nodeName:}" failed. No retries permitted until 2026-09-10 17:36:32.03739398 +0000 UTC m=+36.037224025 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ad03af6b-287a-4b2f-9ea3-ec8c43c17cba-registry-tls") pod "image-registry-8f44f7f46-d2blr" (UID: "ad03af6b-287a-4b2f-9ea3-ec8c43c17cba") : secret "image-registry-tls" not found
Sep 10 17:36:31.537986 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:36:31.537468    2579 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 10 17:36:31.537986 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:36:31.537686    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f1713e2d-701b-435b-8d76-18824228a81e-samples-operator-tls podName:f1713e2d-701b-435b-8d76-18824228a81e nodeName:}" failed. No retries permitted until 2026-09-10 17:36:32.037668538 +0000 UTC m=+36.037498589 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/f1713e2d-701b-435b-8d76-18824228a81e-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-mvd6n" (UID: "f1713e2d-701b-435b-8d76-18824228a81e") : secret "samples-operator-tls" not found
Sep 10 17:36:31.537986 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:31.537725    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpn9k\" (UniqueName: \"kubernetes.io/projected/678c4830-c56e-4af8-85a2-6f26a4e03aa8-kube-api-access-cpn9k\") pod \"router-default-d7f986b9d-xvnsc\" (UID: \"678c4830-c56e-4af8-85a2-6f26a4e03aa8\") " pod="openshift-ingress/router-default-d7f986b9d-xvnsc"
Sep 10 17:36:31.537986 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:31.537761    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c8f186e0-4c0d-4422-be62-96685cca0258-serving-cert\") pod \"kube-storage-version-migrator-operator-8d7dd4675-wt6xs\" (UID: \"c8f186e0-4c0d-4422-be62-96685cca0258\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-wt6xs"
Sep 10 17:36:31.538791 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:31.537786    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/678c4830-c56e-4af8-85a2-6f26a4e03aa8-metrics-certs\") pod \"router-default-d7f986b9d-xvnsc\" (UID: \"678c4830-c56e-4af8-85a2-6f26a4e03aa8\") " pod="openshift-ingress/router-default-d7f986b9d-xvnsc"
Sep 10 17:36:31.538791 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:31.537820    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/678c4830-c56e-4af8-85a2-6f26a4e03aa8-default-certificate\") pod \"router-default-d7f986b9d-xvnsc\" (UID: \"678c4830-c56e-4af8-85a2-6f26a4e03aa8\") " pod="openshift-ingress/router-default-d7f986b9d-xvnsc"
Sep 10 17:36:31.538791 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:31.537858    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/678c4830-c56e-4af8-85a2-6f26a4e03aa8-service-ca-bundle\") pod \"router-default-d7f986b9d-xvnsc\" (UID: \"678c4830-c56e-4af8-85a2-6f26a4e03aa8\") " pod="openshift-ingress/router-default-d7f986b9d-xvnsc"
Sep 10 17:36:31.538791 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:31.537889    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qt6bw\" (UniqueName: \"kubernetes.io/projected/97172b14-b84a-476e-89f1-05d3e70df416-kube-api-access-qt6bw\") pod \"console-operator-85b454874b-rb854\" (UID: \"97172b14-b84a-476e-89f1-05d3e70df416\") " pod="openshift-console-operator/console-operator-85b454874b-rb854"
Sep 10 17:36:31.538791 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:31.538039    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ad03af6b-287a-4b2f-9ea3-ec8c43c17cba-trusted-ca\") pod \"image-registry-8f44f7f46-d2blr\" (UID: \"ad03af6b-287a-4b2f-9ea3-ec8c43c17cba\") " pod="openshift-image-registry/image-registry-8f44f7f46-d2blr"
Sep 10 17:36:31.538791 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:31.537787    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/97172b14-b84a-476e-89f1-05d3e70df416-config\") pod \"console-operator-85b454874b-rb854\" (UID: \"97172b14-b84a-476e-89f1-05d3e70df416\") " pod="openshift-console-operator/console-operator-85b454874b-rb854"
Sep 10 17:36:31.538791 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:31.538158    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fd21a8ca-3f90-4aa5-b1de-78ddebecf928-service-ca-bundle\") pod \"insights-operator-76bcd6ccf5-rh6vv\" (UID: \"fd21a8ca-3f90-4aa5-b1de-78ddebecf928\") " pod="openshift-insights/insights-operator-76bcd6ccf5-rh6vv"
Sep 10 17:36:31.538791 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:31.538183    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c8f186e0-4c0d-4422-be62-96685cca0258-config\") pod \"kube-storage-version-migrator-operator-8d7dd4675-wt6xs\" (UID: \"c8f186e0-4c0d-4422-be62-96685cca0258\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-wt6xs"
Sep 10 17:36:31.538791 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:31.538254    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/6976caf5-0ff1-4604-9e30-aab464b54edf-telemetry-config\") pod \"cluster-monitoring-operator-6cdcc96965-5cbrn\" (UID: \"6976caf5-0ff1-4604-9e30-aab464b54edf\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-5cbrn"
Sep 10 17:36:31.538791 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:36:31.538333    2579 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 10 17:36:31.538791 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:36:31.538383    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6976caf5-0ff1-4604-9e30-aab464b54edf-cluster-monitoring-operator-tls podName:6976caf5-0ff1-4604-9e30-aab464b54edf nodeName:}" failed. No retries permitted until 2026-09-10 17:36:32.038370094 +0000 UTC m=+36.038200158 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/6976caf5-0ff1-4604-9e30-aab464b54edf-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-5cbrn" (UID: "6976caf5-0ff1-4604-9e30-aab464b54edf") : secret "cluster-monitoring-operator-tls" not found
Sep 10 17:36:31.538791 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:36:31.538752    2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 10 17:36:31.539362 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:36:31.539082    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/678c4830-c56e-4af8-85a2-6f26a4e03aa8-metrics-certs podName:678c4830-c56e-4af8-85a2-6f26a4e03aa8 nodeName:}" failed. No retries permitted until 2026-09-10 17:36:32.039066526 +0000 UTC m=+36.038896577 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/678c4830-c56e-4af8-85a2-6f26a4e03aa8-metrics-certs") pod "router-default-d7f986b9d-xvnsc" (UID: "678c4830-c56e-4af8-85a2-6f26a4e03aa8") : secret "router-metrics-certs-default" not found
Sep 10 17:36:31.539362 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:36:31.539180    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/678c4830-c56e-4af8-85a2-6f26a4e03aa8-service-ca-bundle podName:678c4830-c56e-4af8-85a2-6f26a4e03aa8 nodeName:}" failed. No retries permitted until 2026-09-10 17:36:32.03916272 +0000 UTC m=+36.038992780 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/678c4830-c56e-4af8-85a2-6f26a4e03aa8-service-ca-bundle") pod "router-default-d7f986b9d-xvnsc" (UID: "678c4830-c56e-4af8-85a2-6f26a4e03aa8") : configmap references non-existent config key: service-ca.crt
Sep 10 17:36:31.539362 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:31.539270    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5edf557d-b123-458c-aaf8-e9760b5b6cfe-config\") pod \"service-ca-operator-859f94c4b-cmwpc\" (UID: \"5edf557d-b123-458c-aaf8-e9760b5b6cfe\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-cmwpc"
Sep 10 17:36:31.539538 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:31.539477    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/97172b14-b84a-476e-89f1-05d3e70df416-trusted-ca\") pod \"console-operator-85b454874b-rb854\" (UID: \"97172b14-b84a-476e-89f1-05d3e70df416\") " pod="openshift-console-operator/console-operator-85b454874b-rb854"
Sep 10 17:36:31.539887 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:31.539819    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ad03af6b-287a-4b2f-9ea3-ec8c43c17cba-registry-certificates\") pod \"image-registry-8f44f7f46-d2blr\" (UID: \"ad03af6b-287a-4b2f-9ea3-ec8c43c17cba\") " pod="openshift-image-registry/image-registry-8f44f7f46-d2blr"
Sep 10 17:36:31.541945 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:31.541918    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5edf557d-b123-458c-aaf8-e9760b5b6cfe-serving-cert\") pod \"service-ca-operator-859f94c4b-cmwpc\" (UID: \"5edf557d-b123-458c-aaf8-e9760b5b6cfe\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-cmwpc"
Sep 10 17:36:31.542046 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:31.541922    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c8f186e0-4c0d-4422-be62-96685cca0258-serving-cert\") pod \"kube-storage-version-migrator-operator-8d7dd4675-wt6xs\" (UID: \"c8f186e0-4c0d-4422-be62-96685cca0258\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-wt6xs"
Sep 10 17:36:31.542272 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:31.542253    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/678c4830-c56e-4af8-85a2-6f26a4e03aa8-default-certificate\") pod \"router-default-d7f986b9d-xvnsc\" (UID: \"678c4830-c56e-4af8-85a2-6f26a4e03aa8\") " pod="openshift-ingress/router-default-d7f986b9d-xvnsc"
Sep 10 17:36:31.542373 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:31.542353    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/97172b14-b84a-476e-89f1-05d3e70df416-serving-cert\") pod \"console-operator-85b454874b-rb854\" (UID: \"97172b14-b84a-476e-89f1-05d3e70df416\") " pod="openshift-console-operator/console-operator-85b454874b-rb854"
Sep 10 17:36:31.542724 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:31.542704    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ad03af6b-287a-4b2f-9ea3-ec8c43c17cba-image-registry-private-configuration\") pod \"image-registry-8f44f7f46-d2blr\" (UID: \"ad03af6b-287a-4b2f-9ea3-ec8c43c17cba\") " pod="openshift-image-registry/image-registry-8f44f7f46-d2blr"
Sep 10 17:36:31.542830 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:31.542809    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/678c4830-c56e-4af8-85a2-6f26a4e03aa8-stats-auth\") pod \"router-default-d7f986b9d-xvnsc\" (UID: \"678c4830-c56e-4af8-85a2-6f26a4e03aa8\") " pod="openshift-ingress/router-default-d7f986b9d-xvnsc"
Sep 10 17:36:31.543379 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:31.543357    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ad03af6b-287a-4b2f-9ea3-ec8c43c17cba-installation-pull-secrets\") pod \"image-registry-8f44f7f46-d2blr\" (UID: \"ad03af6b-287a-4b2f-9ea3-ec8c43c17cba\") " pod="openshift-image-registry/image-registry-8f44f7f46-d2blr"
Sep 10 17:36:31.549999 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:31.549805    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2x5w\" (UniqueName: \"kubernetes.io/projected/6976caf5-0ff1-4604-9e30-aab464b54edf-kube-api-access-q2x5w\") pod \"cluster-monitoring-operator-6cdcc96965-5cbrn\" (UID: \"6976caf5-0ff1-4604-9e30-aab464b54edf\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-5cbrn"
Sep 10 17:36:31.550534 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:31.550512    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qp685\" (UniqueName: \"kubernetes.io/projected/ad03af6b-287a-4b2f-9ea3-ec8c43c17cba-kube-api-access-qp685\") pod \"image-registry-8f44f7f46-d2blr\" (UID: \"ad03af6b-287a-4b2f-9ea3-ec8c43c17cba\") " pod="openshift-image-registry/image-registry-8f44f7f46-d2blr"
Sep 10 17:36:31.552626 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:31.552529    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ad03af6b-287a-4b2f-9ea3-ec8c43c17cba-bound-sa-token\") pod \"image-registry-8f44f7f46-d2blr\" (UID: \"ad03af6b-287a-4b2f-9ea3-ec8c43c17cba\") " pod="openshift-image-registry/image-registry-8f44f7f46-d2blr"
Sep 10 17:36:31.552720 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:31.552622    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmphc\" (UniqueName: \"kubernetes.io/projected/5edf557d-b123-458c-aaf8-e9760b5b6cfe-kube-api-access-dmphc\") pod \"service-ca-operator-859f94c4b-cmwpc\" (UID: \"5edf557d-b123-458c-aaf8-e9760b5b6cfe\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-cmwpc"
Sep 10 17:36:31.552720 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:31.552697    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-77f7r\" (UniqueName: \"kubernetes.io/projected/7f13ef98-e00e-4fe2-8f80-41eb99b0c7f7-kube-api-access-77f7r\") pod \"volume-data-source-validator-695c895bd6-8dmcm\" (UID: \"7f13ef98-e00e-4fe2-8f80-41eb99b0c7f7\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-8dmcm"
Sep 10 17:36:31.552720 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:31.552705    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vsbfw\" (UniqueName: \"kubernetes.io/projected/f1713e2d-701b-435b-8d76-18824228a81e-kube-api-access-vsbfw\") pod \"cluster-samples-operator-7cdcfcf76c-mvd6n\" (UID: \"f1713e2d-701b-435b-8d76-18824228a81e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-mvd6n"
Sep 10 17:36:31.553175 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:31.553155    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpn9k\" (UniqueName: \"kubernetes.io/projected/678c4830-c56e-4af8-85a2-6f26a4e03aa8-kube-api-access-cpn9k\") pod \"router-default-d7f986b9d-xvnsc\" (UID: \"678c4830-c56e-4af8-85a2-6f26a4e03aa8\") " pod="openshift-ingress/router-default-d7f986b9d-xvnsc"
Sep 10 17:36:31.554836 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:31.554810    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qt6bw\" (UniqueName: \"kubernetes.io/projected/97172b14-b84a-476e-89f1-05d3e70df416-kube-api-access-qt6bw\") pod \"console-operator-85b454874b-rb854\" (UID: \"97172b14-b84a-476e-89f1-05d3e70df416\") " pod="openshift-console-operator/console-operator-85b454874b-rb854"
Sep 10 17:36:31.556087 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:31.556066    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmfg7\" (UniqueName: \"kubernetes.io/projected/c8f186e0-4c0d-4422-be62-96685cca0258-kube-api-access-tmfg7\") pod \"kube-storage-version-migrator-operator-8d7dd4675-wt6xs\" (UID: \"c8f186e0-4c0d-4422-be62-96685cca0258\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-wt6xs"
Sep 10 17:36:31.586253 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:31.586226    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-8dmcm"
Sep 10 17:36:31.625710 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:31.625682    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-85b454874b-rb854"
Sep 10 17:36:31.633817 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:31.633791    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-wt6xs"
Sep 10 17:36:31.639248 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:31.639229    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fd21a8ca-3f90-4aa5-b1de-78ddebecf928-service-ca-bundle\") pod \"insights-operator-76bcd6ccf5-rh6vv\" (UID: \"fd21a8ca-3f90-4aa5-b1de-78ddebecf928\") " pod="openshift-insights/insights-operator-76bcd6ccf5-rh6vv"
Sep 10 17:36:31.639355 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:31.639271    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06ecbee3-6c6f-467b-831a-d9545f4588bd-tmp-dir\") pod \"dns-default-dxqxz\" (UID: \"06ecbee3-6c6f-467b-831a-d9545f4588bd\") " pod="openshift-dns/dns-default-dxqxz"
Sep 10 17:36:31.639355 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:31.639294    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bfa6c6a2-4e67-4c3c-b445-e07834780f92-cert\") pod \"ingress-canary-xctgk\" (UID: \"bfa6c6a2-4e67-4c3c-b445-e07834780f92\") " pod="openshift-ingress-canary/ingress-canary-xctgk"
Sep 10 17:36:31.639355 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:31.639318    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kx8ft\" (UniqueName: \"kubernetes.io/projected/bfa6c6a2-4e67-4c3c-b445-e07834780f92-kube-api-access-kx8ft\") pod \"ingress-canary-xctgk\" (UID: \"bfa6c6a2-4e67-4c3c-b445-e07834780f92\") " pod="openshift-ingress-canary/ingress-canary-xctgk"
Sep 10 17:36:31.639355 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:31.639337    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/fd21a8ca-3f90-4aa5-b1de-78ddebecf928-serving-cert\") pod \"insights-operator-76bcd6ccf5-rh6vv\" (UID: \"fd21a8ca-3f90-4aa5-b1de-78ddebecf928\") " pod="openshift-insights/insights-operator-76bcd6ccf5-rh6vv"
Sep 10 17:36:31.639581 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:31.639369    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fd21a8ca-3f90-4aa5-b1de-78ddebecf928-trusted-ca-bundle\") pod \"insights-operator-76bcd6ccf5-rh6vv\" (UID: \"fd21a8ca-3f90-4aa5-b1de-78ddebecf928\") " pod="openshift-insights/insights-operator-76bcd6ccf5-rh6vv"
Sep 10 17:36:31.639581 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:31.639405    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/fd21a8ca-3f90-4aa5-b1de-78ddebecf928-snapshots\") pod \"insights-operator-76bcd6ccf5-rh6vv\" (UID: \"fd21a8ca-3f90-4aa5-b1de-78ddebecf928\") " pod="openshift-insights/insights-operator-76bcd6ccf5-rh6vv"
Sep 10 17:36:31.639581 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:31.639445    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svcd8\" (UniqueName: \"kubernetes.io/projected/fd21a8ca-3f90-4aa5-b1de-78ddebecf928-kube-api-access-svcd8\") pod \"insights-operator-76bcd6ccf5-rh6vv\" (UID: \"fd21a8ca-3f90-4aa5-b1de-78ddebecf928\") " pod="openshift-insights/insights-operator-76bcd6ccf5-rh6vv"
Sep 10 17:36:31.639581 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:31.639505    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/06ecbee3-6c6f-467b-831a-d9545f4588bd-metrics-tls\") pod \"dns-default-dxqxz\" (UID: \"06ecbee3-6c6f-467b-831a-d9545f4588bd\") " pod="openshift-dns/dns-default-dxqxz"
Sep 10 17:36:31.639581 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:31.639551    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/06ecbee3-6c6f-467b-831a-d9545f4588bd-config-volume\") pod \"dns-default-dxqxz\" (UID: \"06ecbee3-6c6f-467b-831a-d9545f4588bd\") " pod="openshift-dns/dns-default-dxqxz"
Sep 10 17:36:31.639808 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:31.639602    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fd21a8ca-3f90-4aa5-b1de-78ddebecf928-tmp\") pod \"insights-operator-76bcd6ccf5-rh6vv\" (UID: \"fd21a8ca-3f90-4aa5-b1de-78ddebecf928\") " pod="openshift-insights/insights-operator-76bcd6ccf5-rh6vv"
Sep 10 17:36:31.639808 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:31.639648    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2nd4\" (UniqueName: \"kubernetes.io/projected/06ecbee3-6c6f-467b-831a-d9545f4588bd-kube-api-access-w2nd4\") pod \"dns-default-dxqxz\" (UID: \"06ecbee3-6c6f-467b-831a-d9545f4588bd\") " pod="openshift-dns/dns-default-dxqxz"
Sep 10 17:36:31.639808 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:31.639683    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4m67q\" (UniqueName: \"kubernetes.io/projected/1694e06a-5285-4434-b725-3384cc295951-kube-api-access-4m67q\") pod \"network-check-source-685456655f-7798s\" (UID: \"1694e06a-5285-4434-b725-3384cc295951\") " pod="openshift-network-diagnostics/network-check-source-685456655f-7798s"
Sep 10 17:36:31.639955 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:31.639930    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fd21a8ca-3f90-4aa5-b1de-78ddebecf928-service-ca-bundle\") pod \"insights-operator-76bcd6ccf5-rh6vv\" (UID: \"fd21a8ca-3f90-4aa5-b1de-78ddebecf928\") " pod="openshift-insights/insights-operator-76bcd6ccf5-rh6vv"
Sep 10 17:36:31.640738 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:31.640116    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06ecbee3-6c6f-467b-831a-d9545f4588bd-tmp-dir\") pod \"dns-default-dxqxz\" (UID: \"06ecbee3-6c6f-467b-831a-d9545f4588bd\") " pod="openshift-dns/dns-default-dxqxz"
Sep 10 17:36:31.640738 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:31.640323    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/fd21a8ca-3f90-4aa5-b1de-78ddebecf928-snapshots\") pod \"insights-operator-76bcd6ccf5-rh6vv\" (UID: \"fd21a8ca-3f90-4aa5-b1de-78ddebecf928\") " pod="openshift-insights/insights-operator-76bcd6ccf5-rh6vv"
Sep 10 17:36:31.640738 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:36:31.640408    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 17:36:31.640738 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:31.640459    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fd21a8ca-3f90-4aa5-b1de-78ddebecf928-tmp\") pod \"insights-operator-76bcd6ccf5-rh6vv\" (UID: \"fd21a8ca-3f90-4aa5-b1de-78ddebecf928\") " pod="openshift-insights/insights-operator-76bcd6ccf5-rh6vv"
Sep 10 17:36:31.640738 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:36:31.640504    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06ecbee3-6c6f-467b-831a-d9545f4588bd-metrics-tls podName:06ecbee3-6c6f-467b-831a-d9545f4588bd nodeName:}" failed. No retries permitted until 2026-09-10 17:36:32.140485789 +0000 UTC m=+36.140315848 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/06ecbee3-6c6f-467b-831a-d9545f4588bd-metrics-tls") pod "dns-default-dxqxz" (UID: "06ecbee3-6c6f-467b-831a-d9545f4588bd") : secret "dns-default-metrics-tls" not found
Sep 10 17:36:31.640738 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:31.640693    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/06ecbee3-6c6f-467b-831a-d9545f4588bd-config-volume\") pod \"dns-default-dxqxz\" (UID: \"06ecbee3-6c6f-467b-831a-d9545f4588bd\") " pod="openshift-dns/dns-default-dxqxz"
Sep 10 17:36:31.641212 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:31.641190    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fd21a8ca-3f90-4aa5-b1de-78ddebecf928-trusted-ca-bundle\") pod \"insights-operator-76bcd6ccf5-rh6vv\" (UID: \"fd21a8ca-3f90-4aa5-b1de-78ddebecf928\") " pod="openshift-insights/insights-operator-76bcd6ccf5-rh6vv"
Sep 10 17:36:31.642368 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:31.642349    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/fd21a8ca-3f90-4aa5-b1de-78ddebecf928-serving-cert\") pod \"insights-operator-76bcd6ccf5-rh6vv\" (UID: \"fd21a8ca-3f90-4aa5-b1de-78ddebecf928\") " pod="openshift-insights/insights-operator-76bcd6ccf5-rh6vv"
Sep 10 17:36:31.642654 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:31.642631    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bfa6c6a2-4e67-4c3c-b445-e07834780f92-cert\") pod \"ingress-canary-xctgk\" (UID: \"bfa6c6a2-4e67-4c3c-b445-e07834780f92\") " pod="openshift-ingress-canary/ingress-canary-xctgk"
Sep 10 17:36:31.649410 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:31.649390    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-cmwpc"
Sep 10 17:36:31.649501 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:31.649438    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4m67q\" (UniqueName: \"kubernetes.io/projected/1694e06a-5285-4434-b725-3384cc295951-kube-api-access-4m67q\") pod \"network-check-source-685456655f-7798s\" (UID: \"1694e06a-5285-4434-b725-3384cc295951\") " pod="openshift-network-diagnostics/network-check-source-685456655f-7798s"
Sep 10 17:36:31.649826 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:31.649798    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2nd4\" (UniqueName: \"kubernetes.io/projected/06ecbee3-6c6f-467b-831a-d9545f4588bd-kube-api-access-w2nd4\") pod \"dns-default-dxqxz\" (UID: \"06ecbee3-6c6f-467b-831a-d9545f4588bd\") " pod="openshift-dns/dns-default-dxqxz"
Sep 10 17:36:31.649908 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:31.649857    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kx8ft\" (UniqueName: \"kubernetes.io/projected/bfa6c6a2-4e67-4c3c-b445-e07834780f92-kube-api-access-kx8ft\") pod \"ingress-canary-xctgk\" (UID: \"bfa6c6a2-4e67-4c3c-b445-e07834780f92\") " pod="openshift-ingress-canary/ingress-canary-xctgk"
Sep 10 17:36:31.649908 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:31.649862    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-svcd8\" (UniqueName: \"kubernetes.io/projected/fd21a8ca-3f90-4aa5-b1de-78ddebecf928-kube-api-access-svcd8\") pod \"insights-operator-76bcd6ccf5-rh6vv\" (UID: \"fd21a8ca-3f90-4aa5-b1de-78ddebecf928\") " pod="openshift-insights/insights-operator-76bcd6ccf5-rh6vv"
Sep 10 17:36:31.703950 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:31.703723    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-76bcd6ccf5-rh6vv"
Sep 10 17:36:31.716894 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:31.716688    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-xctgk"
Sep 10 17:36:31.735501 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:31.735357    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-685456655f-7798s"
Sep 10 17:36:31.895779 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:31.895747    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-8dmcm"]
Sep 10 17:36:31.898807 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:36:31.898781    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f13ef98_e00e_4fe2_8f80_41eb99b0c7f7.slice/crio-3c7c1634590ecdf60941bba5ec400d03236c158a8964042c9badb3c6d4ffe869 WatchSource:0}: Error finding container 3c7c1634590ecdf60941bba5ec400d03236c158a8964042c9badb3c6d4ffe869: Status 404 returned error can't find the container with id 3c7c1634590ecdf60941bba5ec400d03236c158a8964042c9badb3c6d4ffe869
Sep 10 17:36:31.919349 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:31.919310    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-85b454874b-rb854"]
Sep 10 17:36:31.921716 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:36:31.921682    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97172b14_b84a_476e_89f1_05d3e70df416.slice/crio-949532d71437bbd9d1f2dff40cdbac9c7a499345c6c44ed0470abe791acb56d8 WatchSource:0}: Error finding container 949532d71437bbd9d1f2dff40cdbac9c7a499345c6c44ed0470abe791acb56d8: Status 404 returned error can't find the container with id 949532d71437bbd9d1f2dff40cdbac9c7a499345c6c44ed0470abe791acb56d8
Sep 10 17:36:31.949592 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:31.949443    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-wt6xs"]
Sep 10 17:36:31.951882 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:36:31.951855    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8f186e0_4c0d_4422_be62_96685cca0258.slice/crio-a93188c7374898455849e97c84c0796b7fa7fdbf464c1e05bce8a0be86da7c31 WatchSource:0}: Error finding container a93188c7374898455849e97c84c0796b7fa7fdbf464c1e05bce8a0be86da7c31: Status 404 returned error can't find the container with id a93188c7374898455849e97c84c0796b7fa7fdbf464c1e05bce8a0be86da7c31
Sep 10 17:36:31.954208 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:31.954189    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-859f94c4b-cmwpc"]
Sep 10 17:36:31.957011 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:36:31.956990    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5edf557d_b123_458c_aaf8_e9760b5b6cfe.slice/crio-598115b4f3a36f95c79c916fc6d0651402671e8f3c4d6f892154025d9df6cee1 WatchSource:0}: Error finding container 598115b4f3a36f95c79c916fc6d0651402671e8f3c4d6f892154025d9df6cee1: Status 404 returned error can't find the container with id 598115b4f3a36f95c79c916fc6d0651402671e8f3c4d6f892154025d9df6cee1
Sep 10 17:36:32.015840 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:32.015815    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-xctgk"]
Sep 10 17:36:32.018592 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:36:32.018541    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfa6c6a2_4e67_4c3c_b445_e07834780f92.slice/crio-af7a4bf2b10a030b1ac6ff10a3f0e177b60946981f250d0a9b50406f7d0d2ea1 WatchSource:0}: Error finding container af7a4bf2b10a030b1ac6ff10a3f0e177b60946981f250d0a9b50406f7d0d2ea1: Status 404 returned error can't find the container with id af7a4bf2b10a030b1ac6ff10a3f0e177b60946981f250d0a9b50406f7d0d2ea1
Sep 10 17:36:32.028420 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:32.028394    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-76bcd6ccf5-rh6vv"]
Sep 10 17:36:32.031491 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:36:32.031465    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd21a8ca_3f90_4aa5_b1de_78ddebecf928.slice/crio-b16483d7a2b87d87443996eb3521a389a9ccb807eec49a5346f9465bc1d7497b WatchSource:0}: Error finding container b16483d7a2b87d87443996eb3521a389a9ccb807eec49a5346f9465bc1d7497b: Status 404 returned error can't find the container with id b16483d7a2b87d87443996eb3521a389a9ccb807eec49a5346f9465bc1d7497b
Sep 10 17:36:32.041473 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:32.041387    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-685456655f-7798s"]
Sep 10 17:36:32.043600 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:36:32.043521    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1694e06a_5285_4434_b725_3384cc295951.slice/crio-56a6c4dae1c8b06778d36b1a9ac6a581f4939a076573bda8a59b9000d53e37ea WatchSource:0}: Error finding container 56a6c4dae1c8b06778d36b1a9ac6a581f4939a076573bda8a59b9000d53e37ea: Status 404 returned error can't find the container with id 56a6c4dae1c8b06778d36b1a9ac6a581f4939a076573bda8a59b9000d53e37ea
Sep 10 17:36:32.043693 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:32.043620    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/678c4830-c56e-4af8-85a2-6f26a4e03aa8-metrics-certs\") pod \"router-default-d7f986b9d-xvnsc\" (UID: \"678c4830-c56e-4af8-85a2-6f26a4e03aa8\") " pod="openshift-ingress/router-default-d7f986b9d-xvnsc"
Sep 10 17:36:32.043693 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:32.043648    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/678c4830-c56e-4af8-85a2-6f26a4e03aa8-service-ca-bundle\") pod \"router-default-d7f986b9d-xvnsc\" (UID: \"678c4830-c56e-4af8-85a2-6f26a4e03aa8\") " pod="openshift-ingress/router-default-d7f986b9d-xvnsc"
Sep 10 17:36:32.043798 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:32.043714    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ad03af6b-287a-4b2f-9ea3-ec8c43c17cba-registry-tls\") pod \"image-registry-8f44f7f46-d2blr\" (UID: \"ad03af6b-287a-4b2f-9ea3-ec8c43c17cba\") " pod="openshift-image-registry/image-registry-8f44f7f46-d2blr"
Sep 10 17:36:32.043798 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:32.043747    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6976caf5-0ff1-4604-9e30-aab464b54edf-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-5cbrn\" (UID: \"6976caf5-0ff1-4604-9e30-aab464b54edf\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-5cbrn"
Sep 10 17:36:32.043798 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:36:32.043758    2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 10 17:36:32.043955 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:36:32.043810    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/678c4830-c56e-4af8-85a2-6f26a4e03aa8-metrics-certs podName:678c4830-c56e-4af8-85a2-6f26a4e03aa8 nodeName:}" failed. No retries permitted until 2026-09-10 17:36:33.043793073 +0000 UTC m=+37.043623124 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/678c4830-c56e-4af8-85a2-6f26a4e03aa8-metrics-certs") pod "router-default-d7f986b9d-xvnsc" (UID: "678c4830-c56e-4af8-85a2-6f26a4e03aa8") : secret "router-metrics-certs-default" not found
Sep 10 17:36:32.043955 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:36:32.043880    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/678c4830-c56e-4af8-85a2-6f26a4e03aa8-service-ca-bundle podName:678c4830-c56e-4af8-85a2-6f26a4e03aa8 nodeName:}" failed. No retries permitted until 2026-09-10 17:36:33.043857704 +0000 UTC m=+37.043687766 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/678c4830-c56e-4af8-85a2-6f26a4e03aa8-service-ca-bundle") pod "router-default-d7f986b9d-xvnsc" (UID: "678c4830-c56e-4af8-85a2-6f26a4e03aa8") : configmap references non-existent config key: service-ca.crt
Sep 10 17:36:32.043955 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:36:32.043923    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 10 17:36:32.043955 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:36:32.043935    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-8f44f7f46-d2blr: secret "image-registry-tls" not found
Sep 10 17:36:32.043955 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:32.043941    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f1713e2d-701b-435b-8d76-18824228a81e-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-mvd6n\" (UID: \"f1713e2d-701b-435b-8d76-18824228a81e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-mvd6n"
Sep 10 17:36:32.044211 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:36:32.043971    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ad03af6b-287a-4b2f-9ea3-ec8c43c17cba-registry-tls podName:ad03af6b-287a-4b2f-9ea3-ec8c43c17cba nodeName:}" failed. No retries permitted until 2026-09-10 17:36:33.043960978 +0000 UTC m=+37.043791030 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ad03af6b-287a-4b2f-9ea3-ec8c43c17cba-registry-tls") pod "image-registry-8f44f7f46-d2blr" (UID: "ad03af6b-287a-4b2f-9ea3-ec8c43c17cba") : secret "image-registry-tls" not found
Sep 10 17:36:32.044211 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:36:32.044002    2579 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 10 17:36:32.044211 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:36:32.044033    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6976caf5-0ff1-4604-9e30-aab464b54edf-cluster-monitoring-operator-tls podName:6976caf5-0ff1-4604-9e30-aab464b54edf nodeName:}" failed. No retries permitted until 2026-09-10 17:36:33.044022139 +0000 UTC m=+37.043852184 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/6976caf5-0ff1-4604-9e30-aab464b54edf-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-5cbrn" (UID: "6976caf5-0ff1-4604-9e30-aab464b54edf") : secret "cluster-monitoring-operator-tls" not found
Sep 10 17:36:32.044211 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:36:32.044064    2579 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 10 17:36:32.044211 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:36:32.044099    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f1713e2d-701b-435b-8d76-18824228a81e-samples-operator-tls podName:f1713e2d-701b-435b-8d76-18824228a81e nodeName:}" failed. No retries permitted until 2026-09-10 17:36:33.044089532 +0000 UTC m=+37.043919576 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/f1713e2d-701b-435b-8d76-18824228a81e-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-mvd6n" (UID: "f1713e2d-701b-435b-8d76-18824228a81e") : secret "samples-operator-tls" not found
Sep 10 17:36:32.144908 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:32.144875    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/06ecbee3-6c6f-467b-831a-d9545f4588bd-metrics-tls\") pod \"dns-default-dxqxz\" (UID: \"06ecbee3-6c6f-467b-831a-d9545f4588bd\") " pod="openshift-dns/dns-default-dxqxz"
Sep 10 17:36:32.145048 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:36:32.145029    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 17:36:32.145107 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:36:32.145096    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06ecbee3-6c6f-467b-831a-d9545f4588bd-metrics-tls podName:06ecbee3-6c6f-467b-831a-d9545f4588bd nodeName:}" failed. No retries permitted until 2026-09-10 17:36:33.145077762 +0000 UTC m=+37.144907809 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/06ecbee3-6c6f-467b-831a-d9545f4588bd-metrics-tls") pod "dns-default-dxqxz" (UID: "06ecbee3-6c6f-467b-831a-d9545f4588bd") : secret "dns-default-metrics-tls" not found
Sep 10 17:36:32.472027 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:32.472002    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m4npt"
Sep 10 17:36:32.472490 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:32.472013    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8k8z5"
Sep 10 17:36:32.474850 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:32.474831    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-sh4f6\""
Sep 10 17:36:32.474850 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:32.474850    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 10 17:36:32.827889 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:32.827777    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-xctgk" event={"ID":"bfa6c6a2-4e67-4c3c-b445-e07834780f92","Type":"ContainerStarted","Data":"af7a4bf2b10a030b1ac6ff10a3f0e177b60946981f250d0a9b50406f7d0d2ea1"}
Sep 10 17:36:32.830229 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:32.830163    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-rb854" event={"ID":"97172b14-b84a-476e-89f1-05d3e70df416","Type":"ContainerStarted","Data":"949532d71437bbd9d1f2dff40cdbac9c7a499345c6c44ed0470abe791acb56d8"}
Sep 10 17:36:32.833882 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:32.833803    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-cmwpc" event={"ID":"5edf557d-b123-458c-aaf8-e9760b5b6cfe","Type":"ContainerStarted","Data":"598115b4f3a36f95c79c916fc6d0651402671e8f3c4d6f892154025d9df6cee1"}
Sep 10 17:36:32.836634 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:32.836549    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-8dmcm" event={"ID":"7f13ef98-e00e-4fe2-8f80-41eb99b0c7f7","Type":"ContainerStarted","Data":"3c7c1634590ecdf60941bba5ec400d03236c158a8964042c9badb3c6d4ffe869"}
Sep 10 17:36:32.839404 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:32.839327    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-685456655f-7798s" event={"ID":"1694e06a-5285-4434-b725-3384cc295951","Type":"ContainerStarted","Data":"56a6c4dae1c8b06778d36b1a9ac6a581f4939a076573bda8a59b9000d53e37ea"}
Sep 10 17:36:32.843866 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:32.843807    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-rh6vv" event={"ID":"fd21a8ca-3f90-4aa5-b1de-78ddebecf928","Type":"ContainerStarted","Data":"b16483d7a2b87d87443996eb3521a389a9ccb807eec49a5346f9465bc1d7497b"}
Sep 10 17:36:32.846704 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:32.846641    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-wt6xs" event={"ID":"c8f186e0-4c0d-4422-be62-96685cca0258","Type":"ContainerStarted","Data":"a93188c7374898455849e97c84c0796b7fa7fdbf464c1e05bce8a0be86da7c31"}
Sep 10 17:36:33.054974 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:33.054217    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ad03af6b-287a-4b2f-9ea3-ec8c43c17cba-registry-tls\") pod \"image-registry-8f44f7f46-d2blr\" (UID: \"ad03af6b-287a-4b2f-9ea3-ec8c43c17cba\") " pod="openshift-image-registry/image-registry-8f44f7f46-d2blr"
Sep 10 17:36:33.054974 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:33.054277    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6976caf5-0ff1-4604-9e30-aab464b54edf-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-5cbrn\" (UID: \"6976caf5-0ff1-4604-9e30-aab464b54edf\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-5cbrn"
Sep 10 17:36:33.054974 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:33.054352    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f1713e2d-701b-435b-8d76-18824228a81e-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-mvd6n\" (UID: \"f1713e2d-701b-435b-8d76-18824228a81e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-mvd6n"
Sep 10 17:36:33.054974 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:33.054406    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/678c4830-c56e-4af8-85a2-6f26a4e03aa8-metrics-certs\") pod \"router-default-d7f986b9d-xvnsc\" (UID: \"678c4830-c56e-4af8-85a2-6f26a4e03aa8\") " pod="openshift-ingress/router-default-d7f986b9d-xvnsc"
Sep 10 17:36:33.054974 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:33.054435    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/678c4830-c56e-4af8-85a2-6f26a4e03aa8-service-ca-bundle\") pod \"router-default-d7f986b9d-xvnsc\" (UID: \"678c4830-c56e-4af8-85a2-6f26a4e03aa8\") " pod="openshift-ingress/router-default-d7f986b9d-xvnsc"
Sep 10 17:36:33.054974 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:36:33.054674    2579 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 10 17:36:33.054974 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:36:33.054769    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 10 17:36:33.054974 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:36:33.054779    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-8f44f7f46-d2blr: secret "image-registry-tls" not found
Sep 10 17:36:33.054974 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:36:33.054857    2579 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 10 17:36:33.054974 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:36:33.054931    2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 10 17:36:33.056108 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:36:33.056012    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/678c4830-c56e-4af8-85a2-6f26a4e03aa8-service-ca-bundle podName:678c4830-c56e-4af8-85a2-6f26a4e03aa8 nodeName:}" failed. No retries permitted until 2026-09-10 17:36:35.054552447 +0000 UTC m=+39.054382498 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/678c4830-c56e-4af8-85a2-6f26a4e03aa8-service-ca-bundle") pod "router-default-d7f986b9d-xvnsc" (UID: "678c4830-c56e-4af8-85a2-6f26a4e03aa8") : configmap references non-existent config key: service-ca.crt
Sep 10 17:36:33.056108 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:36:33.056045    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6976caf5-0ff1-4604-9e30-aab464b54edf-cluster-monitoring-operator-tls podName:6976caf5-0ff1-4604-9e30-aab464b54edf nodeName:}" failed. No retries permitted until 2026-09-10 17:36:35.056030823 +0000 UTC m=+39.055860875 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/6976caf5-0ff1-4604-9e30-aab464b54edf-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-5cbrn" (UID: "6976caf5-0ff1-4604-9e30-aab464b54edf") : secret "cluster-monitoring-operator-tls" not found
Sep 10 17:36:33.056108 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:36:33.056062    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ad03af6b-287a-4b2f-9ea3-ec8c43c17cba-registry-tls podName:ad03af6b-287a-4b2f-9ea3-ec8c43c17cba nodeName:}" failed. No retries permitted until 2026-09-10 17:36:35.056051698 +0000 UTC m=+39.055881749 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ad03af6b-287a-4b2f-9ea3-ec8c43c17cba-registry-tls") pod "image-registry-8f44f7f46-d2blr" (UID: "ad03af6b-287a-4b2f-9ea3-ec8c43c17cba") : secret "image-registry-tls" not found
Sep 10 17:36:33.056108 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:36:33.056076    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f1713e2d-701b-435b-8d76-18824228a81e-samples-operator-tls podName:f1713e2d-701b-435b-8d76-18824228a81e nodeName:}" failed. No retries permitted until 2026-09-10 17:36:35.056067836 +0000 UTC m=+39.055897930 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/f1713e2d-701b-435b-8d76-18824228a81e-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-mvd6n" (UID: "f1713e2d-701b-435b-8d76-18824228a81e") : secret "samples-operator-tls" not found
Sep 10 17:36:33.056108 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:36:33.056090    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/678c4830-c56e-4af8-85a2-6f26a4e03aa8-metrics-certs podName:678c4830-c56e-4af8-85a2-6f26a4e03aa8 nodeName:}" failed. No retries permitted until 2026-09-10 17:36:35.056081959 +0000 UTC m=+39.055912047 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/678c4830-c56e-4af8-85a2-6f26a4e03aa8-metrics-certs") pod "router-default-d7f986b9d-xvnsc" (UID: "678c4830-c56e-4af8-85a2-6f26a4e03aa8") : secret "router-metrics-certs-default" not found
Sep 10 17:36:33.155911 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:33.155378    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/06ecbee3-6c6f-467b-831a-d9545f4588bd-metrics-tls\") pod \"dns-default-dxqxz\" (UID: \"06ecbee3-6c6f-467b-831a-d9545f4588bd\") " pod="openshift-dns/dns-default-dxqxz"
Sep 10 17:36:33.155911 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:36:33.155504    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 17:36:33.155911 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:36:33.155597    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06ecbee3-6c6f-467b-831a-d9545f4588bd-metrics-tls podName:06ecbee3-6c6f-467b-831a-d9545f4588bd nodeName:}" failed. No retries permitted until 2026-09-10 17:36:35.155559073 +0000 UTC m=+39.155389122 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/06ecbee3-6c6f-467b-831a-d9545f4588bd-metrics-tls") pod "dns-default-dxqxz" (UID: "06ecbee3-6c6f-467b-831a-d9545f4588bd") : secret "dns-default-metrics-tls" not found
Sep 10 17:36:35.073929 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:35.073889    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ad03af6b-287a-4b2f-9ea3-ec8c43c17cba-registry-tls\") pod \"image-registry-8f44f7f46-d2blr\" (UID: \"ad03af6b-287a-4b2f-9ea3-ec8c43c17cba\") " pod="openshift-image-registry/image-registry-8f44f7f46-d2blr"
Sep 10 17:36:35.074616 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:35.073949    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6976caf5-0ff1-4604-9e30-aab464b54edf-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-5cbrn\" (UID: \"6976caf5-0ff1-4604-9e30-aab464b54edf\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-5cbrn"
Sep 10 17:36:35.074616 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:35.074024    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f1713e2d-701b-435b-8d76-18824228a81e-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-mvd6n\" (UID: \"f1713e2d-701b-435b-8d76-18824228a81e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-mvd6n"
Sep 10 17:36:35.074616 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:36:35.074045    2579 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 10 17:36:35.074616 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:35.074072    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/678c4830-c56e-4af8-85a2-6f26a4e03aa8-metrics-certs\") pod \"router-default-d7f986b9d-xvnsc\" (UID: \"678c4830-c56e-4af8-85a2-6f26a4e03aa8\") " pod="openshift-ingress/router-default-d7f986b9d-xvnsc"
Sep 10 17:36:35.074616 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:35.074101    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/678c4830-c56e-4af8-85a2-6f26a4e03aa8-service-ca-bundle\") pod \"router-default-d7f986b9d-xvnsc\" (UID: \"678c4830-c56e-4af8-85a2-6f26a4e03aa8\") " pod="openshift-ingress/router-default-d7f986b9d-xvnsc"
Sep 10 17:36:35.074616 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:36:35.074116    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6976caf5-0ff1-4604-9e30-aab464b54edf-cluster-monitoring-operator-tls podName:6976caf5-0ff1-4604-9e30-aab464b54edf nodeName:}" failed. No retries permitted until 2026-09-10 17:36:39.074101388 +0000 UTC m=+43.073931437 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/6976caf5-0ff1-4604-9e30-aab464b54edf-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-5cbrn" (UID: "6976caf5-0ff1-4604-9e30-aab464b54edf") : secret "cluster-monitoring-operator-tls" not found
Sep 10 17:36:35.074616 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:36:35.074050    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 10 17:36:35.074616 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:36:35.074160    2579 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 10 17:36:35.074616 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:36:35.074166    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-8f44f7f46-d2blr: secret "image-registry-tls" not found
Sep 10 17:36:35.074616 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:36:35.074185    2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 10 17:36:35.074616 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:36:35.074209    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/678c4830-c56e-4af8-85a2-6f26a4e03aa8-service-ca-bundle podName:678c4830-c56e-4af8-85a2-6f26a4e03aa8 nodeName:}" failed. No retries permitted until 2026-09-10 17:36:39.074193926 +0000 UTC m=+43.074023977 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/678c4830-c56e-4af8-85a2-6f26a4e03aa8-service-ca-bundle") pod "router-default-d7f986b9d-xvnsc" (UID: "678c4830-c56e-4af8-85a2-6f26a4e03aa8") : configmap references non-existent config key: service-ca.crt
Sep 10 17:36:35.074616 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:36:35.074228    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f1713e2d-701b-435b-8d76-18824228a81e-samples-operator-tls podName:f1713e2d-701b-435b-8d76-18824228a81e nodeName:}" failed. No retries permitted until 2026-09-10 17:36:39.074218552 +0000 UTC m=+43.074048597 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/f1713e2d-701b-435b-8d76-18824228a81e-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-mvd6n" (UID: "f1713e2d-701b-435b-8d76-18824228a81e") : secret "samples-operator-tls" not found
Sep 10 17:36:35.074616 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:36:35.074243    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ad03af6b-287a-4b2f-9ea3-ec8c43c17cba-registry-tls podName:ad03af6b-287a-4b2f-9ea3-ec8c43c17cba nodeName:}" failed. No retries permitted until 2026-09-10 17:36:39.074234295 +0000 UTC m=+43.074064347 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ad03af6b-287a-4b2f-9ea3-ec8c43c17cba-registry-tls") pod "image-registry-8f44f7f46-d2blr" (UID: "ad03af6b-287a-4b2f-9ea3-ec8c43c17cba") : secret "image-registry-tls" not found
Sep 10 17:36:35.074616 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:36:35.074257    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/678c4830-c56e-4af8-85a2-6f26a4e03aa8-metrics-certs podName:678c4830-c56e-4af8-85a2-6f26a4e03aa8 nodeName:}" failed. No retries permitted until 2026-09-10 17:36:39.074248787 +0000 UTC m=+43.074078836 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/678c4830-c56e-4af8-85a2-6f26a4e03aa8-metrics-certs") pod "router-default-d7f986b9d-xvnsc" (UID: "678c4830-c56e-4af8-85a2-6f26a4e03aa8") : secret "router-metrics-certs-default" not found
Sep 10 17:36:35.175443 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:35.175400    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/06ecbee3-6c6f-467b-831a-d9545f4588bd-metrics-tls\") pod \"dns-default-dxqxz\" (UID: \"06ecbee3-6c6f-467b-831a-d9545f4588bd\") " pod="openshift-dns/dns-default-dxqxz"
Sep 10 17:36:35.175644 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:36:35.175602    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 17:36:35.175724 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:36:35.175671    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06ecbee3-6c6f-467b-831a-d9545f4588bd-metrics-tls podName:06ecbee3-6c6f-467b-831a-d9545f4588bd nodeName:}" failed. No retries permitted until 2026-09-10 17:36:39.17565081 +0000 UTC m=+43.175480870 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/06ecbee3-6c6f-467b-831a-d9545f4588bd-metrics-tls") pod "dns-default-dxqxz" (UID: "06ecbee3-6c6f-467b-831a-d9545f4588bd") : secret "dns-default-metrics-tls" not found
Sep 10 17:36:35.660382 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:35.659261    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-79977f9844-5klhd"]
Sep 10 17:36:35.677371 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:35.677337    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-79977f9844-5klhd"]
Sep 10 17:36:35.677513 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:35.677433    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-79977f9844-5klhd"
Sep 10 17:36:35.680198 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:35.680175    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Sep 10 17:36:35.680311 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:35.680175    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-9bpk6\""
Sep 10 17:36:35.681279 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:35.681258    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Sep 10 17:36:35.780110 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:35.780070    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/510bea53-93c2-4158-851a-0b6ec3e6bee6-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-5klhd\" (UID: \"510bea53-93c2-4158-851a-0b6ec3e6bee6\") " pod="openshift-network-console/networking-console-plugin-79977f9844-5klhd"
Sep 10 17:36:35.780288 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:35.780123    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/510bea53-93c2-4158-851a-0b6ec3e6bee6-nginx-conf\") pod \"networking-console-plugin-79977f9844-5klhd\" (UID: \"510bea53-93c2-4158-851a-0b6ec3e6bee6\") " pod="openshift-network-console/networking-console-plugin-79977f9844-5klhd"
Sep 10 17:36:35.836170 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:35.836092    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpm4k_1cd8f80e-f5c5-4e80-9abb-b43af15ed1d1/konnectivity-agent/0.log" limit=1024
Sep 10 17:36:35.880850 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:35.880820    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/510bea53-93c2-4158-851a-0b6ec3e6bee6-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-5klhd\" (UID: \"510bea53-93c2-4158-851a-0b6ec3e6bee6\") " pod="openshift-network-console/networking-console-plugin-79977f9844-5klhd"
Sep 10 17:36:35.881027 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:35.880871    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/510bea53-93c2-4158-851a-0b6ec3e6bee6-nginx-conf\") pod \"networking-console-plugin-79977f9844-5klhd\" (UID: \"510bea53-93c2-4158-851a-0b6ec3e6bee6\") " pod="openshift-network-console/networking-console-plugin-79977f9844-5klhd"
Sep 10 17:36:35.881027 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:36:35.880971    2579 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 10 17:36:35.881134 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:36:35.881039    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/510bea53-93c2-4158-851a-0b6ec3e6bee6-networking-console-plugin-cert podName:510bea53-93c2-4158-851a-0b6ec3e6bee6 nodeName:}" failed. No retries permitted until 2026-09-10 17:36:36.381019233 +0000 UTC m=+40.380849295 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/510bea53-93c2-4158-851a-0b6ec3e6bee6-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-5klhd" (UID: "510bea53-93c2-4158-851a-0b6ec3e6bee6") : secret "networking-console-plugin-cert" not found
Sep 10 17:36:35.881642 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:35.881619    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/510bea53-93c2-4158-851a-0b6ec3e6bee6-nginx-conf\") pod \"networking-console-plugin-79977f9844-5klhd\" (UID: \"510bea53-93c2-4158-851a-0b6ec3e6bee6\") " pod="openshift-network-console/networking-console-plugin-79977f9844-5klhd"
Sep 10 17:36:35.946651 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:35.946580    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpm4k_1cd8f80e-f5c5-4e80-9abb-b43af15ed1d1/konnectivity-agent/0.log" limit=1024
Sep 10 17:36:36.386018 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:36.385987    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/510bea53-93c2-4158-851a-0b6ec3e6bee6-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-5klhd\" (UID: \"510bea53-93c2-4158-851a-0b6ec3e6bee6\") " pod="openshift-network-console/networking-console-plugin-79977f9844-5klhd"
Sep 10 17:36:36.386482 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:36:36.386132    2579 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 10 17:36:36.386482 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:36:36.386192    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/510bea53-93c2-4158-851a-0b6ec3e6bee6-networking-console-plugin-cert podName:510bea53-93c2-4158-851a-0b6ec3e6bee6 nodeName:}" failed. No retries permitted until 2026-09-10 17:36:37.386174985 +0000 UTC m=+41.386005030 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/510bea53-93c2-4158-851a-0b6ec3e6bee6-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-5klhd" (UID: "510bea53-93c2-4158-851a-0b6ec3e6bee6") : secret "networking-console-plugin-cert" not found
Sep 10 17:36:37.394780 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:37.394739    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/510bea53-93c2-4158-851a-0b6ec3e6bee6-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-5klhd\" (UID: \"510bea53-93c2-4158-851a-0b6ec3e6bee6\") " pod="openshift-network-console/networking-console-plugin-79977f9844-5klhd"
Sep 10 17:36:37.395192 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:36:37.394925    2579 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 10 17:36:37.395192 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:36:37.394992    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/510bea53-93c2-4158-851a-0b6ec3e6bee6-networking-console-plugin-cert podName:510bea53-93c2-4158-851a-0b6ec3e6bee6 nodeName:}" failed. No retries permitted until 2026-09-10 17:36:39.394976378 +0000 UTC m=+43.394806422 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/510bea53-93c2-4158-851a-0b6ec3e6bee6-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-5klhd" (UID: "510bea53-93c2-4158-851a-0b6ec3e6bee6") : secret "networking-console-plugin-cert" not found
Sep 10 17:36:37.777772 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:37.777689    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpm4k_1cd8f80e-f5c5-4e80-9abb-b43af15ed1d1/konnectivity-agent/0.log" limit=1024
Sep 10 17:36:37.837436 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:37.837400    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpm4k_1cd8f80e-f5c5-4e80-9abb-b43af15ed1d1/konnectivity-agent/0.log" limit=1024
Sep 10 17:36:39.109951 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:39.109917    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ad03af6b-287a-4b2f-9ea3-ec8c43c17cba-registry-tls\") pod \"image-registry-8f44f7f46-d2blr\" (UID: \"ad03af6b-287a-4b2f-9ea3-ec8c43c17cba\") " pod="openshift-image-registry/image-registry-8f44f7f46-d2blr"
Sep 10 17:36:39.109951 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:39.109957    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6976caf5-0ff1-4604-9e30-aab464b54edf-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-5cbrn\" (UID: \"6976caf5-0ff1-4604-9e30-aab464b54edf\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-5cbrn"
Sep 10 17:36:39.110391 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:36:39.110061    2579 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 10 17:36:39.110391 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:36:39.110067    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 10 17:36:39.110391 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:36:39.110088    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-8f44f7f46-d2blr: secret "image-registry-tls" not found
Sep 10 17:36:39.110391 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:36:39.110116    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6976caf5-0ff1-4604-9e30-aab464b54edf-cluster-monitoring-operator-tls podName:6976caf5-0ff1-4604-9e30-aab464b54edf nodeName:}" failed. No retries permitted until 2026-09-10 17:36:47.110099089 +0000 UTC m=+51.109929157 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/6976caf5-0ff1-4604-9e30-aab464b54edf-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-5cbrn" (UID: "6976caf5-0ff1-4604-9e30-aab464b54edf") : secret "cluster-monitoring-operator-tls" not found
Sep 10 17:36:39.110391 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:39.110153    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f1713e2d-701b-435b-8d76-18824228a81e-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-mvd6n\" (UID: \"f1713e2d-701b-435b-8d76-18824228a81e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-mvd6n"
Sep 10 17:36:39.110391 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:36:39.110201    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ad03af6b-287a-4b2f-9ea3-ec8c43c17cba-registry-tls podName:ad03af6b-287a-4b2f-9ea3-ec8c43c17cba nodeName:}" failed. No retries permitted until 2026-09-10 17:36:47.110186509 +0000 UTC m=+51.110016554 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ad03af6b-287a-4b2f-9ea3-ec8c43c17cba-registry-tls") pod "image-registry-8f44f7f46-d2blr" (UID: "ad03af6b-287a-4b2f-9ea3-ec8c43c17cba") : secret "image-registry-tls" not found
Sep 10 17:36:39.110391 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:39.110224    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/678c4830-c56e-4af8-85a2-6f26a4e03aa8-metrics-certs\") pod \"router-default-d7f986b9d-xvnsc\" (UID: \"678c4830-c56e-4af8-85a2-6f26a4e03aa8\") " pod="openshift-ingress/router-default-d7f986b9d-xvnsc"
Sep 10 17:36:39.110391 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:36:39.110239    2579 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 10 17:36:39.110391 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:39.110257    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/678c4830-c56e-4af8-85a2-6f26a4e03aa8-service-ca-bundle\") pod \"router-default-d7f986b9d-xvnsc\" (UID: \"678c4830-c56e-4af8-85a2-6f26a4e03aa8\") " pod="openshift-ingress/router-default-d7f986b9d-xvnsc"
Sep 10 17:36:39.110391 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:36:39.110272    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f1713e2d-701b-435b-8d76-18824228a81e-samples-operator-tls podName:f1713e2d-701b-435b-8d76-18824228a81e nodeName:}" failed. No retries permitted until 2026-09-10 17:36:47.110262249 +0000 UTC m=+51.110092295 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/f1713e2d-701b-435b-8d76-18824228a81e-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-mvd6n" (UID: "f1713e2d-701b-435b-8d76-18824228a81e") : secret "samples-operator-tls" not found
Sep 10 17:36:39.110391 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:36:39.110322    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/678c4830-c56e-4af8-85a2-6f26a4e03aa8-service-ca-bundle podName:678c4830-c56e-4af8-85a2-6f26a4e03aa8 nodeName:}" failed. No retries permitted until 2026-09-10 17:36:47.110313524 +0000 UTC m=+51.110143568 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/678c4830-c56e-4af8-85a2-6f26a4e03aa8-service-ca-bundle") pod "router-default-d7f986b9d-xvnsc" (UID: "678c4830-c56e-4af8-85a2-6f26a4e03aa8") : configmap references non-existent config key: service-ca.crt
Sep 10 17:36:39.110391 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:36:39.110386    2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 10 17:36:39.110840 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:36:39.110461    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/678c4830-c56e-4af8-85a2-6f26a4e03aa8-metrics-certs podName:678c4830-c56e-4af8-85a2-6f26a4e03aa8 nodeName:}" failed. No retries permitted until 2026-09-10 17:36:47.110442167 +0000 UTC m=+51.110272227 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/678c4830-c56e-4af8-85a2-6f26a4e03aa8-metrics-certs") pod "router-default-d7f986b9d-xvnsc" (UID: "678c4830-c56e-4af8-85a2-6f26a4e03aa8") : secret "router-metrics-certs-default" not found
Sep 10 17:36:39.211128 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:39.211103    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/06ecbee3-6c6f-467b-831a-d9545f4588bd-metrics-tls\") pod \"dns-default-dxqxz\" (UID: \"06ecbee3-6c6f-467b-831a-d9545f4588bd\") " pod="openshift-dns/dns-default-dxqxz"
Sep 10 17:36:39.211257 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:36:39.211229    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 17:36:39.211301 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:36:39.211269    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06ecbee3-6c6f-467b-831a-d9545f4588bd-metrics-tls podName:06ecbee3-6c6f-467b-831a-d9545f4588bd nodeName:}" failed. No retries permitted until 2026-09-10 17:36:47.211258043 +0000 UTC m=+51.211088089 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/06ecbee3-6c6f-467b-831a-d9545f4588bd-metrics-tls") pod "dns-default-dxqxz" (UID: "06ecbee3-6c6f-467b-831a-d9545f4588bd") : secret "dns-default-metrics-tls" not found
Sep 10 17:36:39.412974 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:39.412936    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/510bea53-93c2-4158-851a-0b6ec3e6bee6-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-5klhd\" (UID: \"510bea53-93c2-4158-851a-0b6ec3e6bee6\") " pod="openshift-network-console/networking-console-plugin-79977f9844-5klhd"
Sep 10 17:36:39.413156 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:36:39.413120    2579 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 10 17:36:39.413202 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:36:39.413192    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/510bea53-93c2-4158-851a-0b6ec3e6bee6-networking-console-plugin-cert podName:510bea53-93c2-4158-851a-0b6ec3e6bee6 nodeName:}" failed. No retries permitted until 2026-09-10 17:36:43.413176776 +0000 UTC m=+47.413006821 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/510bea53-93c2-4158-851a-0b6ec3e6bee6-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-5klhd" (UID: "510bea53-93c2-4158-851a-0b6ec3e6bee6") : secret "networking-console-plugin-cert" not found
Sep 10 17:36:41.875950 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:41.875676    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-8dmcm" event={"ID":"7f13ef98-e00e-4fe2-8f80-41eb99b0c7f7","Type":"ContainerStarted","Data":"35582c9330fee03332c95936d2b1bde63c1b10c3f7684672278ae363c69feda6"}
Sep 10 17:36:41.877799 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:41.877774    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-wt6xs" event={"ID":"c8f186e0-4c0d-4422-be62-96685cca0258","Type":"ContainerStarted","Data":"5ebcb6d9258d5021f102c0f2fe373ad9afb5f8b629a9d4bee7fd5b2c3e674461"}
Sep 10 17:36:41.936407 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:41.936352    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-8dmcm" podStartSLOduration=6.382580925 podStartE2EDuration="15.9363298s" totalImagesPullingTime="9.553748875s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:36:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:36:41.920223312 +0000 UTC m=+45.920053381" watchObservedRunningTime="2026-09-10 17:36:41.9363298 +0000 UTC m=+45.936159868"
Sep 10 17:36:42.022395 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:42.022323    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpm4k_1cd8f80e-f5c5-4e80-9abb-b43af15ed1d1/konnectivity-agent/0.log" limit=1024
Sep 10 17:36:42.792307 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:42.792254    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-wt6xs" podStartSLOduration=7.217083532 podStartE2EDuration="16.792239001s" totalImagesPullingTime="9.575155469s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:36:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:36:41.936136912 +0000 UTC m=+45.935966980" watchObservedRunningTime="2026-09-10 17:36:42.792239001 +0000 UTC m=+46.792069065"
Sep 10 17:36:42.792831 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:42.792813    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-fdbb8fdff-kzxj2"]
Sep 10 17:36:42.812511 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:42.812472    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-fdbb8fdff-kzxj2"]
Sep 10 17:36:42.812867 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:42.812852    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-kzxj2"
Sep 10 17:36:42.816246 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:42.816223    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Sep 10 17:36:42.817096 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:42.817058    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Sep 10 17:36:42.817096 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:42.817085    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-9294k\""
Sep 10 17:36:42.882233 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:42.882203    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-xctgk" event={"ID":"bfa6c6a2-4e67-4c3c-b445-e07834780f92","Type":"ContainerStarted","Data":"7856163f70ed3da4a43f034de965644149bb74882be76d79f42c92fdeb852769"}
Sep 10 17:36:42.884155 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:42.884131    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-rb854_97172b14-b84a-476e-89f1-05d3e70df416/console-operator/0.log"
Sep 10 17:36:42.884268 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:42.884164    2579 generic.go:356] "Generic (PLEG): container finished" podID="97172b14-b84a-476e-89f1-05d3e70df416" containerID="cbca3777a676905de8013877d9d7fcda83c532d28e8579107703490e22902307" exitCode=255
Sep 10 17:36:42.884268 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:42.884192    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-rb854" event={"ID":"97172b14-b84a-476e-89f1-05d3e70df416","Type":"ContainerDied","Data":"cbca3777a676905de8013877d9d7fcda83c532d28e8579107703490e22902307"}
Sep 10 17:36:42.884451 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:42.884435    2579 scope.go:117] "RemoveContainer" containerID="cbca3777a676905de8013877d9d7fcda83c532d28e8579107703490e22902307"
Sep 10 17:36:42.887473 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:42.887452    2579 generic.go:356] "Generic (PLEG): container finished" podID="48d6ee6d-6025-4ab7-a095-6a31180ac9f7" containerID="f2bd7f6ce232352e597e92678eb09e1b2be52da4be2e430574162e213c4fb7b5" exitCode=0
Sep 10 17:36:42.887586 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:42.887495    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-drhxj" event={"ID":"48d6ee6d-6025-4ab7-a095-6a31180ac9f7","Type":"ContainerDied","Data":"f2bd7f6ce232352e597e92678eb09e1b2be52da4be2e430574162e213c4fb7b5"}
Sep 10 17:36:42.889231 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:42.889211    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-cmwpc" event={"ID":"5edf557d-b123-458c-aaf8-e9760b5b6cfe","Type":"ContainerStarted","Data":"6317aad02cc1fc51f3e98d11813b36a4e860918f7f7b1cd6324a750ccec4514e"}
Sep 10 17:36:42.890927 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:42.890905    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-685456655f-7798s" event={"ID":"1694e06a-5285-4434-b725-3384cc295951","Type":"ContainerStarted","Data":"eb060b7dc89bead2edcceeb79316f65cb40fcaa1bc8194a123f7e9940ec8f9b7"}
Sep 10 17:36:42.892670 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:42.892593    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-rh6vv" event={"ID":"fd21a8ca-3f90-4aa5-b1de-78ddebecf928","Type":"ContainerStarted","Data":"8b65ca7cc317e38b9421f49b647c2e9cc4fb2d7225001dce85fe9997b9beabf0"}
Sep 10 17:36:42.899328 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:42.899285    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-xctgk" podStartSLOduration=2.383186552 podStartE2EDuration="11.899268229s" totalImagesPullingTime="9.516081677s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:36:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:36:42.897801593 +0000 UTC m=+46.897631661" watchObservedRunningTime="2026-09-10 17:36:42.899268229 +0000 UTC m=+46.899098302"
Sep 10 17:36:42.947955 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:42.945888    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w68sx\" (UniqueName: \"kubernetes.io/projected/7ff85e45-8b8f-432b-b51a-f1ebf3019da3-kube-api-access-w68sx\") pod \"migrator-fdbb8fdff-kzxj2\" (UID: \"7ff85e45-8b8f-432b-b51a-f1ebf3019da3\") " pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-kzxj2"
Sep 10 17:36:42.947955 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:42.946864    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-76bcd6ccf5-rh6vv" podStartSLOduration=7.449519266 podStartE2EDuration="16.946823599s" totalImagesPullingTime="9.497304333s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:36:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:36:42.932389786 +0000 UTC m=+46.932219854" watchObservedRunningTime="2026-09-10 17:36:42.946823599 +0000 UTC m=+46.946653670"
Sep 10 17:36:42.948950 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:42.948886    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-cmwpc" podStartSLOduration=7.3791416 podStartE2EDuration="16.948872843s" totalImagesPullingTime="9.569731243s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:36:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:36:42.913186881 +0000 UTC m=+46.913016950" watchObservedRunningTime="2026-09-10 17:36:42.948872843 +0000 UTC m=+46.948702907"
Sep 10 17:36:42.988975 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:42.988800    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-685456655f-7798s" podStartSLOduration=3.165045273 podStartE2EDuration="12.988786592s" totalImagesPullingTime="9.823741319s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:36:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:36:42.963281105 +0000 UTC m=+46.963111172" watchObservedRunningTime="2026-09-10 17:36:42.988786592 +0000 UTC m=+46.988616674"
Sep 10 17:36:43.048387 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:43.048312    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w68sx\" (UniqueName: \"kubernetes.io/projected/7ff85e45-8b8f-432b-b51a-f1ebf3019da3-kube-api-access-w68sx\") pod \"migrator-fdbb8fdff-kzxj2\" (UID: \"7ff85e45-8b8f-432b-b51a-f1ebf3019da3\") " pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-kzxj2"
Sep 10 17:36:43.065184 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:43.065157    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w68sx\" (UniqueName: \"kubernetes.io/projected/7ff85e45-8b8f-432b-b51a-f1ebf3019da3-kube-api-access-w68sx\") pod \"migrator-fdbb8fdff-kzxj2\" (UID: \"7ff85e45-8b8f-432b-b51a-f1ebf3019da3\") " pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-kzxj2"
Sep 10 17:36:43.125503 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:43.125474    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-kzxj2"
Sep 10 17:36:43.408148 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:43.408125    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-fdbb8fdff-kzxj2"]
Sep 10 17:36:43.410060 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:36:43.410030    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ff85e45_8b8f_432b_b51a_f1ebf3019da3.slice/crio-7c7cc9974be7dd36c1345d50498249e52f714e07e62d95e1e1213cb3e273e63d WatchSource:0}: Error finding container 7c7cc9974be7dd36c1345d50498249e52f714e07e62d95e1e1213cb3e273e63d: Status 404 returned error can't find the container with id 7c7cc9974be7dd36c1345d50498249e52f714e07e62d95e1e1213cb3e273e63d
Sep 10 17:36:43.453548 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:43.453520    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/510bea53-93c2-4158-851a-0b6ec3e6bee6-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-5klhd\" (UID: \"510bea53-93c2-4158-851a-0b6ec3e6bee6\") " pod="openshift-network-console/networking-console-plugin-79977f9844-5klhd"
Sep 10 17:36:43.453684 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:36:43.453666    2579 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 10 17:36:43.453737 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:36:43.453726    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/510bea53-93c2-4158-851a-0b6ec3e6bee6-networking-console-plugin-cert podName:510bea53-93c2-4158-851a-0b6ec3e6bee6 nodeName:}" failed. No retries permitted until 2026-09-10 17:36:51.453711659 +0000 UTC m=+55.453541704 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/510bea53-93c2-4158-851a-0b6ec3e6bee6-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-5klhd" (UID: "510bea53-93c2-4158-851a-0b6ec3e6bee6") : secret "networking-console-plugin-cert" not found
Sep 10 17:36:43.775685 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:43.775612    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-sxbzl"]
Sep 10 17:36:43.795629 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:43.795603    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-sxbzl"]
Sep 10 17:36:43.795769 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:43.795661    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-sxbzl"
Sep 10 17:36:43.798100 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:43.798073    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 10 17:36:43.798253 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:43.798078    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 10 17:36:43.798253 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:43.798162    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-gx7jr\""
Sep 10 17:36:43.899397 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:43.899358    2579 generic.go:356] "Generic (PLEG): container finished" podID="48d6ee6d-6025-4ab7-a095-6a31180ac9f7" containerID="72b66ab2028be9868edc2b0927889aab9c4637ba52a07ece0f99c433794b4abc" exitCode=0
Sep 10 17:36:43.899866 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:43.899420    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-drhxj" event={"ID":"48d6ee6d-6025-4ab7-a095-6a31180ac9f7","Type":"ContainerDied","Data":"72b66ab2028be9868edc2b0927889aab9c4637ba52a07ece0f99c433794b4abc"}
Sep 10 17:36:43.900765 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:43.900733    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-kzxj2" event={"ID":"7ff85e45-8b8f-432b-b51a-f1ebf3019da3","Type":"ContainerStarted","Data":"7c7cc9974be7dd36c1345d50498249e52f714e07e62d95e1e1213cb3e273e63d"}
Sep 10 17:36:43.902091 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:43.902076    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-rb854_97172b14-b84a-476e-89f1-05d3e70df416/console-operator/1.log"
Sep 10 17:36:43.902412 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:43.902398    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-rb854_97172b14-b84a-476e-89f1-05d3e70df416/console-operator/0.log"
Sep 10 17:36:43.902481 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:43.902427    2579 generic.go:356] "Generic (PLEG): container finished" podID="97172b14-b84a-476e-89f1-05d3e70df416" containerID="7307b9101dc85645021d36e85099523b0f100cf1bd856a5b1159ca7d207bd745" exitCode=255
Sep 10 17:36:43.902481 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:43.902454    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-rb854" event={"ID":"97172b14-b84a-476e-89f1-05d3e70df416","Type":"ContainerDied","Data":"7307b9101dc85645021d36e85099523b0f100cf1bd856a5b1159ca7d207bd745"}
Sep 10 17:36:43.902610 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:43.902510    2579 scope.go:117] "RemoveContainer" containerID="cbca3777a676905de8013877d9d7fcda83c532d28e8579107703490e22902307"
Sep 10 17:36:43.902748 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:43.902730    2579 scope.go:117] "RemoveContainer" containerID="7307b9101dc85645021d36e85099523b0f100cf1bd856a5b1159ca7d207bd745"
Sep 10 17:36:43.903008 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:36:43.902984    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-85b454874b-rb854_openshift-console-operator(97172b14-b84a-476e-89f1-05d3e70df416)\"" pod="openshift-console-operator/console-operator-85b454874b-rb854" podUID="97172b14-b84a-476e-89f1-05d3e70df416"
Sep 10 17:36:43.958925 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:43.958888    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/98c74a0b-1bd4-46f2-b78c-97a6469856ba-crio-socket\") pod \"insights-runtime-extractor-sxbzl\" (UID: \"98c74a0b-1bd4-46f2-b78c-97a6469856ba\") " pod="openshift-insights/insights-runtime-extractor-sxbzl"
Sep 10 17:36:43.959083 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:43.958998    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/98c74a0b-1bd4-46f2-b78c-97a6469856ba-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sxbzl\" (UID: \"98c74a0b-1bd4-46f2-b78c-97a6469856ba\") " pod="openshift-insights/insights-runtime-extractor-sxbzl"
Sep 10 17:36:43.959083 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:43.959038    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/98c74a0b-1bd4-46f2-b78c-97a6469856ba-data-volume\") pod \"insights-runtime-extractor-sxbzl\" (UID: \"98c74a0b-1bd4-46f2-b78c-97a6469856ba\") " pod="openshift-insights/insights-runtime-extractor-sxbzl"
Sep 10 17:36:43.959190 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:43.959159    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/98c74a0b-1bd4-46f2-b78c-97a6469856ba-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sxbzl\" (UID: \"98c74a0b-1bd4-46f2-b78c-97a6469856ba\") " pod="openshift-insights/insights-runtime-extractor-sxbzl"
Sep 10 17:36:43.959244 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:43.959214    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zg5zw\" (UniqueName: \"kubernetes.io/projected/98c74a0b-1bd4-46f2-b78c-97a6469856ba-kube-api-access-zg5zw\") pod \"insights-runtime-extractor-sxbzl\" (UID: \"98c74a0b-1bd4-46f2-b78c-97a6469856ba\") " pod="openshift-insights/insights-runtime-extractor-sxbzl"
Sep 10 17:36:44.060519 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:44.060446    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/98c74a0b-1bd4-46f2-b78c-97a6469856ba-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sxbzl\" (UID: \"98c74a0b-1bd4-46f2-b78c-97a6469856ba\") " pod="openshift-insights/insights-runtime-extractor-sxbzl"
Sep 10 17:36:44.060733 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:36:44.060554    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 10 17:36:44.060733 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:36:44.060629    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/98c74a0b-1bd4-46f2-b78c-97a6469856ba-insights-runtime-extractor-tls podName:98c74a0b-1bd4-46f2-b78c-97a6469856ba nodeName:}" failed. No retries permitted until 2026-09-10 17:36:44.560607771 +0000 UTC m=+48.560437817 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/98c74a0b-1bd4-46f2-b78c-97a6469856ba-insights-runtime-extractor-tls") pod "insights-runtime-extractor-sxbzl" (UID: "98c74a0b-1bd4-46f2-b78c-97a6469856ba") : secret "insights-runtime-extractor-tls" not found
Sep 10 17:36:44.060864 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:44.060774    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/98c74a0b-1bd4-46f2-b78c-97a6469856ba-data-volume\") pod \"insights-runtime-extractor-sxbzl\" (UID: \"98c74a0b-1bd4-46f2-b78c-97a6469856ba\") " pod="openshift-insights/insights-runtime-extractor-sxbzl"
Sep 10 17:36:44.060965 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:44.060940    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/98c74a0b-1bd4-46f2-b78c-97a6469856ba-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sxbzl\" (UID: \"98c74a0b-1bd4-46f2-b78c-97a6469856ba\") " pod="openshift-insights/insights-runtime-extractor-sxbzl"
Sep 10 17:36:44.061050 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:44.060977    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zg5zw\" (UniqueName: \"kubernetes.io/projected/98c74a0b-1bd4-46f2-b78c-97a6469856ba-kube-api-access-zg5zw\") pod \"insights-runtime-extractor-sxbzl\" (UID: \"98c74a0b-1bd4-46f2-b78c-97a6469856ba\") " pod="openshift-insights/insights-runtime-extractor-sxbzl"
Sep 10 17:36:44.061113 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:36:44.061056    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 10 17:36:44.061113 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:36:44.061100    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/98c74a0b-1bd4-46f2-b78c-97a6469856ba-kube-rbac-proxy-cm podName:98c74a0b-1bd4-46f2-b78c-97a6469856ba nodeName:}" failed. No retries permitted until 2026-09-10 17:36:44.561085359 +0000 UTC m=+48.560915419 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/98c74a0b-1bd4-46f2-b78c-97a6469856ba-kube-rbac-proxy-cm") pod "insights-runtime-extractor-sxbzl" (UID: "98c74a0b-1bd4-46f2-b78c-97a6469856ba") : configmap "kube-rbac-proxy" not found
Sep 10 17:36:44.061260 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:44.061238    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/98c74a0b-1bd4-46f2-b78c-97a6469856ba-data-volume\") pod \"insights-runtime-extractor-sxbzl\" (UID: \"98c74a0b-1bd4-46f2-b78c-97a6469856ba\") " pod="openshift-insights/insights-runtime-extractor-sxbzl"
Sep 10 17:36:44.061324 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:44.061291    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/98c74a0b-1bd4-46f2-b78c-97a6469856ba-crio-socket\") pod \"insights-runtime-extractor-sxbzl\" (UID: \"98c74a0b-1bd4-46f2-b78c-97a6469856ba\") " pod="openshift-insights/insights-runtime-extractor-sxbzl"
Sep 10 17:36:44.061554 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:44.061532    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/98c74a0b-1bd4-46f2-b78c-97a6469856ba-crio-socket\") pod \"insights-runtime-extractor-sxbzl\" (UID: \"98c74a0b-1bd4-46f2-b78c-97a6469856ba\") " pod="openshift-insights/insights-runtime-extractor-sxbzl"
Sep 10 17:36:44.069244 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:44.069219    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zg5zw\" (UniqueName: \"kubernetes.io/projected/98c74a0b-1bd4-46f2-b78c-97a6469856ba-kube-api-access-zg5zw\") pod \"insights-runtime-extractor-sxbzl\" (UID: \"98c74a0b-1bd4-46f2-b78c-97a6469856ba\") " pod="openshift-insights/insights-runtime-extractor-sxbzl"
Sep 10 17:36:44.566714 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:44.566680    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/98c74a0b-1bd4-46f2-b78c-97a6469856ba-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sxbzl\" (UID: \"98c74a0b-1bd4-46f2-b78c-97a6469856ba\") " pod="openshift-insights/insights-runtime-extractor-sxbzl"
Sep 10 17:36:44.566870 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:44.566770    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/98c74a0b-1bd4-46f2-b78c-97a6469856ba-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sxbzl\" (UID: \"98c74a0b-1bd4-46f2-b78c-97a6469856ba\") " pod="openshift-insights/insights-runtime-extractor-sxbzl"
Sep 10 17:36:44.566870 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:36:44.566846    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 10 17:36:44.566945 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:36:44.566884    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 10 17:36:44.566945 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:36:44.566917    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/98c74a0b-1bd4-46f2-b78c-97a6469856ba-insights-runtime-extractor-tls podName:98c74a0b-1bd4-46f2-b78c-97a6469856ba nodeName:}" failed. No retries permitted until 2026-09-10 17:36:45.566898896 +0000 UTC m=+49.566728942 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/98c74a0b-1bd4-46f2-b78c-97a6469856ba-insights-runtime-extractor-tls") pod "insights-runtime-extractor-sxbzl" (UID: "98c74a0b-1bd4-46f2-b78c-97a6469856ba") : secret "insights-runtime-extractor-tls" not found
Sep 10 17:36:44.567011 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:36:44.566971    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/98c74a0b-1bd4-46f2-b78c-97a6469856ba-kube-rbac-proxy-cm podName:98c74a0b-1bd4-46f2-b78c-97a6469856ba nodeName:}" failed. No retries permitted until 2026-09-10 17:36:45.566959066 +0000 UTC m=+49.566789115 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/98c74a0b-1bd4-46f2-b78c-97a6469856ba-kube-rbac-proxy-cm") pod "insights-runtime-extractor-sxbzl" (UID: "98c74a0b-1bd4-46f2-b78c-97a6469856ba") : configmap "kube-rbac-proxy" not found
Sep 10 17:36:44.911856 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:44.911831    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-rb854_97172b14-b84a-476e-89f1-05d3e70df416/console-operator/1.log"
Sep 10 17:36:44.912255 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:44.912157    2579 scope.go:117] "RemoveContainer" containerID="7307b9101dc85645021d36e85099523b0f100cf1bd856a5b1159ca7d207bd745"
Sep 10 17:36:44.912330 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:36:44.912309    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-85b454874b-rb854_openshift-console-operator(97172b14-b84a-476e-89f1-05d3e70df416)\"" pod="openshift-console-operator/console-operator-85b454874b-rb854" podUID="97172b14-b84a-476e-89f1-05d3e70df416"
Sep 10 17:36:44.915503 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:44.915480    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-drhxj" event={"ID":"48d6ee6d-6025-4ab7-a095-6a31180ac9f7","Type":"ContainerStarted","Data":"aeb63f5a80ccdad8394faae61424c7cb3e653b9702d0466cb6197ded0fe131a6"}
Sep 10 17:36:44.953785 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:44.953737    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-drhxj" podStartSLOduration=8.839286088 podStartE2EDuration="48.953722751s" totalImagesPullingTime="40.114436663s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 17:35:56 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:36:44.953010274 +0000 UTC m=+48.952840515" watchObservedRunningTime="2026-09-10 17:36:44.953722751 +0000 UTC m=+48.953552818"
Sep 10 17:36:45.574944 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:45.574914    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/98c74a0b-1bd4-46f2-b78c-97a6469856ba-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sxbzl\" (UID: \"98c74a0b-1bd4-46f2-b78c-97a6469856ba\") " pod="openshift-insights/insights-runtime-extractor-sxbzl"
Sep 10 17:36:45.575107 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:45.575022    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/98c74a0b-1bd4-46f2-b78c-97a6469856ba-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sxbzl\" (UID: \"98c74a0b-1bd4-46f2-b78c-97a6469856ba\") " pod="openshift-insights/insights-runtime-extractor-sxbzl"
Sep 10 17:36:45.575107 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:36:45.575040    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 10 17:36:45.575107 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:36:45.575099    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/98c74a0b-1bd4-46f2-b78c-97a6469856ba-kube-rbac-proxy-cm podName:98c74a0b-1bd4-46f2-b78c-97a6469856ba nodeName:}" failed. No retries permitted until 2026-09-10 17:36:47.575083554 +0000 UTC m=+51.574913599 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/98c74a0b-1bd4-46f2-b78c-97a6469856ba-kube-rbac-proxy-cm") pod "insights-runtime-extractor-sxbzl" (UID: "98c74a0b-1bd4-46f2-b78c-97a6469856ba") : configmap "kube-rbac-proxy" not found
Sep 10 17:36:45.575234 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:36:45.575139    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 10 17:36:45.575234 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:36:45.575193    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/98c74a0b-1bd4-46f2-b78c-97a6469856ba-insights-runtime-extractor-tls podName:98c74a0b-1bd4-46f2-b78c-97a6469856ba nodeName:}" failed. No retries permitted until 2026-09-10 17:36:47.575181746 +0000 UTC m=+51.575011792 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/98c74a0b-1bd4-46f2-b78c-97a6469856ba-insights-runtime-extractor-tls") pod "insights-runtime-extractor-sxbzl" (UID: "98c74a0b-1bd4-46f2-b78c-97a6469856ba") : secret "insights-runtime-extractor-tls" not found
Sep 10 17:36:45.826029 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:45.825948    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-75d99c484-hjpcw"]
Sep 10 17:36:45.827756 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:45.827741    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-75d99c484-hjpcw"
Sep 10 17:36:45.830260 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:45.830229    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Sep 10 17:36:45.830387 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:45.830268    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Sep 10 17:36:45.830387 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:45.830303    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-85fzc\""
Sep 10 17:36:45.830387 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:45.830309    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Sep 10 17:36:45.831331 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:45.831310    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Sep 10 17:36:45.835182 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:45.835150    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-75d99c484-hjpcw"]
Sep 10 17:36:45.920406 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:45.920374    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-kzxj2" event={"ID":"7ff85e45-8b8f-432b-b51a-f1ebf3019da3","Type":"ContainerStarted","Data":"4de5f4da56f9e633c2c0a831b65567d978a4cd395ad53985145417072be00e3d"}
Sep 10 17:36:45.920757 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:45.920429    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-kzxj2" event={"ID":"7ff85e45-8b8f-432b-b51a-f1ebf3019da3","Type":"ContainerStarted","Data":"d2d079a86cf724a86c838eb8251f1edbd9cd68b2829cd723133aeacc4edd54fa"}
Sep 10 17:36:45.936370 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:45.936331    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-kzxj2" podStartSLOduration=2.405748636 podStartE2EDuration="3.936318203s" totalImagesPullingTime="1.530569567s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:36:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:36:45.935660456 +0000 UTC m=+49.935490523" watchObservedRunningTime="2026-09-10 17:36:45.936318203 +0000 UTC m=+49.936148267"
Sep 10 17:36:45.944404 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:45.944387    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-t4l6s_3dc43868-1f6c-48fa-85fc-59607a50c103/dns-node-resolver/0.log"
Sep 10 17:36:45.978749 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:45.978721    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xc2z\" (UniqueName: \"kubernetes.io/projected/35444e5d-4d6d-4a63-b93f-132bbc9f0c30-kube-api-access-5xc2z\") pod \"service-ca-75d99c484-hjpcw\" (UID: \"35444e5d-4d6d-4a63-b93f-132bbc9f0c30\") " pod="openshift-service-ca/service-ca-75d99c484-hjpcw"
Sep 10 17:36:45.978859 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:45.978751    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/35444e5d-4d6d-4a63-b93f-132bbc9f0c30-signing-cabundle\") pod \"service-ca-75d99c484-hjpcw\" (UID: \"35444e5d-4d6d-4a63-b93f-132bbc9f0c30\") " pod="openshift-service-ca/service-ca-75d99c484-hjpcw"
Sep 10 17:36:45.978859 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:45.978790    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/35444e5d-4d6d-4a63-b93f-132bbc9f0c30-signing-key\") pod \"service-ca-75d99c484-hjpcw\" (UID: \"35444e5d-4d6d-4a63-b93f-132bbc9f0c30\") " pod="openshift-service-ca/service-ca-75d99c484-hjpcw"
Sep 10 17:36:46.079923 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:46.079852    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xc2z\" (UniqueName: \"kubernetes.io/projected/35444e5d-4d6d-4a63-b93f-132bbc9f0c30-kube-api-access-5xc2z\") pod \"service-ca-75d99c484-hjpcw\" (UID: \"35444e5d-4d6d-4a63-b93f-132bbc9f0c30\") " pod="openshift-service-ca/service-ca-75d99c484-hjpcw"
Sep 10 17:36:46.079923 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:46.079883    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/35444e5d-4d6d-4a63-b93f-132bbc9f0c30-signing-cabundle\") pod \"service-ca-75d99c484-hjpcw\" (UID: \"35444e5d-4d6d-4a63-b93f-132bbc9f0c30\") " pod="openshift-service-ca/service-ca-75d99c484-hjpcw"
Sep 10 17:36:46.080080 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:46.079943    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/35444e5d-4d6d-4a63-b93f-132bbc9f0c30-signing-key\") pod \"service-ca-75d99c484-hjpcw\" (UID: \"35444e5d-4d6d-4a63-b93f-132bbc9f0c30\") " pod="openshift-service-ca/service-ca-75d99c484-hjpcw"
Sep 10 17:36:46.080528 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:46.080506    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/35444e5d-4d6d-4a63-b93f-132bbc9f0c30-signing-cabundle\") pod \"service-ca-75d99c484-hjpcw\" (UID: \"35444e5d-4d6d-4a63-b93f-132bbc9f0c30\") " pod="openshift-service-ca/service-ca-75d99c484-hjpcw"
Sep 10 17:36:46.082551 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:46.082528    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/35444e5d-4d6d-4a63-b93f-132bbc9f0c30-signing-key\") pod \"service-ca-75d99c484-hjpcw\" (UID: \"35444e5d-4d6d-4a63-b93f-132bbc9f0c30\") " pod="openshift-service-ca/service-ca-75d99c484-hjpcw"
Sep 10 17:36:46.089035 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:46.089001    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xc2z\" (UniqueName: \"kubernetes.io/projected/35444e5d-4d6d-4a63-b93f-132bbc9f0c30-kube-api-access-5xc2z\") pod \"service-ca-75d99c484-hjpcw\" (UID: \"35444e5d-4d6d-4a63-b93f-132bbc9f0c30\") " pod="openshift-service-ca/service-ca-75d99c484-hjpcw"
Sep 10 17:36:46.137242 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:46.137219    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-75d99c484-hjpcw"
Sep 10 17:36:46.409528 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:46.409500    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-75d99c484-hjpcw"]
Sep 10 17:36:46.412844 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:36:46.412821    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35444e5d_4d6d_4a63_b93f_132bbc9f0c30.slice/crio-62a212a8aaff2c0518394195cedde6014ae8317879b71cda8d45570b5efb9adf WatchSource:0}: Error finding container 62a212a8aaff2c0518394195cedde6014ae8317879b71cda8d45570b5efb9adf: Status 404 returned error can't find the container with id 62a212a8aaff2c0518394195cedde6014ae8317879b71cda8d45570b5efb9adf
Sep 10 17:36:46.924973 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:46.924931    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-75d99c484-hjpcw" event={"ID":"35444e5d-4d6d-4a63-b93f-132bbc9f0c30","Type":"ContainerStarted","Data":"b357389a14c0a278daab7f6e5644e3d54bfaf10aa7a7b37b62157cbe73236c1b"}
Sep 10 17:36:46.925428 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:46.925025    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-75d99c484-hjpcw" event={"ID":"35444e5d-4d6d-4a63-b93f-132bbc9f0c30","Type":"ContainerStarted","Data":"62a212a8aaff2c0518394195cedde6014ae8317879b71cda8d45570b5efb9adf"}
Sep 10 17:36:46.939926 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:46.939886    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-75d99c484-hjpcw" podStartSLOduration=1.939872241 podStartE2EDuration="1.939872241s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:36:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:36:46.939134913 +0000 UTC m=+50.938964982" watchObservedRunningTime="2026-09-10 17:36:46.939872241 +0000 UTC m=+50.939702310"
Sep 10 17:36:46.945280 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:46.945258    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-rkzwg_eca4abd0-e29b-4c77-9a3b-c9c32e5b5101/node-ca/0.log"
Sep 10 17:36:47.190956 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:47.190872    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f1713e2d-701b-435b-8d76-18824228a81e-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-mvd6n\" (UID: \"f1713e2d-701b-435b-8d76-18824228a81e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-mvd6n"
Sep 10 17:36:47.190956 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:47.190940    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/678c4830-c56e-4af8-85a2-6f26a4e03aa8-metrics-certs\") pod \"router-default-d7f986b9d-xvnsc\" (UID: \"678c4830-c56e-4af8-85a2-6f26a4e03aa8\") " pod="openshift-ingress/router-default-d7f986b9d-xvnsc"
Sep 10 17:36:47.191145 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:47.190972    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/678c4830-c56e-4af8-85a2-6f26a4e03aa8-service-ca-bundle\") pod \"router-default-d7f986b9d-xvnsc\" (UID: \"678c4830-c56e-4af8-85a2-6f26a4e03aa8\") " pod="openshift-ingress/router-default-d7f986b9d-xvnsc"
Sep 10 17:36:47.191145 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:36:47.191041    2579 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 10 17:36:47.191145 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:36:47.191086    2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 10 17:36:47.191145 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:36:47.191108    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f1713e2d-701b-435b-8d76-18824228a81e-samples-operator-tls podName:f1713e2d-701b-435b-8d76-18824228a81e nodeName:}" failed. No retries permitted until 2026-09-10 17:37:03.191089061 +0000 UTC m=+67.190919128 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/f1713e2d-701b-435b-8d76-18824228a81e-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-mvd6n" (UID: "f1713e2d-701b-435b-8d76-18824228a81e") : secret "samples-operator-tls" not found
Sep 10 17:36:47.191145 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:36:47.191116    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 10 17:36:47.191145 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:36:47.191128    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-8f44f7f46-d2blr: secret "image-registry-tls" not found
Sep 10 17:36:47.191145 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:47.191042    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ad03af6b-287a-4b2f-9ea3-ec8c43c17cba-registry-tls\") pod \"image-registry-8f44f7f46-d2blr\" (UID: \"ad03af6b-287a-4b2f-9ea3-ec8c43c17cba\") " pod="openshift-image-registry/image-registry-8f44f7f46-d2blr"
Sep 10 17:36:47.191392 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:36:47.191149    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/678c4830-c56e-4af8-85a2-6f26a4e03aa8-metrics-certs podName:678c4830-c56e-4af8-85a2-6f26a4e03aa8 nodeName:}" failed. No retries permitted until 2026-09-10 17:37:03.191132862 +0000 UTC m=+67.190962911 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/678c4830-c56e-4af8-85a2-6f26a4e03aa8-metrics-certs") pod "router-default-d7f986b9d-xvnsc" (UID: "678c4830-c56e-4af8-85a2-6f26a4e03aa8") : secret "router-metrics-certs-default" not found
Sep 10 17:36:47.191392 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:36:47.191170    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ad03af6b-287a-4b2f-9ea3-ec8c43c17cba-registry-tls podName:ad03af6b-287a-4b2f-9ea3-ec8c43c17cba nodeName:}" failed. No retries permitted until 2026-09-10 17:37:03.191160806 +0000 UTC m=+67.190990851 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ad03af6b-287a-4b2f-9ea3-ec8c43c17cba-registry-tls") pod "image-registry-8f44f7f46-d2blr" (UID: "ad03af6b-287a-4b2f-9ea3-ec8c43c17cba") : secret "image-registry-tls" not found
Sep 10 17:36:47.191392 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:36:47.191185    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/678c4830-c56e-4af8-85a2-6f26a4e03aa8-service-ca-bundle podName:678c4830-c56e-4af8-85a2-6f26a4e03aa8 nodeName:}" failed. No retries permitted until 2026-09-10 17:37:03.191177099 +0000 UTC m=+67.191007143 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/678c4830-c56e-4af8-85a2-6f26a4e03aa8-service-ca-bundle") pod "router-default-d7f986b9d-xvnsc" (UID: "678c4830-c56e-4af8-85a2-6f26a4e03aa8") : configmap references non-existent config key: service-ca.crt
Sep 10 17:36:47.191392 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:47.191219    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6976caf5-0ff1-4604-9e30-aab464b54edf-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-5cbrn\" (UID: \"6976caf5-0ff1-4604-9e30-aab464b54edf\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-5cbrn"
Sep 10 17:36:47.191392 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:36:47.191311    2579 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 10 17:36:47.191392 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:36:47.191352    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6976caf5-0ff1-4604-9e30-aab464b54edf-cluster-monitoring-operator-tls podName:6976caf5-0ff1-4604-9e30-aab464b54edf nodeName:}" failed. No retries permitted until 2026-09-10 17:37:03.191341141 +0000 UTC m=+67.191171190 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/6976caf5-0ff1-4604-9e30-aab464b54edf-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-5cbrn" (UID: "6976caf5-0ff1-4604-9e30-aab464b54edf") : secret "cluster-monitoring-operator-tls" not found
Sep 10 17:36:47.292381 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:47.292331    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/06ecbee3-6c6f-467b-831a-d9545f4588bd-metrics-tls\") pod \"dns-default-dxqxz\" (UID: \"06ecbee3-6c6f-467b-831a-d9545f4588bd\") " pod="openshift-dns/dns-default-dxqxz"
Sep 10 17:36:47.292582 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:36:47.292505    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 17:36:47.292654 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:36:47.292603    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06ecbee3-6c6f-467b-831a-d9545f4588bd-metrics-tls podName:06ecbee3-6c6f-467b-831a-d9545f4588bd nodeName:}" failed. No retries permitted until 2026-09-10 17:37:03.292579304 +0000 UTC m=+67.292409369 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/06ecbee3-6c6f-467b-831a-d9545f4588bd-metrics-tls") pod "dns-default-dxqxz" (UID: "06ecbee3-6c6f-467b-831a-d9545f4588bd") : secret "dns-default-metrics-tls" not found
Sep 10 17:36:47.345077 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:47.345045    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-xctgk_bfa6c6a2-4e67-4c3c-b445-e07834780f92/serve-healthcheck-canary/0.log"
Sep 10 17:36:47.544327 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:47.544260    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-fdbb8fdff-kzxj2_7ff85e45-8b8f-432b-b51a-f1ebf3019da3/migrator/0.log"
Sep 10 17:36:47.595997 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:47.595962    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/98c74a0b-1bd4-46f2-b78c-97a6469856ba-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sxbzl\" (UID: \"98c74a0b-1bd4-46f2-b78c-97a6469856ba\") " pod="openshift-insights/insights-runtime-extractor-sxbzl"
Sep 10 17:36:47.596159 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:47.596055    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/98c74a0b-1bd4-46f2-b78c-97a6469856ba-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sxbzl\" (UID: \"98c74a0b-1bd4-46f2-b78c-97a6469856ba\") " pod="openshift-insights/insights-runtime-extractor-sxbzl"
Sep 10 17:36:47.596159 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:36:47.596114    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 10 17:36:47.596285 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:36:47.596204    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/98c74a0b-1bd4-46f2-b78c-97a6469856ba-insights-runtime-extractor-tls podName:98c74a0b-1bd4-46f2-b78c-97a6469856ba nodeName:}" failed. No retries permitted until 2026-09-10 17:36:51.596187822 +0000 UTC m=+55.596017867 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/98c74a0b-1bd4-46f2-b78c-97a6469856ba-insights-runtime-extractor-tls") pod "insights-runtime-extractor-sxbzl" (UID: "98c74a0b-1bd4-46f2-b78c-97a6469856ba") : secret "insights-runtime-extractor-tls" not found
Sep 10 17:36:47.596722 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:47.596698    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/98c74a0b-1bd4-46f2-b78c-97a6469856ba-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sxbzl\" (UID: \"98c74a0b-1bd4-46f2-b78c-97a6469856ba\") " pod="openshift-insights/insights-runtime-extractor-sxbzl"
Sep 10 17:36:47.744122 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:47.744090    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-fdbb8fdff-kzxj2_7ff85e45-8b8f-432b-b51a-f1ebf3019da3/graceful-termination/0.log"
Sep 10 17:36:47.945263 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:47.945228    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-8d7dd4675-wt6xs_c8f186e0-4c0d-4422-be62-96685cca0258/kube-storage-version-migrator-operator/0.log"
Sep 10 17:36:51.527164 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:51.527125    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/510bea53-93c2-4158-851a-0b6ec3e6bee6-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-5klhd\" (UID: \"510bea53-93c2-4158-851a-0b6ec3e6bee6\") " pod="openshift-network-console/networking-console-plugin-79977f9844-5klhd"
Sep 10 17:36:51.527635 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:36:51.527288    2579 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 10 17:36:51.527635 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:36:51.527361    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/510bea53-93c2-4158-851a-0b6ec3e6bee6-networking-console-plugin-cert podName:510bea53-93c2-4158-851a-0b6ec3e6bee6 nodeName:}" failed. No retries permitted until 2026-09-10 17:37:07.527342237 +0000 UTC m=+71.527172288 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/510bea53-93c2-4158-851a-0b6ec3e6bee6-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-5klhd" (UID: "510bea53-93c2-4158-851a-0b6ec3e6bee6") : secret "networking-console-plugin-cert" not found
Sep 10 17:36:51.626498 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:51.626464    2579 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-85b454874b-rb854"
Sep 10 17:36:51.626647 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:51.626525    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-85b454874b-rb854"
Sep 10 17:36:51.626940 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:51.626927    2579 scope.go:117] "RemoveContainer" containerID="7307b9101dc85645021d36e85099523b0f100cf1bd856a5b1159ca7d207bd745"
Sep 10 17:36:51.627137 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:36:51.627118    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-85b454874b-rb854_openshift-console-operator(97172b14-b84a-476e-89f1-05d3e70df416)\"" pod="openshift-console-operator/console-operator-85b454874b-rb854" podUID="97172b14-b84a-476e-89f1-05d3e70df416"
Sep 10 17:36:51.627711 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:51.627668    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/98c74a0b-1bd4-46f2-b78c-97a6469856ba-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sxbzl\" (UID: \"98c74a0b-1bd4-46f2-b78c-97a6469856ba\") " pod="openshift-insights/insights-runtime-extractor-sxbzl"
Sep 10 17:36:51.627821 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:36:51.627806    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 10 17:36:51.627871 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:36:51.627860    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/98c74a0b-1bd4-46f2-b78c-97a6469856ba-insights-runtime-extractor-tls podName:98c74a0b-1bd4-46f2-b78c-97a6469856ba nodeName:}" failed. No retries permitted until 2026-09-10 17:36:59.627844181 +0000 UTC m=+63.627674225 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/98c74a0b-1bd4-46f2-b78c-97a6469856ba-insights-runtime-extractor-tls") pod "insights-runtime-extractor-sxbzl" (UID: "98c74a0b-1bd4-46f2-b78c-97a6469856ba") : secret "insights-runtime-extractor-tls" not found
Sep 10 17:36:57.148472 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:57.148436    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-4fsjz"
Sep 10 17:36:59.697709 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:59.697676    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/98c74a0b-1bd4-46f2-b78c-97a6469856ba-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sxbzl\" (UID: \"98c74a0b-1bd4-46f2-b78c-97a6469856ba\") " pod="openshift-insights/insights-runtime-extractor-sxbzl"
Sep 10 17:36:59.700193 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:59.700166    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/98c74a0b-1bd4-46f2-b78c-97a6469856ba-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sxbzl\" (UID: \"98c74a0b-1bd4-46f2-b78c-97a6469856ba\") " pod="openshift-insights/insights-runtime-extractor-sxbzl"
Sep 10 17:36:59.708408 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:59.708389    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-gx7jr\""
Sep 10 17:36:59.716693 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:59.716677    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-sxbzl"
Sep 10 17:36:59.986412 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:36:59.982927    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-sxbzl"]
Sep 10 17:36:59.987685 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:36:59.987659    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98c74a0b_1bd4_46f2_b78c_97a6469856ba.slice/crio-222063b205da171217bed89b058f60f034d82f0db8fdf5cdef41b48541809a8a WatchSource:0}: Error finding container 222063b205da171217bed89b058f60f034d82f0db8fdf5cdef41b48541809a8a: Status 404 returned error can't find the container with id 222063b205da171217bed89b058f60f034d82f0db8fdf5cdef41b48541809a8a
Sep 10 17:37:00.981884 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:00.981849    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-sxbzl" event={"ID":"98c74a0b-1bd4-46f2-b78c-97a6469856ba","Type":"ContainerStarted","Data":"2c6e5812ee001125ff6d73de3fd33c4dcf61d6cec56ebecffbb1384fbb2115ac"}
Sep 10 17:37:00.982196 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:00.981907    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-sxbzl" event={"ID":"98c74a0b-1bd4-46f2-b78c-97a6469856ba","Type":"ContainerStarted","Data":"1a7958bc8ecc6e3701666d196c2e3b6026ab8c62045aad141e0ab5cf0f34f229"}
Sep 10 17:37:00.982196 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:00.981927    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-sxbzl" event={"ID":"98c74a0b-1bd4-46f2-b78c-97a6469856ba","Type":"ContainerStarted","Data":"222063b205da171217bed89b058f60f034d82f0db8fdf5cdef41b48541809a8a"}
Sep 10 17:37:01.310230 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:01.310200    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3fcacac9-c23e-4ba5-8b1d-dfeede6b57e3-metrics-certs\") pod \"network-metrics-daemon-ncmn7\" (UID: \"3fcacac9-c23e-4ba5-8b1d-dfeede6b57e3\") " pod="openshift-multus/network-metrics-daemon-ncmn7"
Sep 10 17:37:01.310388 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:01.310256    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e0fe9300-104f-489d-9962-908000359645-original-pull-secret\") pod \"global-pull-secret-syncer-m4npt\" (UID: \"e0fe9300-104f-489d-9962-908000359645\") " pod="kube-system/global-pull-secret-syncer-m4npt"
Sep 10 17:37:01.312774 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:01.312748    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 10 17:37:01.312893 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:01.312787    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 10 17:37:01.323337 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:01.323321    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e0fe9300-104f-489d-9962-908000359645-original-pull-secret\") pod \"global-pull-secret-syncer-m4npt\" (UID: \"e0fe9300-104f-489d-9962-908000359645\") " pod="kube-system/global-pull-secret-syncer-m4npt"
Sep 10 17:37:01.323408 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:01.323341    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3fcacac9-c23e-4ba5-8b1d-dfeede6b57e3-metrics-certs\") pod \"network-metrics-daemon-ncmn7\" (UID: \"3fcacac9-c23e-4ba5-8b1d-dfeede6b57e3\") " pod="openshift-multus/network-metrics-daemon-ncmn7"
Sep 10 17:37:01.411626 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:01.411595    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9zqd\" (UniqueName: \"kubernetes.io/projected/8436b9d2-f47a-411e-a695-cb52b8277b62-kube-api-access-p9zqd\") pod \"network-check-target-8k8z5\" (UID: \"8436b9d2-f47a-411e-a695-cb52b8277b62\") " pod="openshift-network-diagnostics/network-check-target-8k8z5"
Sep 10 17:37:01.414129 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:01.414108    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9zqd\" (UniqueName: \"kubernetes.io/projected/8436b9d2-f47a-411e-a695-cb52b8277b62-kube-api-access-p9zqd\") pod \"network-check-target-8k8z5\" (UID: \"8436b9d2-f47a-411e-a695-cb52b8277b62\") " pod="openshift-network-diagnostics/network-check-target-8k8z5"
Sep 10 17:37:01.479274 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:01.479245    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-7lms2\""
Sep 10 17:37:01.487387 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:01.487365    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ncmn7"
Sep 10 17:37:01.583963 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:01.583896    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m4npt"
Sep 10 17:37:01.590200 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:01.590181    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-sh4f6\""
Sep 10 17:37:01.598168 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:01.598148    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8k8z5"
Sep 10 17:37:01.760228 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:01.760052    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-ncmn7"]
Sep 10 17:37:01.762636 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:37:01.762606    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fcacac9_c23e_4ba5_8b1d_dfeede6b57e3.slice/crio-86e1f26206d2b199c0381d8d15364a1499765dc391216fa31e02a699ec1b1e1c WatchSource:0}: Error finding container 86e1f26206d2b199c0381d8d15364a1499765dc391216fa31e02a699ec1b1e1c: Status 404 returned error can't find the container with id 86e1f26206d2b199c0381d8d15364a1499765dc391216fa31e02a699ec1b1e1c
Sep 10 17:37:01.852239 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:01.852156    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-m4npt"]
Sep 10 17:37:01.855458 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:37:01.855434    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0fe9300_104f_489d_9962_908000359645.slice/crio-bfeb63f4ee7d9bb476237f4fa039d71c567b25c3ae90a27d1ea67bdf43edec7f WatchSource:0}: Error finding container bfeb63f4ee7d9bb476237f4fa039d71c567b25c3ae90a27d1ea67bdf43edec7f: Status 404 returned error can't find the container with id bfeb63f4ee7d9bb476237f4fa039d71c567b25c3ae90a27d1ea67bdf43edec7f
Sep 10 17:37:01.879082 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:01.879053    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-8k8z5"]
Sep 10 17:37:01.882098 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:37:01.882075    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8436b9d2_f47a_411e_a695_cb52b8277b62.slice/crio-5fcb422cc2e166efad2ee74eb7be22b091d3ec95d07c9bf33835fc8d839a0ac9 WatchSource:0}: Error finding container 5fcb422cc2e166efad2ee74eb7be22b091d3ec95d07c9bf33835fc8d839a0ac9: Status 404 returned error can't find the container with id 5fcb422cc2e166efad2ee74eb7be22b091d3ec95d07c9bf33835fc8d839a0ac9
Sep 10 17:37:01.986806 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:01.986774    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-m4npt" event={"ID":"e0fe9300-104f-489d-9962-908000359645","Type":"ContainerStarted","Data":"bfeb63f4ee7d9bb476237f4fa039d71c567b25c3ae90a27d1ea67bdf43edec7f"}
Sep 10 17:37:01.988690 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:01.988662    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-8k8z5" event={"ID":"8436b9d2-f47a-411e-a695-cb52b8277b62","Type":"ContainerStarted","Data":"53802846a138bd5a207d9d1dc0d7d45bc9019e4da0e9d1255fde58aa6f82cfde"}
Sep 10 17:37:01.988821 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:01.988715    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-8k8z5" event={"ID":"8436b9d2-f47a-411e-a695-cb52b8277b62","Type":"ContainerStarted","Data":"5fcb422cc2e166efad2ee74eb7be22b091d3ec95d07c9bf33835fc8d839a0ac9"}
Sep 10 17:37:01.988821 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:01.988768    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-8k8z5"
Sep 10 17:37:01.990077 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:01.990056    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-ncmn7" event={"ID":"3fcacac9-c23e-4ba5-8b1d-dfeede6b57e3","Type":"ContainerStarted","Data":"86e1f26206d2b199c0381d8d15364a1499765dc391216fa31e02a699ec1b1e1c"}
Sep 10 17:37:02.003699 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:02.003655    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-8k8z5" podStartSLOduration=66.003641942 podStartE2EDuration="1m6.003641942s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:35:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:37:02.003183712 +0000 UTC m=+66.003013778" watchObservedRunningTime="2026-09-10 17:37:02.003641942 +0000 UTC m=+66.003472008"
Sep 10 17:37:02.996802 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:02.996765    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-sxbzl" event={"ID":"98c74a0b-1bd4-46f2-b78c-97a6469856ba","Type":"ContainerStarted","Data":"3d16f5bd31865b5d7fba88797dbcb5c81cf3a87c12512104238ca96f317d8042"}
Sep 10 17:37:03.016409 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:03.016356    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-sxbzl" podStartSLOduration=17.678908881 podStartE2EDuration="20.016337836s" totalImagesPullingTime="2.337428955s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 17:36:43 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:37:03.016113572 +0000 UTC m=+67.015943641" watchObservedRunningTime="2026-09-10 17:37:03.016337836 +0000 UTC m=+67.016167903"
Sep 10 17:37:03.230271 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:03.229703    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f1713e2d-701b-435b-8d76-18824228a81e-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-mvd6n\" (UID: \"f1713e2d-701b-435b-8d76-18824228a81e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-mvd6n"
Sep 10 17:37:03.230271 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:03.229771    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/678c4830-c56e-4af8-85a2-6f26a4e03aa8-metrics-certs\") pod \"router-default-d7f986b9d-xvnsc\" (UID: \"678c4830-c56e-4af8-85a2-6f26a4e03aa8\") " pod="openshift-ingress/router-default-d7f986b9d-xvnsc"
Sep 10 17:37:03.230271 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:03.229793    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/678c4830-c56e-4af8-85a2-6f26a4e03aa8-service-ca-bundle\") pod \"router-default-d7f986b9d-xvnsc\" (UID: \"678c4830-c56e-4af8-85a2-6f26a4e03aa8\") " pod="openshift-ingress/router-default-d7f986b9d-xvnsc"
Sep 10 17:37:03.230271 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:03.229859    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ad03af6b-287a-4b2f-9ea3-ec8c43c17cba-registry-tls\") pod \"image-registry-8f44f7f46-d2blr\" (UID: \"ad03af6b-287a-4b2f-9ea3-ec8c43c17cba\") " pod="openshift-image-registry/image-registry-8f44f7f46-d2blr"
Sep 10 17:37:03.230271 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:03.229915    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6976caf5-0ff1-4604-9e30-aab464b54edf-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-5cbrn\" (UID: \"6976caf5-0ff1-4604-9e30-aab464b54edf\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-5cbrn"
Sep 10 17:37:03.230819 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:03.230789    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/678c4830-c56e-4af8-85a2-6f26a4e03aa8-service-ca-bundle\") pod \"router-default-d7f986b9d-xvnsc\" (UID: \"678c4830-c56e-4af8-85a2-6f26a4e03aa8\") " pod="openshift-ingress/router-default-d7f986b9d-xvnsc"
Sep 10 17:37:03.233308 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:03.233257    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6976caf5-0ff1-4604-9e30-aab464b54edf-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-5cbrn\" (UID: \"6976caf5-0ff1-4604-9e30-aab464b54edf\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-5cbrn"
Sep 10 17:37:03.233463 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:03.233445    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ad03af6b-287a-4b2f-9ea3-ec8c43c17cba-registry-tls\") pod \"image-registry-8f44f7f46-d2blr\" (UID: \"ad03af6b-287a-4b2f-9ea3-ec8c43c17cba\") " pod="openshift-image-registry/image-registry-8f44f7f46-d2blr"
Sep 10 17:37:03.233901 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:03.233862    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/678c4830-c56e-4af8-85a2-6f26a4e03aa8-metrics-certs\") pod \"router-default-d7f986b9d-xvnsc\" (UID: \"678c4830-c56e-4af8-85a2-6f26a4e03aa8\") " pod="openshift-ingress/router-default-d7f986b9d-xvnsc"
Sep 10 17:37:03.236222 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:03.236180    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f1713e2d-701b-435b-8d76-18824228a81e-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-mvd6n\" (UID: \"f1713e2d-701b-435b-8d76-18824228a81e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-mvd6n"
Sep 10 17:37:03.330822 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:03.330623    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/06ecbee3-6c6f-467b-831a-d9545f4588bd-metrics-tls\") pod \"dns-default-dxqxz\" (UID: \"06ecbee3-6c6f-467b-831a-d9545f4588bd\") " pod="openshift-dns/dns-default-dxqxz"
Sep 10 17:37:03.333633 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:03.333554    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/06ecbee3-6c6f-467b-831a-d9545f4588bd-metrics-tls\") pod \"dns-default-dxqxz\" (UID: \"06ecbee3-6c6f-467b-831a-d9545f4588bd\") " pod="openshift-dns/dns-default-dxqxz"
Sep 10 17:37:03.420023 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:03.419990    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-hcd8f\""
Sep 10 17:37:03.427334 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:03.427301    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-8f44f7f46-d2blr"
Sep 10 17:37:03.445092 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:03.445056    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-v4bqr\""
Sep 10 17:37:03.453152 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:03.453126    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-mvd6n"
Sep 10 17:37:03.463592 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:03.463430    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-rf44j\""
Sep 10 17:37:03.471581 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:03.471544    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-d7f986b9d-xvnsc"
Sep 10 17:37:03.476672 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:03.476650    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-wrf4l\""
Sep 10 17:37:03.484336 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:03.484020    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-5cbrn"
Sep 10 17:37:03.493141 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:03.492937    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-wtnr4\""
Sep 10 17:37:03.500437 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:03.500403    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-dxqxz"
Sep 10 17:37:03.708692 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:03.708659    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-8f44f7f46-d2blr"]
Sep 10 17:37:03.712332 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:37:03.712297    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad03af6b_287a_4b2f_9ea3_ec8c43c17cba.slice/crio-c16d59a962807033283b34bde932ecaf65cbff633310e54efdc411cc9f01f874 WatchSource:0}: Error finding container c16d59a962807033283b34bde932ecaf65cbff633310e54efdc411cc9f01f874: Status 404 returned error can't find the container with id c16d59a962807033283b34bde932ecaf65cbff633310e54efdc411cc9f01f874
Sep 10 17:37:03.741760 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:03.741692    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-mvd6n"]
Sep 10 17:37:03.776854 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:03.776797    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-d7f986b9d-xvnsc"]
Sep 10 17:37:03.778128 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:37:03.777941    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod678c4830_c56e_4af8_85a2_6f26a4e03aa8.slice/crio-ffc5518d585d3cae6c063f53ced556396ccde5648f9ac8cafdf10aefbe4922da WatchSource:0}: Error finding container ffc5518d585d3cae6c063f53ced556396ccde5648f9ac8cafdf10aefbe4922da: Status 404 returned error can't find the container with id ffc5518d585d3cae6c063f53ced556396ccde5648f9ac8cafdf10aefbe4922da
Sep 10 17:37:03.789178 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:03.789150    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cdcc96965-5cbrn"]
Sep 10 17:37:03.792968 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:37:03.792938    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6976caf5_0ff1_4604_9e30_aab464b54edf.slice/crio-a3f9287052d508d5d03b00b413283caf3ae2ca4ccde4338edc719de4a267f549 WatchSource:0}: Error finding container a3f9287052d508d5d03b00b413283caf3ae2ca4ccde4338edc719de4a267f549: Status 404 returned error can't find the container with id a3f9287052d508d5d03b00b413283caf3ae2ca4ccde4338edc719de4a267f549
Sep 10 17:37:03.810366 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:03.810337    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-dxqxz"]
Sep 10 17:37:04.005798 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:04.005293    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-d7f986b9d-xvnsc" event={"ID":"678c4830-c56e-4af8-85a2-6f26a4e03aa8","Type":"ContainerStarted","Data":"6f1ba9ffac5e0b4ac82483f9e51dca00804910c911a244161b194d79ee2c7b08"}
Sep 10 17:37:04.005798 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:04.005358    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-d7f986b9d-xvnsc" event={"ID":"678c4830-c56e-4af8-85a2-6f26a4e03aa8","Type":"ContainerStarted","Data":"ffc5518d585d3cae6c063f53ced556396ccde5648f9ac8cafdf10aefbe4922da"}
Sep 10 17:37:04.008075 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:04.008039    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-ncmn7" event={"ID":"3fcacac9-c23e-4ba5-8b1d-dfeede6b57e3","Type":"ContainerStarted","Data":"e1110dd462a0fd80dae3ec1323ca56a6b696d1ec27026a6ae077afa0287fb4ce"}
Sep 10 17:37:04.008200 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:04.008084    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-ncmn7" event={"ID":"3fcacac9-c23e-4ba5-8b1d-dfeede6b57e3","Type":"ContainerStarted","Data":"b2f025f6d156d626d3b8ac44788b8a89954c83b643a5551f1ec73eb234d09490"}
Sep 10 17:37:04.010545 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:04.010485    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-mvd6n" event={"ID":"f1713e2d-701b-435b-8d76-18824228a81e","Type":"ContainerStarted","Data":"ec70e1b196d267af867c4531cbe96ff5ef8c4b99d4e6ca500902d15d7ec3ef27"}
Sep 10 17:37:04.011898 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:04.011865    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-5cbrn" event={"ID":"6976caf5-0ff1-4604-9e30-aab464b54edf","Type":"ContainerStarted","Data":"a3f9287052d508d5d03b00b413283caf3ae2ca4ccde4338edc719de4a267f549"}
Sep 10 17:37:04.013642 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:04.013520    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-8f44f7f46-d2blr" event={"ID":"ad03af6b-287a-4b2f-9ea3-ec8c43c17cba","Type":"ContainerStarted","Data":"fe8211f889292711ccdd10b0cd9051047afaae86afca06b3397f146699d8c7b5"}
Sep 10 17:37:04.013642 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:04.013588    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-8f44f7f46-d2blr" event={"ID":"ad03af6b-287a-4b2f-9ea3-ec8c43c17cba","Type":"ContainerStarted","Data":"c16d59a962807033283b34bde932ecaf65cbff633310e54efdc411cc9f01f874"}
Sep 10 17:37:04.013642 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:04.013617    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-8f44f7f46-d2blr"
Sep 10 17:37:04.015323 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:04.015275    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-dxqxz" event={"ID":"06ecbee3-6c6f-467b-831a-d9545f4588bd","Type":"ContainerStarted","Data":"1e594ed9178d61be921a858a29cc1e8268ebb5122b440a9a6253f1dc0fcffeb3"}
Sep 10 17:37:04.048483 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:04.048372    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-d7f986b9d-xvnsc" podStartSLOduration=38.048351052 podStartE2EDuration="38.048351052s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:36:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:37:04.024808385 +0000 UTC m=+68.024638453" watchObservedRunningTime="2026-09-10 17:37:04.048351052 +0000 UTC m=+68.048181118"
Sep 10 17:37:04.049518 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:04.049477    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-8f44f7f46-d2blr" podStartSLOduration=67.049461806 podStartE2EDuration="1m7.049461806s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:35:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:37:04.047897526 +0000 UTC m=+68.047727590" watchObservedRunningTime="2026-09-10 17:37:04.049461806 +0000 UTC m=+68.049291874"
Sep 10 17:37:04.063239 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:04.063186    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-ncmn7" podStartSLOduration=66.641753402 podStartE2EDuration="1m8.063176834s" totalImagesPullingTime="1.421423432s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:35:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:37:04.063153557 +0000 UTC m=+68.062983629" watchObservedRunningTime="2026-09-10 17:37:04.063176834 +0000 UTC m=+68.063006899"
Sep 10 17:37:04.475893 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:04.475599    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-d7f986b9d-xvnsc"
Sep 10 17:37:04.475893 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:04.475712    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-d7f986b9d-xvnsc"
Sep 10 17:37:05.022293 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:05.022261    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-d7f986b9d-xvnsc"
Sep 10 17:37:05.022956 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:05.022936    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-d7f986b9d-xvnsc"
Sep 10 17:37:05.469442 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:05.469413    2579 scope.go:117] "RemoveContainer" containerID="7307b9101dc85645021d36e85099523b0f100cf1bd856a5b1159ca7d207bd745"
Sep 10 17:37:07.568281 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:07.568227    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/510bea53-93c2-4158-851a-0b6ec3e6bee6-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-5klhd\" (UID: \"510bea53-93c2-4158-851a-0b6ec3e6bee6\") " pod="openshift-network-console/networking-console-plugin-79977f9844-5klhd"
Sep 10 17:37:07.570701 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:07.570674    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/510bea53-93c2-4158-851a-0b6ec3e6bee6-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-5klhd\" (UID: \"510bea53-93c2-4158-851a-0b6ec3e6bee6\") " pod="openshift-network-console/networking-console-plugin-79977f9844-5klhd"
Sep 10 17:37:07.794831 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:07.794799    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-9bpk6\""
Sep 10 17:37:07.802847 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:07.802820    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-79977f9844-5klhd"
Sep 10 17:37:08.822650 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:08.821758    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-79977f9844-5klhd"]
Sep 10 17:37:08.833804 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:37:08.833769    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod510bea53_93c2_4158_851a_0b6ec3e6bee6.slice/crio-ea99847d3afc12f257b0056a7591395772e5642c7048ddcd332d4a3bea763fd6 WatchSource:0}: Error finding container ea99847d3afc12f257b0056a7591395772e5642c7048ddcd332d4a3bea763fd6: Status 404 returned error can't find the container with id ea99847d3afc12f257b0056a7591395772e5642c7048ddcd332d4a3bea763fd6
Sep 10 17:37:09.040184 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:09.040148    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-5cbrn" event={"ID":"6976caf5-0ff1-4604-9e30-aab464b54edf","Type":"ContainerStarted","Data":"33ac027b1d39b0f5b438ac2bf9b386da9ff1efe89d09d12268d8151e9abedfb8"}
Sep 10 17:37:09.041497 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:09.041465    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-79977f9844-5klhd" event={"ID":"510bea53-93c2-4158-851a-0b6ec3e6bee6","Type":"ContainerStarted","Data":"ea99847d3afc12f257b0056a7591395772e5642c7048ddcd332d4a3bea763fd6"}
Sep 10 17:37:09.043443 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:09.043426    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-rb854_97172b14-b84a-476e-89f1-05d3e70df416/console-operator/1.log"
Sep 10 17:37:09.043538 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:09.043506    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-rb854" event={"ID":"97172b14-b84a-476e-89f1-05d3e70df416","Type":"ContainerStarted","Data":"dcaca927c04361be300283c65bf59f7c55d301e7985aa4952f170559996cf2d9"}
Sep 10 17:37:09.044612 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:09.043951    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-85b454874b-rb854"
Sep 10 17:37:09.045298 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:09.045272    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-m4npt" event={"ID":"e0fe9300-104f-489d-9962-908000359645","Type":"ContainerStarted","Data":"6aaba1ccaa7d229ceba00b827d07f6e789ea75f80d01e79231f3c46145142594"}
Sep 10 17:37:09.050583 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:09.047355    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-dxqxz" event={"ID":"06ecbee3-6c6f-467b-831a-d9545f4588bd","Type":"ContainerStarted","Data":"33e2a4d935db25c7a66f3d4c4f1950c75287ca83ed9051a68dbbcfc8fae9b912"}
Sep 10 17:37:09.050583 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:09.047394    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-dxqxz" event={"ID":"06ecbee3-6c6f-467b-831a-d9545f4588bd","Type":"ContainerStarted","Data":"94c1916cce3fee27a6416f333ef47160911f4f1e24886f6dab4562a2a8843e2d"}
Sep 10 17:37:09.050583 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:09.047441    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-dxqxz"
Sep 10 17:37:09.050583 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:09.049499    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-mvd6n" event={"ID":"f1713e2d-701b-435b-8d76-18824228a81e","Type":"ContainerStarted","Data":"bd3d94620fdc6b2daa078fe5563ccc8aa882618fa952061491ba8cadf7aaf829"}
Sep 10 17:37:09.050583 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:09.049536    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-mvd6n" event={"ID":"f1713e2d-701b-435b-8d76-18824228a81e","Type":"ContainerStarted","Data":"52330db8a5a600be91d0d8e641f3e564877f827134a92d8305c0fc2c3288e8e6"}
Sep 10 17:37:09.057883 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:09.057806    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-5cbrn" podStartSLOduration=38.302879146 podStartE2EDuration="43.057792922s" totalImagesPullingTime="4.754913776s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:36:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:37:09.056581169 +0000 UTC m=+73.056411231" watchObservedRunningTime="2026-09-10 17:37:09.057792922 +0000 UTC m=+73.057622987"
Sep 10 17:37:09.078192 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:09.078140    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-mvd6n" podStartSLOduration=38.3033399 podStartE2EDuration="43.078122497s" totalImagesPullingTime="4.774782597s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:36:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:37:09.076459815 +0000 UTC m=+73.076289881" watchObservedRunningTime="2026-09-10 17:37:09.078122497 +0000 UTC m=+73.077952565"
Sep 10 17:37:09.093187 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:09.093146    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-dxqxz" podStartSLOduration=33.359900009 podStartE2EDuration="38.093133425s" totalImagesPullingTime="4.733233416s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:36:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:37:09.091920817 +0000 UTC m=+73.091750884" watchObservedRunningTime="2026-09-10 17:37:09.093133425 +0000 UTC m=+73.092963489"
Sep 10 17:37:09.106950 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:09.106903    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-m4npt" podStartSLOduration=65.413981066 podStartE2EDuration="1m12.106892886s" totalImagesPullingTime="6.69291182s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 17:35:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:37:09.105588561 +0000 UTC m=+73.105418626" watchObservedRunningTime="2026-09-10 17:37:09.106892886 +0000 UTC m=+73.106722951"
Sep 10 17:37:09.121204 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:09.121165    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-85b454874b-rb854" podStartSLOduration=33.515099497 podStartE2EDuration="43.121153396s" totalImagesPullingTime="9.606053899s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:36:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:37:09.120635766 +0000 UTC m=+73.120465832" watchObservedRunningTime="2026-09-10 17:37:09.121153396 +0000 UTC m=+73.120983461"
Sep 10 17:37:09.214599 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:09.214549    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-85b454874b-rb854"
Sep 10 17:37:10.685715 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:10.685684    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-bb59f6db6-5sszs"]
Sep 10 17:37:10.701511 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:10.701487    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-bb59f6db6-5sszs"
Sep 10 17:37:10.701667 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:10.701528    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-767f5d7b89-8hxxf"]
Sep 10 17:37:10.708708 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:10.708674    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 10 17:37:10.708708 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:10.708700    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 10 17:37:10.708907 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:10.708710    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Sep 10 17:37:10.713286 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:10.713264    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-l5vvb\""
Sep 10 17:37:10.714818 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:10.714803    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d47ff5645-ptzsm"]
Sep 10 17:37:10.714941 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:10.714927    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-767f5d7b89-8hxxf"
Sep 10 17:37:10.718075 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:10.718057    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 10 17:37:10.726753 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:10.726732    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Sep 10 17:37:10.731487 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:10.731450    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-bb59f6db6-5sszs"]
Sep 10 17:37:10.731600 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:10.731561    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d47ff5645-ptzsm"
Sep 10 17:37:10.734817 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:10.734781    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Sep 10 17:37:10.734817 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:10.734814    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Sep 10 17:37:10.735117 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:10.735099    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Sep 10 17:37:10.735212 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:10.735122    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Sep 10 17:37:10.747457 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:10.747399    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d47ff5645-ptzsm"]
Sep 10 17:37:10.748624 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:10.748604    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-767f5d7b89-8hxxf"]
Sep 10 17:37:10.771840 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:10.771819    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-6kgsw"]
Sep 10 17:37:10.785024 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:10.785006    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-6kgsw"
Sep 10 17:37:10.787650 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:10.787631    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-6kgsw"]
Sep 10 17:37:10.788881 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:10.788860    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Sep 10 17:37:10.789031 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:10.788939    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-lhpvp\""
Sep 10 17:37:10.799233 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:10.799212    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/e7c5efad-909e-4c5d-94af-5b39d4419cd3-ca\") pod \"cluster-proxy-proxy-agent-7d47ff5645-ptzsm\" (UID: \"e7c5efad-909e-4c5d-94af-5b39d4419cd3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d47ff5645-ptzsm"
Sep 10 17:37:10.799329 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:10.799250    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7c5efad-909e-4c5d-94af-5b39d4419cd3-additional-service-ca\") pod \"cluster-proxy-proxy-agent-7d47ff5645-ptzsm\" (UID: \"e7c5efad-909e-4c5d-94af-5b39d4419cd3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d47ff5645-ptzsm"
Sep 10 17:37:10.799329 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:10.799280    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sf9lr\" (UniqueName: \"kubernetes.io/projected/61d89520-ff3f-4d4d-8359-a114e75fe14d-kube-api-access-sf9lr\") pod \"managed-serviceaccount-addon-agent-bb59f6db6-5sszs\" (UID: \"61d89520-ff3f-4d4d-8359-a114e75fe14d\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-bb59f6db6-5sszs"
Sep 10 17:37:10.799329 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:10.799308    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7cg7\" (UniqueName: \"kubernetes.io/projected/e7c5efad-909e-4c5d-94af-5b39d4419cd3-kube-api-access-r7cg7\") pod \"cluster-proxy-proxy-agent-7d47ff5645-ptzsm\" (UID: \"e7c5efad-909e-4c5d-94af-5b39d4419cd3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d47ff5645-ptzsm"
Sep 10 17:37:10.799440 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:10.799365    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/471cb8d8-1961-442e-b041-7d6ab2367171-klusterlet-config\") pod \"klusterlet-addon-workmgr-767f5d7b89-8hxxf\" (UID: \"471cb8d8-1961-442e-b041-7d6ab2367171\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-767f5d7b89-8hxxf"
Sep 10 17:37:10.799440 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:10.799417    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/61d89520-ff3f-4d4d-8359-a114e75fe14d-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-bb59f6db6-5sszs\" (UID: \"61d89520-ff3f-4d4d-8359-a114e75fe14d\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-bb59f6db6-5sszs"
Sep 10 17:37:10.799510 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:10.799457    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/e7c5efad-909e-4c5d-94af-5b39d4419cd3-hub\") pod \"cluster-proxy-proxy-agent-7d47ff5645-ptzsm\" (UID: \"e7c5efad-909e-4c5d-94af-5b39d4419cd3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d47ff5645-ptzsm"
Sep 10 17:37:10.799510 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:10.799474    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e7c5efad-909e-4c5d-94af-5b39d4419cd3-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7d47ff5645-ptzsm\" (UID: \"e7c5efad-909e-4c5d-94af-5b39d4419cd3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d47ff5645-ptzsm"
Sep 10 17:37:10.799590 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:10.799528    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/471cb8d8-1961-442e-b041-7d6ab2367171-tmp\") pod \"klusterlet-addon-workmgr-767f5d7b89-8hxxf\" (UID: \"471cb8d8-1961-442e-b041-7d6ab2367171\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-767f5d7b89-8hxxf"
Sep 10 17:37:10.799590 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:10.799550    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/e7c5efad-909e-4c5d-94af-5b39d4419cd3-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7d47ff5645-ptzsm\" (UID: \"e7c5efad-909e-4c5d-94af-5b39d4419cd3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d47ff5645-ptzsm"
Sep 10 17:37:10.799665 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:10.799642    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtw4b\" (UniqueName: \"kubernetes.io/projected/471cb8d8-1961-442e-b041-7d6ab2367171-kube-api-access-vtw4b\") pod \"klusterlet-addon-workmgr-767f5d7b89-8hxxf\" (UID: \"471cb8d8-1961-442e-b041-7d6ab2367171\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-767f5d7b89-8hxxf"
Sep 10 17:37:10.856913 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:10.856881    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-85d98c998f-fvbsc"]
Sep 10 17:37:10.871271 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:10.871252    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-85d98c998f-fvbsc"
Sep 10 17:37:10.874010 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:10.873810    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-dkw89\""
Sep 10 17:37:10.874124 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:10.874030    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 10 17:37:10.874124 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:10.874078    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 10 17:37:10.874781 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:10.874763    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-85d98c998f-fvbsc"]
Sep 10 17:37:10.900469 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:10.900327    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/61d89520-ff3f-4d4d-8359-a114e75fe14d-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-bb59f6db6-5sszs\" (UID: \"61d89520-ff3f-4d4d-8359-a114e75fe14d\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-bb59f6db6-5sszs"
Sep 10 17:37:10.900469 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:10.900374    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/e7c5efad-909e-4c5d-94af-5b39d4419cd3-hub\") pod \"cluster-proxy-proxy-agent-7d47ff5645-ptzsm\" (UID: \"e7c5efad-909e-4c5d-94af-5b39d4419cd3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d47ff5645-ptzsm"
Sep 10 17:37:10.900469 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:10.900400    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e7c5efad-909e-4c5d-94af-5b39d4419cd3-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7d47ff5645-ptzsm\" (UID: \"e7c5efad-909e-4c5d-94af-5b39d4419cd3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d47ff5645-ptzsm"
Sep 10 17:37:10.900469 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:10.900430    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/c1fc82a8-9158-4f94-be7e-e5dbea38a73d-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-6kgsw\" (UID: \"c1fc82a8-9158-4f94-be7e-e5dbea38a73d\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-6kgsw"
Sep 10 17:37:10.900469 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:10.900469    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/471cb8d8-1961-442e-b041-7d6ab2367171-tmp\") pod \"klusterlet-addon-workmgr-767f5d7b89-8hxxf\" (UID: \"471cb8d8-1961-442e-b041-7d6ab2367171\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-767f5d7b89-8hxxf"
Sep 10 17:37:10.900831 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:10.900497    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/e7c5efad-909e-4c5d-94af-5b39d4419cd3-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7d47ff5645-ptzsm\" (UID: \"e7c5efad-909e-4c5d-94af-5b39d4419cd3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d47ff5645-ptzsm"
Sep 10 17:37:10.900831 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:10.900542    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtw4b\" (UniqueName: \"kubernetes.io/projected/471cb8d8-1961-442e-b041-7d6ab2367171-kube-api-access-vtw4b\") pod \"klusterlet-addon-workmgr-767f5d7b89-8hxxf\" (UID: \"471cb8d8-1961-442e-b041-7d6ab2367171\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-767f5d7b89-8hxxf"
Sep 10 17:37:10.900831 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:10.900602    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/e7c5efad-909e-4c5d-94af-5b39d4419cd3-ca\") pod \"cluster-proxy-proxy-agent-7d47ff5645-ptzsm\" (UID: \"e7c5efad-909e-4c5d-94af-5b39d4419cd3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d47ff5645-ptzsm"
Sep 10 17:37:10.900831 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:10.900622    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7c5efad-909e-4c5d-94af-5b39d4419cd3-additional-service-ca\") pod \"cluster-proxy-proxy-agent-7d47ff5645-ptzsm\" (UID: \"e7c5efad-909e-4c5d-94af-5b39d4419cd3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d47ff5645-ptzsm"
Sep 10 17:37:10.900831 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:10.900639    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sf9lr\" (UniqueName: \"kubernetes.io/projected/61d89520-ff3f-4d4d-8359-a114e75fe14d-kube-api-access-sf9lr\") pod \"managed-serviceaccount-addon-agent-bb59f6db6-5sszs\" (UID: \"61d89520-ff3f-4d4d-8359-a114e75fe14d\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-bb59f6db6-5sszs"
Sep 10 17:37:10.900831 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:10.900659    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7cg7\" (UniqueName: \"kubernetes.io/projected/e7c5efad-909e-4c5d-94af-5b39d4419cd3-kube-api-access-r7cg7\") pod \"cluster-proxy-proxy-agent-7d47ff5645-ptzsm\" (UID: \"e7c5efad-909e-4c5d-94af-5b39d4419cd3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d47ff5645-ptzsm"
Sep 10 17:37:10.900831 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:10.900693    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/471cb8d8-1961-442e-b041-7d6ab2367171-klusterlet-config\") pod \"klusterlet-addon-workmgr-767f5d7b89-8hxxf\" (UID: \"471cb8d8-1961-442e-b041-7d6ab2367171\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-767f5d7b89-8hxxf"
Sep 10 17:37:10.901533 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:10.901387    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7c5efad-909e-4c5d-94af-5b39d4419cd3-additional-service-ca\") pod \"cluster-proxy-proxy-agent-7d47ff5645-ptzsm\" (UID: \"e7c5efad-909e-4c5d-94af-5b39d4419cd3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d47ff5645-ptzsm"
Sep 10 17:37:10.901533 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:10.901521    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/471cb8d8-1961-442e-b041-7d6ab2367171-tmp\") pod \"klusterlet-addon-workmgr-767f5d7b89-8hxxf\" (UID: \"471cb8d8-1961-442e-b041-7d6ab2367171\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-767f5d7b89-8hxxf"
Sep 10 17:37:10.903627 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:10.903605    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e7c5efad-909e-4c5d-94af-5b39d4419cd3-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7d47ff5645-ptzsm\" (UID: \"e7c5efad-909e-4c5d-94af-5b39d4419cd3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d47ff5645-ptzsm"
Sep 10 17:37:10.903803 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:10.903786    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/e7c5efad-909e-4c5d-94af-5b39d4419cd3-ca\") pod \"cluster-proxy-proxy-agent-7d47ff5645-ptzsm\" (UID: \"e7c5efad-909e-4c5d-94af-5b39d4419cd3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d47ff5645-ptzsm"
Sep 10 17:37:10.903887 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:10.903872    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/e7c5efad-909e-4c5d-94af-5b39d4419cd3-hub\") pod \"cluster-proxy-proxy-agent-7d47ff5645-ptzsm\" (UID: \"e7c5efad-909e-4c5d-94af-5b39d4419cd3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d47ff5645-ptzsm"
Sep 10 17:37:10.903921 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:10.903887    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/471cb8d8-1961-442e-b041-7d6ab2367171-klusterlet-config\") pod \"klusterlet-addon-workmgr-767f5d7b89-8hxxf\" (UID: \"471cb8d8-1961-442e-b041-7d6ab2367171\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-767f5d7b89-8hxxf"
Sep 10 17:37:10.903965 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:10.903946    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/61d89520-ff3f-4d4d-8359-a114e75fe14d-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-bb59f6db6-5sszs\" (UID: \"61d89520-ff3f-4d4d-8359-a114e75fe14d\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-bb59f6db6-5sszs"
Sep 10 17:37:10.904698 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:10.904681    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/e7c5efad-909e-4c5d-94af-5b39d4419cd3-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7d47ff5645-ptzsm\" (UID: \"e7c5efad-909e-4c5d-94af-5b39d4419cd3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d47ff5645-ptzsm"
Sep 10 17:37:10.912617 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:10.912557    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sf9lr\" (UniqueName: \"kubernetes.io/projected/61d89520-ff3f-4d4d-8359-a114e75fe14d-kube-api-access-sf9lr\") pod \"managed-serviceaccount-addon-agent-bb59f6db6-5sszs\" (UID: \"61d89520-ff3f-4d4d-8359-a114e75fe14d\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-bb59f6db6-5sszs"
Sep 10 17:37:10.913353 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:10.913308    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtw4b\" (UniqueName: \"kubernetes.io/projected/471cb8d8-1961-442e-b041-7d6ab2367171-kube-api-access-vtw4b\") pod \"klusterlet-addon-workmgr-767f5d7b89-8hxxf\" (UID: \"471cb8d8-1961-442e-b041-7d6ab2367171\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-767f5d7b89-8hxxf"
Sep 10 17:37:10.914304 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:10.914283    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7cg7\" (UniqueName: \"kubernetes.io/projected/e7c5efad-909e-4c5d-94af-5b39d4419cd3-kube-api-access-r7cg7\") pod \"cluster-proxy-proxy-agent-7d47ff5645-ptzsm\" (UID: \"e7c5efad-909e-4c5d-94af-5b39d4419cd3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d47ff5645-ptzsm"
Sep 10 17:37:11.001949 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:11.001865    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/c1fc82a8-9158-4f94-be7e-e5dbea38a73d-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-6kgsw\" (UID: \"c1fc82a8-9158-4f94-be7e-e5dbea38a73d\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-6kgsw"
Sep 10 17:37:11.001949 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:11.001943    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxbx8\" (UniqueName: \"kubernetes.io/projected/023584ce-2746-4d1c-9163-704c37e4cff1-kube-api-access-gxbx8\") pod \"downloads-85d98c998f-fvbsc\" (UID: \"023584ce-2746-4d1c-9163-704c37e4cff1\") " pod="openshift-console/downloads-85d98c998f-fvbsc"
Sep 10 17:37:11.004208 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:11.004186    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/c1fc82a8-9158-4f94-be7e-e5dbea38a73d-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-6kgsw\" (UID: \"c1fc82a8-9158-4f94-be7e-e5dbea38a73d\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-6kgsw"
Sep 10 17:37:11.019944 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:11.019921    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-bb59f6db6-5sszs"
Sep 10 17:37:11.029090 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:11.029071    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-767f5d7b89-8hxxf"
Sep 10 17:37:11.040347 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:11.040327    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d47ff5645-ptzsm"
Sep 10 17:37:11.061544 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:11.061487    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-79977f9844-5klhd" event={"ID":"510bea53-93c2-4158-851a-0b6ec3e6bee6","Type":"ContainerStarted","Data":"3942605c40e7ff9524cf75d79a8c8709a7481917f3504d460f41294530a92a76"}
Sep 10 17:37:11.083322 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:11.083264    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-79977f9844-5klhd" podStartSLOduration=34.428211516 podStartE2EDuration="36.083242891s" totalImagesPullingTime="1.655031375s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:36:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:37:11.081781642 +0000 UTC m=+75.081611710" watchObservedRunningTime="2026-09-10 17:37:11.083242891 +0000 UTC m=+75.083072958"
Sep 10 17:37:11.100014 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:11.099896    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-6kgsw"
Sep 10 17:37:11.103708 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:11.103223    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxbx8\" (UniqueName: \"kubernetes.io/projected/023584ce-2746-4d1c-9163-704c37e4cff1-kube-api-access-gxbx8\") pod \"downloads-85d98c998f-fvbsc\" (UID: \"023584ce-2746-4d1c-9163-704c37e4cff1\") " pod="openshift-console/downloads-85d98c998f-fvbsc"
Sep 10 17:37:11.114524 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:11.114153    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxbx8\" (UniqueName: \"kubernetes.io/projected/023584ce-2746-4d1c-9163-704c37e4cff1-kube-api-access-gxbx8\") pod \"downloads-85d98c998f-fvbsc\" (UID: \"023584ce-2746-4d1c-9163-704c37e4cff1\") " pod="openshift-console/downloads-85d98c998f-fvbsc"
Sep 10 17:37:11.181021 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:11.180994    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-85d98c998f-fvbsc"
Sep 10 17:37:11.321302 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:11.321279    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-bb59f6db6-5sszs"]
Sep 10 17:37:11.323393 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:37:11.323365    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod61d89520_ff3f_4d4d_8359_a114e75fe14d.slice/crio-fa358a1e7cb2b550d7f0578d2b4be27cc006f0c73e63980233f3523c6d3a64c0 WatchSource:0}: Error finding container fa358a1e7cb2b550d7f0578d2b4be27cc006f0c73e63980233f3523c6d3a64c0: Status 404 returned error can't find the container with id fa358a1e7cb2b550d7f0578d2b4be27cc006f0c73e63980233f3523c6d3a64c0
Sep 10 17:37:11.345505 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:11.345485    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-767f5d7b89-8hxxf"]
Sep 10 17:37:11.347492 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:37:11.347472    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod471cb8d8_1961_442e_b041_7d6ab2367171.slice/crio-0882f9f70ad7d92ed5861985f1841a627fb08b3c07b458a4c64a45308c95f513 WatchSource:0}: Error finding container 0882f9f70ad7d92ed5861985f1841a627fb08b3c07b458a4c64a45308c95f513: Status 404 returned error can't find the container with id 0882f9f70ad7d92ed5861985f1841a627fb08b3c07b458a4c64a45308c95f513
Sep 10 17:37:11.374342 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:11.374320    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d47ff5645-ptzsm"]
Sep 10 17:37:11.376415 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:37:11.376387    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode7c5efad_909e_4c5d_94af_5b39d4419cd3.slice/crio-72232f5a148f10838d0f820b8ca08678b6c5bde1854abe6091165f4ba05d4c48 WatchSource:0}: Error finding container 72232f5a148f10838d0f820b8ca08678b6c5bde1854abe6091165f4ba05d4c48: Status 404 returned error can't find the container with id 72232f5a148f10838d0f820b8ca08678b6c5bde1854abe6091165f4ba05d4c48
Sep 10 17:37:11.395395 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:11.395376    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-6kgsw"]
Sep 10 17:37:11.397166 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:37:11.397145    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1fc82a8_9158_4f94_be7e_e5dbea38a73d.slice/crio-81910ebfaf06ea6ba8c699fe1617e75e5a0077f3ff2eb607b082cdf57bb2a7ef WatchSource:0}: Error finding container 81910ebfaf06ea6ba8c699fe1617e75e5a0077f3ff2eb607b082cdf57bb2a7ef: Status 404 returned error can't find the container with id 81910ebfaf06ea6ba8c699fe1617e75e5a0077f3ff2eb607b082cdf57bb2a7ef
Sep 10 17:37:11.469655 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:11.469590    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-85d98c998f-fvbsc"]
Sep 10 17:37:11.471940 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:37:11.471915    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod023584ce_2746_4d1c_9163_704c37e4cff1.slice/crio-135c5e55b9a7c226cf2f3da64bdd2a98998fadc61c8ad3634c394881687051f3 WatchSource:0}: Error finding container 135c5e55b9a7c226cf2f3da64bdd2a98998fadc61c8ad3634c394881687051f3: Status 404 returned error can't find the container with id 135c5e55b9a7c226cf2f3da64bdd2a98998fadc61c8ad3634c394881687051f3
Sep 10 17:37:12.067673 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:12.067630    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-6kgsw" event={"ID":"c1fc82a8-9158-4f94-be7e-e5dbea38a73d","Type":"ContainerStarted","Data":"81910ebfaf06ea6ba8c699fe1617e75e5a0077f3ff2eb607b082cdf57bb2a7ef"}
Sep 10 17:37:12.069219 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:12.069188    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d47ff5645-ptzsm" event={"ID":"e7c5efad-909e-4c5d-94af-5b39d4419cd3","Type":"ContainerStarted","Data":"72232f5a148f10838d0f820b8ca08678b6c5bde1854abe6091165f4ba05d4c48"}
Sep 10 17:37:12.070545 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:12.070519    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-767f5d7b89-8hxxf" event={"ID":"471cb8d8-1961-442e-b041-7d6ab2367171","Type":"ContainerStarted","Data":"0882f9f70ad7d92ed5861985f1841a627fb08b3c07b458a4c64a45308c95f513"}
Sep 10 17:37:12.072273 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:12.072246    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-85d98c998f-fvbsc" event={"ID":"023584ce-2746-4d1c-9163-704c37e4cff1","Type":"ContainerStarted","Data":"135c5e55b9a7c226cf2f3da64bdd2a98998fadc61c8ad3634c394881687051f3"}
Sep 10 17:37:12.073478 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:12.073450    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-bb59f6db6-5sszs" event={"ID":"61d89520-ff3f-4d4d-8359-a114e75fe14d","Type":"ContainerStarted","Data":"fa358a1e7cb2b550d7f0578d2b4be27cc006f0c73e63980233f3523c6d3a64c0"}
Sep 10 17:37:19.057160 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:19.057124    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-dxqxz"
Sep 10 17:37:20.137019 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:20.136976    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-bb59f6db6-5sszs" event={"ID":"61d89520-ff3f-4d4d-8359-a114e75fe14d","Type":"ContainerStarted","Data":"9aecad4926fc817d3af142e3e57bdd4e325295a677a4a659199dcc58d622de31"}
Sep 10 17:37:20.139310 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:20.139073    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-6kgsw" event={"ID":"c1fc82a8-9158-4f94-be7e-e5dbea38a73d","Type":"ContainerStarted","Data":"8f4e7a5fa786c6061b84d623a97c0237c6a45feb158916799fa80ed2ad66d3b7"}
Sep 10 17:37:20.140070 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:20.139236    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-6kgsw"
Sep 10 17:37:20.144318 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:20.143727    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d47ff5645-ptzsm" event={"ID":"e7c5efad-909e-4c5d-94af-5b39d4419cd3","Type":"ContainerStarted","Data":"60f1c4c94403afe6d50ccb9ad2ee2b86165d02bae485fab7a1007f29cff4cff8"}
Sep 10 17:37:20.144318 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:20.143770    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d47ff5645-ptzsm" event={"ID":"e7c5efad-909e-4c5d-94af-5b39d4419cd3","Type":"ContainerStarted","Data":"71b1d165f0034c9f41a26782c371d4441d22dbf945dc9f52bc49c1c447f92aa5"}
Sep 10 17:37:20.144318 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:20.143791    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d47ff5645-ptzsm" event={"ID":"e7c5efad-909e-4c5d-94af-5b39d4419cd3","Type":"ContainerStarted","Data":"46f51714d81c542facdb584ff9e0c2086fc2f15bf98a7d615852afab6fdd4446"}
Sep 10 17:37:20.149046 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:20.149013    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-767f5d7b89-8hxxf" event={"ID":"471cb8d8-1961-442e-b041-7d6ab2367171","Type":"ContainerStarted","Data":"42e4d56d6f02b807ab0bde662ecd199632382671e386937b82ea933f96d23c83"}
Sep 10 17:37:20.149167 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:20.149156    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-6kgsw"
Sep 10 17:37:20.160903 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:20.160861    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-bb59f6db6-5sszs" podStartSLOduration=1.9110026009999999 podStartE2EDuration="10.160848159s" totalImagesPullingTime="8.249845558s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:37:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:37:20.160450124 +0000 UTC m=+84.160280191" watchObservedRunningTime="2026-09-10 17:37:20.160848159 +0000 UTC m=+84.160678225"
Sep 10 17:37:20.178078 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:20.178037    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-6kgsw" podStartSLOduration=2.001926934 podStartE2EDuration="10.178025238s" totalImagesPullingTime="8.176098304s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:37:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:37:20.175863415 +0000 UTC m=+84.175693483" watchObservedRunningTime="2026-09-10 17:37:20.178025238 +0000 UTC m=+84.177855305"
Sep 10 17:37:20.197013 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:20.196968    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-767f5d7b89-8hxxf" podStartSLOduration=1.941942494 podStartE2EDuration="10.196865279s" totalImagesPullingTime="8.254922785s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:37:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:37:20.195866995 +0000 UTC m=+84.195697063" watchObservedRunningTime="2026-09-10 17:37:20.196865279 +0000 UTC m=+84.196695346"
Sep 10 17:37:20.218830 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:20.218787    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d47ff5645-ptzsm" podStartSLOduration=1.996469061 podStartE2EDuration="10.218772128s" totalImagesPullingTime="8.222303067s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:37:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:37:20.218103247 +0000 UTC m=+84.217933315" watchObservedRunningTime="2026-09-10 17:37:20.218772128 +0000 UTC m=+84.218602247"
Sep 10 17:37:21.154291 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:21.154260    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-767f5d7b89-8hxxf"
Sep 10 17:37:21.155014 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:21.154989    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-767f5d7b89-8hxxf"
Sep 10 17:37:23.431648 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:23.431616    2579 patch_prober.go:28] interesting pod/image-registry-8f44f7f46-d2blr container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 10 17:37:23.432025 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:23.431668    2579 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-8f44f7f46-d2blr" podUID="ad03af6b-287a-4b2f-9ea3-ec8c43c17cba" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 10 17:37:25.027269 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:25.027239    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-8f44f7f46-d2blr"
Sep 10 17:37:26.540073 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:26.540027    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-pjp7g"]
Sep 10 17:37:26.544266 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:26.544230    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-pjp7g"
Sep 10 17:37:26.547269 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:26.547242    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 10 17:37:26.547269 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:26.547260    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 10 17:37:26.547601 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:26.547456    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 10 17:37:26.547601 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:26.547558    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 10 17:37:26.548387 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:26.548363    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-rtt87\""
Sep 10 17:37:26.652034 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:26.651996    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b1681ff4-3ace-4de2-b0de-56cf8551c284-node-exporter-tls\") pod \"node-exporter-pjp7g\" (UID: \"b1681ff4-3ace-4de2-b0de-56cf8551c284\") " pod="openshift-monitoring/node-exporter-pjp7g"
Sep 10 17:37:26.652183 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:26.652045    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b1681ff4-3ace-4de2-b0de-56cf8551c284-node-exporter-accelerators-collector-config\") pod \"node-exporter-pjp7g\" (UID: \"b1681ff4-3ace-4de2-b0de-56cf8551c284\") " pod="openshift-monitoring/node-exporter-pjp7g"
Sep 10 17:37:26.652183 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:26.652078    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b1681ff4-3ace-4de2-b0de-56cf8551c284-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-pjp7g\" (UID: \"b1681ff4-3ace-4de2-b0de-56cf8551c284\") " pod="openshift-monitoring/node-exporter-pjp7g"
Sep 10 17:37:26.652183 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:26.652117    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b1681ff4-3ace-4de2-b0de-56cf8551c284-metrics-client-ca\") pod \"node-exporter-pjp7g\" (UID: \"b1681ff4-3ace-4de2-b0de-56cf8551c284\") " pod="openshift-monitoring/node-exporter-pjp7g"
Sep 10 17:37:26.652183 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:26.652149    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b1681ff4-3ace-4de2-b0de-56cf8551c284-node-exporter-textfile\") pod \"node-exporter-pjp7g\" (UID: \"b1681ff4-3ace-4de2-b0de-56cf8551c284\") " pod="openshift-monitoring/node-exporter-pjp7g"
Sep 10 17:37:26.652183 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:26.652174    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b1681ff4-3ace-4de2-b0de-56cf8551c284-node-exporter-wtmp\") pod \"node-exporter-pjp7g\" (UID: \"b1681ff4-3ace-4de2-b0de-56cf8551c284\") " pod="openshift-monitoring/node-exporter-pjp7g"
Sep 10 17:37:26.652353 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:26.652209    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bz2n\" (UniqueName: \"kubernetes.io/projected/b1681ff4-3ace-4de2-b0de-56cf8551c284-kube-api-access-5bz2n\") pod \"node-exporter-pjp7g\" (UID: \"b1681ff4-3ace-4de2-b0de-56cf8551c284\") " pod="openshift-monitoring/node-exporter-pjp7g"
Sep 10 17:37:26.652353 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:26.652238    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b1681ff4-3ace-4de2-b0de-56cf8551c284-sys\") pod \"node-exporter-pjp7g\" (UID: \"b1681ff4-3ace-4de2-b0de-56cf8551c284\") " pod="openshift-monitoring/node-exporter-pjp7g"
Sep 10 17:37:26.652353 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:26.652264    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b1681ff4-3ace-4de2-b0de-56cf8551c284-root\") pod \"node-exporter-pjp7g\" (UID: \"b1681ff4-3ace-4de2-b0de-56cf8551c284\") " pod="openshift-monitoring/node-exporter-pjp7g"
Sep 10 17:37:26.753327 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:26.753294    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b1681ff4-3ace-4de2-b0de-56cf8551c284-root\") pod \"node-exporter-pjp7g\" (UID: \"b1681ff4-3ace-4de2-b0de-56cf8551c284\") " pod="openshift-monitoring/node-exporter-pjp7g"
Sep 10 17:37:26.753521 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:26.753354    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b1681ff4-3ace-4de2-b0de-56cf8551c284-node-exporter-tls\") pod \"node-exporter-pjp7g\" (UID: \"b1681ff4-3ace-4de2-b0de-56cf8551c284\") " pod="openshift-monitoring/node-exporter-pjp7g"
Sep 10 17:37:26.753521 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:26.753385    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b1681ff4-3ace-4de2-b0de-56cf8551c284-node-exporter-accelerators-collector-config\") pod \"node-exporter-pjp7g\" (UID: \"b1681ff4-3ace-4de2-b0de-56cf8551c284\") " pod="openshift-monitoring/node-exporter-pjp7g"
Sep 10 17:37:26.753521 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:26.753417    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b1681ff4-3ace-4de2-b0de-56cf8551c284-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-pjp7g\" (UID: \"b1681ff4-3ace-4de2-b0de-56cf8551c284\") " pod="openshift-monitoring/node-exporter-pjp7g"
Sep 10 17:37:26.753521 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:26.753452    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b1681ff4-3ace-4de2-b0de-56cf8551c284-metrics-client-ca\") pod \"node-exporter-pjp7g\" (UID: \"b1681ff4-3ace-4de2-b0de-56cf8551c284\") " pod="openshift-monitoring/node-exporter-pjp7g"
Sep 10 17:37:26.753521 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:26.753476    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b1681ff4-3ace-4de2-b0de-56cf8551c284-node-exporter-textfile\") pod \"node-exporter-pjp7g\" (UID: \"b1681ff4-3ace-4de2-b0de-56cf8551c284\") " pod="openshift-monitoring/node-exporter-pjp7g"
Sep 10 17:37:26.753521 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:26.753490    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b1681ff4-3ace-4de2-b0de-56cf8551c284-node-exporter-wtmp\") pod \"node-exporter-pjp7g\" (UID: \"b1681ff4-3ace-4de2-b0de-56cf8551c284\") " pod="openshift-monitoring/node-exporter-pjp7g"
Sep 10 17:37:26.753868 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:26.753526    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bz2n\" (UniqueName: \"kubernetes.io/projected/b1681ff4-3ace-4de2-b0de-56cf8551c284-kube-api-access-5bz2n\") pod \"node-exporter-pjp7g\" (UID: \"b1681ff4-3ace-4de2-b0de-56cf8551c284\") " pod="openshift-monitoring/node-exporter-pjp7g"
Sep 10 17:37:26.753868 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:26.753555    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b1681ff4-3ace-4de2-b0de-56cf8551c284-sys\") pod \"node-exporter-pjp7g\" (UID: \"b1681ff4-3ace-4de2-b0de-56cf8551c284\") " pod="openshift-monitoring/node-exporter-pjp7g"
Sep 10 17:37:26.753868 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:26.753638    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b1681ff4-3ace-4de2-b0de-56cf8551c284-sys\") pod \"node-exporter-pjp7g\" (UID: \"b1681ff4-3ace-4de2-b0de-56cf8551c284\") " pod="openshift-monitoring/node-exporter-pjp7g"
Sep 10 17:37:26.753868 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:26.753676    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b1681ff4-3ace-4de2-b0de-56cf8551c284-root\") pod \"node-exporter-pjp7g\" (UID: \"b1681ff4-3ace-4de2-b0de-56cf8551c284\") " pod="openshift-monitoring/node-exporter-pjp7g"
Sep 10 17:37:26.753868 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:37:26.753765    2579 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 10 17:37:26.753868 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:37:26.753833    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1681ff4-3ace-4de2-b0de-56cf8551c284-node-exporter-tls podName:b1681ff4-3ace-4de2-b0de-56cf8551c284 nodeName:}" failed. No retries permitted until 2026-09-10 17:37:27.253810619 +0000 UTC m=+91.253640684 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/b1681ff4-3ace-4de2-b0de-56cf8551c284-node-exporter-tls") pod "node-exporter-pjp7g" (UID: "b1681ff4-3ace-4de2-b0de-56cf8551c284") : secret "node-exporter-tls" not found
Sep 10 17:37:26.755143 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:26.754207    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b1681ff4-3ace-4de2-b0de-56cf8551c284-node-exporter-wtmp\") pod \"node-exporter-pjp7g\" (UID: \"b1681ff4-3ace-4de2-b0de-56cf8551c284\") " pod="openshift-monitoring/node-exporter-pjp7g"
Sep 10 17:37:26.755143 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:26.754424    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b1681ff4-3ace-4de2-b0de-56cf8551c284-node-exporter-textfile\") pod \"node-exporter-pjp7g\" (UID: \"b1681ff4-3ace-4de2-b0de-56cf8551c284\") " pod="openshift-monitoring/node-exporter-pjp7g"
Sep 10 17:37:26.755596 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:26.755545    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b1681ff4-3ace-4de2-b0de-56cf8551c284-node-exporter-accelerators-collector-config\") pod \"node-exporter-pjp7g\" (UID: \"b1681ff4-3ace-4de2-b0de-56cf8551c284\") " pod="openshift-monitoring/node-exporter-pjp7g"
Sep 10 17:37:26.755858 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:26.755838    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b1681ff4-3ace-4de2-b0de-56cf8551c284-metrics-client-ca\") pod \"node-exporter-pjp7g\" (UID: \"b1681ff4-3ace-4de2-b0de-56cf8551c284\") " pod="openshift-monitoring/node-exporter-pjp7g"
Sep 10 17:37:26.757225 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:26.757182    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b1681ff4-3ace-4de2-b0de-56cf8551c284-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-pjp7g\" (UID: \"b1681ff4-3ace-4de2-b0de-56cf8551c284\") " pod="openshift-monitoring/node-exporter-pjp7g"
Sep 10 17:37:26.765749 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:26.765695    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bz2n\" (UniqueName: \"kubernetes.io/projected/b1681ff4-3ace-4de2-b0de-56cf8551c284-kube-api-access-5bz2n\") pod \"node-exporter-pjp7g\" (UID: \"b1681ff4-3ace-4de2-b0de-56cf8551c284\") " pod="openshift-monitoring/node-exporter-pjp7g"
Sep 10 17:37:27.261950 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:27.261893    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b1681ff4-3ace-4de2-b0de-56cf8551c284-node-exporter-tls\") pod \"node-exporter-pjp7g\" (UID: \"b1681ff4-3ace-4de2-b0de-56cf8551c284\") " pod="openshift-monitoring/node-exporter-pjp7g"
Sep 10 17:37:27.264868 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:27.264841    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b1681ff4-3ace-4de2-b0de-56cf8551c284-node-exporter-tls\") pod \"node-exporter-pjp7g\" (UID: \"b1681ff4-3ace-4de2-b0de-56cf8551c284\") " pod="openshift-monitoring/node-exporter-pjp7g"
Sep 10 17:37:27.460527 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:27.460490    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-pjp7g"
Sep 10 17:37:31.222153 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:37:31.222123    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1681ff4_3ace_4de2_b0de_56cf8551c284.slice/crio-533a951dab374160e6728b0a9f4a0f3d7c7c9e3693ddc3c10dd9e1705fe3fa26 WatchSource:0}: Error finding container 533a951dab374160e6728b0a9f4a0f3d7c7c9e3693ddc3c10dd9e1705fe3fa26: Status 404 returned error can't find the container with id 533a951dab374160e6728b0a9f4a0f3d7c7c9e3693ddc3c10dd9e1705fe3fa26
Sep 10 17:37:32.210336 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:32.207157    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-85d98c998f-fvbsc" event={"ID":"023584ce-2746-4d1c-9163-704c37e4cff1","Type":"ContainerStarted","Data":"1a823a8931ec9e3c5ee88a81240f7ece84dd8860042240cb8d4f71e5a95f3f06"}
Sep 10 17:37:32.210662 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:32.210600    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-85d98c998f-fvbsc"
Sep 10 17:37:32.212462 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:32.212317    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-pjp7g" event={"ID":"b1681ff4-3ace-4de2-b0de-56cf8551c284","Type":"ContainerStarted","Data":"533a951dab374160e6728b0a9f4a0f3d7c7c9e3693ddc3c10dd9e1705fe3fa26"}
Sep 10 17:37:32.228956 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:32.228914    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-85d98c998f-fvbsc" podStartSLOduration=2.136501662 podStartE2EDuration="22.228898016s" totalImagesPullingTime="20.092396354s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:37:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:37:32.228230876 +0000 UTC m=+96.228060944" watchObservedRunningTime="2026-09-10 17:37:32.228898016 +0000 UTC m=+96.228728082"
Sep 10 17:37:32.234923 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:32.234899    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-85d98c998f-fvbsc"
Sep 10 17:37:33.000089 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:33.000051    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-8k8z5"
Sep 10 17:37:33.219356 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:33.219313    2579 generic.go:356] "Generic (PLEG): container finished" podID="b1681ff4-3ace-4de2-b0de-56cf8551c284" containerID="50994beaf71ddcef516f38c1c816e53ef2cadcd352c75ef6f1601ef13d01e7e9" exitCode=0
Sep 10 17:37:33.219671 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:33.219607    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-pjp7g" event={"ID":"b1681ff4-3ace-4de2-b0de-56cf8551c284","Type":"ContainerDied","Data":"50994beaf71ddcef516f38c1c816e53ef2cadcd352c75ef6f1601ef13d01e7e9"}
Sep 10 17:37:34.227790 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:34.227749    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-pjp7g" event={"ID":"b1681ff4-3ace-4de2-b0de-56cf8551c284","Type":"ContainerStarted","Data":"7bd4612f08d19bbd1d58d34f56c2441e2190590d03b1e41e4bf41f0cb26575c1"}
Sep 10 17:37:34.228254 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:34.227818    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-pjp7g" event={"ID":"b1681ff4-3ace-4de2-b0de-56cf8551c284","Type":"ContainerStarted","Data":"fae921184b4504242d0b98d490f083a5ff6b48489ea33230a9c1aec581ca05c4"}
Sep 10 17:37:34.252388 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:34.252340    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-pjp7g" podStartSLOduration=6.769294743 podStartE2EDuration="8.252320993s" totalImagesPullingTime="1.392073967s" totalInitContainerRuntime="90.952283ms" isStatefulPod=true podCreationTimestamp="2026-09-10 17:37:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:37:34.251106849 +0000 UTC m=+98.250936918" watchObservedRunningTime="2026-09-10 17:37:34.252320993 +0000 UTC m=+98.252151062"
Sep 10 17:37:37.178418 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:37.178374    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-8f44f7f46-d2blr"]
Sep 10 17:37:58.329304 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:58.329267    2579 generic.go:356] "Generic (PLEG): container finished" podID="fd21a8ca-3f90-4aa5-b1de-78ddebecf928" containerID="8b65ca7cc317e38b9421f49b647c2e9cc4fb2d7225001dce85fe9997b9beabf0" exitCode=0
Sep 10 17:37:58.329772 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:58.329341    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-rh6vv" event={"ID":"fd21a8ca-3f90-4aa5-b1de-78ddebecf928","Type":"ContainerDied","Data":"8b65ca7cc317e38b9421f49b647c2e9cc4fb2d7225001dce85fe9997b9beabf0"}
Sep 10 17:37:58.329772 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:58.329755    2579 scope.go:117] "RemoveContainer" containerID="8b65ca7cc317e38b9421f49b647c2e9cc4fb2d7225001dce85fe9997b9beabf0"
Sep 10 17:37:58.331058 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:58.331038    2579 generic.go:356] "Generic (PLEG): container finished" podID="5edf557d-b123-458c-aaf8-e9760b5b6cfe" containerID="6317aad02cc1fc51f3e98d11813b36a4e860918f7f7b1cd6324a750ccec4514e" exitCode=0
Sep 10 17:37:58.331141 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:58.331066    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-cmwpc" event={"ID":"5edf557d-b123-458c-aaf8-e9760b5b6cfe","Type":"ContainerDied","Data":"6317aad02cc1fc51f3e98d11813b36a4e860918f7f7b1cd6324a750ccec4514e"}
Sep 10 17:37:58.331294 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:58.331283    2579 scope.go:117] "RemoveContainer" containerID="6317aad02cc1fc51f3e98d11813b36a4e860918f7f7b1cd6324a750ccec4514e"
Sep 10 17:37:59.338496 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:59.338459    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-rh6vv" event={"ID":"fd21a8ca-3f90-4aa5-b1de-78ddebecf928","Type":"ContainerStarted","Data":"1554135a020930a356693ab68f92257abf6584366c2c6dd01fa4d6ff5b72e47d"}
Sep 10 17:37:59.340425 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:37:59.340401    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-cmwpc" event={"ID":"5edf557d-b123-458c-aaf8-e9760b5b6cfe","Type":"ContainerStarted","Data":"26a6687bf26f8c9e0b9267e6d7cf7b039675484cc530e65b10cff86299cef1a0"}
Sep 10 17:38:02.204308 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:38:02.204224    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-8f44f7f46-d2blr" podUID="ad03af6b-287a-4b2f-9ea3-ec8c43c17cba" containerName="registry" containerID="cri-o://fe8211f889292711ccdd10b0cd9051047afaae86afca06b3397f146699d8c7b5" gracePeriod=30
Sep 10 17:38:02.352904 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:38:02.352875    2579 generic.go:356] "Generic (PLEG): container finished" podID="ad03af6b-287a-4b2f-9ea3-ec8c43c17cba" containerID="fe8211f889292711ccdd10b0cd9051047afaae86afca06b3397f146699d8c7b5" exitCode=0
Sep 10 17:38:02.353054 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:38:02.352945    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-8f44f7f46-d2blr" event={"ID":"ad03af6b-287a-4b2f-9ea3-ec8c43c17cba","Type":"ContainerDied","Data":"fe8211f889292711ccdd10b0cd9051047afaae86afca06b3397f146699d8c7b5"}
Sep 10 17:38:02.486094 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:38:02.486055    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-8f44f7f46-d2blr"
Sep 10 17:38:02.567082 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:38:02.567038    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ad03af6b-287a-4b2f-9ea3-ec8c43c17cba-registry-tls\") pod \"ad03af6b-287a-4b2f-9ea3-ec8c43c17cba\" (UID: \"ad03af6b-287a-4b2f-9ea3-ec8c43c17cba\") "
Sep 10 17:38:02.567082 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:38:02.567084    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ad03af6b-287a-4b2f-9ea3-ec8c43c17cba-image-registry-private-configuration\") pod \"ad03af6b-287a-4b2f-9ea3-ec8c43c17cba\" (UID: \"ad03af6b-287a-4b2f-9ea3-ec8c43c17cba\") "
Sep 10 17:38:02.567317 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:38:02.567119    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ad03af6b-287a-4b2f-9ea3-ec8c43c17cba-registry-certificates\") pod \"ad03af6b-287a-4b2f-9ea3-ec8c43c17cba\" (UID: \"ad03af6b-287a-4b2f-9ea3-ec8c43c17cba\") "
Sep 10 17:38:02.567317 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:38:02.567153    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ad03af6b-287a-4b2f-9ea3-ec8c43c17cba-bound-sa-token\") pod \"ad03af6b-287a-4b2f-9ea3-ec8c43c17cba\" (UID: \"ad03af6b-287a-4b2f-9ea3-ec8c43c17cba\") "
Sep 10 17:38:02.567317 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:38:02.567170    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ad03af6b-287a-4b2f-9ea3-ec8c43c17cba-installation-pull-secrets\") pod \"ad03af6b-287a-4b2f-9ea3-ec8c43c17cba\" (UID: \"ad03af6b-287a-4b2f-9ea3-ec8c43c17cba\") "
Sep 10 17:38:02.567317 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:38:02.567215    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ad03af6b-287a-4b2f-9ea3-ec8c43c17cba-ca-trust-extracted\") pod \"ad03af6b-287a-4b2f-9ea3-ec8c43c17cba\" (UID: \"ad03af6b-287a-4b2f-9ea3-ec8c43c17cba\") "
Sep 10 17:38:02.567317 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:38:02.567265    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qp685\" (UniqueName: \"kubernetes.io/projected/ad03af6b-287a-4b2f-9ea3-ec8c43c17cba-kube-api-access-qp685\") pod \"ad03af6b-287a-4b2f-9ea3-ec8c43c17cba\" (UID: \"ad03af6b-287a-4b2f-9ea3-ec8c43c17cba\") "
Sep 10 17:38:02.567317 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:38:02.567289    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ad03af6b-287a-4b2f-9ea3-ec8c43c17cba-trusted-ca\") pod \"ad03af6b-287a-4b2f-9ea3-ec8c43c17cba\" (UID: \"ad03af6b-287a-4b2f-9ea3-ec8c43c17cba\") "
Sep 10 17:38:02.567770 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:38:02.567684    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad03af6b-287a-4b2f-9ea3-ec8c43c17cba-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "ad03af6b-287a-4b2f-9ea3-ec8c43c17cba" (UID: "ad03af6b-287a-4b2f-9ea3-ec8c43c17cba"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 17:38:02.567876 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:38:02.567850    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad03af6b-287a-4b2f-9ea3-ec8c43c17cba-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "ad03af6b-287a-4b2f-9ea3-ec8c43c17cba" (UID: "ad03af6b-287a-4b2f-9ea3-ec8c43c17cba"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 17:38:02.569954 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:38:02.569925    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad03af6b-287a-4b2f-9ea3-ec8c43c17cba-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "ad03af6b-287a-4b2f-9ea3-ec8c43c17cba" (UID: "ad03af6b-287a-4b2f-9ea3-ec8c43c17cba"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:38:02.569954 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:38:02.569934    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad03af6b-287a-4b2f-9ea3-ec8c43c17cba-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "ad03af6b-287a-4b2f-9ea3-ec8c43c17cba" (UID: "ad03af6b-287a-4b2f-9ea3-ec8c43c17cba"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 17:38:02.570233 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:38:02.570211    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad03af6b-287a-4b2f-9ea3-ec8c43c17cba-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "ad03af6b-287a-4b2f-9ea3-ec8c43c17cba" (UID: "ad03af6b-287a-4b2f-9ea3-ec8c43c17cba"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:38:02.570313 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:38:02.570280    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad03af6b-287a-4b2f-9ea3-ec8c43c17cba-kube-api-access-qp685" (OuterVolumeSpecName: "kube-api-access-qp685") pod "ad03af6b-287a-4b2f-9ea3-ec8c43c17cba" (UID: "ad03af6b-287a-4b2f-9ea3-ec8c43c17cba"). InnerVolumeSpecName "kube-api-access-qp685". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:38:02.570432 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:38:02.570399    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad03af6b-287a-4b2f-9ea3-ec8c43c17cba-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "ad03af6b-287a-4b2f-9ea3-ec8c43c17cba" (UID: "ad03af6b-287a-4b2f-9ea3-ec8c43c17cba"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 17:38:02.575902 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:38:02.575861    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad03af6b-287a-4b2f-9ea3-ec8c43c17cba-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "ad03af6b-287a-4b2f-9ea3-ec8c43c17cba" (UID: "ad03af6b-287a-4b2f-9ea3-ec8c43c17cba"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:38:02.668849 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:38:02.668802    2579 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ad03af6b-287a-4b2f-9ea3-ec8c43c17cba-bound-sa-token\") on node \"ip-10-0-129-122.ec2.internal\" DevicePath \"\""
Sep 10 17:38:02.668849 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:38:02.668837    2579 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ad03af6b-287a-4b2f-9ea3-ec8c43c17cba-installation-pull-secrets\") on node \"ip-10-0-129-122.ec2.internal\" DevicePath \"\""
Sep 10 17:38:02.668849 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:38:02.668848    2579 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ad03af6b-287a-4b2f-9ea3-ec8c43c17cba-ca-trust-extracted\") on node \"ip-10-0-129-122.ec2.internal\" DevicePath \"\""
Sep 10 17:38:02.668849 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:38:02.668857    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qp685\" (UniqueName: \"kubernetes.io/projected/ad03af6b-287a-4b2f-9ea3-ec8c43c17cba-kube-api-access-qp685\") on node \"ip-10-0-129-122.ec2.internal\" DevicePath \"\""
Sep 10 17:38:02.669108 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:38:02.668866    2579 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ad03af6b-287a-4b2f-9ea3-ec8c43c17cba-trusted-ca\") on node \"ip-10-0-129-122.ec2.internal\" DevicePath \"\""
Sep 10 17:38:02.669108 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:38:02.668874    2579 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ad03af6b-287a-4b2f-9ea3-ec8c43c17cba-registry-tls\") on node \"ip-10-0-129-122.ec2.internal\" DevicePath \"\""
Sep 10 17:38:02.669108 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:38:02.668883    2579 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ad03af6b-287a-4b2f-9ea3-ec8c43c17cba-image-registry-private-configuration\") on node \"ip-10-0-129-122.ec2.internal\" DevicePath \"\""
Sep 10 17:38:02.669108 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:38:02.668892    2579 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ad03af6b-287a-4b2f-9ea3-ec8c43c17cba-registry-certificates\") on node \"ip-10-0-129-122.ec2.internal\" DevicePath \"\""
Sep 10 17:38:03.358449 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:38:03.358418    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-8f44f7f46-d2blr"
Sep 10 17:38:03.358449 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:38:03.358445    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-8f44f7f46-d2blr" event={"ID":"ad03af6b-287a-4b2f-9ea3-ec8c43c17cba","Type":"ContainerDied","Data":"c16d59a962807033283b34bde932ecaf65cbff633310e54efdc411cc9f01f874"}
Sep 10 17:38:03.358983 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:38:03.358497    2579 scope.go:117] "RemoveContainer" containerID="fe8211f889292711ccdd10b0cd9051047afaae86afca06b3397f146699d8c7b5"
Sep 10 17:38:03.379530 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:38:03.379499    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-8f44f7f46-d2blr"]
Sep 10 17:38:03.384930 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:38:03.384909    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-8f44f7f46-d2blr"]
Sep 10 17:38:04.473762 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:38:04.473726    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ad03af6b-287a-4b2f-9ea3-ec8c43c17cba" path="/var/lib/kubelet/pods/ad03af6b-287a-4b2f-9ea3-ec8c43c17cba/volumes"
Sep 10 17:38:12.404789 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:38:12.404758    2579 generic.go:356] "Generic (PLEG): container finished" podID="c8f186e0-4c0d-4422-be62-96685cca0258" containerID="5ebcb6d9258d5021f102c0f2fe373ad9afb5f8b629a9d4bee7fd5b2c3e674461" exitCode=0
Sep 10 17:38:12.405160 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:38:12.404827    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-wt6xs" event={"ID":"c8f186e0-4c0d-4422-be62-96685cca0258","Type":"ContainerDied","Data":"5ebcb6d9258d5021f102c0f2fe373ad9afb5f8b629a9d4bee7fd5b2c3e674461"}
Sep 10 17:38:12.405160 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:38:12.405154    2579 scope.go:117] "RemoveContainer" containerID="5ebcb6d9258d5021f102c0f2fe373ad9afb5f8b629a9d4bee7fd5b2c3e674461"
Sep 10 17:38:13.410662 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:38:13.410626    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-wt6xs" event={"ID":"c8f186e0-4c0d-4422-be62-96685cca0258","Type":"ContainerStarted","Data":"bfe2343c57b221ac9301be81cf71de9155f7c73d74a5a19c4b825e92d665414b"}
Sep 10 17:40:33.885331 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:40:33.885300    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-dbf885bd4-txcxf"]
Sep 10 17:40:33.885761 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:40:33.885616    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad03af6b-287a-4b2f-9ea3-ec8c43c17cba" containerName="registry"
Sep 10 17:40:33.885761 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:40:33.885628    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad03af6b-287a-4b2f-9ea3-ec8c43c17cba" containerName="registry"
Sep 10 17:40:33.885761 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:40:33.885692    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="ad03af6b-287a-4b2f-9ea3-ec8c43c17cba" containerName="registry"
Sep 10 17:40:33.887849 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:40:33.887830    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-dbf885bd4-txcxf"
Sep 10 17:40:33.890313 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:40:33.890292    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 10 17:40:33.890439 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:40:33.890298    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 10 17:40:33.891520 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:40:33.891325    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 10 17:40:33.891520 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:40:33.891407    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-c9dnv\""
Sep 10 17:40:33.891520 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:40:33.891493    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 10 17:40:33.907354 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:40:33.907336    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-dbf885bd4-txcxf"]
Sep 10 17:40:33.961668 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:40:33.961645    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/19472691-efec-4186-9c11-10dc71432e71-manifests\") pod \"opendatahub-operator-controller-manager-dbf885bd4-txcxf\" (UID: \"19472691-efec-4186-9c11-10dc71432e71\") " pod="opendatahub/opendatahub-operator-controller-manager-dbf885bd4-txcxf"
Sep 10 17:40:33.961793 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:40:33.961676    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r69z2\" (UniqueName: \"kubernetes.io/projected/19472691-efec-4186-9c11-10dc71432e71-kube-api-access-r69z2\") pod \"opendatahub-operator-controller-manager-dbf885bd4-txcxf\" (UID: \"19472691-efec-4186-9c11-10dc71432e71\") " pod="opendatahub/opendatahub-operator-controller-manager-dbf885bd4-txcxf"
Sep 10 17:40:33.961793 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:40:33.961710    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/19472691-efec-4186-9c11-10dc71432e71-webhook-cert\") pod \"opendatahub-operator-controller-manager-dbf885bd4-txcxf\" (UID: \"19472691-efec-4186-9c11-10dc71432e71\") " pod="opendatahub/opendatahub-operator-controller-manager-dbf885bd4-txcxf"
Sep 10 17:40:33.961793 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:40:33.961777    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/19472691-efec-4186-9c11-10dc71432e71-tmp\") pod \"opendatahub-operator-controller-manager-dbf885bd4-txcxf\" (UID: \"19472691-efec-4186-9c11-10dc71432e71\") " pod="opendatahub/opendatahub-operator-controller-manager-dbf885bd4-txcxf"
Sep 10 17:40:33.961893 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:40:33.961806    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/19472691-efec-4186-9c11-10dc71432e71-apiservice-cert\") pod \"opendatahub-operator-controller-manager-dbf885bd4-txcxf\" (UID: \"19472691-efec-4186-9c11-10dc71432e71\") " pod="opendatahub/opendatahub-operator-controller-manager-dbf885bd4-txcxf"
Sep 10 17:40:34.062508 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:40:34.062429    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/19472691-efec-4186-9c11-10dc71432e71-webhook-cert\") pod \"opendatahub-operator-controller-manager-dbf885bd4-txcxf\" (UID: \"19472691-efec-4186-9c11-10dc71432e71\") " pod="opendatahub/opendatahub-operator-controller-manager-dbf885bd4-txcxf"
Sep 10 17:40:34.062508 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:40:34.062478    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/19472691-efec-4186-9c11-10dc71432e71-tmp\") pod \"opendatahub-operator-controller-manager-dbf885bd4-txcxf\" (UID: \"19472691-efec-4186-9c11-10dc71432e71\") " pod="opendatahub/opendatahub-operator-controller-manager-dbf885bd4-txcxf"
Sep 10 17:40:34.062508 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:40:34.062506    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/19472691-efec-4186-9c11-10dc71432e71-apiservice-cert\") pod \"opendatahub-operator-controller-manager-dbf885bd4-txcxf\" (UID: \"19472691-efec-4186-9c11-10dc71432e71\") " pod="opendatahub/opendatahub-operator-controller-manager-dbf885bd4-txcxf"
Sep 10 17:40:34.062772 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:40:34.062532    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/19472691-efec-4186-9c11-10dc71432e71-manifests\") pod \"opendatahub-operator-controller-manager-dbf885bd4-txcxf\" (UID: \"19472691-efec-4186-9c11-10dc71432e71\") " pod="opendatahub/opendatahub-operator-controller-manager-dbf885bd4-txcxf"
Sep 10 17:40:34.062772 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:40:34.062549    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r69z2\" (UniqueName: \"kubernetes.io/projected/19472691-efec-4186-9c11-10dc71432e71-kube-api-access-r69z2\") pod \"opendatahub-operator-controller-manager-dbf885bd4-txcxf\" (UID: \"19472691-efec-4186-9c11-10dc71432e71\") " pod="opendatahub/opendatahub-operator-controller-manager-dbf885bd4-txcxf"
Sep 10 17:40:34.062956 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:40:34.062933    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/19472691-efec-4186-9c11-10dc71432e71-manifests\") pod \"opendatahub-operator-controller-manager-dbf885bd4-txcxf\" (UID: \"19472691-efec-4186-9c11-10dc71432e71\") " pod="opendatahub/opendatahub-operator-controller-manager-dbf885bd4-txcxf"
Sep 10 17:40:34.064951 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:40:34.064930    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/19472691-efec-4186-9c11-10dc71432e71-tmp\") pod \"opendatahub-operator-controller-manager-dbf885bd4-txcxf\" (UID: \"19472691-efec-4186-9c11-10dc71432e71\") " pod="opendatahub/opendatahub-operator-controller-manager-dbf885bd4-txcxf"
Sep 10 17:40:34.065046 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:40:34.065022    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/19472691-efec-4186-9c11-10dc71432e71-apiservice-cert\") pod \"opendatahub-operator-controller-manager-dbf885bd4-txcxf\" (UID: \"19472691-efec-4186-9c11-10dc71432e71\") " pod="opendatahub/opendatahub-operator-controller-manager-dbf885bd4-txcxf"
Sep 10 17:40:34.065244 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:40:34.065224    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/19472691-efec-4186-9c11-10dc71432e71-webhook-cert\") pod \"opendatahub-operator-controller-manager-dbf885bd4-txcxf\" (UID: \"19472691-efec-4186-9c11-10dc71432e71\") " pod="opendatahub/opendatahub-operator-controller-manager-dbf885bd4-txcxf"
Sep 10 17:40:34.071023 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:40:34.071000    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r69z2\" (UniqueName: \"kubernetes.io/projected/19472691-efec-4186-9c11-10dc71432e71-kube-api-access-r69z2\") pod \"opendatahub-operator-controller-manager-dbf885bd4-txcxf\" (UID: \"19472691-efec-4186-9c11-10dc71432e71\") " pod="opendatahub/opendatahub-operator-controller-manager-dbf885bd4-txcxf"
Sep 10 17:40:34.198971 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:40:34.198945    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-dbf885bd4-txcxf"
Sep 10 17:40:34.474706 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:40:34.474677    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19472691_efec_4186_9c11_10dc71432e71.slice/crio-5954be0bb9b332644cb94a59a2851b5c2d560e4e3428408bd3d4f3ed21c1c2ef WatchSource:0}: Error finding container 5954be0bb9b332644cb94a59a2851b5c2d560e4e3428408bd3d4f3ed21c1c2ef: Status 404 returned error can't find the container with id 5954be0bb9b332644cb94a59a2851b5c2d560e4e3428408bd3d4f3ed21c1c2ef
Sep 10 17:40:34.474828 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:40:34.474705    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-dbf885bd4-txcxf"]
Sep 10 17:40:34.949144 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:40:34.949102    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-dbf885bd4-txcxf" event={"ID":"19472691-efec-4186-9c11-10dc71432e71","Type":"ContainerStarted","Data":"5954be0bb9b332644cb94a59a2851b5c2d560e4e3428408bd3d4f3ed21c1c2ef"}
Sep 10 17:40:37.964599 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:40:37.964543    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-dbf885bd4-txcxf" event={"ID":"19472691-efec-4186-9c11-10dc71432e71","Type":"ContainerStarted","Data":"f1b7490a75f13c425b127fda37cba0fd5974533a30cae6287cfdb22649e630fd"}
Sep 10 17:40:38.970167 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:40:38.970133    2579 generic.go:356] "Generic (PLEG): container finished" podID="19472691-efec-4186-9c11-10dc71432e71" containerID="f1b7490a75f13c425b127fda37cba0fd5974533a30cae6287cfdb22649e630fd" exitCode=0
Sep 10 17:40:38.970519 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:40:38.970222    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-dbf885bd4-txcxf" event={"ID":"19472691-efec-4186-9c11-10dc71432e71","Type":"ContainerDied","Data":"f1b7490a75f13c425b127fda37cba0fd5974533a30cae6287cfdb22649e630fd"}
Sep 10 17:40:39.978297 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:40:39.978259    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-dbf885bd4-txcxf" event={"ID":"19472691-efec-4186-9c11-10dc71432e71","Type":"ContainerStarted","Data":"ce00bfd69bc6ff6991a0b4f237e6fe82da8b42d23cd4dfe47b2ed2e6e5f73279"}
Sep 10 17:40:39.978729 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:40:39.978495    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-dbf885bd4-txcxf"
Sep 10 17:40:40.005860 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:40:40.005801    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-dbf885bd4-txcxf" podStartSLOduration=2.594995939 podStartE2EDuration="7.005782162s" totalImagesPullingTime="3.364172602s" totalInitContainerRuntime="1.046613621s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:40:33 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:40:40.003229456 +0000 UTC m=+284.003059521" watchObservedRunningTime="2026-09-10 17:40:40.005782162 +0000 UTC m=+284.005612232"
Sep 10 17:40:50.984256 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:40:50.984227    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-dbf885bd4-txcxf"
Sep 10 17:40:56.380390 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:40:56.380366    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-rb854_97172b14-b84a-476e-89f1-05d3e70df416/console-operator/1.log"
Sep 10 17:40:56.382737 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:40:56.382713    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-rb854_97172b14-b84a-476e-89f1-05d3e70df416/console-operator/1.log"
Sep 10 17:40:56.392284 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:40:56.392262    2579 kubelet.go:1648] "Image garbage collection succeeded"
Sep 10 17:41:11.706503 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:41:11.706466    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-57bb875478-k5dn6"]
Sep 10 17:41:11.709859 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:41:11.709843    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-57bb875478-k5dn6"
Sep 10 17:41:11.712966 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:41:11.712946    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 10 17:41:11.714001 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:41:11.713983    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 10 17:41:11.714084 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:41:11.713995    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 10 17:41:11.714134 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:41:11.714104    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 10 17:41:11.714176 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:41:11.714104    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-tkbtv\""
Sep 10 17:41:11.714269 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:41:11.714252    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 10 17:41:11.721526 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:41:11.721504    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-57bb875478-k5dn6"]
Sep 10 17:41:11.743149 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:41:11.743123    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/719743db-5de0-4ea1-a365-75fca5745d64-cert\") pod \"lws-controller-manager-57bb875478-k5dn6\" (UID: \"719743db-5de0-4ea1-a365-75fca5745d64\") " pod="openshift-lws-operator/lws-controller-manager-57bb875478-k5dn6"
Sep 10 17:41:11.743260 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:41:11.743197    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c62zw\" (UniqueName: \"kubernetes.io/projected/719743db-5de0-4ea1-a365-75fca5745d64-kube-api-access-c62zw\") pod \"lws-controller-manager-57bb875478-k5dn6\" (UID: \"719743db-5de0-4ea1-a365-75fca5745d64\") " pod="openshift-lws-operator/lws-controller-manager-57bb875478-k5dn6"
Sep 10 17:41:11.743325 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:41:11.743256    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/719743db-5de0-4ea1-a365-75fca5745d64-manager-config\") pod \"lws-controller-manager-57bb875478-k5dn6\" (UID: \"719743db-5de0-4ea1-a365-75fca5745d64\") " pod="openshift-lws-operator/lws-controller-manager-57bb875478-k5dn6"
Sep 10 17:41:11.743325 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:41:11.743300    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/719743db-5de0-4ea1-a365-75fca5745d64-metrics-cert\") pod \"lws-controller-manager-57bb875478-k5dn6\" (UID: \"719743db-5de0-4ea1-a365-75fca5745d64\") " pod="openshift-lws-operator/lws-controller-manager-57bb875478-k5dn6"
Sep 10 17:41:11.843820 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:41:11.843794    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c62zw\" (UniqueName: \"kubernetes.io/projected/719743db-5de0-4ea1-a365-75fca5745d64-kube-api-access-c62zw\") pod \"lws-controller-manager-57bb875478-k5dn6\" (UID: \"719743db-5de0-4ea1-a365-75fca5745d64\") " pod="openshift-lws-operator/lws-controller-manager-57bb875478-k5dn6"
Sep 10 17:41:11.843934 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:41:11.843833    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/719743db-5de0-4ea1-a365-75fca5745d64-manager-config\") pod \"lws-controller-manager-57bb875478-k5dn6\" (UID: \"719743db-5de0-4ea1-a365-75fca5745d64\") " pod="openshift-lws-operator/lws-controller-manager-57bb875478-k5dn6"
Sep 10 17:41:11.843934 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:41:11.843853    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/719743db-5de0-4ea1-a365-75fca5745d64-metrics-cert\") pod \"lws-controller-manager-57bb875478-k5dn6\" (UID: \"719743db-5de0-4ea1-a365-75fca5745d64\") " pod="openshift-lws-operator/lws-controller-manager-57bb875478-k5dn6"
Sep 10 17:41:11.843934 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:41:11.843882    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/719743db-5de0-4ea1-a365-75fca5745d64-cert\") pod \"lws-controller-manager-57bb875478-k5dn6\" (UID: \"719743db-5de0-4ea1-a365-75fca5745d64\") " pod="openshift-lws-operator/lws-controller-manager-57bb875478-k5dn6"
Sep 10 17:41:11.844519 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:41:11.844497    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/719743db-5de0-4ea1-a365-75fca5745d64-manager-config\") pod \"lws-controller-manager-57bb875478-k5dn6\" (UID: \"719743db-5de0-4ea1-a365-75fca5745d64\") " pod="openshift-lws-operator/lws-controller-manager-57bb875478-k5dn6"
Sep 10 17:41:11.846414 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:41:11.846393    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/719743db-5de0-4ea1-a365-75fca5745d64-metrics-cert\") pod \"lws-controller-manager-57bb875478-k5dn6\" (UID: \"719743db-5de0-4ea1-a365-75fca5745d64\") " pod="openshift-lws-operator/lws-controller-manager-57bb875478-k5dn6"
Sep 10 17:41:11.846503 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:41:11.846431    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/719743db-5de0-4ea1-a365-75fca5745d64-cert\") pod \"lws-controller-manager-57bb875478-k5dn6\" (UID: \"719743db-5de0-4ea1-a365-75fca5745d64\") " pod="openshift-lws-operator/lws-controller-manager-57bb875478-k5dn6"
Sep 10 17:41:11.853922 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:41:11.853900    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c62zw\" (UniqueName: \"kubernetes.io/projected/719743db-5de0-4ea1-a365-75fca5745d64-kube-api-access-c62zw\") pod \"lws-controller-manager-57bb875478-k5dn6\" (UID: \"719743db-5de0-4ea1-a365-75fca5745d64\") " pod="openshift-lws-operator/lws-controller-manager-57bb875478-k5dn6"
Sep 10 17:41:12.019544 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:41:12.019469    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-57bb875478-k5dn6"
Sep 10 17:41:12.299100 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:41:12.299011    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-57bb875478-k5dn6"]
Sep 10 17:41:12.301966 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:41:12.301937    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod719743db_5de0_4ea1_a365_75fca5745d64.slice/crio-3e43af70e573fba75ae1f5580ab0da7cf8c1c5562193e8165464d934fbffdba3 WatchSource:0}: Error finding container 3e43af70e573fba75ae1f5580ab0da7cf8c1c5562193e8165464d934fbffdba3: Status 404 returned error can't find the container with id 3e43af70e573fba75ae1f5580ab0da7cf8c1c5562193e8165464d934fbffdba3
Sep 10 17:41:12.303897 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:41:12.303882    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 17:41:13.128268 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:41:13.128205    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-57bb875478-k5dn6" event={"ID":"719743db-5de0-4ea1-a365-75fca5745d64","Type":"ContainerStarted","Data":"3e43af70e573fba75ae1f5580ab0da7cf8c1c5562193e8165464d934fbffdba3"}
Sep 10 17:41:15.138821 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:41:15.138790    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-57bb875478-k5dn6" event={"ID":"719743db-5de0-4ea1-a365-75fca5745d64","Type":"ContainerStarted","Data":"99ff783fd2c2fe5181f12717b42648e5bdf380783179bdf49e4512c4992bb301"}
Sep 10 17:41:15.139195 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:41:15.138923    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-57bb875478-k5dn6"
Sep 10 17:41:15.158881 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:41:15.158770    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-57bb875478-k5dn6" podStartSLOduration=1.5567142299999999 podStartE2EDuration="4.158755347s" totalImagesPullingTime="2.602041117s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:41:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:41:15.157889564 +0000 UTC m=+319.157719632" watchObservedRunningTime="2026-09-10 17:41:15.158755347 +0000 UTC m=+319.158585414"
Sep 10 17:41:26.145319 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:41:26.145288    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-57bb875478-k5dn6"
Sep 10 17:42:22.201458 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:42:22.201421    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4nvzt"]
Sep 10 17:42:22.204681 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:42:22.204664    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4nvzt"
Sep 10 17:42:22.211311 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:42:22.211295    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-85c54\""
Sep 10 17:42:22.211379 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:42:22.211350    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 10 17:42:22.212437 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:42:22.212422    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 10 17:42:22.223634 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:42:22.223613    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4nvzt"]
Sep 10 17:42:22.321294 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:42:22.321255    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b4e06d5c-8f4d-426b-be4f-5a8f5a038b8a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-4nvzt\" (UID: \"b4e06d5c-8f4d-426b-be4f-5a8f5a038b8a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4nvzt"
Sep 10 17:42:22.321445 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:42:22.321307    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qxzn\" (UniqueName: \"kubernetes.io/projected/b4e06d5c-8f4d-426b-be4f-5a8f5a038b8a-kube-api-access-5qxzn\") pod \"kuadrant-operator-controller-manager-76f8f686bf-4nvzt\" (UID: \"b4e06d5c-8f4d-426b-be4f-5a8f5a038b8a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4nvzt"
Sep 10 17:42:22.422814 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:42:22.422773    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b4e06d5c-8f4d-426b-be4f-5a8f5a038b8a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-4nvzt\" (UID: \"b4e06d5c-8f4d-426b-be4f-5a8f5a038b8a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4nvzt"
Sep 10 17:42:22.422972 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:42:22.422822    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qxzn\" (UniqueName: \"kubernetes.io/projected/b4e06d5c-8f4d-426b-be4f-5a8f5a038b8a-kube-api-access-5qxzn\") pod \"kuadrant-operator-controller-manager-76f8f686bf-4nvzt\" (UID: \"b4e06d5c-8f4d-426b-be4f-5a8f5a038b8a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4nvzt"
Sep 10 17:42:22.423166 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:42:22.423143    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b4e06d5c-8f4d-426b-be4f-5a8f5a038b8a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-4nvzt\" (UID: \"b4e06d5c-8f4d-426b-be4f-5a8f5a038b8a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4nvzt"
Sep 10 17:42:22.446906 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:42:22.446871    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qxzn\" (UniqueName: \"kubernetes.io/projected/b4e06d5c-8f4d-426b-be4f-5a8f5a038b8a-kube-api-access-5qxzn\") pod \"kuadrant-operator-controller-manager-76f8f686bf-4nvzt\" (UID: \"b4e06d5c-8f4d-426b-be4f-5a8f5a038b8a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4nvzt"
Sep 10 17:42:22.514493 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:42:22.514387    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4nvzt"
Sep 10 17:42:22.798639 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:42:22.798611    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4e06d5c_8f4d_426b_be4f_5a8f5a038b8a.slice/crio-29cd933f9b64bc49ca57606130724406322c8c30ba4dd822897361f0961fa70c WatchSource:0}: Error finding container 29cd933f9b64bc49ca57606130724406322c8c30ba4dd822897361f0961fa70c: Status 404 returned error can't find the container with id 29cd933f9b64bc49ca57606130724406322c8c30ba4dd822897361f0961fa70c
Sep 10 17:42:22.799512 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:42:22.799490    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4nvzt"]
Sep 10 17:42:23.446073 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:42:23.446038    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4nvzt" event={"ID":"b4e06d5c-8f4d-426b-be4f-5a8f5a038b8a","Type":"ContainerStarted","Data":"29cd933f9b64bc49ca57606130724406322c8c30ba4dd822897361f0961fa70c"}
Sep 10 17:42:27.467802 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:42:27.467759    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4nvzt" event={"ID":"b4e06d5c-8f4d-426b-be4f-5a8f5a038b8a","Type":"ContainerStarted","Data":"78e06b973374a4ea17040e73e12f034b6d4e22ce0498749d053003fd3e85e842"}
Sep 10 17:42:27.468252 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:42:27.467883    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4nvzt"
Sep 10 17:42:27.511872 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:42:27.511809    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4nvzt" podStartSLOduration=1.592634983 podStartE2EDuration="5.511794469s" totalImagesPullingTime="3.919159486s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:42:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:42:27.509610615 +0000 UTC m=+391.509440687" watchObservedRunningTime="2026-09-10 17:42:27.511794469 +0000 UTC m=+391.511624536"
Sep 10 17:42:38.475462 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:42:38.475429    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4nvzt"
Sep 10 17:42:40.845348 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:42:40.845308    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4nvzt"]
Sep 10 17:42:40.845780 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:42:40.845510    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4nvzt" podUID="b4e06d5c-8f4d-426b-be4f-5a8f5a038b8a" containerName="manager" containerID="cri-o://78e06b973374a4ea17040e73e12f034b6d4e22ce0498749d053003fd3e85e842" gracePeriod=2
Sep 10 17:42:40.859595 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:42:40.859542    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4nvzt"]
Sep 10 17:42:40.875603 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:42:40.875542    2579 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"b4e06d5c-8f4d-426b-be4f-5a8f5a038b8a\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"kuadrant-operator-controller-manager-76f8f686bf-4nvzt\": pods \"kuadrant-operator-controller-manager-76f8f686bf-4nvzt\" not found" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4nvzt"
Sep 10 17:42:40.886725 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:42:40.886686    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-hf9bs"]
Sep 10 17:42:40.887096 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:42:40.887073    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4e06d5c-8f4d-426b-be4f-5a8f5a038b8a" containerName="manager"
Sep 10 17:42:40.887096 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:42:40.887097    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4e06d5c-8f4d-426b-be4f-5a8f5a038b8a" containerName="manager"
Sep 10 17:42:40.887247 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:42:40.887171    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="b4e06d5c-8f4d-426b-be4f-5a8f5a038b8a" containerName="manager"
Sep 10 17:42:40.890125 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:42:40.890103    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-hf9bs"
Sep 10 17:42:40.894164 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:42:40.894131    2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-4nvzt\" is forbidden: User \"system:node:ip-10-0-129-122.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-122.ec2.internal' and this object" podUID="b4e06d5c-8f4d-426b-be4f-5a8f5a038b8a" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4nvzt"
Sep 10 17:42:40.916106 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:42:40.916073    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-hf9bs"]
Sep 10 17:42:40.984536 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:42:40.984503    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ff33bd78-f06c-424f-af64-8b5c23705f29-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-hf9bs\" (UID: \"ff33bd78-f06c-424f-af64-8b5c23705f29\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-hf9bs"
Sep 10 17:42:40.984720 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:42:40.984545    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nsstp\" (UniqueName: \"kubernetes.io/projected/ff33bd78-f06c-424f-af64-8b5c23705f29-kube-api-access-nsstp\") pod \"kuadrant-operator-controller-manager-76f8f686bf-hf9bs\" (UID: \"ff33bd78-f06c-424f-af64-8b5c23705f29\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-hf9bs"
Sep 10 17:42:41.085179 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:42:41.085148    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ff33bd78-f06c-424f-af64-8b5c23705f29-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-hf9bs\" (UID: \"ff33bd78-f06c-424f-af64-8b5c23705f29\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-hf9bs"
Sep 10 17:42:41.085329 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:42:41.085195    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nsstp\" (UniqueName: \"kubernetes.io/projected/ff33bd78-f06c-424f-af64-8b5c23705f29-kube-api-access-nsstp\") pod \"kuadrant-operator-controller-manager-76f8f686bf-hf9bs\" (UID: \"ff33bd78-f06c-424f-af64-8b5c23705f29\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-hf9bs"
Sep 10 17:42:41.085532 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:42:41.085510    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ff33bd78-f06c-424f-af64-8b5c23705f29-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-hf9bs\" (UID: \"ff33bd78-f06c-424f-af64-8b5c23705f29\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-hf9bs"
Sep 10 17:42:41.088433 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:42:41.088415    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4nvzt"
Sep 10 17:42:41.090771 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:42:41.090738    2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-4nvzt\" is forbidden: User \"system:node:ip-10-0-129-122.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-122.ec2.internal' and this object" podUID="b4e06d5c-8f4d-426b-be4f-5a8f5a038b8a" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4nvzt"
Sep 10 17:42:41.094394 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:42:41.094370    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nsstp\" (UniqueName: \"kubernetes.io/projected/ff33bd78-f06c-424f-af64-8b5c23705f29-kube-api-access-nsstp\") pod \"kuadrant-operator-controller-manager-76f8f686bf-hf9bs\" (UID: \"ff33bd78-f06c-424f-af64-8b5c23705f29\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-hf9bs"
Sep 10 17:42:41.186042 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:42:41.186007    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b4e06d5c-8f4d-426b-be4f-5a8f5a038b8a-extensions-socket-volume\") pod \"b4e06d5c-8f4d-426b-be4f-5a8f5a038b8a\" (UID: \"b4e06d5c-8f4d-426b-be4f-5a8f5a038b8a\") "
Sep 10 17:42:41.186208 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:42:41.186097    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5qxzn\" (UniqueName: \"kubernetes.io/projected/b4e06d5c-8f4d-426b-be4f-5a8f5a038b8a-kube-api-access-5qxzn\") pod \"b4e06d5c-8f4d-426b-be4f-5a8f5a038b8a\" (UID: \"b4e06d5c-8f4d-426b-be4f-5a8f5a038b8a\") "
Sep 10 17:42:41.186470 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:42:41.186441    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4e06d5c-8f4d-426b-be4f-5a8f5a038b8a-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "b4e06d5c-8f4d-426b-be4f-5a8f5a038b8a" (UID: "b4e06d5c-8f4d-426b-be4f-5a8f5a038b8a"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:42:41.188432 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:42:41.188405    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4e06d5c-8f4d-426b-be4f-5a8f5a038b8a-kube-api-access-5qxzn" (OuterVolumeSpecName: "kube-api-access-5qxzn") pod "b4e06d5c-8f4d-426b-be4f-5a8f5a038b8a" (UID: "b4e06d5c-8f4d-426b-be4f-5a8f5a038b8a"). InnerVolumeSpecName "kube-api-access-5qxzn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:42:41.227386 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:42:41.227344    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-hf9bs"
Sep 10 17:42:41.287346 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:42:41.287288    2579 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b4e06d5c-8f4d-426b-be4f-5a8f5a038b8a-extensions-socket-volume\") on node \"ip-10-0-129-122.ec2.internal\" DevicePath \"\""
Sep 10 17:42:41.287346 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:42:41.287325    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5qxzn\" (UniqueName: \"kubernetes.io/projected/b4e06d5c-8f4d-426b-be4f-5a8f5a038b8a-kube-api-access-5qxzn\") on node \"ip-10-0-129-122.ec2.internal\" DevicePath \"\""
Sep 10 17:42:41.516440 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:42:41.516417    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-hf9bs"]
Sep 10 17:42:41.518937 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:42:41.518909    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff33bd78_f06c_424f_af64_8b5c23705f29.slice/crio-eb42747a8fc11bbc8289663535a4d32596dfb1789ab587606870ea71d7e71061 WatchSource:0}: Error finding container eb42747a8fc11bbc8289663535a4d32596dfb1789ab587606870ea71d7e71061: Status 404 returned error can't find the container with id eb42747a8fc11bbc8289663535a4d32596dfb1789ab587606870ea71d7e71061
Sep 10 17:42:41.532029 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:42:41.531997    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-hf9bs" event={"ID":"ff33bd78-f06c-424f-af64-8b5c23705f29","Type":"ContainerStarted","Data":"eb42747a8fc11bbc8289663535a4d32596dfb1789ab587606870ea71d7e71061"}
Sep 10 17:42:41.533401 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:42:41.533373    2579 generic.go:356] "Generic (PLEG): container finished" podID="b4e06d5c-8f4d-426b-be4f-5a8f5a038b8a" containerID="78e06b973374a4ea17040e73e12f034b6d4e22ce0498749d053003fd3e85e842" exitCode=0
Sep 10 17:42:41.533509 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:42:41.533423    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4nvzt"
Sep 10 17:42:41.533509 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:42:41.533439    2579 scope.go:117] "RemoveContainer" containerID="78e06b973374a4ea17040e73e12f034b6d4e22ce0498749d053003fd3e85e842"
Sep 10 17:42:41.535974 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:42:41.535952    2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-4nvzt\" is forbidden: User \"system:node:ip-10-0-129-122.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-122.ec2.internal' and this object" podUID="b4e06d5c-8f4d-426b-be4f-5a8f5a038b8a" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4nvzt"
Sep 10 17:42:41.543354 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:42:41.543324    2579 scope.go:117] "RemoveContainer" containerID="78e06b973374a4ea17040e73e12f034b6d4e22ce0498749d053003fd3e85e842"
Sep 10 17:42:41.543705 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:42:41.543683    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78e06b973374a4ea17040e73e12f034b6d4e22ce0498749d053003fd3e85e842\": container with ID starting with 78e06b973374a4ea17040e73e12f034b6d4e22ce0498749d053003fd3e85e842 not found: ID does not exist" containerID="78e06b973374a4ea17040e73e12f034b6d4e22ce0498749d053003fd3e85e842"
Sep 10 17:42:41.543777 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:42:41.543714    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78e06b973374a4ea17040e73e12f034b6d4e22ce0498749d053003fd3e85e842"} err="failed to get container status \"78e06b973374a4ea17040e73e12f034b6d4e22ce0498749d053003fd3e85e842\": rpc error: code = NotFound desc = could not find container \"78e06b973374a4ea17040e73e12f034b6d4e22ce0498749d053003fd3e85e842\": container with ID starting with 78e06b973374a4ea17040e73e12f034b6d4e22ce0498749d053003fd3e85e842 not found: ID does not exist"
Sep 10 17:42:41.565676 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:42:41.565651    2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-4nvzt\" is forbidden: User \"system:node:ip-10-0-129-122.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-122.ec2.internal' and this object" podUID="b4e06d5c-8f4d-426b-be4f-5a8f5a038b8a" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4nvzt"
Sep 10 17:42:42.216751 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:42:42.216717    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-2m796"]
Sep 10 17:42:42.219919 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:42:42.219903    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-2m796"
Sep 10 17:42:42.237618 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:42:42.237592    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-2m796"]
Sep 10 17:42:42.259208 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:42:42.259175    2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-4nvzt\" is forbidden: User \"system:node:ip-10-0-129-122.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-122.ec2.internal' and this object" podUID="b4e06d5c-8f4d-426b-be4f-5a8f5a038b8a" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4nvzt"
Sep 10 17:42:42.396684 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:42:42.396637    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/14e9bbf5-88d2-4de8-a033-cebd387ee6b7-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-2m796\" (UID: \"14e9bbf5-88d2-4de8-a033-cebd387ee6b7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-2m796"
Sep 10 17:42:42.396684 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:42:42.396684    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fnt9n\" (UniqueName: \"kubernetes.io/projected/14e9bbf5-88d2-4de8-a033-cebd387ee6b7-kube-api-access-fnt9n\") pod \"kuadrant-operator-controller-manager-5769f99d9f-2m796\" (UID: \"14e9bbf5-88d2-4de8-a033-cebd387ee6b7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-2m796"
Sep 10 17:42:42.474644 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:42:42.474528    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b4e06d5c-8f4d-426b-be4f-5a8f5a038b8a" path="/var/lib/kubelet/pods/b4e06d5c-8f4d-426b-be4f-5a8f5a038b8a/volumes"
Sep 10 17:42:42.497677 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:42:42.497639    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/14e9bbf5-88d2-4de8-a033-cebd387ee6b7-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-2m796\" (UID: \"14e9bbf5-88d2-4de8-a033-cebd387ee6b7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-2m796"
Sep 10 17:42:42.497677 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:42:42.497677    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnt9n\" (UniqueName: \"kubernetes.io/projected/14e9bbf5-88d2-4de8-a033-cebd387ee6b7-kube-api-access-fnt9n\") pod \"kuadrant-operator-controller-manager-5769f99d9f-2m796\" (UID: \"14e9bbf5-88d2-4de8-a033-cebd387ee6b7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-2m796"
Sep 10 17:42:42.498057 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:42:42.498037    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/14e9bbf5-88d2-4de8-a033-cebd387ee6b7-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-2m796\" (UID: \"14e9bbf5-88d2-4de8-a033-cebd387ee6b7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-2m796"
Sep 10 17:42:42.512956 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:42:42.512926    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fnt9n\" (UniqueName: \"kubernetes.io/projected/14e9bbf5-88d2-4de8-a033-cebd387ee6b7-kube-api-access-fnt9n\") pod \"kuadrant-operator-controller-manager-5769f99d9f-2m796\" (UID: \"14e9bbf5-88d2-4de8-a033-cebd387ee6b7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-2m796"
Sep 10 17:42:42.530399 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:42:42.530370    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-2m796"
Sep 10 17:42:42.540256 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:42:42.540223    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-hf9bs" event={"ID":"ff33bd78-f06c-424f-af64-8b5c23705f29","Type":"ContainerStarted","Data":"8e67290d5bde523702ab1f48ea5755fdbc9501736f76601001fbde2cab17586a"}
Sep 10 17:42:42.540375 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:42:42.540326    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-hf9bs"
Sep 10 17:42:42.585613 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:42:42.585272    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-hf9bs" podStartSLOduration=2.5852480829999998 podStartE2EDuration="2.585248083s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:42:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:42:42.584374045 +0000 UTC m=+406.584204151" watchObservedRunningTime="2026-09-10 17:42:42.585248083 +0000 UTC m=+406.585078150"
Sep 10 17:42:42.810745 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:42:42.810659    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-2m796"]
Sep 10 17:42:42.814335 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:42:42.814297    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14e9bbf5_88d2_4de8_a033_cebd387ee6b7.slice/crio-657e978ee644b4d10cf47bc81fe3175a95fa91c4a19ea24c13a029ddfe0f6d87 WatchSource:0}: Error finding container 657e978ee644b4d10cf47bc81fe3175a95fa91c4a19ea24c13a029ddfe0f6d87: Status 404 returned error can't find the container with id 657e978ee644b4d10cf47bc81fe3175a95fa91c4a19ea24c13a029ddfe0f6d87
Sep 10 17:42:43.548172 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:42:43.548135    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-2m796" event={"ID":"14e9bbf5-88d2-4de8-a033-cebd387ee6b7","Type":"ContainerStarted","Data":"f70116414612aad2d72e73d9596438dd8a47ff803eeac0f85d038636dbe147be"}
Sep 10 17:42:43.548549 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:42:43.548193    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-2m796" event={"ID":"14e9bbf5-88d2-4de8-a033-cebd387ee6b7","Type":"ContainerStarted","Data":"657e978ee644b4d10cf47bc81fe3175a95fa91c4a19ea24c13a029ddfe0f6d87"}
Sep 10 17:42:43.548549 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:42:43.548326    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-2m796"
Sep 10 17:42:43.595348 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:42:43.595290    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-2m796" podStartSLOduration=1.595272747 podStartE2EDuration="1.595272747s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:42:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:42:43.593730198 +0000 UTC m=+407.593560271" watchObservedRunningTime="2026-09-10 17:42:43.595272747 +0000 UTC m=+407.595102813"
Sep 10 17:42:53.550987 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:42:53.550951    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-hf9bs"
Sep 10 17:42:54.555372 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:42:54.555338    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-2m796"
Sep 10 17:42:54.630068 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:42:54.630036    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-hf9bs"]
Sep 10 17:42:54.630260 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:42:54.630241    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-hf9bs" podUID="ff33bd78-f06c-424f-af64-8b5c23705f29" containerName="manager" containerID="cri-o://8e67290d5bde523702ab1f48ea5755fdbc9501736f76601001fbde2cab17586a" gracePeriod=10
Sep 10 17:42:54.882911 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:42:54.882886    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-hf9bs"
Sep 10 17:42:54.902199 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:42:54.902169    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ff33bd78-f06c-424f-af64-8b5c23705f29-extensions-socket-volume\") pod \"ff33bd78-f06c-424f-af64-8b5c23705f29\" (UID: \"ff33bd78-f06c-424f-af64-8b5c23705f29\") "
Sep 10 17:42:54.902363 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:42:54.902293    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nsstp\" (UniqueName: \"kubernetes.io/projected/ff33bd78-f06c-424f-af64-8b5c23705f29-kube-api-access-nsstp\") pod \"ff33bd78-f06c-424f-af64-8b5c23705f29\" (UID: \"ff33bd78-f06c-424f-af64-8b5c23705f29\") "
Sep 10 17:42:54.902614 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:42:54.902586    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff33bd78-f06c-424f-af64-8b5c23705f29-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "ff33bd78-f06c-424f-af64-8b5c23705f29" (UID: "ff33bd78-f06c-424f-af64-8b5c23705f29"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:42:54.904471 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:42:54.904451    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff33bd78-f06c-424f-af64-8b5c23705f29-kube-api-access-nsstp" (OuterVolumeSpecName: "kube-api-access-nsstp") pod "ff33bd78-f06c-424f-af64-8b5c23705f29" (UID: "ff33bd78-f06c-424f-af64-8b5c23705f29"). InnerVolumeSpecName "kube-api-access-nsstp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:42:55.003127 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:42:55.003096    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nsstp\" (UniqueName: \"kubernetes.io/projected/ff33bd78-f06c-424f-af64-8b5c23705f29-kube-api-access-nsstp\") on node \"ip-10-0-129-122.ec2.internal\" DevicePath \"\""
Sep 10 17:42:55.003127 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:42:55.003125    2579 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ff33bd78-f06c-424f-af64-8b5c23705f29-extensions-socket-volume\") on node \"ip-10-0-129-122.ec2.internal\" DevicePath \"\""
Sep 10 17:42:55.613581 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:42:55.613526    2579 generic.go:356] "Generic (PLEG): container finished" podID="ff33bd78-f06c-424f-af64-8b5c23705f29" containerID="8e67290d5bde523702ab1f48ea5755fdbc9501736f76601001fbde2cab17586a" exitCode=0
Sep 10 17:42:55.613995 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:42:55.613591    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-hf9bs"
Sep 10 17:42:55.613995 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:42:55.613615    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-hf9bs" event={"ID":"ff33bd78-f06c-424f-af64-8b5c23705f29","Type":"ContainerDied","Data":"8e67290d5bde523702ab1f48ea5755fdbc9501736f76601001fbde2cab17586a"}
Sep 10 17:42:55.613995 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:42:55.613676    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-hf9bs" event={"ID":"ff33bd78-f06c-424f-af64-8b5c23705f29","Type":"ContainerDied","Data":"eb42747a8fc11bbc8289663535a4d32596dfb1789ab587606870ea71d7e71061"}
Sep 10 17:42:55.613995 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:42:55.613700    2579 scope.go:117] "RemoveContainer" containerID="8e67290d5bde523702ab1f48ea5755fdbc9501736f76601001fbde2cab17586a"
Sep 10 17:42:55.623510 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:42:55.623492    2579 scope.go:117] "RemoveContainer" containerID="8e67290d5bde523702ab1f48ea5755fdbc9501736f76601001fbde2cab17586a"
Sep 10 17:42:55.623848 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:42:55.623820    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e67290d5bde523702ab1f48ea5755fdbc9501736f76601001fbde2cab17586a\": container with ID starting with 8e67290d5bde523702ab1f48ea5755fdbc9501736f76601001fbde2cab17586a not found: ID does not exist" containerID="8e67290d5bde523702ab1f48ea5755fdbc9501736f76601001fbde2cab17586a"
Sep 10 17:42:55.623918 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:42:55.623854    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e67290d5bde523702ab1f48ea5755fdbc9501736f76601001fbde2cab17586a"} err="failed to get container status \"8e67290d5bde523702ab1f48ea5755fdbc9501736f76601001fbde2cab17586a\": rpc error: code = NotFound desc = could not find container \"8e67290d5bde523702ab1f48ea5755fdbc9501736f76601001fbde2cab17586a\": container with ID starting with 8e67290d5bde523702ab1f48ea5755fdbc9501736f76601001fbde2cab17586a not found: ID does not exist"
Sep 10 17:42:55.638004 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:42:55.637976    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-hf9bs"]
Sep 10 17:42:55.644584 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:42:55.644534    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-hf9bs"]
Sep 10 17:42:56.474921 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:42:56.474894    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ff33bd78-f06c-424f-af64-8b5c23705f29" path="/var/lib/kubelet/pods/ff33bd78-f06c-424f-af64-8b5c23705f29/volumes"
Sep 10 17:43:12.920785 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:43:12.920751    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-d97b4"]
Sep 10 17:43:12.921142 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:43:12.921110    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff33bd78-f06c-424f-af64-8b5c23705f29" containerName="manager"
Sep 10 17:43:12.921142 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:43:12.921123    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff33bd78-f06c-424f-af64-8b5c23705f29" containerName="manager"
Sep 10 17:43:12.921214 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:43:12.921182    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="ff33bd78-f06c-424f-af64-8b5c23705f29" containerName="manager"
Sep 10 17:43:12.924855 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:43:12.924835    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-d97b4"
Sep 10 17:43:12.927458 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:43:12.927432    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-jrtvh\""
Sep 10 17:43:12.927601 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:43:12.927437    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 10 17:43:12.934038 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:43:12.934015    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-d97b4"]
Sep 10 17:43:13.017528 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:43:13.017492    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-d97b4"]
Sep 10 17:43:13.050322 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:43:13.050289    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/d125c989-71c8-46c7-a568-6c64f74624bf-config-file\") pod \"limitador-limitador-8588749977-d97b4\" (UID: \"d125c989-71c8-46c7-a568-6c64f74624bf\") " pod="kuadrant-system/limitador-limitador-8588749977-d97b4"
Sep 10 17:43:13.050479 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:43:13.050389    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rfv9b\" (UniqueName: \"kubernetes.io/projected/d125c989-71c8-46c7-a568-6c64f74624bf-kube-api-access-rfv9b\") pod \"limitador-limitador-8588749977-d97b4\" (UID: \"d125c989-71c8-46c7-a568-6c64f74624bf\") " pod="kuadrant-system/limitador-limitador-8588749977-d97b4"
Sep 10 17:43:13.151208 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:43:13.151178    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfv9b\" (UniqueName: \"kubernetes.io/projected/d125c989-71c8-46c7-a568-6c64f74624bf-kube-api-access-rfv9b\") pod \"limitador-limitador-8588749977-d97b4\" (UID: \"d125c989-71c8-46c7-a568-6c64f74624bf\") " pod="kuadrant-system/limitador-limitador-8588749977-d97b4"
Sep 10 17:43:13.151461 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:43:13.151241    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/d125c989-71c8-46c7-a568-6c64f74624bf-config-file\") pod \"limitador-limitador-8588749977-d97b4\" (UID: \"d125c989-71c8-46c7-a568-6c64f74624bf\") " pod="kuadrant-system/limitador-limitador-8588749977-d97b4"
Sep 10 17:43:13.151882 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:43:13.151860    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/d125c989-71c8-46c7-a568-6c64f74624bf-config-file\") pod \"limitador-limitador-8588749977-d97b4\" (UID: \"d125c989-71c8-46c7-a568-6c64f74624bf\") " pod="kuadrant-system/limitador-limitador-8588749977-d97b4"
Sep 10 17:43:13.168133 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:43:13.168107    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfv9b\" (UniqueName: \"kubernetes.io/projected/d125c989-71c8-46c7-a568-6c64f74624bf-kube-api-access-rfv9b\") pod \"limitador-limitador-8588749977-d97b4\" (UID: \"d125c989-71c8-46c7-a568-6c64f74624bf\") " pod="kuadrant-system/limitador-limitador-8588749977-d97b4"
Sep 10 17:43:13.236218 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:43:13.236143    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-d97b4"
Sep 10 17:43:13.523145 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:43:13.523119    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-d97b4"]
Sep 10 17:43:13.525793 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:43:13.525745    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd125c989_71c8_46c7_a568_6c64f74624bf.slice/crio-c6d19d5f420df55d1740904f2d3725f67e13e2f3dc7f60267f5145e61b09f0ae WatchSource:0}: Error finding container c6d19d5f420df55d1740904f2d3725f67e13e2f3dc7f60267f5145e61b09f0ae: Status 404 returned error can't find the container with id c6d19d5f420df55d1740904f2d3725f67e13e2f3dc7f60267f5145e61b09f0ae
Sep 10 17:43:13.690101 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:43:13.690071    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-d97b4" event={"ID":"d125c989-71c8-46c7-a568-6c64f74624bf","Type":"ContainerStarted","Data":"c6d19d5f420df55d1740904f2d3725f67e13e2f3dc7f60267f5145e61b09f0ae"}
Sep 10 17:43:14.193054 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:43:14.193024    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-59c974bd48-bdh7g"]
Sep 10 17:43:14.196043 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:43:14.196023    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-bdh7g"
Sep 10 17:43:14.198475 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:43:14.198456    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-dhx2d\""
Sep 10 17:43:14.202588 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:43:14.202549    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-bdh7g"]
Sep 10 17:43:14.260778 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:43:14.260743    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkzw6\" (UniqueName: \"kubernetes.io/projected/1ccace62-5280-42be-8164-48131433f066-kube-api-access-xkzw6\") pod \"authorino-59c974bd48-bdh7g\" (UID: \"1ccace62-5280-42be-8164-48131433f066\") " pod="kuadrant-system/authorino-59c974bd48-bdh7g"
Sep 10 17:43:14.361367 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:43:14.361328    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkzw6\" (UniqueName: \"kubernetes.io/projected/1ccace62-5280-42be-8164-48131433f066-kube-api-access-xkzw6\") pod \"authorino-59c974bd48-bdh7g\" (UID: \"1ccace62-5280-42be-8164-48131433f066\") " pod="kuadrant-system/authorino-59c974bd48-bdh7g"
Sep 10 17:43:14.370458 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:43:14.370433    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkzw6\" (UniqueName: \"kubernetes.io/projected/1ccace62-5280-42be-8164-48131433f066-kube-api-access-xkzw6\") pod \"authorino-59c974bd48-bdh7g\" (UID: \"1ccace62-5280-42be-8164-48131433f066\") " pod="kuadrant-system/authorino-59c974bd48-bdh7g"
Sep 10 17:43:14.506515 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:43:14.506452    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-bdh7g"
Sep 10 17:43:14.914977 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:43:14.914948    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-bdh7g"]
Sep 10 17:43:14.917915 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:43:14.917885    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1ccace62_5280_42be_8164_48131433f066.slice/crio-d40c59f7ed2fc0094456b817ceda3a35cc5fba1bd9e877805ee97a170a24d2cc WatchSource:0}: Error finding container d40c59f7ed2fc0094456b817ceda3a35cc5fba1bd9e877805ee97a170a24d2cc: Status 404 returned error can't find the container with id d40c59f7ed2fc0094456b817ceda3a35cc5fba1bd9e877805ee97a170a24d2cc
Sep 10 17:43:15.702356 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:43:15.702311    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-bdh7g" event={"ID":"1ccace62-5280-42be-8164-48131433f066","Type":"ContainerStarted","Data":"d40c59f7ed2fc0094456b817ceda3a35cc5fba1bd9e877805ee97a170a24d2cc"}
Sep 10 17:43:15.704083 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:43:15.704059    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-d97b4" event={"ID":"d125c989-71c8-46c7-a568-6c64f74624bf","Type":"ContainerStarted","Data":"57795eea1b204716a03325cb6130cd3181bfe70986a319310cc16c2de142263c"}
Sep 10 17:43:15.704215 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:43:15.704191    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-8588749977-d97b4"
Sep 10 17:43:15.736520 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:43:15.736470    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-8588749977-d97b4" podStartSLOduration=2.584839337 podStartE2EDuration="3.736450344s" totalImagesPullingTime="1.151611007s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:43:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:43:15.734868005 +0000 UTC m=+439.734698073" watchObservedRunningTime="2026-09-10 17:43:15.736450344 +0000 UTC m=+439.736280412"
Sep 10 17:43:16.710770 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:43:16.710733    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-bdh7g" event={"ID":"1ccace62-5280-42be-8164-48131433f066","Type":"ContainerStarted","Data":"91ad5dd3d9855ca18f7bc19e3f438b266a3d93ce059e20b30586356b1fa3607e"}
Sep 10 17:43:16.727625 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:43:16.727557    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-59c974bd48-bdh7g" podStartSLOduration=1.362742935 podStartE2EDuration="2.727541887s" totalImagesPullingTime="1.364798952s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:43:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:43:16.72615236 +0000 UTC m=+440.725982427" watchObservedRunningTime="2026-09-10 17:43:16.727541887 +0000 UTC m=+440.727371953"
Sep 10 17:43:26.711783 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:43:26.711708    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-8588749977-d97b4"
Sep 10 17:43:26.941155 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:43:26.941118    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-d97b4"]
Sep 10 17:43:26.941369 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:43:26.941332    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-8588749977-d97b4" podUID="d125c989-71c8-46c7-a568-6c64f74624bf" containerName="limitador" containerID="cri-o://57795eea1b204716a03325cb6130cd3181bfe70986a319310cc16c2de142263c" gracePeriod=30
Sep 10 17:43:27.485517 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:43:27.485495    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-d97b4"
Sep 10 17:43:27.568083 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:43:27.568019    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rfv9b\" (UniqueName: \"kubernetes.io/projected/d125c989-71c8-46c7-a568-6c64f74624bf-kube-api-access-rfv9b\") pod \"d125c989-71c8-46c7-a568-6c64f74624bf\" (UID: \"d125c989-71c8-46c7-a568-6c64f74624bf\") "
Sep 10 17:43:27.568083 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:43:27.568060    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/d125c989-71c8-46c7-a568-6c64f74624bf-config-file\") pod \"d125c989-71c8-46c7-a568-6c64f74624bf\" (UID: \"d125c989-71c8-46c7-a568-6c64f74624bf\") "
Sep 10 17:43:27.568423 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:43:27.568404    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d125c989-71c8-46c7-a568-6c64f74624bf-config-file" (OuterVolumeSpecName: "config-file") pod "d125c989-71c8-46c7-a568-6c64f74624bf" (UID: "d125c989-71c8-46c7-a568-6c64f74624bf"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 17:43:27.570215 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:43:27.570182    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d125c989-71c8-46c7-a568-6c64f74624bf-kube-api-access-rfv9b" (OuterVolumeSpecName: "kube-api-access-rfv9b") pod "d125c989-71c8-46c7-a568-6c64f74624bf" (UID: "d125c989-71c8-46c7-a568-6c64f74624bf"). InnerVolumeSpecName "kube-api-access-rfv9b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:43:27.668765 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:43:27.668741    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rfv9b\" (UniqueName: \"kubernetes.io/projected/d125c989-71c8-46c7-a568-6c64f74624bf-kube-api-access-rfv9b\") on node \"ip-10-0-129-122.ec2.internal\" DevicePath \"\""
Sep 10 17:43:27.668765 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:43:27.668764    2579 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/d125c989-71c8-46c7-a568-6c64f74624bf-config-file\") on node \"ip-10-0-129-122.ec2.internal\" DevicePath \"\""
Sep 10 17:43:27.767603 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:43:27.767554    2579 generic.go:356] "Generic (PLEG): container finished" podID="d125c989-71c8-46c7-a568-6c64f74624bf" containerID="57795eea1b204716a03325cb6130cd3181bfe70986a319310cc16c2de142263c" exitCode=0
Sep 10 17:43:27.767957 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:43:27.767607    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-d97b4"
Sep 10 17:43:27.767957 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:43:27.767633    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-d97b4" event={"ID":"d125c989-71c8-46c7-a568-6c64f74624bf","Type":"ContainerDied","Data":"57795eea1b204716a03325cb6130cd3181bfe70986a319310cc16c2de142263c"}
Sep 10 17:43:27.767957 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:43:27.767672    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-d97b4" event={"ID":"d125c989-71c8-46c7-a568-6c64f74624bf","Type":"ContainerDied","Data":"c6d19d5f420df55d1740904f2d3725f67e13e2f3dc7f60267f5145e61b09f0ae"}
Sep 10 17:43:27.767957 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:43:27.767695    2579 scope.go:117] "RemoveContainer" containerID="57795eea1b204716a03325cb6130cd3181bfe70986a319310cc16c2de142263c"
Sep 10 17:43:27.776513 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:43:27.776497    2579 scope.go:117] "RemoveContainer" containerID="57795eea1b204716a03325cb6130cd3181bfe70986a319310cc16c2de142263c"
Sep 10 17:43:27.776798 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:43:27.776779    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57795eea1b204716a03325cb6130cd3181bfe70986a319310cc16c2de142263c\": container with ID starting with 57795eea1b204716a03325cb6130cd3181bfe70986a319310cc16c2de142263c not found: ID does not exist" containerID="57795eea1b204716a03325cb6130cd3181bfe70986a319310cc16c2de142263c"
Sep 10 17:43:27.776871 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:43:27.776811    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57795eea1b204716a03325cb6130cd3181bfe70986a319310cc16c2de142263c"} err="failed to get container status \"57795eea1b204716a03325cb6130cd3181bfe70986a319310cc16c2de142263c\": rpc error: code = NotFound desc = could not find container \"57795eea1b204716a03325cb6130cd3181bfe70986a319310cc16c2de142263c\": container with ID starting with 57795eea1b204716a03325cb6130cd3181bfe70986a319310cc16c2de142263c not found: ID does not exist"
Sep 10 17:43:27.805721 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:43:27.805695    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-d97b4"]
Sep 10 17:43:27.810126 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:43:27.810105    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-d97b4"]
Sep 10 17:43:28.474753 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:43:28.474714    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d125c989-71c8-46c7-a568-6c64f74624bf" path="/var/lib/kubelet/pods/d125c989-71c8-46c7-a568-6c64f74624bf/volumes"
Sep 10 17:43:29.651989 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:43:29.651951    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-f765s"]
Sep 10 17:43:29.652458 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:43:29.652436    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d125c989-71c8-46c7-a568-6c64f74624bf" containerName="limitador"
Sep 10 17:43:29.652530 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:43:29.652461    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="d125c989-71c8-46c7-a568-6c64f74624bf" containerName="limitador"
Sep 10 17:43:29.652621 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:43:29.652607    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="d125c989-71c8-46c7-a568-6c64f74624bf" containerName="limitador"
Sep 10 17:43:29.656883 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:43:29.656862    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-f765s"
Sep 10 17:43:29.661037 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:43:29.661017    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Sep 10 17:43:29.661186 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:43:29.661170    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 17:43:29.662213 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:43:29.662192    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 10 17:43:29.662314 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:43:29.662239    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-7khj4\""
Sep 10 17:43:29.668441 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:43:29.668419    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-f765s"]
Sep 10 17:43:29.784699 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:43:29.784666    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/bf74a5e6-2847-46e9-8131-f44fad693339-data\") pod \"postgres-69fb4594cf-f765s\" (UID: \"bf74a5e6-2847-46e9-8131-f44fad693339\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-f765s"
Sep 10 17:43:29.784884 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:43:29.784754    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49mjb\" (UniqueName: \"kubernetes.io/projected/bf74a5e6-2847-46e9-8131-f44fad693339-kube-api-access-49mjb\") pod \"postgres-69fb4594cf-f765s\" (UID: \"bf74a5e6-2847-46e9-8131-f44fad693339\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-f765s"
Sep 10 17:43:29.885775 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:43:29.885734    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/bf74a5e6-2847-46e9-8131-f44fad693339-data\") pod \"postgres-69fb4594cf-f765s\" (UID: \"bf74a5e6-2847-46e9-8131-f44fad693339\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-f765s"
Sep 10 17:43:29.885933 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:43:29.885804    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49mjb\" (UniqueName: \"kubernetes.io/projected/bf74a5e6-2847-46e9-8131-f44fad693339-kube-api-access-49mjb\") pod \"postgres-69fb4594cf-f765s\" (UID: \"bf74a5e6-2847-46e9-8131-f44fad693339\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-f765s"
Sep 10 17:43:29.886143 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:43:29.886120    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/bf74a5e6-2847-46e9-8131-f44fad693339-data\") pod \"postgres-69fb4594cf-f765s\" (UID: \"bf74a5e6-2847-46e9-8131-f44fad693339\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-f765s"
Sep 10 17:43:29.899794 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:43:29.899765    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-49mjb\" (UniqueName: \"kubernetes.io/projected/bf74a5e6-2847-46e9-8131-f44fad693339-kube-api-access-49mjb\") pod \"postgres-69fb4594cf-f765s\" (UID: \"bf74a5e6-2847-46e9-8131-f44fad693339\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-f765s"
Sep 10 17:43:29.966702 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:43:29.966643    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-f765s"
Sep 10 17:43:30.240503 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:43:30.240478    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-f765s"]
Sep 10 17:43:30.242679 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:43:30.242646    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf74a5e6_2847_46e9_8131_f44fad693339.slice/crio-99cbf9b7d8f8a41067053f017bcf8c891c668b8a8f09e3e2785b2f8530c3f704 WatchSource:0}: Error finding container 99cbf9b7d8f8a41067053f017bcf8c891c668b8a8f09e3e2785b2f8530c3f704: Status 404 returned error can't find the container with id 99cbf9b7d8f8a41067053f017bcf8c891c668b8a8f09e3e2785b2f8530c3f704
Sep 10 17:43:30.781213 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:43:30.781184    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-f765s" event={"ID":"bf74a5e6-2847-46e9-8131-f44fad693339","Type":"ContainerStarted","Data":"99cbf9b7d8f8a41067053f017bcf8c891c668b8a8f09e3e2785b2f8530c3f704"}
Sep 10 17:43:35.811680 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:43:35.811641    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-f765s" event={"ID":"bf74a5e6-2847-46e9-8131-f44fad693339","Type":"ContainerStarted","Data":"e21a04ee966f17b7cb7f4b5653f3d8f9a96431c7c81023752a909786b57a9d72"}
Sep 10 17:43:35.812135 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:43:35.811752    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-f765s"
Sep 10 17:43:35.831736 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:43:35.831687    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-f765s" podStartSLOduration=2.003529322 podStartE2EDuration="6.831673219s" totalImagesPullingTime="4.828143897s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:43:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:43:35.830299248 +0000 UTC m=+459.830129315" watchObservedRunningTime="2026-09-10 17:43:35.831673219 +0000 UTC m=+459.831503286"
Sep 10 17:43:41.846458 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:43:41.846425    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-f765s"
Sep 10 17:43:42.578363 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:43:42.578331    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-2kncf"]
Sep 10 17:43:42.584207 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:43:42.584188    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-2kncf"
Sep 10 17:43:42.586959 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:43:42.586939    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Sep 10 17:43:42.588331 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:43:42.588299    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-2kncf"]
Sep 10 17:43:42.671875 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:43:42.671851    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-2kncf"]
Sep 10 17:43:42.672072 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:43:42.672055    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-gsbm6 tls-cert], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-6d4dfc649f-2kncf" podUID="a15f9e3e-ed2b-4a4b-975c-53368db46a92"
Sep 10 17:43:42.690627 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:43:42.690592    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a15f9e3e-ed2b-4a4b-975c-53368db46a92-tls-cert\") pod \"authorino-6d4dfc649f-2kncf\" (UID: \"a15f9e3e-ed2b-4a4b-975c-53368db46a92\") " pod="kuadrant-system/authorino-6d4dfc649f-2kncf"
Sep 10 17:43:42.690747 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:43:42.690727    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gsbm6\" (UniqueName: \"kubernetes.io/projected/a15f9e3e-ed2b-4a4b-975c-53368db46a92-kube-api-access-gsbm6\") pod \"authorino-6d4dfc649f-2kncf\" (UID: \"a15f9e3e-ed2b-4a4b-975c-53368db46a92\") " pod="kuadrant-system/authorino-6d4dfc649f-2kncf"
Sep 10 17:43:42.705009 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:43:42.704986    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-9d8b447bf-7dl75"]
Sep 10 17:43:42.709115 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:43:42.709090    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-9d8b447bf-7dl75"
Sep 10 17:43:42.716112 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:43:42.716088    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-9d8b447bf-7dl75"]
Sep 10 17:43:42.791807 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:43:42.791766    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gsbm6\" (UniqueName: \"kubernetes.io/projected/a15f9e3e-ed2b-4a4b-975c-53368db46a92-kube-api-access-gsbm6\") pod \"authorino-6d4dfc649f-2kncf\" (UID: \"a15f9e3e-ed2b-4a4b-975c-53368db46a92\") " pod="kuadrant-system/authorino-6d4dfc649f-2kncf"
Sep 10 17:43:42.791807 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:43:42.791813    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a15f9e3e-ed2b-4a4b-975c-53368db46a92-tls-cert\") pod \"authorino-6d4dfc649f-2kncf\" (UID: \"a15f9e3e-ed2b-4a4b-975c-53368db46a92\") " pod="kuadrant-system/authorino-6d4dfc649f-2kncf"
Sep 10 17:43:42.794382 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:43:42.794360    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a15f9e3e-ed2b-4a4b-975c-53368db46a92-tls-cert\") pod \"authorino-6d4dfc649f-2kncf\" (UID: \"a15f9e3e-ed2b-4a4b-975c-53368db46a92\") " pod="kuadrant-system/authorino-6d4dfc649f-2kncf"
Sep 10 17:43:42.800112 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:43:42.800089    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gsbm6\" (UniqueName: \"kubernetes.io/projected/a15f9e3e-ed2b-4a4b-975c-53368db46a92-kube-api-access-gsbm6\") pod \"authorino-6d4dfc649f-2kncf\" (UID: \"a15f9e3e-ed2b-4a4b-975c-53368db46a92\") " pod="kuadrant-system/authorino-6d4dfc649f-2kncf"
Sep 10 17:43:42.841420 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:43:42.841368    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-2kncf"
Sep 10 17:43:42.848607 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:43:42.848589    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-2kncf"
Sep 10 17:43:42.892277 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:43:42.892258    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/80ecaacc-72d8-42b4-b3d0-f41f6e27391b-tls-cert\") pod \"authorino-9d8b447bf-7dl75\" (UID: \"80ecaacc-72d8-42b4-b3d0-f41f6e27391b\") " pod="kuadrant-system/authorino-9d8b447bf-7dl75"
Sep 10 17:43:42.892377 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:43:42.892294    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8dfw9\" (UniqueName: \"kubernetes.io/projected/80ecaacc-72d8-42b4-b3d0-f41f6e27391b-kube-api-access-8dfw9\") pod \"authorino-9d8b447bf-7dl75\" (UID: \"80ecaacc-72d8-42b4-b3d0-f41f6e27391b\") " pod="kuadrant-system/authorino-9d8b447bf-7dl75"
Sep 10 17:43:42.993505 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:43:42.993483    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gsbm6\" (UniqueName: \"kubernetes.io/projected/a15f9e3e-ed2b-4a4b-975c-53368db46a92-kube-api-access-gsbm6\") pod \"a15f9e3e-ed2b-4a4b-975c-53368db46a92\" (UID: \"a15f9e3e-ed2b-4a4b-975c-53368db46a92\") "
Sep 10 17:43:42.993622 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:43:42.993528    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a15f9e3e-ed2b-4a4b-975c-53368db46a92-tls-cert\") pod \"a15f9e3e-ed2b-4a4b-975c-53368db46a92\" (UID: \"a15f9e3e-ed2b-4a4b-975c-53368db46a92\") "
Sep 10 17:43:42.993733 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:43:42.993716    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/80ecaacc-72d8-42b4-b3d0-f41f6e27391b-tls-cert\") pod \"authorino-9d8b447bf-7dl75\" (UID: \"80ecaacc-72d8-42b4-b3d0-f41f6e27391b\") " pod="kuadrant-system/authorino-9d8b447bf-7dl75"
Sep 10 17:43:42.993793 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:43:42.993777    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8dfw9\" (UniqueName: \"kubernetes.io/projected/80ecaacc-72d8-42b4-b3d0-f41f6e27391b-kube-api-access-8dfw9\") pod \"authorino-9d8b447bf-7dl75\" (UID: \"80ecaacc-72d8-42b4-b3d0-f41f6e27391b\") " pod="kuadrant-system/authorino-9d8b447bf-7dl75"
Sep 10 17:43:42.995633 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:43:42.995609    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a15f9e3e-ed2b-4a4b-975c-53368db46a92-kube-api-access-gsbm6" (OuterVolumeSpecName: "kube-api-access-gsbm6") pod "a15f9e3e-ed2b-4a4b-975c-53368db46a92" (UID: "a15f9e3e-ed2b-4a4b-975c-53368db46a92"). InnerVolumeSpecName "kube-api-access-gsbm6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:43:42.995713 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:43:42.995677    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a15f9e3e-ed2b-4a4b-975c-53368db46a92-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "a15f9e3e-ed2b-4a4b-975c-53368db46a92" (UID: "a15f9e3e-ed2b-4a4b-975c-53368db46a92"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 17:43:42.996132 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:43:42.996117    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/80ecaacc-72d8-42b4-b3d0-f41f6e27391b-tls-cert\") pod \"authorino-9d8b447bf-7dl75\" (UID: \"80ecaacc-72d8-42b4-b3d0-f41f6e27391b\") " pod="kuadrant-system/authorino-9d8b447bf-7dl75"
Sep 10 17:43:43.002775 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:43:43.002755    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8dfw9\" (UniqueName: \"kubernetes.io/projected/80ecaacc-72d8-42b4-b3d0-f41f6e27391b-kube-api-access-8dfw9\") pod \"authorino-9d8b447bf-7dl75\" (UID: \"80ecaacc-72d8-42b4-b3d0-f41f6e27391b\") " pod="kuadrant-system/authorino-9d8b447bf-7dl75"
Sep 10 17:43:43.020616 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:43:43.020593    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-9d8b447bf-7dl75"
Sep 10 17:43:43.094869 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:43:43.094758    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gsbm6\" (UniqueName: \"kubernetes.io/projected/a15f9e3e-ed2b-4a4b-975c-53368db46a92-kube-api-access-gsbm6\") on node \"ip-10-0-129-122.ec2.internal\" DevicePath \"\""
Sep 10 17:43:43.094869 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:43:43.094799    2579 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a15f9e3e-ed2b-4a4b-975c-53368db46a92-tls-cert\") on node \"ip-10-0-129-122.ec2.internal\" DevicePath \"\""
Sep 10 17:43:43.291029 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:43:43.290990    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-9d8b447bf-7dl75"]
Sep 10 17:43:43.293509 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:43:43.293483    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod80ecaacc_72d8_42b4_b3d0_f41f6e27391b.slice/crio-55e03d68e3cc7bb1e58a091c04b1ca4f86a890718e78c74552b6175ec9a173e6 WatchSource:0}: Error finding container 55e03d68e3cc7bb1e58a091c04b1ca4f86a890718e78c74552b6175ec9a173e6: Status 404 returned error can't find the container with id 55e03d68e3cc7bb1e58a091c04b1ca4f86a890718e78c74552b6175ec9a173e6
Sep 10 17:43:43.846848 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:43:43.846821    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-9d8b447bf-7dl75" event={"ID":"80ecaacc-72d8-42b4-b3d0-f41f6e27391b","Type":"ContainerStarted","Data":"55e03d68e3cc7bb1e58a091c04b1ca4f86a890718e78c74552b6175ec9a173e6"}
Sep 10 17:43:43.847042 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:43:43.847022    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-2kncf"
Sep 10 17:43:43.893060 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:43:43.893022    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-2kncf"]
Sep 10 17:43:43.894480 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:43:43.894455    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-2kncf"]
Sep 10 17:43:44.478956 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:43:44.478915    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a15f9e3e-ed2b-4a4b-975c-53368db46a92" path="/var/lib/kubelet/pods/a15f9e3e-ed2b-4a4b-975c-53368db46a92/volumes"
Sep 10 17:43:44.852849 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:43:44.852763    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-9d8b447bf-7dl75" event={"ID":"80ecaacc-72d8-42b4-b3d0-f41f6e27391b","Type":"ContainerStarted","Data":"5168f49548e6f3313113ed4c085c3ae03468f4c8475cc55b5dcbdeb53957f931"}
Sep 10 17:43:44.872261 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:43:44.872206    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-9d8b447bf-7dl75" podStartSLOduration=2.326255647 podStartE2EDuration="2.872186759s" totalImagesPullingTime="545.931112ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:43:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:43:44.870611683 +0000 UTC m=+468.870441750" watchObservedRunningTime="2026-09-10 17:43:44.872186759 +0000 UTC m=+468.872016833"
Sep 10 17:43:44.899630 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:43:44.899603    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-bdh7g"]
Sep 10 17:43:44.900029 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:43:44.899759    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-59c974bd48-bdh7g" podUID="1ccace62-5280-42be-8164-48131433f066" containerName="authorino" containerID="cri-o://91ad5dd3d9855ca18f7bc19e3f438b266a3d93ce059e20b30586356b1fa3607e" gracePeriod=30
Sep 10 17:43:45.153455 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:43:45.153431    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-bdh7g"
Sep 10 17:43:45.312921 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:43:45.312889    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xkzw6\" (UniqueName: \"kubernetes.io/projected/1ccace62-5280-42be-8164-48131433f066-kube-api-access-xkzw6\") pod \"1ccace62-5280-42be-8164-48131433f066\" (UID: \"1ccace62-5280-42be-8164-48131433f066\") "
Sep 10 17:43:45.315059 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:43:45.315030    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ccace62-5280-42be-8164-48131433f066-kube-api-access-xkzw6" (OuterVolumeSpecName: "kube-api-access-xkzw6") pod "1ccace62-5280-42be-8164-48131433f066" (UID: "1ccace62-5280-42be-8164-48131433f066"). InnerVolumeSpecName "kube-api-access-xkzw6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:43:45.414012 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:43:45.413941    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xkzw6\" (UniqueName: \"kubernetes.io/projected/1ccace62-5280-42be-8164-48131433f066-kube-api-access-xkzw6\") on node \"ip-10-0-129-122.ec2.internal\" DevicePath \"\""
Sep 10 17:43:45.858114 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:43:45.858025    2579 generic.go:356] "Generic (PLEG): container finished" podID="1ccace62-5280-42be-8164-48131433f066" containerID="91ad5dd3d9855ca18f7bc19e3f438b266a3d93ce059e20b30586356b1fa3607e" exitCode=0
Sep 10 17:43:45.858114 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:43:45.858071    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-bdh7g"
Sep 10 17:43:45.858333 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:43:45.858111    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-bdh7g" event={"ID":"1ccace62-5280-42be-8164-48131433f066","Type":"ContainerDied","Data":"91ad5dd3d9855ca18f7bc19e3f438b266a3d93ce059e20b30586356b1fa3607e"}
Sep 10 17:43:45.858333 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:43:45.858154    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-bdh7g" event={"ID":"1ccace62-5280-42be-8164-48131433f066","Type":"ContainerDied","Data":"d40c59f7ed2fc0094456b817ceda3a35cc5fba1bd9e877805ee97a170a24d2cc"}
Sep 10 17:43:45.858333 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:43:45.858171    2579 scope.go:117] "RemoveContainer" containerID="91ad5dd3d9855ca18f7bc19e3f438b266a3d93ce059e20b30586356b1fa3607e"
Sep 10 17:43:45.867225 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:43:45.867208    2579 scope.go:117] "RemoveContainer" containerID="91ad5dd3d9855ca18f7bc19e3f438b266a3d93ce059e20b30586356b1fa3607e"
Sep 10 17:43:45.867505 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:43:45.867480    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91ad5dd3d9855ca18f7bc19e3f438b266a3d93ce059e20b30586356b1fa3607e\": container with ID starting with 91ad5dd3d9855ca18f7bc19e3f438b266a3d93ce059e20b30586356b1fa3607e not found: ID does not exist" containerID="91ad5dd3d9855ca18f7bc19e3f438b266a3d93ce059e20b30586356b1fa3607e"
Sep 10 17:43:45.867624 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:43:45.867511    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91ad5dd3d9855ca18f7bc19e3f438b266a3d93ce059e20b30586356b1fa3607e"} err="failed to get container status \"91ad5dd3d9855ca18f7bc19e3f438b266a3d93ce059e20b30586356b1fa3607e\": rpc error: code = NotFound desc = could not find container \"91ad5dd3d9855ca18f7bc19e3f438b266a3d93ce059e20b30586356b1fa3607e\": container with ID starting with 91ad5dd3d9855ca18f7bc19e3f438b266a3d93ce059e20b30586356b1fa3607e not found: ID does not exist"
Sep 10 17:43:45.879192 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:43:45.879170    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-bdh7g"]
Sep 10 17:43:45.883432 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:43:45.883411    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-59c974bd48-bdh7g"]
Sep 10 17:43:46.474664 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:43:46.474632    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1ccace62-5280-42be-8164-48131433f066" path="/var/lib/kubelet/pods/1ccace62-5280-42be-8164-48131433f066/volumes"
Sep 10 17:43:50.096498 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:43:50.096462    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-776dc79cbc-zm94q"]
Sep 10 17:43:50.097022 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:43:50.097005    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ccace62-5280-42be-8164-48131433f066" containerName="authorino"
Sep 10 17:43:50.097080 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:43:50.097026    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ccace62-5280-42be-8164-48131433f066" containerName="authorino"
Sep 10 17:43:50.097116 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:43:50.097105    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="1ccace62-5280-42be-8164-48131433f066" containerName="authorino"
Sep 10 17:43:50.101632 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:43:50.101612    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-776dc79cbc-zm94q"
Sep 10 17:43:50.104738 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:43:50.104718    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 10 17:43:50.105073 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:43:50.105055    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-bbp7v\""
Sep 10 17:43:50.105218 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:43:50.105200    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 10 17:43:50.105931 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:43:50.105916    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 10 17:43:50.116644 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:43:50.116617    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-776dc79cbc-zm94q"]
Sep 10 17:43:50.251661 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:43:50.251629    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8db4\" (UniqueName: \"kubernetes.io/projected/bd7efc4c-bbc9-4c30-8f32-d55f15b4d9c4-kube-api-access-f8db4\") pod \"maas-controller-776dc79cbc-zm94q\" (UID: \"bd7efc4c-bbc9-4c30-8f32-d55f15b4d9c4\") " pod="opendatahub/maas-controller-776dc79cbc-zm94q"
Sep 10 17:43:50.251824 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:43:50.251697    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd7efc4c-bbc9-4c30-8f32-d55f15b4d9c4-metrics-certs\") pod \"maas-controller-776dc79cbc-zm94q\" (UID: \"bd7efc4c-bbc9-4c30-8f32-d55f15b4d9c4\") " pod="opendatahub/maas-controller-776dc79cbc-zm94q"
Sep 10 17:43:50.251824 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:43:50.251725    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/bd7efc4c-bbc9-4c30-8f32-d55f15b4d9c4-webhook-cert\") pod \"maas-controller-776dc79cbc-zm94q\" (UID: \"bd7efc4c-bbc9-4c30-8f32-d55f15b4d9c4\") " pod="opendatahub/maas-controller-776dc79cbc-zm94q"
Sep 10 17:43:50.352321 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:43:50.352242    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8db4\" (UniqueName: \"kubernetes.io/projected/bd7efc4c-bbc9-4c30-8f32-d55f15b4d9c4-kube-api-access-f8db4\") pod \"maas-controller-776dc79cbc-zm94q\" (UID: \"bd7efc4c-bbc9-4c30-8f32-d55f15b4d9c4\") " pod="opendatahub/maas-controller-776dc79cbc-zm94q"
Sep 10 17:43:50.352321 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:43:50.352291    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd7efc4c-bbc9-4c30-8f32-d55f15b4d9c4-metrics-certs\") pod \"maas-controller-776dc79cbc-zm94q\" (UID: \"bd7efc4c-bbc9-4c30-8f32-d55f15b4d9c4\") " pod="opendatahub/maas-controller-776dc79cbc-zm94q"
Sep 10 17:43:50.352321 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:43:50.352310    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/bd7efc4c-bbc9-4c30-8f32-d55f15b4d9c4-webhook-cert\") pod \"maas-controller-776dc79cbc-zm94q\" (UID: \"bd7efc4c-bbc9-4c30-8f32-d55f15b4d9c4\") " pod="opendatahub/maas-controller-776dc79cbc-zm94q"
Sep 10 17:43:50.352626 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:43:50.352423    2579 secret.go:189] Couldn't get secret opendatahub/maas-controller-metrics-tls: secret "maas-controller-metrics-tls" not found
Sep 10 17:43:50.352626 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:43:50.352528    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bd7efc4c-bbc9-4c30-8f32-d55f15b4d9c4-metrics-certs podName:bd7efc4c-bbc9-4c30-8f32-d55f15b4d9c4 nodeName:}" failed. No retries permitted until 2026-09-10 17:43:50.852505748 +0000 UTC m=+474.852335807 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bd7efc4c-bbc9-4c30-8f32-d55f15b4d9c4-metrics-certs") pod "maas-controller-776dc79cbc-zm94q" (UID: "bd7efc4c-bbc9-4c30-8f32-d55f15b4d9c4") : secret "maas-controller-metrics-tls" not found
Sep 10 17:43:50.354935 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:43:50.354912    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/bd7efc4c-bbc9-4c30-8f32-d55f15b4d9c4-webhook-cert\") pod \"maas-controller-776dc79cbc-zm94q\" (UID: \"bd7efc4c-bbc9-4c30-8f32-d55f15b4d9c4\") " pod="opendatahub/maas-controller-776dc79cbc-zm94q"
Sep 10 17:43:50.361190 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:43:50.361166    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8db4\" (UniqueName: \"kubernetes.io/projected/bd7efc4c-bbc9-4c30-8f32-d55f15b4d9c4-kube-api-access-f8db4\") pod \"maas-controller-776dc79cbc-zm94q\" (UID: \"bd7efc4c-bbc9-4c30-8f32-d55f15b4d9c4\") " pod="opendatahub/maas-controller-776dc79cbc-zm94q"
Sep 10 17:43:50.857661 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:43:50.857628    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd7efc4c-bbc9-4c30-8f32-d55f15b4d9c4-metrics-certs\") pod \"maas-controller-776dc79cbc-zm94q\" (UID: \"bd7efc4c-bbc9-4c30-8f32-d55f15b4d9c4\") " pod="opendatahub/maas-controller-776dc79cbc-zm94q"
Sep 10 17:43:50.860094 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:43:50.860072    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd7efc4c-bbc9-4c30-8f32-d55f15b4d9c4-metrics-certs\") pod \"maas-controller-776dc79cbc-zm94q\" (UID: \"bd7efc4c-bbc9-4c30-8f32-d55f15b4d9c4\") " pod="opendatahub/maas-controller-776dc79cbc-zm94q"
Sep 10 17:43:51.011596 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:43:51.011545    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-776dc79cbc-zm94q"
Sep 10 17:43:51.283474 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:43:51.283446    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-776dc79cbc-zm94q"]
Sep 10 17:43:51.285348 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:43:51.285311    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd7efc4c_bbc9_4c30_8f32_d55f15b4d9c4.slice/crio-268f4b7cfbe9aa94679d7a63503ca95c9fc4f6656d336d60a6421530f9d66297 WatchSource:0}: Error finding container 268f4b7cfbe9aa94679d7a63503ca95c9fc4f6656d336d60a6421530f9d66297: Status 404 returned error can't find the container with id 268f4b7cfbe9aa94679d7a63503ca95c9fc4f6656d336d60a6421530f9d66297
Sep 10 17:43:51.888906 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:43:51.888871    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-776dc79cbc-zm94q" event={"ID":"bd7efc4c-bbc9-4c30-8f32-d55f15b4d9c4","Type":"ContainerStarted","Data":"268f4b7cfbe9aa94679d7a63503ca95c9fc4f6656d336d60a6421530f9d66297"}
Sep 10 17:43:53.901413 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:43:53.901373    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-776dc79cbc-zm94q" event={"ID":"bd7efc4c-bbc9-4c30-8f32-d55f15b4d9c4","Type":"ContainerStarted","Data":"d227b539bfa41ebe422b0e30f2ecdea65c79e5fb83ceaa747d40af7dee5c782c"}
Sep 10 17:43:53.901804 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:43:53.901441    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-776dc79cbc-zm94q"
Sep 10 17:43:53.926139 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:43:53.926054    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-776dc79cbc-zm94q" podStartSLOduration=1.535724027 podStartE2EDuration="3.926038949s" totalImagesPullingTime="2.390314922s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:43:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:43:53.925515989 +0000 UTC m=+477.925346058" watchObservedRunningTime="2026-09-10 17:43:53.926038949 +0000 UTC m=+477.925869015"
Sep 10 17:44:04.908715 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:44:04.908683    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-776dc79cbc-zm94q"
Sep 10 17:44:05.192712 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:44:05.192627    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-776dc79cbc-zm94q"]
Sep 10 17:44:05.192855 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:44:05.192800    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-776dc79cbc-zm94q" podUID="bd7efc4c-bbc9-4c30-8f32-d55f15b4d9c4" containerName="manager" containerID="cri-o://d227b539bfa41ebe422b0e30f2ecdea65c79e5fb83ceaa747d40af7dee5c782c" gracePeriod=30
Sep 10 17:44:05.442431 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:44:05.442401    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-776dc79cbc-zm94q"
Sep 10 17:44:05.465421 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:44:05.465350    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f8db4\" (UniqueName: \"kubernetes.io/projected/bd7efc4c-bbc9-4c30-8f32-d55f15b4d9c4-kube-api-access-f8db4\") pod \"bd7efc4c-bbc9-4c30-8f32-d55f15b4d9c4\" (UID: \"bd7efc4c-bbc9-4c30-8f32-d55f15b4d9c4\") "
Sep 10 17:44:05.465421 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:44:05.465414    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/bd7efc4c-bbc9-4c30-8f32-d55f15b4d9c4-webhook-cert\") pod \"bd7efc4c-bbc9-4c30-8f32-d55f15b4d9c4\" (UID: \"bd7efc4c-bbc9-4c30-8f32-d55f15b4d9c4\") "
Sep 10 17:44:05.465642 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:44:05.465458    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd7efc4c-bbc9-4c30-8f32-d55f15b4d9c4-metrics-certs\") pod \"bd7efc4c-bbc9-4c30-8f32-d55f15b4d9c4\" (UID: \"bd7efc4c-bbc9-4c30-8f32-d55f15b4d9c4\") "
Sep 10 17:44:05.468181 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:44:05.468138    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd7efc4c-bbc9-4c30-8f32-d55f15b4d9c4-kube-api-access-f8db4" (OuterVolumeSpecName: "kube-api-access-f8db4") pod "bd7efc4c-bbc9-4c30-8f32-d55f15b4d9c4" (UID: "bd7efc4c-bbc9-4c30-8f32-d55f15b4d9c4"). InnerVolumeSpecName "kube-api-access-f8db4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:44:05.468181 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:44:05.468152    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd7efc4c-bbc9-4c30-8f32-d55f15b4d9c4-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "bd7efc4c-bbc9-4c30-8f32-d55f15b4d9c4" (UID: "bd7efc4c-bbc9-4c30-8f32-d55f15b4d9c4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 17:44:05.468347 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:44:05.468215    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd7efc4c-bbc9-4c30-8f32-d55f15b4d9c4-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "bd7efc4c-bbc9-4c30-8f32-d55f15b4d9c4" (UID: "bd7efc4c-bbc9-4c30-8f32-d55f15b4d9c4"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 17:44:05.566819 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:44:05.566781    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f8db4\" (UniqueName: \"kubernetes.io/projected/bd7efc4c-bbc9-4c30-8f32-d55f15b4d9c4-kube-api-access-f8db4\") on node \"ip-10-0-129-122.ec2.internal\" DevicePath \"\""
Sep 10 17:44:05.566819 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:44:05.566808    2579 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/bd7efc4c-bbc9-4c30-8f32-d55f15b4d9c4-webhook-cert\") on node \"ip-10-0-129-122.ec2.internal\" DevicePath \"\""
Sep 10 17:44:05.566819 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:44:05.566819    2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd7efc4c-bbc9-4c30-8f32-d55f15b4d9c4-metrics-certs\") on node \"ip-10-0-129-122.ec2.internal\" DevicePath \"\""
Sep 10 17:44:05.961560 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:44:05.961526    2579 generic.go:356] "Generic (PLEG): container finished" podID="bd7efc4c-bbc9-4c30-8f32-d55f15b4d9c4" containerID="d227b539bfa41ebe422b0e30f2ecdea65c79e5fb83ceaa747d40af7dee5c782c" exitCode=0
Sep 10 17:44:05.961985 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:44:05.961598    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-776dc79cbc-zm94q"
Sep 10 17:44:05.961985 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:44:05.961616    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-776dc79cbc-zm94q" event={"ID":"bd7efc4c-bbc9-4c30-8f32-d55f15b4d9c4","Type":"ContainerDied","Data":"d227b539bfa41ebe422b0e30f2ecdea65c79e5fb83ceaa747d40af7dee5c782c"}
Sep 10 17:44:05.961985 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:44:05.961666    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-776dc79cbc-zm94q" event={"ID":"bd7efc4c-bbc9-4c30-8f32-d55f15b4d9c4","Type":"ContainerDied","Data":"268f4b7cfbe9aa94679d7a63503ca95c9fc4f6656d336d60a6421530f9d66297"}
Sep 10 17:44:05.961985 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:44:05.961689    2579 scope.go:117] "RemoveContainer" containerID="d227b539bfa41ebe422b0e30f2ecdea65c79e5fb83ceaa747d40af7dee5c782c"
Sep 10 17:44:05.971193 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:44:05.971174    2579 scope.go:117] "RemoveContainer" containerID="d227b539bfa41ebe422b0e30f2ecdea65c79e5fb83ceaa747d40af7dee5c782c"
Sep 10 17:44:05.971467 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:44:05.971447    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d227b539bfa41ebe422b0e30f2ecdea65c79e5fb83ceaa747d40af7dee5c782c\": container with ID starting with d227b539bfa41ebe422b0e30f2ecdea65c79e5fb83ceaa747d40af7dee5c782c not found: ID does not exist" containerID="d227b539bfa41ebe422b0e30f2ecdea65c79e5fb83ceaa747d40af7dee5c782c"
Sep 10 17:44:05.971535 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:44:05.971479    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d227b539bfa41ebe422b0e30f2ecdea65c79e5fb83ceaa747d40af7dee5c782c"} err="failed to get container status \"d227b539bfa41ebe422b0e30f2ecdea65c79e5fb83ceaa747d40af7dee5c782c\": rpc error: code = NotFound desc = could not find container \"d227b539bfa41ebe422b0e30f2ecdea65c79e5fb83ceaa747d40af7dee5c782c\": container with ID starting with d227b539bfa41ebe422b0e30f2ecdea65c79e5fb83ceaa747d40af7dee5c782c not found: ID does not exist"
Sep 10 17:44:05.986894 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:44:05.986864    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-776dc79cbc-zm94q"]
Sep 10 17:44:05.990218 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:44:05.990195    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-776dc79cbc-zm94q"]
Sep 10 17:44:06.475445 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:44:06.475413    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bd7efc4c-bbc9-4c30-8f32-d55f15b4d9c4" path="/var/lib/kubelet/pods/bd7efc4c-bbc9-4c30-8f32-d55f15b4d9c4/volumes"
Sep 10 17:44:27.610294 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:44:27.610260    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6ff67d7f48-b46rw"]
Sep 10 17:44:27.610774 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:44:27.610590    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd7efc4c-bbc9-4c30-8f32-d55f15b4d9c4" containerName="manager"
Sep 10 17:44:27.610774 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:44:27.610601    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd7efc4c-bbc9-4c30-8f32-d55f15b4d9c4" containerName="manager"
Sep 10 17:44:27.610774 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:44:27.610679    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="bd7efc4c-bbc9-4c30-8f32-d55f15b4d9c4" containerName="manager"
Sep 10 17:44:27.616745 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:44:27.616728    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6ff67d7f48-b46rw"
Sep 10 17:44:27.619253 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:44:27.619232    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Sep 10 17:44:27.620639 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:44:27.620622    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 10 17:44:27.640755 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:44:27.640734    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6ff67d7f48-b46rw"]
Sep 10 17:44:27.749035 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:44:27.749000    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4ba160dc-d064-4f2d-8bd1-dcebd0855f10-metrics-certs\") pod \"maas-api-6ff67d7f48-b46rw\" (UID: \"4ba160dc-d064-4f2d-8bd1-dcebd0855f10\") " pod="odh-ai-gateway-infra/maas-api-6ff67d7f48-b46rw"
Sep 10 17:44:27.749207 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:44:27.749048    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bhb6\" (UniqueName: \"kubernetes.io/projected/4ba160dc-d064-4f2d-8bd1-dcebd0855f10-kube-api-access-6bhb6\") pod \"maas-api-6ff67d7f48-b46rw\" (UID: \"4ba160dc-d064-4f2d-8bd1-dcebd0855f10\") " pod="odh-ai-gateway-infra/maas-api-6ff67d7f48-b46rw"
Sep 10 17:44:27.749207 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:44:27.749110    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4ba160dc-d064-4f2d-8bd1-dcebd0855f10-maas-api-tls\") pod \"maas-api-6ff67d7f48-b46rw\" (UID: \"4ba160dc-d064-4f2d-8bd1-dcebd0855f10\") " pod="odh-ai-gateway-infra/maas-api-6ff67d7f48-b46rw"
Sep 10 17:44:27.749207 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:44:27.749155    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ba160dc-d064-4f2d-8bd1-dcebd0855f10-openshift-service-ca\") pod \"maas-api-6ff67d7f48-b46rw\" (UID: \"4ba160dc-d064-4f2d-8bd1-dcebd0855f10\") " pod="odh-ai-gateway-infra/maas-api-6ff67d7f48-b46rw"
Sep 10 17:44:27.850180 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:44:27.850142    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4ba160dc-d064-4f2d-8bd1-dcebd0855f10-metrics-certs\") pod \"maas-api-6ff67d7f48-b46rw\" (UID: \"4ba160dc-d064-4f2d-8bd1-dcebd0855f10\") " pod="odh-ai-gateway-infra/maas-api-6ff67d7f48-b46rw"
Sep 10 17:44:27.850348 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:44:27.850186    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bhb6\" (UniqueName: \"kubernetes.io/projected/4ba160dc-d064-4f2d-8bd1-dcebd0855f10-kube-api-access-6bhb6\") pod \"maas-api-6ff67d7f48-b46rw\" (UID: \"4ba160dc-d064-4f2d-8bd1-dcebd0855f10\") " pod="odh-ai-gateway-infra/maas-api-6ff67d7f48-b46rw"
Sep 10 17:44:27.850348 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:44:27.850221    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4ba160dc-d064-4f2d-8bd1-dcebd0855f10-maas-api-tls\") pod \"maas-api-6ff67d7f48-b46rw\" (UID: \"4ba160dc-d064-4f2d-8bd1-dcebd0855f10\") " pod="odh-ai-gateway-infra/maas-api-6ff67d7f48-b46rw"
Sep 10 17:44:27.850348 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:44:27.850259    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ba160dc-d064-4f2d-8bd1-dcebd0855f10-openshift-service-ca\") pod \"maas-api-6ff67d7f48-b46rw\" (UID: \"4ba160dc-d064-4f2d-8bd1-dcebd0855f10\") " pod="odh-ai-gateway-infra/maas-api-6ff67d7f48-b46rw"
Sep 10 17:44:27.850909 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:44:27.850885    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ba160dc-d064-4f2d-8bd1-dcebd0855f10-openshift-service-ca\") pod \"maas-api-6ff67d7f48-b46rw\" (UID: \"4ba160dc-d064-4f2d-8bd1-dcebd0855f10\") " pod="odh-ai-gateway-infra/maas-api-6ff67d7f48-b46rw"
Sep 10 17:44:27.852663 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:44:27.852639    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4ba160dc-d064-4f2d-8bd1-dcebd0855f10-maas-api-tls\") pod \"maas-api-6ff67d7f48-b46rw\" (UID: \"4ba160dc-d064-4f2d-8bd1-dcebd0855f10\") " pod="odh-ai-gateway-infra/maas-api-6ff67d7f48-b46rw"
Sep 10 17:44:27.852748 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:44:27.852690    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4ba160dc-d064-4f2d-8bd1-dcebd0855f10-metrics-certs\") pod \"maas-api-6ff67d7f48-b46rw\" (UID: \"4ba160dc-d064-4f2d-8bd1-dcebd0855f10\") " pod="odh-ai-gateway-infra/maas-api-6ff67d7f48-b46rw"
Sep 10 17:44:27.863455 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:44:27.863403    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bhb6\" (UniqueName: \"kubernetes.io/projected/4ba160dc-d064-4f2d-8bd1-dcebd0855f10-kube-api-access-6bhb6\") pod \"maas-api-6ff67d7f48-b46rw\" (UID: \"4ba160dc-d064-4f2d-8bd1-dcebd0855f10\") " pod="odh-ai-gateway-infra/maas-api-6ff67d7f48-b46rw"
Sep 10 17:44:27.928025 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:44:27.928000    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6ff67d7f48-b46rw"
Sep 10 17:44:28.243055 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:44:28.243023    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6ff67d7f48-b46rw"]
Sep 10 17:44:28.246592 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:44:28.246543    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ba160dc_d064_4f2d_8bd1_dcebd0855f10.slice/crio-eb0329a74c16ab13600d2a950f7c4f99e36974b4e691601944f1bab5b3996d98 WatchSource:0}: Error finding container eb0329a74c16ab13600d2a950f7c4f99e36974b4e691601944f1bab5b3996d98: Status 404 returned error can't find the container with id eb0329a74c16ab13600d2a950f7c4f99e36974b4e691601944f1bab5b3996d98
Sep 10 17:44:28.498068 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:44:28.497991    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-kpwl9"]
Sep 10 17:44:28.503061 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:44:28.503034    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-kpwl9"
Sep 10 17:44:28.506126 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:44:28.506098    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Sep 10 17:44:28.506263 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:44:28.506241    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-n66ts\""
Sep 10 17:44:28.516530 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:44:28.516486    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-kpwl9"]
Sep 10 17:44:28.557115 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:44:28.557087    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5nrn\" (UniqueName: \"kubernetes.io/projected/fcfe8a95-b4b5-4997-8d4b-717a1f3d8149-kube-api-access-x5nrn\") pod \"payload-processing-5b54b7fbb7-kpwl9\" (UID: \"fcfe8a95-b4b5-4997-8d4b-717a1f3d8149\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-kpwl9"
Sep 10 17:44:28.557260 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:44:28.557147    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fcfe8a95-b4b5-4997-8d4b-717a1f3d8149-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-kpwl9\" (UID: \"fcfe8a95-b4b5-4997-8d4b-717a1f3d8149\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-kpwl9"
Sep 10 17:44:28.658533 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:44:28.658497    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fcfe8a95-b4b5-4997-8d4b-717a1f3d8149-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-kpwl9\" (UID: \"fcfe8a95-b4b5-4997-8d4b-717a1f3d8149\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-kpwl9"
Sep 10 17:44:28.659029 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:44:28.658636    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5nrn\" (UniqueName: \"kubernetes.io/projected/fcfe8a95-b4b5-4997-8d4b-717a1f3d8149-kube-api-access-x5nrn\") pod \"payload-processing-5b54b7fbb7-kpwl9\" (UID: \"fcfe8a95-b4b5-4997-8d4b-717a1f3d8149\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-kpwl9"
Sep 10 17:44:28.659516 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:44:28.659488    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fcfe8a95-b4b5-4997-8d4b-717a1f3d8149-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-kpwl9\" (UID: \"fcfe8a95-b4b5-4997-8d4b-717a1f3d8149\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-kpwl9"
Sep 10 17:44:28.668299 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:44:28.668277    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5nrn\" (UniqueName: \"kubernetes.io/projected/fcfe8a95-b4b5-4997-8d4b-717a1f3d8149-kube-api-access-x5nrn\") pod \"payload-processing-5b54b7fbb7-kpwl9\" (UID: \"fcfe8a95-b4b5-4997-8d4b-717a1f3d8149\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-kpwl9"
Sep 10 17:44:28.814425 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:44:28.814343    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-kpwl9"
Sep 10 17:44:29.063299 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:44:29.063266    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6ff67d7f48-b46rw" event={"ID":"4ba160dc-d064-4f2d-8bd1-dcebd0855f10","Type":"ContainerStarted","Data":"eb0329a74c16ab13600d2a950f7c4f99e36974b4e691601944f1bab5b3996d98"}
Sep 10 17:44:29.099581 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:44:29.099536    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-kpwl9"]
Sep 10 17:44:29.101344 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:44:29.101319    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcfe8a95_b4b5_4997_8d4b_717a1f3d8149.slice/crio-9c7b8e39da4ba5e64ae96a536d2e10c5f00682a552da75fe36b502d365e8e450 WatchSource:0}: Error finding container 9c7b8e39da4ba5e64ae96a536d2e10c5f00682a552da75fe36b502d365e8e450: Status 404 returned error can't find the container with id 9c7b8e39da4ba5e64ae96a536d2e10c5f00682a552da75fe36b502d365e8e450
Sep 10 17:44:30.071773 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:44:30.071730    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-kpwl9" event={"ID":"fcfe8a95-b4b5-4997-8d4b-717a1f3d8149","Type":"ContainerStarted","Data":"9c7b8e39da4ba5e64ae96a536d2e10c5f00682a552da75fe36b502d365e8e450"}
Sep 10 17:44:32.082468 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:44:32.082433    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-kpwl9" event={"ID":"fcfe8a95-b4b5-4997-8d4b-717a1f3d8149","Type":"ContainerStarted","Data":"3d820a9dd4b707a45bc2dc90d48d416ebdc94106c22ceaf3ea26fb63f1c493fa"}
Sep 10 17:44:32.082929 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:44:32.082616    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-kpwl9"
Sep 10 17:44:32.084064 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:44:32.084045    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6ff67d7f48-b46rw" event={"ID":"4ba160dc-d064-4f2d-8bd1-dcebd0855f10","Type":"ContainerStarted","Data":"25c5e471389bfaccb91cb9cc72f2eaffd805f7798757f2eabefe0978b7b2fa31"}
Sep 10 17:44:32.084185 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:44:32.084169    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6ff67d7f48-b46rw"
Sep 10 17:44:32.110946 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:44:32.110902    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5b54b7fbb7-kpwl9" podStartSLOduration=1.939001478 podStartE2EDuration="4.110889781s" totalImagesPullingTime="2.171888303s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:44:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:44:32.109127316 +0000 UTC m=+516.108957381" watchObservedRunningTime="2026-09-10 17:44:32.110889781 +0000 UTC m=+516.110719848"
Sep 10 17:44:32.137366 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:44:32.137319    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6ff67d7f48-b46rw" podStartSLOduration=3.142205395 podStartE2EDuration="5.137305526s" totalImagesPullingTime="1.995100131s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:44:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:44:32.137258422 +0000 UTC m=+516.137088488" watchObservedRunningTime="2026-09-10 17:44:32.137305526 +0000 UTC m=+516.137135593"
Sep 10 17:44:38.094473 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:44:38.094441    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6ff67d7f48-b46rw"
Sep 10 17:44:43.091006 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:44:43.090972    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-kpwl9"
Sep 10 17:45:26.728668 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:45:26.728635    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-4jfr8"]
Sep 10 17:45:26.730998 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:45:26.730981    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4jfr8"
Sep 10 17:45:26.734961 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:45:26.734926    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 10 17:45:26.735221 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:45:26.734941    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Sep 10 17:45:26.735221 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:45:26.734953    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-pntqt\""
Sep 10 17:45:26.735406 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:45:26.734964    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 10 17:45:26.744810 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:45:26.744789    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-4jfr8"]
Sep 10 17:45:26.842139 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:45:26.842109    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/254aaaed-8976-4976-88e6-d0aa5cb2a84e-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4jfr8\" (UID: \"254aaaed-8976-4976-88e6-d0aa5cb2a84e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4jfr8"
Sep 10 17:45:26.842289 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:45:26.842195    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sd6k6\" (UniqueName: \"kubernetes.io/projected/254aaaed-8976-4976-88e6-d0aa5cb2a84e-kube-api-access-sd6k6\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4jfr8\" (UID: \"254aaaed-8976-4976-88e6-d0aa5cb2a84e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4jfr8"
Sep 10 17:45:26.842289 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:45:26.842234    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/254aaaed-8976-4976-88e6-d0aa5cb2a84e-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4jfr8\" (UID: \"254aaaed-8976-4976-88e6-d0aa5cb2a84e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4jfr8"
Sep 10 17:45:26.842289 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:45:26.842253    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/254aaaed-8976-4976-88e6-d0aa5cb2a84e-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4jfr8\" (UID: \"254aaaed-8976-4976-88e6-d0aa5cb2a84e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4jfr8"
Sep 10 17:45:26.842289 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:45:26.842275    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/254aaaed-8976-4976-88e6-d0aa5cb2a84e-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4jfr8\" (UID: \"254aaaed-8976-4976-88e6-d0aa5cb2a84e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4jfr8"
Sep 10 17:45:26.842424 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:45:26.842296    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/254aaaed-8976-4976-88e6-d0aa5cb2a84e-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4jfr8\" (UID: \"254aaaed-8976-4976-88e6-d0aa5cb2a84e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4jfr8"
Sep 10 17:45:26.943508 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:45:26.943470    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/254aaaed-8976-4976-88e6-d0aa5cb2a84e-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4jfr8\" (UID: \"254aaaed-8976-4976-88e6-d0aa5cb2a84e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4jfr8"
Sep 10 17:45:26.943702 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:45:26.943560    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sd6k6\" (UniqueName: \"kubernetes.io/projected/254aaaed-8976-4976-88e6-d0aa5cb2a84e-kube-api-access-sd6k6\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4jfr8\" (UID: \"254aaaed-8976-4976-88e6-d0aa5cb2a84e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4jfr8"
Sep 10 17:45:26.943702 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:45:26.943632    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/254aaaed-8976-4976-88e6-d0aa5cb2a84e-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4jfr8\" (UID: \"254aaaed-8976-4976-88e6-d0aa5cb2a84e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4jfr8"
Sep 10 17:45:26.943702 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:45:26.943658    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/254aaaed-8976-4976-88e6-d0aa5cb2a84e-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4jfr8\" (UID: \"254aaaed-8976-4976-88e6-d0aa5cb2a84e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4jfr8"
Sep 10 17:45:26.943702 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:45:26.943697    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/254aaaed-8976-4976-88e6-d0aa5cb2a84e-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4jfr8\" (UID: \"254aaaed-8976-4976-88e6-d0aa5cb2a84e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4jfr8"
Sep 10 17:45:26.943913 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:45:26.943731    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/254aaaed-8976-4976-88e6-d0aa5cb2a84e-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4jfr8\" (UID: \"254aaaed-8976-4976-88e6-d0aa5cb2a84e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4jfr8"
Sep 10 17:45:26.943913 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:45:26.943888    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/254aaaed-8976-4976-88e6-d0aa5cb2a84e-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4jfr8\" (UID: \"254aaaed-8976-4976-88e6-d0aa5cb2a84e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4jfr8"
Sep 10 17:45:26.944029 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:45:26.944007    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/254aaaed-8976-4976-88e6-d0aa5cb2a84e-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4jfr8\" (UID: \"254aaaed-8976-4976-88e6-d0aa5cb2a84e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4jfr8"
Sep 10 17:45:26.944134 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:45:26.944076    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/254aaaed-8976-4976-88e6-d0aa5cb2a84e-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4jfr8\" (UID: \"254aaaed-8976-4976-88e6-d0aa5cb2a84e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4jfr8"
Sep 10 17:45:26.945967 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:45:26.945938    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/254aaaed-8976-4976-88e6-d0aa5cb2a84e-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4jfr8\" (UID: \"254aaaed-8976-4976-88e6-d0aa5cb2a84e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4jfr8"
Sep 10 17:45:26.946175 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:45:26.946159    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/254aaaed-8976-4976-88e6-d0aa5cb2a84e-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4jfr8\" (UID: \"254aaaed-8976-4976-88e6-d0aa5cb2a84e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4jfr8"
Sep 10 17:45:26.952171 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:45:26.952152    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sd6k6\" (UniqueName: \"kubernetes.io/projected/254aaaed-8976-4976-88e6-d0aa5cb2a84e-kube-api-access-sd6k6\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4jfr8\" (UID: \"254aaaed-8976-4976-88e6-d0aa5cb2a84e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4jfr8"
Sep 10 17:45:27.043313 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:45:27.043239    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4jfr8"
Sep 10 17:45:27.313096 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:45:27.312966    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-4jfr8"]
Sep 10 17:45:27.315447 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:45:27.315416    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod254aaaed_8976_4976_88e6_d0aa5cb2a84e.slice/crio-07696ff1eb263717717b377e087570f63ac089e390c002a4880abef6ef1e13ef WatchSource:0}: Error finding container 07696ff1eb263717717b377e087570f63ac089e390c002a4880abef6ef1e13ef: Status 404 returned error can't find the container with id 07696ff1eb263717717b377e087570f63ac089e390c002a4880abef6ef1e13ef
Sep 10 17:45:27.338588 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:45:27.338534    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4jfr8" event={"ID":"254aaaed-8976-4976-88e6-d0aa5cb2a84e","Type":"ContainerStarted","Data":"07696ff1eb263717717b377e087570f63ac089e390c002a4880abef6ef1e13ef"}
Sep 10 17:45:29.351341 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:45:29.351299    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4jfr8" event={"ID":"254aaaed-8976-4976-88e6-d0aa5cb2a84e","Type":"ContainerStarted","Data":"068c4c474672b9c3c8e04d9bd447c59542faf85324214827bb217e33cf30ada2"}
Sep 10 17:45:29.375213 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:45:29.375139    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4jfr8" podStartSLOduration=2.289772387 podStartE2EDuration="3.375119186s" totalImagesPullingTime="1.085346799s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:45:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:45:29.37267151 +0000 UTC m=+573.372501576" watchObservedRunningTime="2026-09-10 17:45:29.375119186 +0000 UTC m=+573.374949253"
Sep 10 17:45:37.043862 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:45:37.043823    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4jfr8"
Sep 10 17:45:37.044239 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:45:37.043910    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4jfr8"
Sep 10 17:45:37.056033 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:45:37.056012    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4jfr8"
Sep 10 17:45:37.396556 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:45:37.396523    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4jfr8"
Sep 10 17:45:47.817382 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:45:47.817351    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f76js2"]
Sep 10 17:45:47.821231 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:45:47.821208    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f76js2"
Sep 10 17:45:47.823782 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:45:47.823764    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Sep 10 17:45:47.834678 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:45:47.834656    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f76js2"]
Sep 10 17:45:47.918156 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:45:47.918125    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/98b3e5d2-a3c9-41fe-9a22-e36ba7760daa-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f76js2\" (UID: \"98b3e5d2-a3c9-41fe-9a22-e36ba7760daa\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f76js2"
Sep 10 17:45:47.918303 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:45:47.918174    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zbdt\" (UniqueName: \"kubernetes.io/projected/98b3e5d2-a3c9-41fe-9a22-e36ba7760daa-kube-api-access-8zbdt\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f76js2\" (UID: \"98b3e5d2-a3c9-41fe-9a22-e36ba7760daa\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f76js2"
Sep 10 17:45:47.918303 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:45:47.918254    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/98b3e5d2-a3c9-41fe-9a22-e36ba7760daa-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f76js2\" (UID: \"98b3e5d2-a3c9-41fe-9a22-e36ba7760daa\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f76js2"
Sep 10 17:45:47.918303 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:45:47.918290    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/98b3e5d2-a3c9-41fe-9a22-e36ba7760daa-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f76js2\" (UID: \"98b3e5d2-a3c9-41fe-9a22-e36ba7760daa\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f76js2"
Sep 10 17:45:47.918410 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:45:47.918310    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/98b3e5d2-a3c9-41fe-9a22-e36ba7760daa-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f76js2\" (UID: \"98b3e5d2-a3c9-41fe-9a22-e36ba7760daa\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f76js2"
Sep 10 17:45:47.918410 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:45:47.918347    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/98b3e5d2-a3c9-41fe-9a22-e36ba7760daa-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f76js2\" (UID: \"98b3e5d2-a3c9-41fe-9a22-e36ba7760daa\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f76js2"
Sep 10 17:45:48.018728 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:45:48.018690    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zbdt\" (UniqueName: \"kubernetes.io/projected/98b3e5d2-a3c9-41fe-9a22-e36ba7760daa-kube-api-access-8zbdt\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f76js2\" (UID: \"98b3e5d2-a3c9-41fe-9a22-e36ba7760daa\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f76js2"
Sep 10 17:45:48.018892 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:45:48.018748    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/98b3e5d2-a3c9-41fe-9a22-e36ba7760daa-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f76js2\" (UID: \"98b3e5d2-a3c9-41fe-9a22-e36ba7760daa\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f76js2"
Sep 10 17:45:48.018892 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:45:48.018777    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/98b3e5d2-a3c9-41fe-9a22-e36ba7760daa-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f76js2\" (UID: \"98b3e5d2-a3c9-41fe-9a22-e36ba7760daa\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f76js2"
Sep 10 17:45:48.018892 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:45:48.018808    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/98b3e5d2-a3c9-41fe-9a22-e36ba7760daa-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f76js2\" (UID: \"98b3e5d2-a3c9-41fe-9a22-e36ba7760daa\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f76js2"
Sep 10 17:45:48.018892 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:45:48.018839    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/98b3e5d2-a3c9-41fe-9a22-e36ba7760daa-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f76js2\" (UID: \"98b3e5d2-a3c9-41fe-9a22-e36ba7760daa\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f76js2"
Sep 10 17:45:48.018892 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:45:48.018873    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/98b3e5d2-a3c9-41fe-9a22-e36ba7760daa-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f76js2\" (UID: \"98b3e5d2-a3c9-41fe-9a22-e36ba7760daa\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f76js2"
Sep 10 17:45:48.019207 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:45:48.019183    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/98b3e5d2-a3c9-41fe-9a22-e36ba7760daa-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f76js2\" (UID: \"98b3e5d2-a3c9-41fe-9a22-e36ba7760daa\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f76js2"
Sep 10 17:45:48.019266 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:45:48.019234    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/98b3e5d2-a3c9-41fe-9a22-e36ba7760daa-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f76js2\" (UID: \"98b3e5d2-a3c9-41fe-9a22-e36ba7760daa\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f76js2"
Sep 10 17:45:48.019319 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:45:48.019281    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/98b3e5d2-a3c9-41fe-9a22-e36ba7760daa-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f76js2\" (UID: \"98b3e5d2-a3c9-41fe-9a22-e36ba7760daa\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f76js2"
Sep 10 17:45:48.021261 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:45:48.021233    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/98b3e5d2-a3c9-41fe-9a22-e36ba7760daa-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f76js2\" (UID: \"98b3e5d2-a3c9-41fe-9a22-e36ba7760daa\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f76js2"
Sep 10 17:45:48.021604 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:45:48.021555    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/98b3e5d2-a3c9-41fe-9a22-e36ba7760daa-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f76js2\" (UID: \"98b3e5d2-a3c9-41fe-9a22-e36ba7760daa\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f76js2"
Sep 10 17:45:48.027591 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:45:48.027547    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zbdt\" (UniqueName: \"kubernetes.io/projected/98b3e5d2-a3c9-41fe-9a22-e36ba7760daa-kube-api-access-8zbdt\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f76js2\" (UID: \"98b3e5d2-a3c9-41fe-9a22-e36ba7760daa\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f76js2"
Sep 10 17:45:48.131941 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:45:48.131911    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f76js2"
Sep 10 17:45:48.407299 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:45:48.407271    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f76js2"]
Sep 10 17:45:48.409946 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:45:48.409920    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98b3e5d2_a3c9_41fe_9a22_e36ba7760daa.slice/crio-d3956e87fd33638fab00c52a185c4991ef2040f24b525b2416a354e2ddfea263 WatchSource:0}: Error finding container d3956e87fd33638fab00c52a185c4991ef2040f24b525b2416a354e2ddfea263: Status 404 returned error can't find the container with id d3956e87fd33638fab00c52a185c4991ef2040f24b525b2416a354e2ddfea263
Sep 10 17:45:48.436344 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:45:48.436314    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f76js2" event={"ID":"98b3e5d2-a3c9-41fe-9a22-e36ba7760daa","Type":"ContainerStarted","Data":"d3956e87fd33638fab00c52a185c4991ef2040f24b525b2416a354e2ddfea263"}
Sep 10 17:45:49.443985 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:45:49.443942    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f76js2" event={"ID":"98b3e5d2-a3c9-41fe-9a22-e36ba7760daa","Type":"ContainerStarted","Data":"61aea9add2a3b2bc6d9d7f1f58197f0f6e383bfc2c806497130887416efa6e67"}
Sep 10 17:45:49.464074 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:45:49.464024    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f76js2" podStartSLOduration=2.253662262 podStartE2EDuration="2.464008629s" totalImagesPullingTime="210.346367ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:45:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:45:49.462895089 +0000 UTC m=+593.462725168" watchObservedRunningTime="2026-09-10 17:45:49.464008629 +0000 UTC m=+593.463838694"
Sep 10 17:45:56.420673 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:45:56.420642    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-rb854_97172b14-b84a-476e-89f1-05d3e70df416/console-operator/1.log"
Sep 10 17:45:56.421095 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:45:56.420757    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-rb854_97172b14-b84a-476e-89f1-05d3e70df416/console-operator/1.log"
Sep 10 17:45:58.132393 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:45:58.132353    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f76js2"
Sep 10 17:45:58.132907 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:45:58.132424    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f76js2"
Sep 10 17:45:58.145242 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:45:58.145217    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f76js2"
Sep 10 17:45:58.497011 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:45:58.496926    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f76js2"
Sep 10 17:47:38.028364 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:47:38.028323    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-9d8b447bf-7dl75"]
Sep 10 17:47:38.028831 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:47:38.028540    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-9d8b447bf-7dl75" podUID="80ecaacc-72d8-42b4-b3d0-f41f6e27391b" containerName="authorino" containerID="cri-o://5168f49548e6f3313113ed4c085c3ae03468f4c8475cc55b5dcbdeb53957f931" gracePeriod=30
Sep 10 17:47:38.287901 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:47:38.287848    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-9d8b447bf-7dl75"
Sep 10 17:47:38.371977 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:47:38.371948    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8dfw9\" (UniqueName: \"kubernetes.io/projected/80ecaacc-72d8-42b4-b3d0-f41f6e27391b-kube-api-access-8dfw9\") pod \"80ecaacc-72d8-42b4-b3d0-f41f6e27391b\" (UID: \"80ecaacc-72d8-42b4-b3d0-f41f6e27391b\") "
Sep 10 17:47:38.372140 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:47:38.371997    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/80ecaacc-72d8-42b4-b3d0-f41f6e27391b-tls-cert\") pod \"80ecaacc-72d8-42b4-b3d0-f41f6e27391b\" (UID: \"80ecaacc-72d8-42b4-b3d0-f41f6e27391b\") "
Sep 10 17:47:38.374277 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:47:38.374250    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80ecaacc-72d8-42b4-b3d0-f41f6e27391b-kube-api-access-8dfw9" (OuterVolumeSpecName: "kube-api-access-8dfw9") pod "80ecaacc-72d8-42b4-b3d0-f41f6e27391b" (UID: "80ecaacc-72d8-42b4-b3d0-f41f6e27391b"). InnerVolumeSpecName "kube-api-access-8dfw9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:47:38.382321 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:47:38.382288    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80ecaacc-72d8-42b4-b3d0-f41f6e27391b-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "80ecaacc-72d8-42b4-b3d0-f41f6e27391b" (UID: "80ecaacc-72d8-42b4-b3d0-f41f6e27391b"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 17:47:38.472981 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:47:38.472949    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8dfw9\" (UniqueName: \"kubernetes.io/projected/80ecaacc-72d8-42b4-b3d0-f41f6e27391b-kube-api-access-8dfw9\") on node \"ip-10-0-129-122.ec2.internal\" DevicePath \"\""
Sep 10 17:47:38.472981 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:47:38.472983    2579 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/80ecaacc-72d8-42b4-b3d0-f41f6e27391b-tls-cert\") on node \"ip-10-0-129-122.ec2.internal\" DevicePath \"\""
Sep 10 17:47:38.953967 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:47:38.953933    2579 generic.go:356] "Generic (PLEG): container finished" podID="80ecaacc-72d8-42b4-b3d0-f41f6e27391b" containerID="5168f49548e6f3313113ed4c085c3ae03468f4c8475cc55b5dcbdeb53957f931" exitCode=0
Sep 10 17:47:38.954249 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:47:38.953976    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-9d8b447bf-7dl75"
Sep 10 17:47:38.954249 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:47:38.954032    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-9d8b447bf-7dl75" event={"ID":"80ecaacc-72d8-42b4-b3d0-f41f6e27391b","Type":"ContainerDied","Data":"5168f49548e6f3313113ed4c085c3ae03468f4c8475cc55b5dcbdeb53957f931"}
Sep 10 17:47:38.954249 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:47:38.954073    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-9d8b447bf-7dl75" event={"ID":"80ecaacc-72d8-42b4-b3d0-f41f6e27391b","Type":"ContainerDied","Data":"55e03d68e3cc7bb1e58a091c04b1ca4f86a890718e78c74552b6175ec9a173e6"}
Sep 10 17:47:38.954249 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:47:38.954089    2579 scope.go:117] "RemoveContainer" containerID="5168f49548e6f3313113ed4c085c3ae03468f4c8475cc55b5dcbdeb53957f931"
Sep 10 17:47:38.962547 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:47:38.962532    2579 scope.go:117] "RemoveContainer" containerID="5168f49548e6f3313113ed4c085c3ae03468f4c8475cc55b5dcbdeb53957f931"
Sep 10 17:47:38.962791 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:47:38.962773    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5168f49548e6f3313113ed4c085c3ae03468f4c8475cc55b5dcbdeb53957f931\": container with ID starting with 5168f49548e6f3313113ed4c085c3ae03468f4c8475cc55b5dcbdeb53957f931 not found: ID does not exist" containerID="5168f49548e6f3313113ed4c085c3ae03468f4c8475cc55b5dcbdeb53957f931"
Sep 10 17:47:38.962841 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:47:38.962800    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5168f49548e6f3313113ed4c085c3ae03468f4c8475cc55b5dcbdeb53957f931"} err="failed to get container status \"5168f49548e6f3313113ed4c085c3ae03468f4c8475cc55b5dcbdeb53957f931\": rpc error: code = NotFound desc = could not find container \"5168f49548e6f3313113ed4c085c3ae03468f4c8475cc55b5dcbdeb53957f931\": container with ID starting with 5168f49548e6f3313113ed4c085c3ae03468f4c8475cc55b5dcbdeb53957f931 not found: ID does not exist"
Sep 10 17:47:38.984629 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:47:38.984600    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-9d8b447bf-7dl75"]
Sep 10 17:47:38.993990 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:47:38.993967    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-9d8b447bf-7dl75"]
Sep 10 17:47:40.475052 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:47:40.475015    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="80ecaacc-72d8-42b4-b3d0-f41f6e27391b" path="/var/lib/kubelet/pods/80ecaacc-72d8-42b4-b3d0-f41f6e27391b/volumes"
Sep 10 17:48:06.280282 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:06.280244    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7708-b77b12"]
Sep 10 17:48:06.280713 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:06.280627    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="80ecaacc-72d8-42b4-b3d0-f41f6e27391b" containerName="authorino"
Sep 10 17:48:06.280713 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:06.280640    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="80ecaacc-72d8-42b4-b3d0-f41f6e27391b" containerName="authorino"
Sep 10 17:48:06.280713 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:06.280703    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="80ecaacc-72d8-42b4-b3d0-f41f6e27391b" containerName="authorino"
Sep 10 17:48:06.284968 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:06.284944    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7708-b77b12"
Sep 10 17:48:06.287730 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:06.287706    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-w4zxj\""
Sep 10 17:48:06.292682 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:06.292651    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7708-b77b12"]
Sep 10 17:48:06.420443 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:06.420404    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdmdk\" (UniqueName: \"kubernetes.io/projected/92daab40-6b16-4d3a-9bea-08e4db55e67c-kube-api-access-bdmdk\") pod \"test-curl-7708-b77b12\" (UID: \"92daab40-6b16-4d3a-9bea-08e4db55e67c\") " pod="opendatahub/test-curl-7708-b77b12"
Sep 10 17:48:06.521491 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:06.521460    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdmdk\" (UniqueName: \"kubernetes.io/projected/92daab40-6b16-4d3a-9bea-08e4db55e67c-kube-api-access-bdmdk\") pod \"test-curl-7708-b77b12\" (UID: \"92daab40-6b16-4d3a-9bea-08e4db55e67c\") " pod="opendatahub/test-curl-7708-b77b12"
Sep 10 17:48:06.530741 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:06.530677    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdmdk\" (UniqueName: \"kubernetes.io/projected/92daab40-6b16-4d3a-9bea-08e4db55e67c-kube-api-access-bdmdk\") pod \"test-curl-7708-b77b12\" (UID: \"92daab40-6b16-4d3a-9bea-08e4db55e67c\") " pod="opendatahub/test-curl-7708-b77b12"
Sep 10 17:48:06.597272 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:06.597248    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7708-b77b12"
Sep 10 17:48:06.867762 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:06.867680    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7708-b77b12"]
Sep 10 17:48:06.871382 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:48:06.871352    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod92daab40_6b16_4d3a_9bea_08e4db55e67c.slice/crio-7595ef345aaf5813fb2c6d1a4a1ab06a3e046a092fd8cda70d19d83d78f04c2c WatchSource:0}: Error finding container 7595ef345aaf5813fb2c6d1a4a1ab06a3e046a092fd8cda70d19d83d78f04c2c: Status 404 returned error can't find the container with id 7595ef345aaf5813fb2c6d1a4a1ab06a3e046a092fd8cda70d19d83d78f04c2c
Sep 10 17:48:06.873433 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:06.873409    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 17:48:07.083127 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:07.083092    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7708-b77b12" event={"ID":"92daab40-6b16-4d3a-9bea-08e4db55e67c","Type":"ContainerStarted","Data":"7595ef345aaf5813fb2c6d1a4a1ab06a3e046a092fd8cda70d19d83d78f04c2c"}
Sep 10 17:48:08.824994 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:08.824957    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-48d00b-54fc5bb588-sntsg"]
Sep 10 17:48:08.828302 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:08.828275    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-48d00b-54fc5bb588-sntsg"
Sep 10 17:48:08.831233 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:08.831206    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-48d00b-serving-cert\""
Sep 10 17:48:08.831366 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:08.831216    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-tdclw\""
Sep 10 17:48:08.841727 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:08.841702    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-48d00b-54fc5bb588-sntsg"]
Sep 10 17:48:08.945096 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:08.945068    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8dc46ba-3e39-4014-86ff-ea9e12e38c16-openshift-service-ca\") pod \"maas-api-e2e-inf-a-48d00b-54fc5bb588-sntsg\" (UID: \"b8dc46ba-3e39-4014-86ff-ea9e12e38c16\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-48d00b-54fc5bb588-sntsg"
Sep 10 17:48:08.945197 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:08.945105    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b8dc46ba-3e39-4014-86ff-ea9e12e38c16-maas-api-tls\") pod \"maas-api-e2e-inf-a-48d00b-54fc5bb588-sntsg\" (UID: \"b8dc46ba-3e39-4014-86ff-ea9e12e38c16\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-48d00b-54fc5bb588-sntsg"
Sep 10 17:48:08.945197 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:08.945133    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhnbj\" (UniqueName: \"kubernetes.io/projected/b8dc46ba-3e39-4014-86ff-ea9e12e38c16-kube-api-access-jhnbj\") pod \"maas-api-e2e-inf-a-48d00b-54fc5bb588-sntsg\" (UID: \"b8dc46ba-3e39-4014-86ff-ea9e12e38c16\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-48d00b-54fc5bb588-sntsg"
Sep 10 17:48:08.945280 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:08.945222    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b8dc46ba-3e39-4014-86ff-ea9e12e38c16-metrics-certs\") pod \"maas-api-e2e-inf-a-48d00b-54fc5bb588-sntsg\" (UID: \"b8dc46ba-3e39-4014-86ff-ea9e12e38c16\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-48d00b-54fc5bb588-sntsg"
Sep 10 17:48:09.045834 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:09.045806    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8dc46ba-3e39-4014-86ff-ea9e12e38c16-openshift-service-ca\") pod \"maas-api-e2e-inf-a-48d00b-54fc5bb588-sntsg\" (UID: \"b8dc46ba-3e39-4014-86ff-ea9e12e38c16\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-48d00b-54fc5bb588-sntsg"
Sep 10 17:48:09.045973 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:09.045845    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b8dc46ba-3e39-4014-86ff-ea9e12e38c16-maas-api-tls\") pod \"maas-api-e2e-inf-a-48d00b-54fc5bb588-sntsg\" (UID: \"b8dc46ba-3e39-4014-86ff-ea9e12e38c16\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-48d00b-54fc5bb588-sntsg"
Sep 10 17:48:09.045973 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:09.045886    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhnbj\" (UniqueName: \"kubernetes.io/projected/b8dc46ba-3e39-4014-86ff-ea9e12e38c16-kube-api-access-jhnbj\") pod \"maas-api-e2e-inf-a-48d00b-54fc5bb588-sntsg\" (UID: \"b8dc46ba-3e39-4014-86ff-ea9e12e38c16\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-48d00b-54fc5bb588-sntsg"
Sep 10 17:48:09.045973 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:09.045929    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b8dc46ba-3e39-4014-86ff-ea9e12e38c16-metrics-certs\") pod \"maas-api-e2e-inf-a-48d00b-54fc5bb588-sntsg\" (UID: \"b8dc46ba-3e39-4014-86ff-ea9e12e38c16\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-48d00b-54fc5bb588-sntsg"
Sep 10 17:48:09.046132 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:48:09.046039    2579 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-48d00b-serving-cert: secret "maas-api-e2e-inf-a-48d00b-serving-cert" not found
Sep 10 17:48:09.046132 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:48:09.046109    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b8dc46ba-3e39-4014-86ff-ea9e12e38c16-maas-api-tls podName:b8dc46ba-3e39-4014-86ff-ea9e12e38c16 nodeName:}" failed. No retries permitted until 2026-09-10 17:48:09.546088184 +0000 UTC m=+733.545918232 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b8dc46ba-3e39-4014-86ff-ea9e12e38c16-maas-api-tls") pod "maas-api-e2e-inf-a-48d00b-54fc5bb588-sntsg" (UID: "b8dc46ba-3e39-4014-86ff-ea9e12e38c16") : secret "maas-api-e2e-inf-a-48d00b-serving-cert" not found
Sep 10 17:48:09.046597 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:09.046541    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8dc46ba-3e39-4014-86ff-ea9e12e38c16-openshift-service-ca\") pod \"maas-api-e2e-inf-a-48d00b-54fc5bb588-sntsg\" (UID: \"b8dc46ba-3e39-4014-86ff-ea9e12e38c16\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-48d00b-54fc5bb588-sntsg"
Sep 10 17:48:09.048391 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:09.048371    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b8dc46ba-3e39-4014-86ff-ea9e12e38c16-metrics-certs\") pod \"maas-api-e2e-inf-a-48d00b-54fc5bb588-sntsg\" (UID: \"b8dc46ba-3e39-4014-86ff-ea9e12e38c16\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-48d00b-54fc5bb588-sntsg"
Sep 10 17:48:09.056675 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:09.056655    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhnbj\" (UniqueName: \"kubernetes.io/projected/b8dc46ba-3e39-4014-86ff-ea9e12e38c16-kube-api-access-jhnbj\") pod \"maas-api-e2e-inf-a-48d00b-54fc5bb588-sntsg\" (UID: \"b8dc46ba-3e39-4014-86ff-ea9e12e38c16\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-48d00b-54fc5bb588-sntsg"
Sep 10 17:48:09.095618 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:09.095530    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7708-b77b12" event={"ID":"92daab40-6b16-4d3a-9bea-08e4db55e67c","Type":"ContainerStarted","Data":"860a7f77eba31f9866970d9633e3a68826eeacd87f7b285d301eeaeffedc264b"}
Sep 10 17:48:09.115068 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:09.115019    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7708-b77b12" podStartSLOduration=1.142883858 podStartE2EDuration="3.115003463s" totalImagesPullingTime="1.972119605s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:48:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:48:09.114867958 +0000 UTC m=+733.114698025" watchObservedRunningTime="2026-09-10 17:48:09.115003463 +0000 UTC m=+733.114833531"
Sep 10 17:48:09.550701 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:09.550675    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b8dc46ba-3e39-4014-86ff-ea9e12e38c16-maas-api-tls\") pod \"maas-api-e2e-inf-a-48d00b-54fc5bb588-sntsg\" (UID: \"b8dc46ba-3e39-4014-86ff-ea9e12e38c16\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-48d00b-54fc5bb588-sntsg"
Sep 10 17:48:09.553675 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:09.553650    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b8dc46ba-3e39-4014-86ff-ea9e12e38c16-maas-api-tls\") pod \"maas-api-e2e-inf-a-48d00b-54fc5bb588-sntsg\" (UID: \"b8dc46ba-3e39-4014-86ff-ea9e12e38c16\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-48d00b-54fc5bb588-sntsg"
Sep 10 17:48:09.743229 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:09.743193    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-48d00b-54fc5bb588-sntsg"
Sep 10 17:48:09.777140 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:09.777050    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7708-b77b12"]
Sep 10 17:48:09.779705 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:09.779674    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7708-b77b12"]
Sep 10 17:48:09.909727 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:09.909695    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7708-42cbc1"]
Sep 10 17:48:09.910111 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:09.910068    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="92daab40-6b16-4d3a-9bea-08e4db55e67c" containerName="test-curl-7708-b77b12"
Sep 10 17:48:09.910111 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:09.910079    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="92daab40-6b16-4d3a-9bea-08e4db55e67c" containerName="test-curl-7708-b77b12"
Sep 10 17:48:09.910192 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:09.910155    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="92daab40-6b16-4d3a-9bea-08e4db55e67c" containerName="test-curl-7708-b77b12"
Sep 10 17:48:09.914133 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:09.914116    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7708-42cbc1"
Sep 10 17:48:09.934411 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:09.934381    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7708-42cbc1"]
Sep 10 17:48:10.030709 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:10.030676    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-48d00b-54fc5bb588-sntsg"]
Sep 10 17:48:10.034436 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:48:10.034404    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8dc46ba_3e39_4014_86ff_ea9e12e38c16.slice/crio-aa4914eced937d10789fb0b789698bc3e8430fccb7839867adc9785bf4f6e729 WatchSource:0}: Error finding container aa4914eced937d10789fb0b789698bc3e8430fccb7839867adc9785bf4f6e729: Status 404 returned error can't find the container with id aa4914eced937d10789fb0b789698bc3e8430fccb7839867adc9785bf4f6e729
Sep 10 17:48:10.055163 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:10.055093    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ns5t7\" (UniqueName: \"kubernetes.io/projected/ca70f0e6-7da0-49f1-9779-9c4b367b87ca-kube-api-access-ns5t7\") pod \"test-curl-7708-42cbc1\" (UID: \"ca70f0e6-7da0-49f1-9779-9c4b367b87ca\") " pod="opendatahub/test-curl-7708-42cbc1"
Sep 10 17:48:10.101633 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:10.101598    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-48d00b-54fc5bb588-sntsg" event={"ID":"b8dc46ba-3e39-4014-86ff-ea9e12e38c16","Type":"ContainerStarted","Data":"aa4914eced937d10789fb0b789698bc3e8430fccb7839867adc9785bf4f6e729"}
Sep 10 17:48:10.101764 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:48:10.101738    2579 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7708-b77b12" container="test-curl-7708-b77b12"
Sep 10 17:48:10.155699 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:10.155672    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ns5t7\" (UniqueName: \"kubernetes.io/projected/ca70f0e6-7da0-49f1-9779-9c4b367b87ca-kube-api-access-ns5t7\") pod \"test-curl-7708-42cbc1\" (UID: \"ca70f0e6-7da0-49f1-9779-9c4b367b87ca\") " pod="opendatahub/test-curl-7708-42cbc1"
Sep 10 17:48:10.163609 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:10.163579    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ns5t7\" (UniqueName: \"kubernetes.io/projected/ca70f0e6-7da0-49f1-9779-9c4b367b87ca-kube-api-access-ns5t7\") pod \"test-curl-7708-42cbc1\" (UID: \"ca70f0e6-7da0-49f1-9779-9c4b367b87ca\") " pod="opendatahub/test-curl-7708-42cbc1"
Sep 10 17:48:10.231411 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:10.231389    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7708-42cbc1"
Sep 10 17:48:10.532520 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:10.532486    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7708-42cbc1"]
Sep 10 17:48:10.533901 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:48:10.533876    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podca70f0e6_7da0_49f1_9779_9c4b367b87ca.slice/crio-bf362ab02d90d2682a5766eeb06bf20b3a98109792a2115dce80e03bf6fda944 WatchSource:0}: Error finding container bf362ab02d90d2682a5766eeb06bf20b3a98109792a2115dce80e03bf6fda944: Status 404 returned error can't find the container with id bf362ab02d90d2682a5766eeb06bf20b3a98109792a2115dce80e03bf6fda944
Sep 10 17:48:11.107503 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:11.107466    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7708-42cbc1" event={"ID":"ca70f0e6-7da0-49f1-9779-9c4b367b87ca","Type":"ContainerStarted","Data":"da03de0a65a90322fe92ca2de52835cb81870cf0e3cf658768628a331de30fa4"}
Sep 10 17:48:11.107986 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:11.107509    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7708-42cbc1" event={"ID":"ca70f0e6-7da0-49f1-9779-9c4b367b87ca","Type":"ContainerStarted","Data":"bf362ab02d90d2682a5766eeb06bf20b3a98109792a2115dce80e03bf6fda944"}
Sep 10 17:48:11.109224 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:11.109194    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-48d00b-54fc5bb588-sntsg" event={"ID":"b8dc46ba-3e39-4014-86ff-ea9e12e38c16","Type":"ContainerStarted","Data":"4f91330bcc42e66c0e26e3aa8e454a4f99e1ad6e733bc91361ebc36f12232996"}
Sep 10 17:48:11.109350 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:11.109264    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-48d00b-54fc5bb588-sntsg"
Sep 10 17:48:11.109431 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:11.109390    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7708-b77b12" podUID="92daab40-6b16-4d3a-9bea-08e4db55e67c" containerName="test-curl-7708-b77b12" containerID="cri-o://860a7f77eba31f9866970d9633e3a68826eeacd87f7b285d301eeaeffedc264b" gracePeriod=2
Sep 10 17:48:11.127456 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:11.127403    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7708-42cbc1" podStartSLOduration=2.127386166 podStartE2EDuration="2.127386166s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:48:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:48:11.124934414 +0000 UTC m=+735.124764482" watchObservedRunningTime="2026-09-10 17:48:11.127386166 +0000 UTC m=+735.127216233"
Sep 10 17:48:11.147692 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:11.147646    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-48d00b-54fc5bb588-sntsg" podStartSLOduration=2.694656434 podStartE2EDuration="3.147632491s" totalImagesPullingTime="452.976057ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:48:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:48:11.147247933 +0000 UTC m=+735.147078001" watchObservedRunningTime="2026-09-10 17:48:11.147632491 +0000 UTC m=+735.147462558"
Sep 10 17:48:11.773939 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:11.773907    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7708-42cbc1"]
Sep 10 17:48:11.780911 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:11.780827    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7708-42cbc1"]
Sep 10 17:48:12.113757 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:48:12.113668    2579 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7708-42cbc1" container="test-curl-7708-42cbc1"
Sep 10 17:48:13.100583 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:13.100535    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-4601bf83-openshift-default-7d5654d96d-g2lmw"]
Sep 10 17:48:13.100968 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:13.100953    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca70f0e6-7da0-49f1-9779-9c4b367b87ca" containerName="test-curl-7708-42cbc1"
Sep 10 17:48:13.101020 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:13.100971    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca70f0e6-7da0-49f1-9779-9c4b367b87ca" containerName="test-curl-7708-42cbc1"
Sep 10 17:48:13.101064 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:13.101033    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="ca70f0e6-7da0-49f1-9779-9c4b367b87ca" containerName="test-curl-7708-42cbc1"
Sep 10 17:48:13.104365 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:13.104345    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-4601bf83-openshift-default-7d5654d96d-g2lmw"
Sep 10 17:48:13.107154 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:13.107134    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 10 17:48:13.107323 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:13.107305    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-4601bf83-openshift-default-dockercfg-zd8t7\""
Sep 10 17:48:13.119737 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:13.119705    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7708-42cbc1" podUID="ca70f0e6-7da0-49f1-9779-9c4b367b87ca" containerName="test-curl-7708-42cbc1" containerID="cri-o://da03de0a65a90322fe92ca2de52835cb81870cf0e3cf658768628a331de30fa4" gracePeriod=2
Sep 10 17:48:13.120928 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:13.120902    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-4601bf83-openshift-default-7d5654d96d-g2lmw"]
Sep 10 17:48:13.285870 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:13.285835    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7774ff67-4869-4805-95f2-b6aad0e5303a-istio-data\") pod \"e2e-ait-4601bf83-openshift-default-7d5654d96d-g2lmw\" (UID: \"7774ff67-4869-4805-95f2-b6aad0e5303a\") " pod="openshift-ingress/e2e-ait-4601bf83-openshift-default-7d5654d96d-g2lmw"
Sep 10 17:48:13.286016 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:13.285873    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7774ff67-4869-4805-95f2-b6aad0e5303a-istiod-ca-cert\") pod \"e2e-ait-4601bf83-openshift-default-7d5654d96d-g2lmw\" (UID: \"7774ff67-4869-4805-95f2-b6aad0e5303a\") " pod="openshift-ingress/e2e-ait-4601bf83-openshift-default-7d5654d96d-g2lmw"
Sep 10 17:48:13.286016 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:13.285963    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7774ff67-4869-4805-95f2-b6aad0e5303a-istio-podinfo\") pod \"e2e-ait-4601bf83-openshift-default-7d5654d96d-g2lmw\" (UID: \"7774ff67-4869-4805-95f2-b6aad0e5303a\") " pod="openshift-ingress/e2e-ait-4601bf83-openshift-default-7d5654d96d-g2lmw"
Sep 10 17:48:13.286016 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:13.286008    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7774ff67-4869-4805-95f2-b6aad0e5303a-workload-certs\") pod \"e2e-ait-4601bf83-openshift-default-7d5654d96d-g2lmw\" (UID: \"7774ff67-4869-4805-95f2-b6aad0e5303a\") " pod="openshift-ingress/e2e-ait-4601bf83-openshift-default-7d5654d96d-g2lmw"
Sep 10 17:48:13.286164 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:13.286033    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7774ff67-4869-4805-95f2-b6aad0e5303a-istio-envoy\") pod \"e2e-ait-4601bf83-openshift-default-7d5654d96d-g2lmw\" (UID: \"7774ff67-4869-4805-95f2-b6aad0e5303a\") " pod="openshift-ingress/e2e-ait-4601bf83-openshift-default-7d5654d96d-g2lmw"
Sep 10 17:48:13.286164 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:13.286087    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjvn7\" (UniqueName: \"kubernetes.io/projected/7774ff67-4869-4805-95f2-b6aad0e5303a-kube-api-access-qjvn7\") pod \"e2e-ait-4601bf83-openshift-default-7d5654d96d-g2lmw\" (UID: \"7774ff67-4869-4805-95f2-b6aad0e5303a\") " pod="openshift-ingress/e2e-ait-4601bf83-openshift-default-7d5654d96d-g2lmw"
Sep 10 17:48:13.286164 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:13.286113    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7774ff67-4869-4805-95f2-b6aad0e5303a-credential-socket\") pod \"e2e-ait-4601bf83-openshift-default-7d5654d96d-g2lmw\" (UID: \"7774ff67-4869-4805-95f2-b6aad0e5303a\") " pod="openshift-ingress/e2e-ait-4601bf83-openshift-default-7d5654d96d-g2lmw"
Sep 10 17:48:13.286164 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:13.286149    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7774ff67-4869-4805-95f2-b6aad0e5303a-istio-token\") pod \"e2e-ait-4601bf83-openshift-default-7d5654d96d-g2lmw\" (UID: \"7774ff67-4869-4805-95f2-b6aad0e5303a\") " pod="openshift-ingress/e2e-ait-4601bf83-openshift-default-7d5654d96d-g2lmw"
Sep 10 17:48:13.286356 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:13.286286    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7774ff67-4869-4805-95f2-b6aad0e5303a-workload-socket\") pod \"e2e-ait-4601bf83-openshift-default-7d5654d96d-g2lmw\" (UID: \"7774ff67-4869-4805-95f2-b6aad0e5303a\") " pod="openshift-ingress/e2e-ait-4601bf83-openshift-default-7d5654d96d-g2lmw"
Sep 10 17:48:13.290860 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:13.290841    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7708-b77b12"
Sep 10 17:48:13.387301 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:13.387278    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7774ff67-4869-4805-95f2-b6aad0e5303a-istio-data\") pod \"e2e-ait-4601bf83-openshift-default-7d5654d96d-g2lmw\" (UID: \"7774ff67-4869-4805-95f2-b6aad0e5303a\") " pod="openshift-ingress/e2e-ait-4601bf83-openshift-default-7d5654d96d-g2lmw"
Sep 10 17:48:13.387449 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:13.387310    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7774ff67-4869-4805-95f2-b6aad0e5303a-istiod-ca-cert\") pod \"e2e-ait-4601bf83-openshift-default-7d5654d96d-g2lmw\" (UID: \"7774ff67-4869-4805-95f2-b6aad0e5303a\") " pod="openshift-ingress/e2e-ait-4601bf83-openshift-default-7d5654d96d-g2lmw"
Sep 10 17:48:13.387449 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:13.387329    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7774ff67-4869-4805-95f2-b6aad0e5303a-istio-podinfo\") pod \"e2e-ait-4601bf83-openshift-default-7d5654d96d-g2lmw\" (UID: \"7774ff67-4869-4805-95f2-b6aad0e5303a\") " pod="openshift-ingress/e2e-ait-4601bf83-openshift-default-7d5654d96d-g2lmw"
Sep 10 17:48:13.387449 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:13.387353    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7774ff67-4869-4805-95f2-b6aad0e5303a-workload-certs\") pod \"e2e-ait-4601bf83-openshift-default-7d5654d96d-g2lmw\" (UID: \"7774ff67-4869-4805-95f2-b6aad0e5303a\") " pod="openshift-ingress/e2e-ait-4601bf83-openshift-default-7d5654d96d-g2lmw"
Sep 10 17:48:13.387641 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:13.387489    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7774ff67-4869-4805-95f2-b6aad0e5303a-istio-envoy\") pod \"e2e-ait-4601bf83-openshift-default-7d5654d96d-g2lmw\" (UID: \"7774ff67-4869-4805-95f2-b6aad0e5303a\") " pod="openshift-ingress/e2e-ait-4601bf83-openshift-default-7d5654d96d-g2lmw"
Sep 10 17:48:13.387641 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:13.387540    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjvn7\" (UniqueName: \"kubernetes.io/projected/7774ff67-4869-4805-95f2-b6aad0e5303a-kube-api-access-qjvn7\") pod \"e2e-ait-4601bf83-openshift-default-7d5654d96d-g2lmw\" (UID: \"7774ff67-4869-4805-95f2-b6aad0e5303a\") " pod="openshift-ingress/e2e-ait-4601bf83-openshift-default-7d5654d96d-g2lmw"
Sep 10 17:48:13.387641 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:13.387583    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7774ff67-4869-4805-95f2-b6aad0e5303a-credential-socket\") pod \"e2e-ait-4601bf83-openshift-default-7d5654d96d-g2lmw\" (UID: \"7774ff67-4869-4805-95f2-b6aad0e5303a\") " pod="openshift-ingress/e2e-ait-4601bf83-openshift-default-7d5654d96d-g2lmw"
Sep 10 17:48:13.387889 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:13.387669    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7774ff67-4869-4805-95f2-b6aad0e5303a-istio-token\") pod \"e2e-ait-4601bf83-openshift-default-7d5654d96d-g2lmw\" (UID: \"7774ff67-4869-4805-95f2-b6aad0e5303a\") " pod="openshift-ingress/e2e-ait-4601bf83-openshift-default-7d5654d96d-g2lmw"
Sep 10 17:48:13.387889 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:13.387730    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7774ff67-4869-4805-95f2-b6aad0e5303a-istio-data\") pod \"e2e-ait-4601bf83-openshift-default-7d5654d96d-g2lmw\" (UID: \"7774ff67-4869-4805-95f2-b6aad0e5303a\") " pod="openshift-ingress/e2e-ait-4601bf83-openshift-default-7d5654d96d-g2lmw"
Sep 10 17:48:13.387889 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:13.387744    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7774ff67-4869-4805-95f2-b6aad0e5303a-workload-certs\") pod \"e2e-ait-4601bf83-openshift-default-7d5654d96d-g2lmw\" (UID: \"7774ff67-4869-4805-95f2-b6aad0e5303a\") " pod="openshift-ingress/e2e-ait-4601bf83-openshift-default-7d5654d96d-g2lmw"
Sep 10 17:48:13.387889 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:13.387732    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7774ff67-4869-4805-95f2-b6aad0e5303a-workload-socket\") pod \"e2e-ait-4601bf83-openshift-default-7d5654d96d-g2lmw\" (UID: \"7774ff67-4869-4805-95f2-b6aad0e5303a\") " pod="openshift-ingress/e2e-ait-4601bf83-openshift-default-7d5654d96d-g2lmw"
Sep 10 17:48:13.388288 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:13.388013    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7774ff67-4869-4805-95f2-b6aad0e5303a-credential-socket\") pod \"e2e-ait-4601bf83-openshift-default-7d5654d96d-g2lmw\" (UID: \"7774ff67-4869-4805-95f2-b6aad0e5303a\") " pod="openshift-ingress/e2e-ait-4601bf83-openshift-default-7d5654d96d-g2lmw"
Sep 10 17:48:13.388288 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:13.388056    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7774ff67-4869-4805-95f2-b6aad0e5303a-workload-socket\") pod \"e2e-ait-4601bf83-openshift-default-7d5654d96d-g2lmw\" (UID: \"7774ff67-4869-4805-95f2-b6aad0e5303a\") " pod="openshift-ingress/e2e-ait-4601bf83-openshift-default-7d5654d96d-g2lmw"
Sep 10 17:48:13.388288 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:13.388082    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7774ff67-4869-4805-95f2-b6aad0e5303a-istiod-ca-cert\") pod \"e2e-ait-4601bf83-openshift-default-7d5654d96d-g2lmw\" (UID: \"7774ff67-4869-4805-95f2-b6aad0e5303a\") " pod="openshift-ingress/e2e-ait-4601bf83-openshift-default-7d5654d96d-g2lmw"
Sep 10 17:48:13.389760 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:13.389735    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7774ff67-4869-4805-95f2-b6aad0e5303a-istio-envoy\") pod \"e2e-ait-4601bf83-openshift-default-7d5654d96d-g2lmw\" (UID: \"7774ff67-4869-4805-95f2-b6aad0e5303a\") " pod="openshift-ingress/e2e-ait-4601bf83-openshift-default-7d5654d96d-g2lmw"
Sep 10 17:48:13.390037 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:13.390021    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7774ff67-4869-4805-95f2-b6aad0e5303a-istio-podinfo\") pod \"e2e-ait-4601bf83-openshift-default-7d5654d96d-g2lmw\" (UID: \"7774ff67-4869-4805-95f2-b6aad0e5303a\") " pod="openshift-ingress/e2e-ait-4601bf83-openshift-default-7d5654d96d-g2lmw"
Sep 10 17:48:13.397466 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:13.397443    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7774ff67-4869-4805-95f2-b6aad0e5303a-istio-token\") pod \"e2e-ait-4601bf83-openshift-default-7d5654d96d-g2lmw\" (UID: \"7774ff67-4869-4805-95f2-b6aad0e5303a\") " pod="openshift-ingress/e2e-ait-4601bf83-openshift-default-7d5654d96d-g2lmw"
Sep 10 17:48:13.397589 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:13.397454    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjvn7\" (UniqueName: \"kubernetes.io/projected/7774ff67-4869-4805-95f2-b6aad0e5303a-kube-api-access-qjvn7\") pod \"e2e-ait-4601bf83-openshift-default-7d5654d96d-g2lmw\" (UID: \"7774ff67-4869-4805-95f2-b6aad0e5303a\") " pod="openshift-ingress/e2e-ait-4601bf83-openshift-default-7d5654d96d-g2lmw"
Sep 10 17:48:13.416356 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:13.416337    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-4601bf83-openshift-default-7d5654d96d-g2lmw"
Sep 10 17:48:13.488534 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:13.488509    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bdmdk\" (UniqueName: \"kubernetes.io/projected/92daab40-6b16-4d3a-9bea-08e4db55e67c-kube-api-access-bdmdk\") pod \"92daab40-6b16-4d3a-9bea-08e4db55e67c\" (UID: \"92daab40-6b16-4d3a-9bea-08e4db55e67c\") "
Sep 10 17:48:13.491126 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:13.491105    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92daab40-6b16-4d3a-9bea-08e4db55e67c-kube-api-access-bdmdk" (OuterVolumeSpecName: "kube-api-access-bdmdk") pod "92daab40-6b16-4d3a-9bea-08e4db55e67c" (UID: "92daab40-6b16-4d3a-9bea-08e4db55e67c"). InnerVolumeSpecName "kube-api-access-bdmdk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:48:13.589735 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:13.589705    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bdmdk\" (UniqueName: \"kubernetes.io/projected/92daab40-6b16-4d3a-9bea-08e4db55e67c-kube-api-access-bdmdk\") on node \"ip-10-0-129-122.ec2.internal\" DevicePath \"\""
Sep 10 17:48:13.697949 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:13.697924    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-4601bf83-openshift-default-7d5654d96d-g2lmw"]
Sep 10 17:48:13.699540 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:48:13.699516    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7774ff67_4869_4805_95f2_b6aad0e5303a.slice/crio-86cd1d559dff7f5b3c34dc4b98abaf9401b81dc1a1ad64fb4f70e15d7fcf3a3f WatchSource:0}: Error finding container 86cd1d559dff7f5b3c34dc4b98abaf9401b81dc1a1ad64fb4f70e15d7fcf3a3f: Status 404 returned error can't find the container with id 86cd1d559dff7f5b3c34dc4b98abaf9401b81dc1a1ad64fb4f70e15d7fcf3a3f
Sep 10 17:48:14.126457 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:14.126425    2579 generic.go:356] "Generic (PLEG): container finished" podID="92daab40-6b16-4d3a-9bea-08e4db55e67c" containerID="860a7f77eba31f9866970d9633e3a68826eeacd87f7b285d301eeaeffedc264b" exitCode=137
Sep 10 17:48:14.126801 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:14.126529    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7708-b77b12"
Sep 10 17:48:14.126801 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:14.126599    2579 scope.go:117] "RemoveContainer" containerID="860a7f77eba31f9866970d9633e3a68826eeacd87f7b285d301eeaeffedc264b"
Sep 10 17:48:14.127956 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:14.127866    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-4601bf83-openshift-default-7d5654d96d-g2lmw" event={"ID":"7774ff67-4869-4805-95f2-b6aad0e5303a","Type":"ContainerStarted","Data":"86cd1d559dff7f5b3c34dc4b98abaf9401b81dc1a1ad64fb4f70e15d7fcf3a3f"}
Sep 10 17:48:14.141170 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:48:14.141143    2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7708-b77b12\" is forbidden: User \"system:node:ip-10-0-129-122.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-122.ec2.internal' and this object" podUID="92daab40-6b16-4d3a-9bea-08e4db55e67c" pod="opendatahub/test-curl-7708-b77b12"
Sep 10 17:48:14.141609 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:14.141595    2579 scope.go:117] "RemoveContainer" containerID="860a7f77eba31f9866970d9633e3a68826eeacd87f7b285d301eeaeffedc264b"
Sep 10 17:48:14.141898 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:48:14.141876    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"860a7f77eba31f9866970d9633e3a68826eeacd87f7b285d301eeaeffedc264b\": container with ID starting with 860a7f77eba31f9866970d9633e3a68826eeacd87f7b285d301eeaeffedc264b not found: ID does not exist" containerID="860a7f77eba31f9866970d9633e3a68826eeacd87f7b285d301eeaeffedc264b"
Sep 10 17:48:14.141954 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:14.141905    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"860a7f77eba31f9866970d9633e3a68826eeacd87f7b285d301eeaeffedc264b"} err="failed to get container status \"860a7f77eba31f9866970d9633e3a68826eeacd87f7b285d301eeaeffedc264b\": rpc error: code = NotFound desc = could not find container \"860a7f77eba31f9866970d9633e3a68826eeacd87f7b285d301eeaeffedc264b\": container with ID starting with 860a7f77eba31f9866970d9633e3a68826eeacd87f7b285d301eeaeffedc264b not found: ID does not exist"
Sep 10 17:48:14.477926 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:14.477893    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="92daab40-6b16-4d3a-9bea-08e4db55e67c" path="/var/lib/kubelet/pods/92daab40-6b16-4d3a-9bea-08e4db55e67c/volumes"
Sep 10 17:48:15.315496 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:15.315473    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7708-42cbc1"
Sep 10 17:48:15.403712 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:15.403678    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ns5t7\" (UniqueName: \"kubernetes.io/projected/ca70f0e6-7da0-49f1-9779-9c4b367b87ca-kube-api-access-ns5t7\") pod \"ca70f0e6-7da0-49f1-9779-9c4b367b87ca\" (UID: \"ca70f0e6-7da0-49f1-9779-9c4b367b87ca\") "
Sep 10 17:48:15.406155 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:15.406121    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca70f0e6-7da0-49f1-9779-9c4b367b87ca-kube-api-access-ns5t7" (OuterVolumeSpecName: "kube-api-access-ns5t7") pod "ca70f0e6-7da0-49f1-9779-9c4b367b87ca" (UID: "ca70f0e6-7da0-49f1-9779-9c4b367b87ca"). InnerVolumeSpecName "kube-api-access-ns5t7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:48:15.505038 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:15.504999    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ns5t7\" (UniqueName: \"kubernetes.io/projected/ca70f0e6-7da0-49f1-9779-9c4b367b87ca-kube-api-access-ns5t7\") on node \"ip-10-0-129-122.ec2.internal\" DevicePath \"\""
Sep 10 17:48:16.142553 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:16.142507    2579 generic.go:356] "Generic (PLEG): container finished" podID="ca70f0e6-7da0-49f1-9779-9c4b367b87ca" containerID="da03de0a65a90322fe92ca2de52835cb81870cf0e3cf658768628a331de30fa4" exitCode=137
Sep 10 17:48:16.142553 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:16.142554    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7708-42cbc1"
Sep 10 17:48:16.142796 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:16.142641    2579 scope.go:117] "RemoveContainer" containerID="da03de0a65a90322fe92ca2de52835cb81870cf0e3cf658768628a331de30fa4"
Sep 10 17:48:16.154375 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:48:16.154342    2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7708-42cbc1\" is forbidden: User \"system:node:ip-10-0-129-122.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-122.ec2.internal' and this object" podUID="ca70f0e6-7da0-49f1-9779-9c4b367b87ca" pod="opendatahub/test-curl-7708-42cbc1"
Sep 10 17:48:16.309654 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:16.309483    2579 scope.go:117] "RemoveContainer" containerID="da03de0a65a90322fe92ca2de52835cb81870cf0e3cf658768628a331de30fa4"
Sep 10 17:48:16.309929 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:48:16.309907    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da03de0a65a90322fe92ca2de52835cb81870cf0e3cf658768628a331de30fa4\": container with ID starting with da03de0a65a90322fe92ca2de52835cb81870cf0e3cf658768628a331de30fa4 not found: ID does not exist" containerID="da03de0a65a90322fe92ca2de52835cb81870cf0e3cf658768628a331de30fa4"
Sep 10 17:48:16.310001 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:16.309941    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da03de0a65a90322fe92ca2de52835cb81870cf0e3cf658768628a331de30fa4"} err="failed to get container status \"da03de0a65a90322fe92ca2de52835cb81870cf0e3cf658768628a331de30fa4\": rpc error: code = NotFound desc = could not find container \"da03de0a65a90322fe92ca2de52835cb81870cf0e3cf658768628a331de30fa4\": container with ID starting with da03de0a65a90322fe92ca2de52835cb81870cf0e3cf658768628a331de30fa4 not found: ID does not exist"
Sep 10 17:48:16.370127 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:16.370088    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 10 17:48:16.370403 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:16.370159    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 10 17:48:16.370403 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:16.370191    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 10 17:48:16.474829 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:48:16.474786    2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7708-42cbc1\" is forbidden: User \"system:node:ip-10-0-129-122.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-122.ec2.internal' and this object" podUID="ca70f0e6-7da0-49f1-9779-9c4b367b87ca" pod="opendatahub/test-curl-7708-42cbc1"
Sep 10 17:48:16.476041 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:16.476015    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ca70f0e6-7da0-49f1-9779-9c4b367b87ca" path="/var/lib/kubelet/pods/ca70f0e6-7da0-49f1-9779-9c4b367b87ca/volumes"
Sep 10 17:48:16.819101 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:16.819014    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-51ed8d-6964d796d5-qjg6w"]
Sep 10 17:48:16.823242 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:16.823215    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-51ed8d-6964d796d5-qjg6w"
Sep 10 17:48:16.826682 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:16.826649    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-autores-51ed8d-serving-cert\""
Sep 10 17:48:16.840617 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:16.840585    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-51ed8d-6964d796d5-qjg6w"]
Sep 10 17:48:16.913339 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:16.913308    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8zpl\" (UniqueName: \"kubernetes.io/projected/faae58be-d518-4d27-96bb-7545e8de34ad-kube-api-access-k8zpl\") pod \"maas-api-e2e-autores-51ed8d-6964d796d5-qjg6w\" (UID: \"faae58be-d518-4d27-96bb-7545e8de34ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-51ed8d-6964d796d5-qjg6w"
Sep 10 17:48:16.913339 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:16.913343    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/faae58be-d518-4d27-96bb-7545e8de34ad-metrics-certs\") pod \"maas-api-e2e-autores-51ed8d-6964d796d5-qjg6w\" (UID: \"faae58be-d518-4d27-96bb-7545e8de34ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-51ed8d-6964d796d5-qjg6w"
Sep 10 17:48:16.913508 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:16.913377    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/faae58be-d518-4d27-96bb-7545e8de34ad-maas-api-tls\") pod \"maas-api-e2e-autores-51ed8d-6964d796d5-qjg6w\" (UID: \"faae58be-d518-4d27-96bb-7545e8de34ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-51ed8d-6964d796d5-qjg6w"
Sep 10 17:48:16.913508 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:16.913403    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/faae58be-d518-4d27-96bb-7545e8de34ad-openshift-service-ca\") pod \"maas-api-e2e-autores-51ed8d-6964d796d5-qjg6w\" (UID: \"faae58be-d518-4d27-96bb-7545e8de34ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-51ed8d-6964d796d5-qjg6w"
Sep 10 17:48:17.014696 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:17.014662    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8zpl\" (UniqueName: \"kubernetes.io/projected/faae58be-d518-4d27-96bb-7545e8de34ad-kube-api-access-k8zpl\") pod \"maas-api-e2e-autores-51ed8d-6964d796d5-qjg6w\" (UID: \"faae58be-d518-4d27-96bb-7545e8de34ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-51ed8d-6964d796d5-qjg6w"
Sep 10 17:48:17.014872 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:17.014705    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/faae58be-d518-4d27-96bb-7545e8de34ad-metrics-certs\") pod \"maas-api-e2e-autores-51ed8d-6964d796d5-qjg6w\" (UID: \"faae58be-d518-4d27-96bb-7545e8de34ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-51ed8d-6964d796d5-qjg6w"
Sep 10 17:48:17.014872 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:17.014756    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/faae58be-d518-4d27-96bb-7545e8de34ad-maas-api-tls\") pod \"maas-api-e2e-autores-51ed8d-6964d796d5-qjg6w\" (UID: \"faae58be-d518-4d27-96bb-7545e8de34ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-51ed8d-6964d796d5-qjg6w"
Sep 10 17:48:17.014872 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:17.014798    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/faae58be-d518-4d27-96bb-7545e8de34ad-openshift-service-ca\") pod \"maas-api-e2e-autores-51ed8d-6964d796d5-qjg6w\" (UID: \"faae58be-d518-4d27-96bb-7545e8de34ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-51ed8d-6964d796d5-qjg6w"
Sep 10 17:48:17.015440 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:17.015416    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/faae58be-d518-4d27-96bb-7545e8de34ad-openshift-service-ca\") pod \"maas-api-e2e-autores-51ed8d-6964d796d5-qjg6w\" (UID: \"faae58be-d518-4d27-96bb-7545e8de34ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-51ed8d-6964d796d5-qjg6w"
Sep 10 17:48:17.017182 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:17.017163    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/faae58be-d518-4d27-96bb-7545e8de34ad-maas-api-tls\") pod \"maas-api-e2e-autores-51ed8d-6964d796d5-qjg6w\" (UID: \"faae58be-d518-4d27-96bb-7545e8de34ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-51ed8d-6964d796d5-qjg6w"
Sep 10 17:48:17.017328 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:17.017308    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/faae58be-d518-4d27-96bb-7545e8de34ad-metrics-certs\") pod \"maas-api-e2e-autores-51ed8d-6964d796d5-qjg6w\" (UID: \"faae58be-d518-4d27-96bb-7545e8de34ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-51ed8d-6964d796d5-qjg6w"
Sep 10 17:48:17.023595 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:17.023549    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8zpl\" (UniqueName: \"kubernetes.io/projected/faae58be-d518-4d27-96bb-7545e8de34ad-kube-api-access-k8zpl\") pod \"maas-api-e2e-autores-51ed8d-6964d796d5-qjg6w\" (UID: \"faae58be-d518-4d27-96bb-7545e8de34ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-51ed8d-6964d796d5-qjg6w"
Sep 10 17:48:17.119215 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:17.119152    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-48d00b-54fc5bb588-sntsg"
Sep 10 17:48:17.136044 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:17.136019    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-51ed8d-6964d796d5-qjg6w"
Sep 10 17:48:17.154113 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:17.153873    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-4601bf83-openshift-default-7d5654d96d-g2lmw" event={"ID":"7774ff67-4869-4805-95f2-b6aad0e5303a","Type":"ContainerStarted","Data":"9029c48809e4762ca38c894b7e889fc51f06ef992d25503f4b852ff51530e529"}
Sep 10 17:48:17.182324 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:17.180756    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-4601bf83-openshift-default-7d5654d96d-g2lmw" podStartSLOduration=1.5124644809999999 podStartE2EDuration="4.180732463s" totalImagesPullingTime="2.668267982s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:48:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:48:17.177731032 +0000 UTC m=+741.177561099" watchObservedRunningTime="2026-09-10 17:48:17.180732463 +0000 UTC m=+741.180562531"
Sep 10 17:48:17.417185 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:17.417156    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-4601bf83-openshift-default-7d5654d96d-g2lmw"
Sep 10 17:48:17.422400 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:17.422372    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-51ed8d-6964d796d5-qjg6w"]
Sep 10 17:48:17.422498 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:17.422476    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-4601bf83-openshift-default-7d5654d96d-g2lmw"
Sep 10 17:48:17.424305 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:48:17.424281    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfaae58be_d518_4d27_96bb_7545e8de34ad.slice/crio-5d8bbe1e0105e1351031a92f33b2abc597e10ff7d6d388cd22ec94d24d5fd500 WatchSource:0}: Error finding container 5d8bbe1e0105e1351031a92f33b2abc597e10ff7d6d388cd22ec94d24d5fd500: Status 404 returned error can't find the container with id 5d8bbe1e0105e1351031a92f33b2abc597e10ff7d6d388cd22ec94d24d5fd500
Sep 10 17:48:17.553069 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:17.553043    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-51ed8d-79cf8c5678-6rwc4"]
Sep 10 17:48:17.557656 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:17.557626    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-51ed8d-79cf8c5678-6rwc4"
Sep 10 17:48:17.561528 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:17.561508    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-51ed8d-dockercfg-sdqfk\""
Sep 10 17:48:17.561650 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:17.561528    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-51ed8d\""
Sep 10 17:48:17.571853 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:17.571824    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-51ed8d-79cf8c5678-6rwc4"]
Sep 10 17:48:17.618883 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:17.618850    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwhd7\" (UniqueName: \"kubernetes.io/projected/82f61180-278f-47de-a4ab-a5225f7e76a4-kube-api-access-rwhd7\") pod \"payload-pre-processing-e2e-autores-51ed8d-79cf8c5678-6rwc4\" (UID: \"82f61180-278f-47de-a4ab-a5225f7e76a4\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-51ed8d-79cf8c5678-6rwc4"
Sep 10 17:48:17.619044 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:17.618910    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82f61180-278f-47de-a4ab-a5225f7e76a4-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-51ed8d-79cf8c5678-6rwc4\" (UID: \"82f61180-278f-47de-a4ab-a5225f7e76a4\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-51ed8d-79cf8c5678-6rwc4"
Sep 10 17:48:17.720270 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:17.720186    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwhd7\" (UniqueName: \"kubernetes.io/projected/82f61180-278f-47de-a4ab-a5225f7e76a4-kube-api-access-rwhd7\") pod \"payload-pre-processing-e2e-autores-51ed8d-79cf8c5678-6rwc4\" (UID: \"82f61180-278f-47de-a4ab-a5225f7e76a4\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-51ed8d-79cf8c5678-6rwc4"
Sep 10 17:48:17.720270 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:17.720240    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82f61180-278f-47de-a4ab-a5225f7e76a4-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-51ed8d-79cf8c5678-6rwc4\" (UID: \"82f61180-278f-47de-a4ab-a5225f7e76a4\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-51ed8d-79cf8c5678-6rwc4"
Sep 10 17:48:17.720897 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:17.720875    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82f61180-278f-47de-a4ab-a5225f7e76a4-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-51ed8d-79cf8c5678-6rwc4\" (UID: \"82f61180-278f-47de-a4ab-a5225f7e76a4\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-51ed8d-79cf8c5678-6rwc4"
Sep 10 17:48:17.735948 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:17.735924    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwhd7\" (UniqueName: \"kubernetes.io/projected/82f61180-278f-47de-a4ab-a5225f7e76a4-kube-api-access-rwhd7\") pod \"payload-pre-processing-e2e-autores-51ed8d-79cf8c5678-6rwc4\" (UID: \"82f61180-278f-47de-a4ab-a5225f7e76a4\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-51ed8d-79cf8c5678-6rwc4"
Sep 10 17:48:17.867908 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:17.867876    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-51ed8d-79cf8c5678-6rwc4"
Sep 10 17:48:18.145729 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:18.145705    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-51ed8d-79cf8c5678-6rwc4"]
Sep 10 17:48:18.147785 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:48:18.147754    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82f61180_278f_47de_a4ab_a5225f7e76a4.slice/crio-ef4fe80f4755da9250cc11f80274e189df6f66e2dc1331e3e926fad61e04dfc8 WatchSource:0}: Error finding container ef4fe80f4755da9250cc11f80274e189df6f66e2dc1331e3e926fad61e04dfc8: Status 404 returned error can't find the container with id ef4fe80f4755da9250cc11f80274e189df6f66e2dc1331e3e926fad61e04dfc8
Sep 10 17:48:18.160586 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:18.160529    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-51ed8d-79cf8c5678-6rwc4" event={"ID":"82f61180-278f-47de-a4ab-a5225f7e76a4","Type":"ContainerStarted","Data":"ef4fe80f4755da9250cc11f80274e189df6f66e2dc1331e3e926fad61e04dfc8"}
Sep 10 17:48:18.162188 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:18.162142    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-51ed8d-6964d796d5-qjg6w" event={"ID":"faae58be-d518-4d27-96bb-7545e8de34ad","Type":"ContainerStarted","Data":"0ddff592df0ee023a7cdf1e0b498b78d1dca2cee6fd0034e0f34462ae732159b"}
Sep 10 17:48:18.162304 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:18.162194    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-51ed8d-6964d796d5-qjg6w" event={"ID":"faae58be-d518-4d27-96bb-7545e8de34ad","Type":"ContainerStarted","Data":"5d8bbe1e0105e1351031a92f33b2abc597e10ff7d6d388cd22ec94d24d5fd500"}
Sep 10 17:48:18.162373 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:18.162353    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-4601bf83-openshift-default-7d5654d96d-g2lmw"
Sep 10 17:48:18.162425 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:18.162419    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-51ed8d-6964d796d5-qjg6w"
Sep 10 17:48:18.163264 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:18.163246    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-4601bf83-openshift-default-7d5654d96d-g2lmw"
Sep 10 17:48:18.186405 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:18.186365    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-autores-51ed8d-6964d796d5-qjg6w" podStartSLOduration=1.822634788 podStartE2EDuration="2.186351658s" totalImagesPullingTime="363.71687ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:48:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:48:18.184204637 +0000 UTC m=+742.184034713" watchObservedRunningTime="2026-09-10 17:48:18.186351658 +0000 UTC m=+742.186181725"
Sep 10 17:48:19.167972 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:19.167935    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-51ed8d-79cf8c5678-6rwc4" event={"ID":"82f61180-278f-47de-a4ab-a5225f7e76a4","Type":"ContainerStarted","Data":"8c3826913dae7223b3c25dfec9fc0c687617e50a56d1b76e1b605cf0a37eda4c"}
Sep 10 17:48:19.168417 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:19.168181    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-51ed8d-79cf8c5678-6rwc4"
Sep 10 17:48:19.187458 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:19.187405    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-autores-51ed8d-79cf8c5678-6rwc4" podStartSLOduration=2.187387409 podStartE2EDuration="2.187387409s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:48:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:48:19.186630549 +0000 UTC m=+743.186460615" watchObservedRunningTime="2026-09-10 17:48:19.187387409 +0000 UTC m=+743.187217478"
Sep 10 17:48:20.241016 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:20.240983    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4601bf83-777667899c-rt2qh"]
Sep 10 17:48:20.246017 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:20.245992    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4601bf83-777667899c-rt2qh"
Sep 10 17:48:20.249417 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:20.249396    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-4601bf83-serving-cert\""
Sep 10 17:48:20.257038 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:20.257003    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4601bf83-777667899c-rt2qh"]
Sep 10 17:48:20.439058 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:20.439025    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b657c988-b137-47c2-aaf0-3a41dbdf7a56-openshift-service-ca\") pod \"maas-api-e2e-ait-4601bf83-777667899c-rt2qh\" (UID: \"b657c988-b137-47c2-aaf0-3a41dbdf7a56\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4601bf83-777667899c-rt2qh"
Sep 10 17:48:20.439058 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:20.439062    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dxvl\" (UniqueName: \"kubernetes.io/projected/b657c988-b137-47c2-aaf0-3a41dbdf7a56-kube-api-access-4dxvl\") pod \"maas-api-e2e-ait-4601bf83-777667899c-rt2qh\" (UID: \"b657c988-b137-47c2-aaf0-3a41dbdf7a56\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4601bf83-777667899c-rt2qh"
Sep 10 17:48:20.439275 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:20.439156    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b657c988-b137-47c2-aaf0-3a41dbdf7a56-metrics-certs\") pod \"maas-api-e2e-ait-4601bf83-777667899c-rt2qh\" (UID: \"b657c988-b137-47c2-aaf0-3a41dbdf7a56\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4601bf83-777667899c-rt2qh"
Sep 10 17:48:20.439275 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:20.439215    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b657c988-b137-47c2-aaf0-3a41dbdf7a56-maas-api-tls\") pod \"maas-api-e2e-ait-4601bf83-777667899c-rt2qh\" (UID: \"b657c988-b137-47c2-aaf0-3a41dbdf7a56\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4601bf83-777667899c-rt2qh"
Sep 10 17:48:20.540525 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:20.540440    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b657c988-b137-47c2-aaf0-3a41dbdf7a56-metrics-certs\") pod \"maas-api-e2e-ait-4601bf83-777667899c-rt2qh\" (UID: \"b657c988-b137-47c2-aaf0-3a41dbdf7a56\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4601bf83-777667899c-rt2qh"
Sep 10 17:48:20.540525 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:20.540495    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b657c988-b137-47c2-aaf0-3a41dbdf7a56-maas-api-tls\") pod \"maas-api-e2e-ait-4601bf83-777667899c-rt2qh\" (UID: \"b657c988-b137-47c2-aaf0-3a41dbdf7a56\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4601bf83-777667899c-rt2qh"
Sep 10 17:48:20.540525 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:20.540524    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b657c988-b137-47c2-aaf0-3a41dbdf7a56-openshift-service-ca\") pod \"maas-api-e2e-ait-4601bf83-777667899c-rt2qh\" (UID: \"b657c988-b137-47c2-aaf0-3a41dbdf7a56\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4601bf83-777667899c-rt2qh"
Sep 10 17:48:20.540782 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:20.540546    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dxvl\" (UniqueName: \"kubernetes.io/projected/b657c988-b137-47c2-aaf0-3a41dbdf7a56-kube-api-access-4dxvl\") pod \"maas-api-e2e-ait-4601bf83-777667899c-rt2qh\" (UID: \"b657c988-b137-47c2-aaf0-3a41dbdf7a56\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4601bf83-777667899c-rt2qh"
Sep 10 17:48:20.541206 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:20.541185    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b657c988-b137-47c2-aaf0-3a41dbdf7a56-openshift-service-ca\") pod \"maas-api-e2e-ait-4601bf83-777667899c-rt2qh\" (UID: \"b657c988-b137-47c2-aaf0-3a41dbdf7a56\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4601bf83-777667899c-rt2qh"
Sep 10 17:48:20.543156 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:20.543135    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b657c988-b137-47c2-aaf0-3a41dbdf7a56-maas-api-tls\") pod \"maas-api-e2e-ait-4601bf83-777667899c-rt2qh\" (UID: \"b657c988-b137-47c2-aaf0-3a41dbdf7a56\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4601bf83-777667899c-rt2qh"
Sep 10 17:48:20.543247 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:20.543142    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b657c988-b137-47c2-aaf0-3a41dbdf7a56-metrics-certs\") pod \"maas-api-e2e-ait-4601bf83-777667899c-rt2qh\" (UID: \"b657c988-b137-47c2-aaf0-3a41dbdf7a56\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4601bf83-777667899c-rt2qh"
Sep 10 17:48:20.554656 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:20.554631    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dxvl\" (UniqueName: \"kubernetes.io/projected/b657c988-b137-47c2-aaf0-3a41dbdf7a56-kube-api-access-4dxvl\") pod \"maas-api-e2e-ait-4601bf83-777667899c-rt2qh\" (UID: \"b657c988-b137-47c2-aaf0-3a41dbdf7a56\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4601bf83-777667899c-rt2qh"
Sep 10 17:48:20.560427 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:20.560408    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4601bf83-777667899c-rt2qh"
Sep 10 17:48:20.835466 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:20.835444    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4601bf83-777667899c-rt2qh"]
Sep 10 17:48:20.837612 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:48:20.837580    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb657c988_b137_47c2_aaf0_3a41dbdf7a56.slice/crio-8ce2eba30aed07dc3dc697fa0b6a10886064ed48d83703c9fdbbc0dfa33ecd44 WatchSource:0}: Error finding container 8ce2eba30aed07dc3dc697fa0b6a10886064ed48d83703c9fdbbc0dfa33ecd44: Status 404 returned error can't find the container with id 8ce2eba30aed07dc3dc697fa0b6a10886064ed48d83703c9fdbbc0dfa33ecd44
Sep 10 17:48:21.050959 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:21.050929    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-4601bf83-845b644dd6-sgjrb"]
Sep 10 17:48:21.055629 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:21.055611    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-4601bf83-845b644dd6-sgjrb"
Sep 10 17:48:21.058440 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:21.058417    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-4601bf83\""
Sep 10 17:48:21.058550 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:21.058453    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-4601bf83-dockercfg-zjjmx\""
Sep 10 17:48:21.064225 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:21.064203    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-4601bf83-845b644dd6-sgjrb"]
Sep 10 17:48:21.145879 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:21.145854    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be27a3df-82ca-42bc-8bf5-75078ae9879a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-4601bf83-845b644dd6-sgjrb\" (UID: \"be27a3df-82ca-42bc-8bf5-75078ae9879a\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-4601bf83-845b644dd6-sgjrb"
Sep 10 17:48:21.146015 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:21.145948    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v79st\" (UniqueName: \"kubernetes.io/projected/be27a3df-82ca-42bc-8bf5-75078ae9879a-kube-api-access-v79st\") pod \"payload-pre-processing-e2e-ait-4601bf83-845b644dd6-sgjrb\" (UID: \"be27a3df-82ca-42bc-8bf5-75078ae9879a\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-4601bf83-845b644dd6-sgjrb"
Sep 10 17:48:21.181630 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:21.181598    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4601bf83-777667899c-rt2qh" event={"ID":"b657c988-b137-47c2-aaf0-3a41dbdf7a56","Type":"ContainerStarted","Data":"8ce2eba30aed07dc3dc697fa0b6a10886064ed48d83703c9fdbbc0dfa33ecd44"}
Sep 10 17:48:21.246481 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:21.246448    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v79st\" (UniqueName: \"kubernetes.io/projected/be27a3df-82ca-42bc-8bf5-75078ae9879a-kube-api-access-v79st\") pod \"payload-pre-processing-e2e-ait-4601bf83-845b644dd6-sgjrb\" (UID: \"be27a3df-82ca-42bc-8bf5-75078ae9879a\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-4601bf83-845b644dd6-sgjrb"
Sep 10 17:48:21.246855 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:21.246541    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be27a3df-82ca-42bc-8bf5-75078ae9879a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-4601bf83-845b644dd6-sgjrb\" (UID: \"be27a3df-82ca-42bc-8bf5-75078ae9879a\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-4601bf83-845b644dd6-sgjrb"
Sep 10 17:48:21.247237 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:21.247219    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be27a3df-82ca-42bc-8bf5-75078ae9879a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-4601bf83-845b644dd6-sgjrb\" (UID: \"be27a3df-82ca-42bc-8bf5-75078ae9879a\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-4601bf83-845b644dd6-sgjrb"
Sep 10 17:48:21.256383 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:21.256355    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v79st\" (UniqueName: \"kubernetes.io/projected/be27a3df-82ca-42bc-8bf5-75078ae9879a-kube-api-access-v79st\") pod \"payload-pre-processing-e2e-ait-4601bf83-845b644dd6-sgjrb\" (UID: \"be27a3df-82ca-42bc-8bf5-75078ae9879a\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-4601bf83-845b644dd6-sgjrb"
Sep 10 17:48:21.367832 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:21.367797    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-4601bf83-845b644dd6-sgjrb"
Sep 10 17:48:21.649393 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:21.649366    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-4601bf83-845b644dd6-sgjrb"]
Sep 10 17:48:21.652603 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:48:21.652552    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe27a3df_82ca_42bc_8bf5_75078ae9879a.slice/crio-7986d2fd3ec4597848823b8c25dab460d122127b259eabc24573011a26d157c6 WatchSource:0}: Error finding container 7986d2fd3ec4597848823b8c25dab460d122127b259eabc24573011a26d157c6: Status 404 returned error can't find the container with id 7986d2fd3ec4597848823b8c25dab460d122127b259eabc24573011a26d157c6
Sep 10 17:48:22.189492 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:22.189458    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4601bf83-777667899c-rt2qh" event={"ID":"b657c988-b137-47c2-aaf0-3a41dbdf7a56","Type":"ContainerStarted","Data":"d6752795364805237172438bdbc5111110b40822ab14ed2481bc1f7cf12812b8"}
Sep 10 17:48:22.189696 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:22.189671    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4601bf83-777667899c-rt2qh"
Sep 10 17:48:22.191716 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:22.191685    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-4601bf83-845b644dd6-sgjrb" event={"ID":"be27a3df-82ca-42bc-8bf5-75078ae9879a","Type":"ContainerStarted","Data":"99949a42df6d04dcce59469a2b4091766825fe83f801df7dc7e124903f1342b1"}
Sep 10 17:48:22.191901 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:22.191735    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-4601bf83-845b644dd6-sgjrb" event={"ID":"be27a3df-82ca-42bc-8bf5-75078ae9879a","Type":"ContainerStarted","Data":"7986d2fd3ec4597848823b8c25dab460d122127b259eabc24573011a26d157c6"}
Sep 10 17:48:22.191901 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:22.191777    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-4601bf83-845b644dd6-sgjrb"
Sep 10 17:48:22.211655 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:22.211613    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4601bf83-777667899c-rt2qh" podStartSLOduration=1.913475464 podStartE2EDuration="2.211599741s" totalImagesPullingTime="298.124277ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:48:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:48:22.209896015 +0000 UTC m=+746.209726085" watchObservedRunningTime="2026-09-10 17:48:22.211599741 +0000 UTC m=+746.211429806"
Sep 10 17:48:22.235783 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:22.235741    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-4601bf83-845b644dd6-sgjrb" podStartSLOduration=1.235728874 podStartE2EDuration="1.235728874s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:48:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:48:22.234846058 +0000 UTC m=+746.234676126" watchObservedRunningTime="2026-09-10 17:48:22.235728874 +0000 UTC m=+746.235558939"
Sep 10 17:48:24.173372 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:24.173343    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-51ed8d-6964d796d5-qjg6w"
Sep 10 17:48:28.202523 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:28.202495    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4601bf83-777667899c-rt2qh"
Sep 10 17:48:30.176973 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:30.176943    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-51ed8d-79cf8c5678-6rwc4"
Sep 10 17:48:33.199309 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:33.199278    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-4601bf83-845b644dd6-sgjrb"
Sep 10 17:48:39.036981 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:39.036949    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-783a5e-77dd655589-szhsh"]
Sep 10 17:48:39.044735 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:39.044710    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-783a5e-77dd655589-szhsh"
Sep 10 17:48:39.049536 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:39.049516    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-783a5e\""
Sep 10 17:48:39.049683 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:39.049623    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-783a5e-dockercfg-4864v\""
Sep 10 17:48:39.056251 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:39.056230    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-783a5e-77dd655589-szhsh"]
Sep 10 17:48:39.116513 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:39.116482    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52tp9\" (UniqueName: \"kubernetes.io/projected/91e65bf9-acd8-45c9-9d04-16f8cf6f4286-kube-api-access-52tp9\") pod \"payload-processing-e2e-inf-b-783a5e-77dd655589-szhsh\" (UID: \"91e65bf9-acd8-45c9-9d04-16f8cf6f4286\") " pod="openshift-ingress/payload-processing-e2e-inf-b-783a5e-77dd655589-szhsh"
Sep 10 17:48:39.118509 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:39.116635    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91e65bf9-acd8-45c9-9d04-16f8cf6f4286-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-783a5e-77dd655589-szhsh\" (UID: \"91e65bf9-acd8-45c9-9d04-16f8cf6f4286\") " pod="openshift-ingress/payload-processing-e2e-inf-b-783a5e-77dd655589-szhsh"
Sep 10 17:48:39.218084 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:39.218051    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91e65bf9-acd8-45c9-9d04-16f8cf6f4286-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-783a5e-77dd655589-szhsh\" (UID: \"91e65bf9-acd8-45c9-9d04-16f8cf6f4286\") " pod="openshift-ingress/payload-processing-e2e-inf-b-783a5e-77dd655589-szhsh"
Sep 10 17:48:39.218202 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:39.218125    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52tp9\" (UniqueName: \"kubernetes.io/projected/91e65bf9-acd8-45c9-9d04-16f8cf6f4286-kube-api-access-52tp9\") pod \"payload-processing-e2e-inf-b-783a5e-77dd655589-szhsh\" (UID: \"91e65bf9-acd8-45c9-9d04-16f8cf6f4286\") " pod="openshift-ingress/payload-processing-e2e-inf-b-783a5e-77dd655589-szhsh"
Sep 10 17:48:39.218709 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:39.218691    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91e65bf9-acd8-45c9-9d04-16f8cf6f4286-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-783a5e-77dd655589-szhsh\" (UID: \"91e65bf9-acd8-45c9-9d04-16f8cf6f4286\") " pod="openshift-ingress/payload-processing-e2e-inf-b-783a5e-77dd655589-szhsh"
Sep 10 17:48:39.227399 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:39.227368    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-52tp9\" (UniqueName: \"kubernetes.io/projected/91e65bf9-acd8-45c9-9d04-16f8cf6f4286-kube-api-access-52tp9\") pod \"payload-processing-e2e-inf-b-783a5e-77dd655589-szhsh\" (UID: \"91e65bf9-acd8-45c9-9d04-16f8cf6f4286\") " pod="openshift-ingress/payload-processing-e2e-inf-b-783a5e-77dd655589-szhsh"
Sep 10 17:48:39.289643 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:39.289581    2579 generic.go:356] "Generic (PLEG): container finished" podID="7774ff67-4869-4805-95f2-b6aad0e5303a" containerID="9029c48809e4762ca38c894b7e889fc51f06ef992d25503f4b852ff51530e529" exitCode=0
Sep 10 17:48:39.289739 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:39.289648    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-4601bf83-openshift-default-7d5654d96d-g2lmw" event={"ID":"7774ff67-4869-4805-95f2-b6aad0e5303a","Type":"ContainerDied","Data":"9029c48809e4762ca38c894b7e889fc51f06ef992d25503f4b852ff51530e529"}
Sep 10 17:48:39.290082 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:39.290067    2579 scope.go:117] "RemoveContainer" containerID="9029c48809e4762ca38c894b7e889fc51f06ef992d25503f4b852ff51530e529"
Sep 10 17:48:39.291212 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:39.291185    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 10 17:48:39.291270 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:39.291251    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 10 17:48:39.291304 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:39.291281    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 10 17:48:39.356862 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:39.356839    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-783a5e-77dd655589-szhsh"
Sep 10 17:48:39.639124 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:39.639102    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-783a5e-77dd655589-szhsh"]
Sep 10 17:48:39.640913 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:48:39.640886    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91e65bf9_acd8_45c9_9d04_16f8cf6f4286.slice/crio-36d3340bec1283cacd92e56c003c030ef4c1418fd69b0b13c064524d7f742705 WatchSource:0}: Error finding container 36d3340bec1283cacd92e56c003c030ef4c1418fd69b0b13c064524d7f742705: Status 404 returned error can't find the container with id 36d3340bec1283cacd92e56c003c030ef4c1418fd69b0b13c064524d7f742705
Sep 10 17:48:40.299765 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:40.299724    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-4601bf83-openshift-default-7d5654d96d-g2lmw" event={"ID":"7774ff67-4869-4805-95f2-b6aad0e5303a","Type":"ContainerStarted","Data":"8b2222d9a3fa5b990505d5878632f8c2f87544ad26d2ecdfb83d44d0a1aace2b"}
Sep 10 17:48:40.301811 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:40.301770    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-783a5e-77dd655589-szhsh" event={"ID":"91e65bf9-acd8-45c9-9d04-16f8cf6f4286","Type":"ContainerStarted","Data":"0141e1b4122a6a79b5fd728cc02e071cec4c8753a68384b43193f462018d5b9f"}
Sep 10 17:48:40.301930 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:40.301843    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-783a5e-77dd655589-szhsh"
Sep 10 17:48:40.301930 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:40.301862    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-783a5e-77dd655589-szhsh" event={"ID":"91e65bf9-acd8-45c9-9d04-16f8cf6f4286","Type":"ContainerStarted","Data":"36d3340bec1283cacd92e56c003c030ef4c1418fd69b0b13c064524d7f742705"}
Sep 10 17:48:40.363967 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:40.363910    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-783a5e-77dd655589-szhsh" podStartSLOduration=1.3638909 podStartE2EDuration="1.3638909s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:48:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:48:40.36130822 +0000 UTC m=+764.361138288" watchObservedRunningTime="2026-09-10 17:48:40.3638909 +0000 UTC m=+764.363720967"
Sep 10 17:48:40.416714 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:40.416674    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-4601bf83-openshift-default-7d5654d96d-g2lmw"
Sep 10 17:48:40.416900 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:40.416851    2579 patch_prober.go:28] interesting pod/e2e-ait-4601bf83-openshift-default-7d5654d96d-g2lmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body=
Sep 10 17:48:40.416959 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:40.416898    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-4601bf83-openshift-default-7d5654d96d-g2lmw" podUID="7774ff67-4869-4805-95f2-b6aad0e5303a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused"
Sep 10 17:48:41.417476 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:41.417438    2579 patch_prober.go:28] interesting pod/e2e-ait-4601bf83-openshift-default-7d5654d96d-g2lmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body=
Sep 10 17:48:41.418043 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:41.417494    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-4601bf83-openshift-default-7d5654d96d-g2lmw" podUID="7774ff67-4869-4805-95f2-b6aad0e5303a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused"
Sep 10 17:48:42.417637 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:42.417596    2579 patch_prober.go:28] interesting pod/e2e-ait-4601bf83-openshift-default-7d5654d96d-g2lmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body=
Sep 10 17:48:42.418059 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:42.417654    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-4601bf83-openshift-default-7d5654d96d-g2lmw" podUID="7774ff67-4869-4805-95f2-b6aad0e5303a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused"
Sep 10 17:48:43.417387 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:43.417347    2579 patch_prober.go:28] interesting pod/e2e-ait-4601bf83-openshift-default-7d5654d96d-g2lmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body=
Sep 10 17:48:43.417591 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:43.417405    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-4601bf83-openshift-default-7d5654d96d-g2lmw" podUID="7774ff67-4869-4805-95f2-b6aad0e5303a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused"
Sep 10 17:48:44.416968 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:44.416936    2579 patch_prober.go:28] interesting pod/e2e-ait-4601bf83-openshift-default-7d5654d96d-g2lmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body=
Sep 10 17:48:44.417333 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:44.416990    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-4601bf83-openshift-default-7d5654d96d-g2lmw" podUID="7774ff67-4869-4805-95f2-b6aad0e5303a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused"
Sep 10 17:48:45.421683 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:45.421653    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-4601bf83-openshift-default-7d5654d96d-g2lmw"
Sep 10 17:48:45.422129 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:45.421923    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-4601bf83-openshift-default-7d5654d96d-g2lmw"
Sep 10 17:48:45.422661 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:45.422642    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-4601bf83-openshift-default-7d5654d96d-g2lmw"
Sep 10 17:48:51.309001 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:48:51.308970    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-783a5e-77dd655589-szhsh"
Sep 10 17:49:00.275728 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:00.275697    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4601bf83-777667899c-rt2qh"]
Sep 10 17:49:00.276156 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:00.275906    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4601bf83-777667899c-rt2qh" podUID="b657c988-b137-47c2-aaf0-3a41dbdf7a56" containerName="maas-api" containerID="cri-o://d6752795364805237172438bdbc5111110b40822ab14ed2481bc1f7cf12812b8" gracePeriod=30
Sep 10 17:49:00.359954 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:00.359921    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-4601bf83-845b644dd6-sgjrb"]
Sep 10 17:49:00.360217 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:00.360170    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-4601bf83-845b644dd6-sgjrb" podUID="be27a3df-82ca-42bc-8bf5-75078ae9879a" containerName="payload-pre-processing" containerID="cri-o://99949a42df6d04dcce59469a2b4091766825fe83f801df7dc7e124903f1342b1" gracePeriod=30
Sep 10 17:49:00.403807 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:00.403778    2579 generic.go:356] "Generic (PLEG): container finished" podID="b657c988-b137-47c2-aaf0-3a41dbdf7a56" containerID="d6752795364805237172438bdbc5111110b40822ab14ed2481bc1f7cf12812b8" exitCode=0
Sep 10 17:49:00.403932 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:00.403852    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4601bf83-777667899c-rt2qh" event={"ID":"b657c988-b137-47c2-aaf0-3a41dbdf7a56","Type":"ContainerDied","Data":"d6752795364805237172438bdbc5111110b40822ab14ed2481bc1f7cf12812b8"}
Sep 10 17:49:00.615370 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:00.615346    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4601bf83-777667899c-rt2qh"
Sep 10 17:49:00.648463 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:00.648442    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-4601bf83-845b644dd6-sgjrb"
Sep 10 17:49:00.806008 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:00.805976    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b657c988-b137-47c2-aaf0-3a41dbdf7a56-openshift-service-ca\") pod \"b657c988-b137-47c2-aaf0-3a41dbdf7a56\" (UID: \"b657c988-b137-47c2-aaf0-3a41dbdf7a56\") "
Sep 10 17:49:00.806155 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:00.806015    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v79st\" (UniqueName: \"kubernetes.io/projected/be27a3df-82ca-42bc-8bf5-75078ae9879a-kube-api-access-v79st\") pod \"be27a3df-82ca-42bc-8bf5-75078ae9879a\" (UID: \"be27a3df-82ca-42bc-8bf5-75078ae9879a\") "
Sep 10 17:49:00.806155 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:00.806086    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b657c988-b137-47c2-aaf0-3a41dbdf7a56-metrics-certs\") pod \"b657c988-b137-47c2-aaf0-3a41dbdf7a56\" (UID: \"b657c988-b137-47c2-aaf0-3a41dbdf7a56\") "
Sep 10 17:49:00.806155 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:00.806146    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b657c988-b137-47c2-aaf0-3a41dbdf7a56-maas-api-tls\") pod \"b657c988-b137-47c2-aaf0-3a41dbdf7a56\" (UID: \"b657c988-b137-47c2-aaf0-3a41dbdf7a56\") "
Sep 10 17:49:00.806309 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:00.806170    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be27a3df-82ca-42bc-8bf5-75078ae9879a-plugins-config-volume\") pod \"be27a3df-82ca-42bc-8bf5-75078ae9879a\" (UID: \"be27a3df-82ca-42bc-8bf5-75078ae9879a\") "
Sep 10 17:49:00.806362 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:00.806324    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4dxvl\" (UniqueName: \"kubernetes.io/projected/b657c988-b137-47c2-aaf0-3a41dbdf7a56-kube-api-access-4dxvl\") pod \"b657c988-b137-47c2-aaf0-3a41dbdf7a56\" (UID: \"b657c988-b137-47c2-aaf0-3a41dbdf7a56\") "
Sep 10 17:49:00.806484 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:00.806442    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b657c988-b137-47c2-aaf0-3a41dbdf7a56-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b657c988-b137-47c2-aaf0-3a41dbdf7a56" (UID: "b657c988-b137-47c2-aaf0-3a41dbdf7a56"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 17:49:00.806645 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:00.806529    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be27a3df-82ca-42bc-8bf5-75078ae9879a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "be27a3df-82ca-42bc-8bf5-75078ae9879a" (UID: "be27a3df-82ca-42bc-8bf5-75078ae9879a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 17:49:00.806798 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:00.806738    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b657c988-b137-47c2-aaf0-3a41dbdf7a56-openshift-service-ca\") on node \"ip-10-0-129-122.ec2.internal\" DevicePath \"\""
Sep 10 17:49:00.806798 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:00.806757    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be27a3df-82ca-42bc-8bf5-75078ae9879a-plugins-config-volume\") on node \"ip-10-0-129-122.ec2.internal\" DevicePath \"\""
Sep 10 17:49:00.808419 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:00.808391    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be27a3df-82ca-42bc-8bf5-75078ae9879a-kube-api-access-v79st" (OuterVolumeSpecName: "kube-api-access-v79st") pod "be27a3df-82ca-42bc-8bf5-75078ae9879a" (UID: "be27a3df-82ca-42bc-8bf5-75078ae9879a"). InnerVolumeSpecName "kube-api-access-v79st". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:49:00.808527 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:00.808477    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b657c988-b137-47c2-aaf0-3a41dbdf7a56-kube-api-access-4dxvl" (OuterVolumeSpecName: "kube-api-access-4dxvl") pod "b657c988-b137-47c2-aaf0-3a41dbdf7a56" (UID: "b657c988-b137-47c2-aaf0-3a41dbdf7a56"). InnerVolumeSpecName "kube-api-access-4dxvl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:49:00.808758 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:00.808739    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b657c988-b137-47c2-aaf0-3a41dbdf7a56-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b657c988-b137-47c2-aaf0-3a41dbdf7a56" (UID: "b657c988-b137-47c2-aaf0-3a41dbdf7a56"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 17:49:00.808817 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:00.808773    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b657c988-b137-47c2-aaf0-3a41dbdf7a56-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "b657c988-b137-47c2-aaf0-3a41dbdf7a56" (UID: "b657c988-b137-47c2-aaf0-3a41dbdf7a56"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 17:49:00.907450 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:00.907419    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b657c988-b137-47c2-aaf0-3a41dbdf7a56-maas-api-tls\") on node \"ip-10-0-129-122.ec2.internal\" DevicePath \"\""
Sep 10 17:49:00.907450 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:00.907453    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4dxvl\" (UniqueName: \"kubernetes.io/projected/b657c988-b137-47c2-aaf0-3a41dbdf7a56-kube-api-access-4dxvl\") on node \"ip-10-0-129-122.ec2.internal\" DevicePath \"\""
Sep 10 17:49:00.907632 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:00.907468    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v79st\" (UniqueName: \"kubernetes.io/projected/be27a3df-82ca-42bc-8bf5-75078ae9879a-kube-api-access-v79st\") on node \"ip-10-0-129-122.ec2.internal\" DevicePath \"\""
Sep 10 17:49:00.907632 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:00.907482    2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b657c988-b137-47c2-aaf0-3a41dbdf7a56-metrics-certs\") on node \"ip-10-0-129-122.ec2.internal\" DevicePath \"\""
Sep 10 17:49:00.990581 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:00.990540    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-4601bf83-openshift-default-7d5654d96d-g2lmw"]
Sep 10 17:49:00.990797 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:00.990774    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-4601bf83-openshift-default-7d5654d96d-g2lmw" podUID="7774ff67-4869-4805-95f2-b6aad0e5303a" containerName="istio-proxy" containerID="cri-o://8b2222d9a3fa5b990505d5878632f8c2f87544ad26d2ecdfb83d44d0a1aace2b" gracePeriod=30
Sep 10 17:49:01.410394 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:01.410363    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4601bf83-777667899c-rt2qh"
Sep 10 17:49:01.410394 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:01.410369    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4601bf83-777667899c-rt2qh" event={"ID":"b657c988-b137-47c2-aaf0-3a41dbdf7a56","Type":"ContainerDied","Data":"8ce2eba30aed07dc3dc697fa0b6a10886064ed48d83703c9fdbbc0dfa33ecd44"}
Sep 10 17:49:01.410947 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:01.410434    2579 scope.go:117] "RemoveContainer" containerID="d6752795364805237172438bdbc5111110b40822ab14ed2481bc1f7cf12812b8"
Sep 10 17:49:01.411920 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:01.411835    2579 generic.go:356] "Generic (PLEG): container finished" podID="be27a3df-82ca-42bc-8bf5-75078ae9879a" containerID="99949a42df6d04dcce59469a2b4091766825fe83f801df7dc7e124903f1342b1" exitCode=0
Sep 10 17:49:01.411920 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:01.411862    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-4601bf83-845b644dd6-sgjrb" event={"ID":"be27a3df-82ca-42bc-8bf5-75078ae9879a","Type":"ContainerDied","Data":"99949a42df6d04dcce59469a2b4091766825fe83f801df7dc7e124903f1342b1"}
Sep 10 17:49:01.411920 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:01.411872    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-4601bf83-845b644dd6-sgjrb"
Sep 10 17:49:01.411920 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:01.411911    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-4601bf83-845b644dd6-sgjrb" event={"ID":"be27a3df-82ca-42bc-8bf5-75078ae9879a","Type":"ContainerDied","Data":"7986d2fd3ec4597848823b8c25dab460d122127b259eabc24573011a26d157c6"}
Sep 10 17:49:01.419718 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:01.419555    2579 scope.go:117] "RemoveContainer" containerID="99949a42df6d04dcce59469a2b4091766825fe83f801df7dc7e124903f1342b1"
Sep 10 17:49:01.427443 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:01.427424    2579 scope.go:117] "RemoveContainer" containerID="99949a42df6d04dcce59469a2b4091766825fe83f801df7dc7e124903f1342b1"
Sep 10 17:49:01.427710 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:49:01.427693    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99949a42df6d04dcce59469a2b4091766825fe83f801df7dc7e124903f1342b1\": container with ID starting with 99949a42df6d04dcce59469a2b4091766825fe83f801df7dc7e124903f1342b1 not found: ID does not exist" containerID="99949a42df6d04dcce59469a2b4091766825fe83f801df7dc7e124903f1342b1"
Sep 10 17:49:01.427761 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:01.427718    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99949a42df6d04dcce59469a2b4091766825fe83f801df7dc7e124903f1342b1"} err="failed to get container status \"99949a42df6d04dcce59469a2b4091766825fe83f801df7dc7e124903f1342b1\": rpc error: code = NotFound desc = could not find container \"99949a42df6d04dcce59469a2b4091766825fe83f801df7dc7e124903f1342b1\": container with ID starting with 99949a42df6d04dcce59469a2b4091766825fe83f801df7dc7e124903f1342b1 not found: ID does not exist"
Sep 10 17:49:01.473116 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:01.473085    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4601bf83-777667899c-rt2qh"]
Sep 10 17:49:01.483073 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:01.483051    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4601bf83-777667899c-rt2qh"]
Sep 10 17:49:01.505182 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:01.505158    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-4601bf83-845b644dd6-sgjrb"]
Sep 10 17:49:01.511904 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:01.511885    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-4601bf83-845b644dd6-sgjrb"]
Sep 10 17:49:01.982232 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:01.982200    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-624b5c35-gw-openshift-default-ccdbc8bfb-s8wkj"]
Sep 10 17:49:01.982787 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:01.982772    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b657c988-b137-47c2-aaf0-3a41dbdf7a56" containerName="maas-api"
Sep 10 17:49:01.982840 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:01.982791    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="b657c988-b137-47c2-aaf0-3a41dbdf7a56" containerName="maas-api"
Sep 10 17:49:01.982840 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:01.982808    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be27a3df-82ca-42bc-8bf5-75078ae9879a" containerName="payload-pre-processing"
Sep 10 17:49:01.982840 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:01.982817    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="be27a3df-82ca-42bc-8bf5-75078ae9879a" containerName="payload-pre-processing"
Sep 10 17:49:01.982928 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:01.982918    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="b657c988-b137-47c2-aaf0-3a41dbdf7a56" containerName="maas-api"
Sep 10 17:49:01.982960 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:01.982933    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="be27a3df-82ca-42bc-8bf5-75078ae9879a" containerName="payload-pre-processing"
Sep 10 17:49:01.987858 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:01.987836    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-624b5c35-gw-openshift-default-ccdbc8bfb-s8wkj"
Sep 10 17:49:01.991115 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:01.991091    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-624b5c35-gw-openshift-default-dockercfg-ns89r\""
Sep 10 17:49:02.001307 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:02.001279    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-624b5c35-gw-openshift-default-ccdbc8bfb-s8wkj"]
Sep 10 17:49:02.117809 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:02.117778    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93e67e8a-cf2c-483a-8c4a-3ca7db13856f-istio-podinfo\") pod \"e2e-migrate-624b5c35-gw-openshift-default-ccdbc8bfb-s8wkj\" (UID: \"93e67e8a-cf2c-483a-8c4a-3ca7db13856f\") " pod="openshift-ingress/e2e-migrate-624b5c35-gw-openshift-default-ccdbc8bfb-s8wkj"
Sep 10 17:49:02.117979 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:02.117832    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94mmr\" (UniqueName: \"kubernetes.io/projected/93e67e8a-cf2c-483a-8c4a-3ca7db13856f-kube-api-access-94mmr\") pod \"e2e-migrate-624b5c35-gw-openshift-default-ccdbc8bfb-s8wkj\" (UID: \"93e67e8a-cf2c-483a-8c4a-3ca7db13856f\") " pod="openshift-ingress/e2e-migrate-624b5c35-gw-openshift-default-ccdbc8bfb-s8wkj"
Sep 10 17:49:02.117979 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:02.117899    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93e67e8a-cf2c-483a-8c4a-3ca7db13856f-istio-data\") pod \"e2e-migrate-624b5c35-gw-openshift-default-ccdbc8bfb-s8wkj\" (UID: \"93e67e8a-cf2c-483a-8c4a-3ca7db13856f\") " pod="openshift-ingress/e2e-migrate-624b5c35-gw-openshift-default-ccdbc8bfb-s8wkj"
Sep 10 17:49:02.117979 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:02.117941    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93e67e8a-cf2c-483a-8c4a-3ca7db13856f-workload-socket\") pod \"e2e-migrate-624b5c35-gw-openshift-default-ccdbc8bfb-s8wkj\" (UID: \"93e67e8a-cf2c-483a-8c4a-3ca7db13856f\") " pod="openshift-ingress/e2e-migrate-624b5c35-gw-openshift-default-ccdbc8bfb-s8wkj"
Sep 10 17:49:02.117979 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:02.117966    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93e67e8a-cf2c-483a-8c4a-3ca7db13856f-credential-socket\") pod \"e2e-migrate-624b5c35-gw-openshift-default-ccdbc8bfb-s8wkj\" (UID: \"93e67e8a-cf2c-483a-8c4a-3ca7db13856f\") " pod="openshift-ingress/e2e-migrate-624b5c35-gw-openshift-default-ccdbc8bfb-s8wkj"
Sep 10 17:49:02.118167 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:02.117994    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93e67e8a-cf2c-483a-8c4a-3ca7db13856f-istiod-ca-cert\") pod \"e2e-migrate-624b5c35-gw-openshift-default-ccdbc8bfb-s8wkj\" (UID: \"93e67e8a-cf2c-483a-8c4a-3ca7db13856f\") " pod="openshift-ingress/e2e-migrate-624b5c35-gw-openshift-default-ccdbc8bfb-s8wkj"
Sep 10 17:49:02.118167 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:02.118019    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93e67e8a-cf2c-483a-8c4a-3ca7db13856f-istio-token\") pod \"e2e-migrate-624b5c35-gw-openshift-default-ccdbc8bfb-s8wkj\" (UID: \"93e67e8a-cf2c-483a-8c4a-3ca7db13856f\") " pod="openshift-ingress/e2e-migrate-624b5c35-gw-openshift-default-ccdbc8bfb-s8wkj"
Sep 10 17:49:02.118167 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:02.118045    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93e67e8a-cf2c-483a-8c4a-3ca7db13856f-istio-envoy\") pod \"e2e-migrate-624b5c35-gw-openshift-default-ccdbc8bfb-s8wkj\" (UID: \"93e67e8a-cf2c-483a-8c4a-3ca7db13856f\") " pod="openshift-ingress/e2e-migrate-624b5c35-gw-openshift-default-ccdbc8bfb-s8wkj"
Sep 10 17:49:02.118283 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:02.118173    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93e67e8a-cf2c-483a-8c4a-3ca7db13856f-workload-certs\") pod \"e2e-migrate-624b5c35-gw-openshift-default-ccdbc8bfb-s8wkj\" (UID: \"93e67e8a-cf2c-483a-8c4a-3ca7db13856f\") " pod="openshift-ingress/e2e-migrate-624b5c35-gw-openshift-default-ccdbc8bfb-s8wkj"
Sep 10 17:49:02.218878 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:02.218844    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93e67e8a-cf2c-483a-8c4a-3ca7db13856f-istiod-ca-cert\") pod \"e2e-migrate-624b5c35-gw-openshift-default-ccdbc8bfb-s8wkj\" (UID: \"93e67e8a-cf2c-483a-8c4a-3ca7db13856f\") " pod="openshift-ingress/e2e-migrate-624b5c35-gw-openshift-default-ccdbc8bfb-s8wkj"
Sep 10 17:49:02.218878 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:02.218878    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93e67e8a-cf2c-483a-8c4a-3ca7db13856f-istio-token\") pod \"e2e-migrate-624b5c35-gw-openshift-default-ccdbc8bfb-s8wkj\" (UID: \"93e67e8a-cf2c-483a-8c4a-3ca7db13856f\") " pod="openshift-ingress/e2e-migrate-624b5c35-gw-openshift-default-ccdbc8bfb-s8wkj"
Sep 10 17:49:02.219120 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:02.218901    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93e67e8a-cf2c-483a-8c4a-3ca7db13856f-istio-envoy\") pod \"e2e-migrate-624b5c35-gw-openshift-default-ccdbc8bfb-s8wkj\" (UID: \"93e67e8a-cf2c-483a-8c4a-3ca7db13856f\") " pod="openshift-ingress/e2e-migrate-624b5c35-gw-openshift-default-ccdbc8bfb-s8wkj"
Sep 10 17:49:02.219120 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:02.218953    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93e67e8a-cf2c-483a-8c4a-3ca7db13856f-workload-certs\") pod \"e2e-migrate-624b5c35-gw-openshift-default-ccdbc8bfb-s8wkj\" (UID: \"93e67e8a-cf2c-483a-8c4a-3ca7db13856f\") " pod="openshift-ingress/e2e-migrate-624b5c35-gw-openshift-default-ccdbc8bfb-s8wkj"
Sep 10 17:49:02.219120 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:02.218972    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93e67e8a-cf2c-483a-8c4a-3ca7db13856f-istio-podinfo\") pod \"e2e-migrate-624b5c35-gw-openshift-default-ccdbc8bfb-s8wkj\" (UID: \"93e67e8a-cf2c-483a-8c4a-3ca7db13856f\") " pod="openshift-ingress/e2e-migrate-624b5c35-gw-openshift-default-ccdbc8bfb-s8wkj"
Sep 10 17:49:02.219120 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:02.219001    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94mmr\" (UniqueName: \"kubernetes.io/projected/93e67e8a-cf2c-483a-8c4a-3ca7db13856f-kube-api-access-94mmr\") pod \"e2e-migrate-624b5c35-gw-openshift-default-ccdbc8bfb-s8wkj\" (UID: \"93e67e8a-cf2c-483a-8c4a-3ca7db13856f\") " pod="openshift-ingress/e2e-migrate-624b5c35-gw-openshift-default-ccdbc8bfb-s8wkj"
Sep 10 17:49:02.219120 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:02.219051    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93e67e8a-cf2c-483a-8c4a-3ca7db13856f-istio-data\") pod \"e2e-migrate-624b5c35-gw-openshift-default-ccdbc8bfb-s8wkj\" (UID: \"93e67e8a-cf2c-483a-8c4a-3ca7db13856f\") " pod="openshift-ingress/e2e-migrate-624b5c35-gw-openshift-default-ccdbc8bfb-s8wkj"
Sep 10 17:49:02.219120 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:02.219093    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93e67e8a-cf2c-483a-8c4a-3ca7db13856f-workload-socket\") pod \"e2e-migrate-624b5c35-gw-openshift-default-ccdbc8bfb-s8wkj\" (UID: \"93e67e8a-cf2c-483a-8c4a-3ca7db13856f\") " pod="openshift-ingress/e2e-migrate-624b5c35-gw-openshift-default-ccdbc8bfb-s8wkj"
Sep 10 17:49:02.219120 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:02.219117    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93e67e8a-cf2c-483a-8c4a-3ca7db13856f-credential-socket\") pod \"e2e-migrate-624b5c35-gw-openshift-default-ccdbc8bfb-s8wkj\" (UID: \"93e67e8a-cf2c-483a-8c4a-3ca7db13856f\") " pod="openshift-ingress/e2e-migrate-624b5c35-gw-openshift-default-ccdbc8bfb-s8wkj"
Sep 10 17:49:02.220013 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:02.219919    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93e67e8a-cf2c-483a-8c4a-3ca7db13856f-istio-data\") pod \"e2e-migrate-624b5c35-gw-openshift-default-ccdbc8bfb-s8wkj\" (UID: \"93e67e8a-cf2c-483a-8c4a-3ca7db13856f\") " pod="openshift-ingress/e2e-migrate-624b5c35-gw-openshift-default-ccdbc8bfb-s8wkj"
Sep 10 17:49:02.220013 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:02.219936    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93e67e8a-cf2c-483a-8c4a-3ca7db13856f-credential-socket\") pod \"e2e-migrate-624b5c35-gw-openshift-default-ccdbc8bfb-s8wkj\" (UID: \"93e67e8a-cf2c-483a-8c4a-3ca7db13856f\") " pod="openshift-ingress/e2e-migrate-624b5c35-gw-openshift-default-ccdbc8bfb-s8wkj"
Sep 10 17:49:02.220013 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:02.219988    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93e67e8a-cf2c-483a-8c4a-3ca7db13856f-workload-certs\") pod \"e2e-migrate-624b5c35-gw-openshift-default-ccdbc8bfb-s8wkj\" (UID: \"93e67e8a-cf2c-483a-8c4a-3ca7db13856f\") " pod="openshift-ingress/e2e-migrate-624b5c35-gw-openshift-default-ccdbc8bfb-s8wkj"
Sep 10 17:49:02.220267 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:02.220136    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93e67e8a-cf2c-483a-8c4a-3ca7db13856f-workload-socket\") pod \"e2e-migrate-624b5c35-gw-openshift-default-ccdbc8bfb-s8wkj\" (UID: \"93e67e8a-cf2c-483a-8c4a-3ca7db13856f\") " pod="openshift-ingress/e2e-migrate-624b5c35-gw-openshift-default-ccdbc8bfb-s8wkj"
Sep 10 17:49:02.222730 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:02.222705    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93e67e8a-cf2c-483a-8c4a-3ca7db13856f-istio-envoy\") pod \"e2e-migrate-624b5c35-gw-openshift-default-ccdbc8bfb-s8wkj\" (UID: \"93e67e8a-cf2c-483a-8c4a-3ca7db13856f\") " pod="openshift-ingress/e2e-migrate-624b5c35-gw-openshift-default-ccdbc8bfb-s8wkj"
Sep 10 17:49:02.222851 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:02.222834    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93e67e8a-cf2c-483a-8c4a-3ca7db13856f-istiod-ca-cert\") pod \"e2e-migrate-624b5c35-gw-openshift-default-ccdbc8bfb-s8wkj\" (UID: \"93e67e8a-cf2c-483a-8c4a-3ca7db13856f\") " pod="openshift-ingress/e2e-migrate-624b5c35-gw-openshift-default-ccdbc8bfb-s8wkj"
Sep 10 17:49:02.228858 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:02.228829    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93e67e8a-cf2c-483a-8c4a-3ca7db13856f-istio-token\") pod \"e2e-migrate-624b5c35-gw-openshift-default-ccdbc8bfb-s8wkj\" (UID: \"93e67e8a-cf2c-483a-8c4a-3ca7db13856f\") " pod="openshift-ingress/e2e-migrate-624b5c35-gw-openshift-default-ccdbc8bfb-s8wkj"
Sep 10 17:49:02.228978 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:02.228873    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93e67e8a-cf2c-483a-8c4a-3ca7db13856f-istio-podinfo\") pod \"e2e-migrate-624b5c35-gw-openshift-default-ccdbc8bfb-s8wkj\" (UID: \"93e67e8a-cf2c-483a-8c4a-3ca7db13856f\") " pod="openshift-ingress/e2e-migrate-624b5c35-gw-openshift-default-ccdbc8bfb-s8wkj"
Sep 10 17:49:02.231024 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:02.231004    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-94mmr\" (UniqueName: \"kubernetes.io/projected/93e67e8a-cf2c-483a-8c4a-3ca7db13856f-kube-api-access-94mmr\") pod \"e2e-migrate-624b5c35-gw-openshift-default-ccdbc8bfb-s8wkj\" (UID: \"93e67e8a-cf2c-483a-8c4a-3ca7db13856f\") " pod="openshift-ingress/e2e-migrate-624b5c35-gw-openshift-default-ccdbc8bfb-s8wkj"
Sep 10 17:49:02.298804 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:02.298720    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-624b5c35-gw-openshift-default-ccdbc8bfb-s8wkj"
Sep 10 17:49:02.475110 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:02.475077    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b657c988-b137-47c2-aaf0-3a41dbdf7a56" path="/var/lib/kubelet/pods/b657c988-b137-47c2-aaf0-3a41dbdf7a56/volumes"
Sep 10 17:49:02.475462 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:02.475406    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="be27a3df-82ca-42bc-8bf5-75078ae9879a" path="/var/lib/kubelet/pods/be27a3df-82ca-42bc-8bf5-75078ae9879a/volumes"
Sep 10 17:49:02.578640 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:02.578384    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-624b5c35-gw-openshift-default-ccdbc8bfb-s8wkj"]
Sep 10 17:49:02.581722 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:49:02.581689    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93e67e8a_cf2c_483a_8c4a_3ca7db13856f.slice/crio-80dce0acd99276c461800828e7400fd6535476335fffc728f1bcf09d81aff167 WatchSource:0}: Error finding container 80dce0acd99276c461800828e7400fd6535476335fffc728f1bcf09d81aff167: Status 404 returned error can't find the container with id 80dce0acd99276c461800828e7400fd6535476335fffc728f1bcf09d81aff167
Sep 10 17:49:02.584243 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:02.584196    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 10 17:49:02.584354 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:02.584283    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 10 17:49:02.584354 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:02.584346    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 10 17:49:03.425788 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:03.425752    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-624b5c35-gw-openshift-default-ccdbc8bfb-s8wkj" event={"ID":"93e67e8a-cf2c-483a-8c4a-3ca7db13856f","Type":"ContainerStarted","Data":"2b7d2e8dfc8388d05ab76f2e01a7a6c0b447e7b3df7cbb7befc59e7cd47ca125"}
Sep 10 17:49:03.425959 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:03.425803    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-624b5c35-gw-openshift-default-ccdbc8bfb-s8wkj" event={"ID":"93e67e8a-cf2c-483a-8c4a-3ca7db13856f","Type":"ContainerStarted","Data":"80dce0acd99276c461800828e7400fd6535476335fffc728f1bcf09d81aff167"}
Sep 10 17:49:03.448672 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:03.448625    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-624b5c35-gw-openshift-default-ccdbc8bfb-s8wkj" podStartSLOduration=2.448605283 podStartE2EDuration="2.448605283s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:49:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:49:03.447854156 +0000 UTC m=+787.447684222" watchObservedRunningTime="2026-09-10 17:49:03.448605283 +0000 UTC m=+787.448435350"
Sep 10 17:49:04.299352 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:04.299321    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-624b5c35-gw-openshift-default-ccdbc8bfb-s8wkj"
Sep 10 17:49:04.303946 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:04.303921    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-624b5c35-gw-openshift-default-ccdbc8bfb-s8wkj"
Sep 10 17:49:04.430675 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:04.430645    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-624b5c35-gw-openshift-default-ccdbc8bfb-s8wkj"
Sep 10 17:49:04.431533 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:04.431512    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-624b5c35-gw-openshift-default-ccdbc8bfb-s8wkj"
Sep 10 17:49:06.013044 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:06.013022    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-624b5c35-5657b6d4cd-6fcv6"]
Sep 10 17:49:06.016371 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:06.016355    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-624b5c35-5657b6d4cd-6fcv6"
Sep 10 17:49:06.019599 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:06.019557    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-624b5c35\""
Sep 10 17:49:06.034443 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:06.034417    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-624b5c35-5657b6d4cd-6fcv6"]
Sep 10 17:49:06.054861 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:06.054838    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a25eadc9-13ff-4713-8f1a-ba4aab03721e-plugins-config-volume\") pod \"payload-processing-e2e-migrate-624b5c35-5657b6d4cd-6fcv6\" (UID: \"a25eadc9-13ff-4713-8f1a-ba4aab03721e\") " pod="openshift-ingress/payload-processing-e2e-migrate-624b5c35-5657b6d4cd-6fcv6"
Sep 10 17:49:06.054959 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:06.054903    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7m792\" (UniqueName: \"kubernetes.io/projected/a25eadc9-13ff-4713-8f1a-ba4aab03721e-kube-api-access-7m792\") pod \"payload-processing-e2e-migrate-624b5c35-5657b6d4cd-6fcv6\" (UID: \"a25eadc9-13ff-4713-8f1a-ba4aab03721e\") " pod="openshift-ingress/payload-processing-e2e-migrate-624b5c35-5657b6d4cd-6fcv6"
Sep 10 17:49:06.087574 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:06.087537    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-624b5c35-7dddf7598-vk9vm"]
Sep 10 17:49:06.091516 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:06.091497    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-624b5c35-7dddf7598-vk9vm"
Sep 10 17:49:06.094233 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:06.094213    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-624b5c35-dockercfg-fctvc\""
Sep 10 17:49:06.101313 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:06.101293    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-624b5c35-7dddf7598-vk9vm"]
Sep 10 17:49:06.156057 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:06.155996    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7m792\" (UniqueName: \"kubernetes.io/projected/a25eadc9-13ff-4713-8f1a-ba4aab03721e-kube-api-access-7m792\") pod \"payload-processing-e2e-migrate-624b5c35-5657b6d4cd-6fcv6\" (UID: \"a25eadc9-13ff-4713-8f1a-ba4aab03721e\") " pod="openshift-ingress/payload-processing-e2e-migrate-624b5c35-5657b6d4cd-6fcv6"
Sep 10 17:49:06.156250 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:06.156085    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7nwzd\" (UniqueName: \"kubernetes.io/projected/a40c0ae9-962b-42da-b407-336d90b2987a-kube-api-access-7nwzd\") pod \"payload-pre-processing-e2e-migrate-624b5c35-7dddf7598-vk9vm\" (UID: \"a40c0ae9-962b-42da-b407-336d90b2987a\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-624b5c35-7dddf7598-vk9vm"
Sep 10 17:49:06.156250 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:06.156142    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a25eadc9-13ff-4713-8f1a-ba4aab03721e-plugins-config-volume\") pod \"payload-processing-e2e-migrate-624b5c35-5657b6d4cd-6fcv6\" (UID: \"a25eadc9-13ff-4713-8f1a-ba4aab03721e\") " pod="openshift-ingress/payload-processing-e2e-migrate-624b5c35-5657b6d4cd-6fcv6"
Sep 10 17:49:06.156250 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:06.156218    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a40c0ae9-962b-42da-b407-336d90b2987a-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-624b5c35-7dddf7598-vk9vm\" (UID: \"a40c0ae9-962b-42da-b407-336d90b2987a\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-624b5c35-7dddf7598-vk9vm"
Sep 10 17:49:06.157448 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:06.157395    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a25eadc9-13ff-4713-8f1a-ba4aab03721e-plugins-config-volume\") pod \"payload-processing-e2e-migrate-624b5c35-5657b6d4cd-6fcv6\" (UID: \"a25eadc9-13ff-4713-8f1a-ba4aab03721e\") " pod="openshift-ingress/payload-processing-e2e-migrate-624b5c35-5657b6d4cd-6fcv6"
Sep 10 17:49:06.165530 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:06.165505    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7m792\" (UniqueName: \"kubernetes.io/projected/a25eadc9-13ff-4713-8f1a-ba4aab03721e-kube-api-access-7m792\") pod \"payload-processing-e2e-migrate-624b5c35-5657b6d4cd-6fcv6\" (UID: \"a25eadc9-13ff-4713-8f1a-ba4aab03721e\") " pod="openshift-ingress/payload-processing-e2e-migrate-624b5c35-5657b6d4cd-6fcv6"
Sep 10 17:49:06.247882 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:06.247858    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-4601bf83-openshift-default-7d5654d96d-g2lmw"
Sep 10 17:49:06.257506 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:06.257455    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7774ff67-4869-4805-95f2-b6aad0e5303a-istio-envoy\") pod \"7774ff67-4869-4805-95f2-b6aad0e5303a\" (UID: \"7774ff67-4869-4805-95f2-b6aad0e5303a\") "
Sep 10 17:49:06.257623 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:06.257508    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7774ff67-4869-4805-95f2-b6aad0e5303a-istio-data\") pod \"7774ff67-4869-4805-95f2-b6aad0e5303a\" (UID: \"7774ff67-4869-4805-95f2-b6aad0e5303a\") "
Sep 10 17:49:06.257623 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:06.257525    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qjvn7\" (UniqueName: \"kubernetes.io/projected/7774ff67-4869-4805-95f2-b6aad0e5303a-kube-api-access-qjvn7\") pod \"7774ff67-4869-4805-95f2-b6aad0e5303a\" (UID: \"7774ff67-4869-4805-95f2-b6aad0e5303a\") "
Sep 10 17:49:06.257623 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:06.257554    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7774ff67-4869-4805-95f2-b6aad0e5303a-workload-socket\") pod \"7774ff67-4869-4805-95f2-b6aad0e5303a\" (UID: \"7774ff67-4869-4805-95f2-b6aad0e5303a\") "
Sep 10 17:49:06.257623 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:06.257603    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7774ff67-4869-4805-95f2-b6aad0e5303a-istio-podinfo\") pod \"7774ff67-4869-4805-95f2-b6aad0e5303a\" (UID: \"7774ff67-4869-4805-95f2-b6aad0e5303a\") "
Sep 10 17:49:06.257911 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:06.257719    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7774ff67-4869-4805-95f2-b6aad0e5303a-istio-token\") pod \"7774ff67-4869-4805-95f2-b6aad0e5303a\" (UID: \"7774ff67-4869-4805-95f2-b6aad0e5303a\") "
Sep 10 17:49:06.257911 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:06.257793    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7774ff67-4869-4805-95f2-b6aad0e5303a-credential-socket\") pod \"7774ff67-4869-4805-95f2-b6aad0e5303a\" (UID: \"7774ff67-4869-4805-95f2-b6aad0e5303a\") "
Sep 10 17:49:06.257911 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:06.257833    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7774ff67-4869-4805-95f2-b6aad0e5303a-workload-certs\") pod \"7774ff67-4869-4805-95f2-b6aad0e5303a\" (UID: \"7774ff67-4869-4805-95f2-b6aad0e5303a\") "
Sep 10 17:49:06.257911 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:06.257863    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7774ff67-4869-4805-95f2-b6aad0e5303a-istiod-ca-cert\") pod \"7774ff67-4869-4805-95f2-b6aad0e5303a\" (UID: \"7774ff67-4869-4805-95f2-b6aad0e5303a\") "
Sep 10 17:49:06.257911 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:06.257832    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7774ff67-4869-4805-95f2-b6aad0e5303a-istio-data" (OuterVolumeSpecName: "istio-data") pod "7774ff67-4869-4805-95f2-b6aad0e5303a" (UID: "7774ff67-4869-4805-95f2-b6aad0e5303a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:49:06.257911 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:06.257850    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7774ff67-4869-4805-95f2-b6aad0e5303a-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "7774ff67-4869-4805-95f2-b6aad0e5303a" (UID: "7774ff67-4869-4805-95f2-b6aad0e5303a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:49:06.258192 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:06.258016    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7774ff67-4869-4805-95f2-b6aad0e5303a-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "7774ff67-4869-4805-95f2-b6aad0e5303a" (UID: "7774ff67-4869-4805-95f2-b6aad0e5303a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:49:06.258192 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:06.258043    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7774ff67-4869-4805-95f2-b6aad0e5303a-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "7774ff67-4869-4805-95f2-b6aad0e5303a" (UID: "7774ff67-4869-4805-95f2-b6aad0e5303a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:49:06.258192 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:06.258073    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a40c0ae9-962b-42da-b407-336d90b2987a-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-624b5c35-7dddf7598-vk9vm\" (UID: \"a40c0ae9-962b-42da-b407-336d90b2987a\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-624b5c35-7dddf7598-vk9vm"
Sep 10 17:49:06.258192 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:06.258168    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7nwzd\" (UniqueName: \"kubernetes.io/projected/a40c0ae9-962b-42da-b407-336d90b2987a-kube-api-access-7nwzd\") pod \"payload-pre-processing-e2e-migrate-624b5c35-7dddf7598-vk9vm\" (UID: \"a40c0ae9-962b-42da-b407-336d90b2987a\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-624b5c35-7dddf7598-vk9vm"
Sep 10 17:49:06.258390 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:06.258250    2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7774ff67-4869-4805-95f2-b6aad0e5303a-credential-socket\") on node \"ip-10-0-129-122.ec2.internal\" DevicePath \"\""
Sep 10 17:49:06.258390 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:06.258294    2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7774ff67-4869-4805-95f2-b6aad0e5303a-workload-certs\") on node \"ip-10-0-129-122.ec2.internal\" DevicePath \"\""
Sep 10 17:49:06.258390 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:06.258308    2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7774ff67-4869-4805-95f2-b6aad0e5303a-istio-data\") on node \"ip-10-0-129-122.ec2.internal\" DevicePath \"\""
Sep 10 17:49:06.258390 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:06.258321    2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7774ff67-4869-4805-95f2-b6aad0e5303a-workload-socket\") on node \"ip-10-0-129-122.ec2.internal\" DevicePath \"\""
Sep 10 17:49:06.258618 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:06.258480    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7774ff67-4869-4805-95f2-b6aad0e5303a-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "7774ff67-4869-4805-95f2-b6aad0e5303a" (UID: "7774ff67-4869-4805-95f2-b6aad0e5303a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 17:49:06.258901 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:06.258880    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a40c0ae9-962b-42da-b407-336d90b2987a-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-624b5c35-7dddf7598-vk9vm\" (UID: \"a40c0ae9-962b-42da-b407-336d90b2987a\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-624b5c35-7dddf7598-vk9vm"
Sep 10 17:49:06.259840 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:06.259823    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7774ff67-4869-4805-95f2-b6aad0e5303a-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "7774ff67-4869-4805-95f2-b6aad0e5303a" (UID: "7774ff67-4869-4805-95f2-b6aad0e5303a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:49:06.260384 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:06.260365    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7774ff67-4869-4805-95f2-b6aad0e5303a-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "7774ff67-4869-4805-95f2-b6aad0e5303a" (UID: "7774ff67-4869-4805-95f2-b6aad0e5303a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 10 17:49:06.260384 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:06.260364    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7774ff67-4869-4805-95f2-b6aad0e5303a-kube-api-access-qjvn7" (OuterVolumeSpecName: "kube-api-access-qjvn7") pod "7774ff67-4869-4805-95f2-b6aad0e5303a" (UID: "7774ff67-4869-4805-95f2-b6aad0e5303a"). InnerVolumeSpecName "kube-api-access-qjvn7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:49:06.260470 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:06.260391    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7774ff67-4869-4805-95f2-b6aad0e5303a-istio-token" (OuterVolumeSpecName: "istio-token") pod "7774ff67-4869-4805-95f2-b6aad0e5303a" (UID: "7774ff67-4869-4805-95f2-b6aad0e5303a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:49:06.266577 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:06.266545    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7nwzd\" (UniqueName: \"kubernetes.io/projected/a40c0ae9-962b-42da-b407-336d90b2987a-kube-api-access-7nwzd\") pod \"payload-pre-processing-e2e-migrate-624b5c35-7dddf7598-vk9vm\" (UID: \"a40c0ae9-962b-42da-b407-336d90b2987a\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-624b5c35-7dddf7598-vk9vm"
Sep 10 17:49:06.358772 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:06.358743    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qjvn7\" (UniqueName: \"kubernetes.io/projected/7774ff67-4869-4805-95f2-b6aad0e5303a-kube-api-access-qjvn7\") on node \"ip-10-0-129-122.ec2.internal\" DevicePath \"\""
Sep 10 17:49:06.358772 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:06.358769    2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7774ff67-4869-4805-95f2-b6aad0e5303a-istio-podinfo\") on node \"ip-10-0-129-122.ec2.internal\" DevicePath \"\""
Sep 10 17:49:06.358933 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:06.358778    2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7774ff67-4869-4805-95f2-b6aad0e5303a-istio-token\") on node \"ip-10-0-129-122.ec2.internal\" DevicePath \"\""
Sep 10 17:49:06.358933 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:06.358786    2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7774ff67-4869-4805-95f2-b6aad0e5303a-istiod-ca-cert\") on node \"ip-10-0-129-122.ec2.internal\" DevicePath \"\""
Sep 10 17:49:06.358933 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:06.358794    2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7774ff67-4869-4805-95f2-b6aad0e5303a-istio-envoy\") on node \"ip-10-0-129-122.ec2.internal\" DevicePath \"\""
Sep 10 17:49:06.392621 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:06.392590    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-624b5c35-5657b6d4cd-6fcv6"
Sep 10 17:49:06.401053 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:06.401029    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-624b5c35-7dddf7598-vk9vm"
Sep 10 17:49:06.442558 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:06.442521    2579 generic.go:356] "Generic (PLEG): container finished" podID="7774ff67-4869-4805-95f2-b6aad0e5303a" containerID="8b2222d9a3fa5b990505d5878632f8c2f87544ad26d2ecdfb83d44d0a1aace2b" exitCode=0
Sep 10 17:49:06.442763 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:06.442708    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-4601bf83-openshift-default-7d5654d96d-g2lmw"
Sep 10 17:49:06.442763 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:06.442727    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-4601bf83-openshift-default-7d5654d96d-g2lmw" event={"ID":"7774ff67-4869-4805-95f2-b6aad0e5303a","Type":"ContainerDied","Data":"8b2222d9a3fa5b990505d5878632f8c2f87544ad26d2ecdfb83d44d0a1aace2b"}
Sep 10 17:49:06.442878 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:06.442816    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-4601bf83-openshift-default-7d5654d96d-g2lmw" event={"ID":"7774ff67-4869-4805-95f2-b6aad0e5303a","Type":"ContainerDied","Data":"86cd1d559dff7f5b3c34dc4b98abaf9401b81dc1a1ad64fb4f70e15d7fcf3a3f"}
Sep 10 17:49:06.442878 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:06.442845    2579 scope.go:117] "RemoveContainer" containerID="8b2222d9a3fa5b990505d5878632f8c2f87544ad26d2ecdfb83d44d0a1aace2b"
Sep 10 17:49:06.457114 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:06.456953    2579 scope.go:117] "RemoveContainer" containerID="9029c48809e4762ca38c894b7e889fc51f06ef992d25503f4b852ff51530e529"
Sep 10 17:49:06.472807 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:06.471774    2579 scope.go:117] "RemoveContainer" containerID="8b2222d9a3fa5b990505d5878632f8c2f87544ad26d2ecdfb83d44d0a1aace2b"
Sep 10 17:49:06.475485 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:49:06.475356    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b2222d9a3fa5b990505d5878632f8c2f87544ad26d2ecdfb83d44d0a1aace2b\": container with ID starting with 8b2222d9a3fa5b990505d5878632f8c2f87544ad26d2ecdfb83d44d0a1aace2b not found: ID does not exist" containerID="8b2222d9a3fa5b990505d5878632f8c2f87544ad26d2ecdfb83d44d0a1aace2b"
Sep 10 17:49:06.475485 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:06.475399    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b2222d9a3fa5b990505d5878632f8c2f87544ad26d2ecdfb83d44d0a1aace2b"} err="failed to get container status \"8b2222d9a3fa5b990505d5878632f8c2f87544ad26d2ecdfb83d44d0a1aace2b\": rpc error: code = NotFound desc = could not find container \"8b2222d9a3fa5b990505d5878632f8c2f87544ad26d2ecdfb83d44d0a1aace2b\": container with ID starting with 8b2222d9a3fa5b990505d5878632f8c2f87544ad26d2ecdfb83d44d0a1aace2b not found: ID does not exist"
Sep 10 17:49:06.475485 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:06.475423    2579 scope.go:117] "RemoveContainer" containerID="9029c48809e4762ca38c894b7e889fc51f06ef992d25503f4b852ff51530e529"
Sep 10 17:49:06.479150 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:49:06.477799    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9029c48809e4762ca38c894b7e889fc51f06ef992d25503f4b852ff51530e529\": container with ID starting with 9029c48809e4762ca38c894b7e889fc51f06ef992d25503f4b852ff51530e529 not found: ID does not exist" containerID="9029c48809e4762ca38c894b7e889fc51f06ef992d25503f4b852ff51530e529"
Sep 10 17:49:06.479150 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:06.477834    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9029c48809e4762ca38c894b7e889fc51f06ef992d25503f4b852ff51530e529"} err="failed to get container status \"9029c48809e4762ca38c894b7e889fc51f06ef992d25503f4b852ff51530e529\": rpc error: code = NotFound desc = could not find container \"9029c48809e4762ca38c894b7e889fc51f06ef992d25503f4b852ff51530e529\": container with ID starting with 9029c48809e4762ca38c894b7e889fc51f06ef992d25503f4b852ff51530e529 not found: ID does not exist"
Sep 10 17:49:06.480418 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:06.480384    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-4601bf83-openshift-default-7d5654d96d-g2lmw"]
Sep 10 17:49:06.482757 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:06.482738    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-4601bf83-openshift-default-7d5654d96d-g2lmw"]
Sep 10 17:49:06.678828 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:06.678804    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-624b5c35-5657b6d4cd-6fcv6"]
Sep 10 17:49:06.681148 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:49:06.681117    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda25eadc9_13ff_4713_8f1a_ba4aab03721e.slice/crio-c2313813e72c2a342a68bcb8ad5043e295acf31f380d9e0cae9c223975b0a489 WatchSource:0}: Error finding container c2313813e72c2a342a68bcb8ad5043e295acf31f380d9e0cae9c223975b0a489: Status 404 returned error can't find the container with id c2313813e72c2a342a68bcb8ad5043e295acf31f380d9e0cae9c223975b0a489
Sep 10 17:49:06.682381 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:06.682360    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-624b5c35-7dddf7598-vk9vm"]
Sep 10 17:49:06.685146 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:49:06.685121    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda40c0ae9_962b_42da_b407_336d90b2987a.slice/crio-85912e95ea0d924e3b1db207a63e70edf5bc4190d38c6f28dd1ecb3226e611d4 WatchSource:0}: Error finding container 85912e95ea0d924e3b1db207a63e70edf5bc4190d38c6f28dd1ecb3226e611d4: Status 404 returned error can't find the container with id 85912e95ea0d924e3b1db207a63e70edf5bc4190d38c6f28dd1ecb3226e611d4
Sep 10 17:49:07.448787 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:07.448750    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-624b5c35-7dddf7598-vk9vm" event={"ID":"a40c0ae9-962b-42da-b407-336d90b2987a","Type":"ContainerStarted","Data":"74bbac718ac63f2692351671fd8a5bea39e045389452256242eb4d2598fe1730"}
Sep 10 17:49:07.449253 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:07.448819    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-624b5c35-7dddf7598-vk9vm"
Sep 10 17:49:07.449253 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:07.448845    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-624b5c35-7dddf7598-vk9vm" event={"ID":"a40c0ae9-962b-42da-b407-336d90b2987a","Type":"ContainerStarted","Data":"85912e95ea0d924e3b1db207a63e70edf5bc4190d38c6f28dd1ecb3226e611d4"}
Sep 10 17:49:07.451278 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:07.451255    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-624b5c35-5657b6d4cd-6fcv6" event={"ID":"a25eadc9-13ff-4713-8f1a-ba4aab03721e","Type":"ContainerStarted","Data":"d06e63194c5517e6e0926530cea7f522bc882442cce14a2bec0d827996d3d23b"}
Sep 10 17:49:07.451389 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:07.451300    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-624b5c35-5657b6d4cd-6fcv6"
Sep 10 17:49:07.451389 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:07.451316    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-624b5c35-5657b6d4cd-6fcv6" event={"ID":"a25eadc9-13ff-4713-8f1a-ba4aab03721e","Type":"ContainerStarted","Data":"c2313813e72c2a342a68bcb8ad5043e295acf31f380d9e0cae9c223975b0a489"}
Sep 10 17:49:07.469404 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:07.469351    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-624b5c35-7dddf7598-vk9vm" podStartSLOduration=1.469332475 podStartE2EDuration="1.469332475s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:49:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:49:07.468875503 +0000 UTC m=+791.468705563" watchObservedRunningTime="2026-09-10 17:49:07.469332475 +0000 UTC m=+791.469162542"
Sep 10 17:49:07.491806 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:07.491761    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-624b5c35-5657b6d4cd-6fcv6" podStartSLOduration=2.491747579 podStartE2EDuration="2.491747579s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:49:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:49:07.489422646 +0000 UTC m=+791.489252714" watchObservedRunningTime="2026-09-10 17:49:07.491747579 +0000 UTC m=+791.491577645"
Sep 10 17:49:08.475873 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:08.475844    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7774ff67-4869-4805-95f2-b6aad0e5303a" path="/var/lib/kubelet/pods/7774ff67-4869-4805-95f2-b6aad0e5303a/volumes"
Sep 10 17:49:17.511968 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:17.511863    2579 generic.go:356] "Generic (PLEG): container finished" podID="93e67e8a-cf2c-483a-8c4a-3ca7db13856f" containerID="2b7d2e8dfc8388d05ab76f2e01a7a6c0b447e7b3df7cbb7befc59e7cd47ca125" exitCode=0
Sep 10 17:49:17.511968 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:17.511932    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-624b5c35-gw-openshift-default-ccdbc8bfb-s8wkj" event={"ID":"93e67e8a-cf2c-483a-8c4a-3ca7db13856f","Type":"ContainerDied","Data":"2b7d2e8dfc8388d05ab76f2e01a7a6c0b447e7b3df7cbb7befc59e7cd47ca125"}
Sep 10 17:49:17.512386 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:17.512317    2579 scope.go:117] "RemoveContainer" containerID="2b7d2e8dfc8388d05ab76f2e01a7a6c0b447e7b3df7cbb7befc59e7cd47ca125"
Sep 10 17:49:17.513365 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:17.513339    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 10 17:49:17.513425 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:17.513412    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 10 17:49:17.513460 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:17.513449    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 10 17:49:18.458635 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:18.458559    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-624b5c35-5657b6d4cd-6fcv6"
Sep 10 17:49:18.458803 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:18.458678    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-624b5c35-7dddf7598-vk9vm"
Sep 10 17:49:18.522127 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:18.521989    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-624b5c35-gw-openshift-default-ccdbc8bfb-s8wkj" event={"ID":"93e67e8a-cf2c-483a-8c4a-3ca7db13856f","Type":"ContainerStarted","Data":"d0230d30b92790296334cf4a1effcdd9fd49d2bd3d9b933068b15302ccfdd94c"}
Sep 10 17:49:19.299429 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:19.299392    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-624b5c35-gw-openshift-default-ccdbc8bfb-s8wkj"
Sep 10 17:49:19.300498 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:19.300466    2579 patch_prober.go:28] interesting pod/e2e-migrate-624b5c35-gw-openshift-default-ccdbc8bfb-s8wkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body=
Sep 10 17:49:19.300656 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:19.300516    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-624b5c35-gw-openshift-default-ccdbc8bfb-s8wkj" podUID="93e67e8a-cf2c-483a-8c4a-3ca7db13856f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused"
Sep 10 17:49:19.430971 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:19.430937    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-624b5c35-gw-openshift-default-ccdbc8bfb-s8wkj"
Sep 10 17:49:20.299924 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:20.299890    2579 patch_prober.go:28] interesting pod/e2e-migrate-624b5c35-gw-openshift-default-ccdbc8bfb-s8wkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body=
Sep 10 17:49:20.300329 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:20.299944    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-624b5c35-gw-openshift-default-ccdbc8bfb-s8wkj" podUID="93e67e8a-cf2c-483a-8c4a-3ca7db13856f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused"
Sep 10 17:49:21.299461 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:21.299423    2579 patch_prober.go:28] interesting pod/e2e-migrate-624b5c35-gw-openshift-default-ccdbc8bfb-s8wkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body=
Sep 10 17:49:21.299652 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:21.299478    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-624b5c35-gw-openshift-default-ccdbc8bfb-s8wkj" podUID="93e67e8a-cf2c-483a-8c4a-3ca7db13856f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused"
Sep 10 17:49:22.299956 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:22.299920    2579 patch_prober.go:28] interesting pod/e2e-migrate-624b5c35-gw-openshift-default-ccdbc8bfb-s8wkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body=
Sep 10 17:49:22.300455 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:22.299987    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-624b5c35-gw-openshift-default-ccdbc8bfb-s8wkj" podUID="93e67e8a-cf2c-483a-8c4a-3ca7db13856f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused"
Sep 10 17:49:23.299895 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:23.299855    2579 patch_prober.go:28] interesting pod/e2e-migrate-624b5c35-gw-openshift-default-ccdbc8bfb-s8wkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body=
Sep 10 17:49:23.300077 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:23.299940    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-624b5c35-gw-openshift-default-ccdbc8bfb-s8wkj" podUID="93e67e8a-cf2c-483a-8c4a-3ca7db13856f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused"
Sep 10 17:49:24.303703 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:24.303671    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-624b5c35-gw-openshift-default-ccdbc8bfb-s8wkj"
Sep 10 17:49:24.304688 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:24.304667    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-624b5c35-gw-openshift-default-ccdbc8bfb-s8wkj"
Sep 10 17:49:37.002555 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:37.002516    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-624b5c35-5657b6d4cd-6fcv6"]
Sep 10 17:49:37.003060 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:37.002843    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-624b5c35-5657b6d4cd-6fcv6" podUID="a25eadc9-13ff-4713-8f1a-ba4aab03721e" containerName="payload-processing" containerID="cri-o://d06e63194c5517e6e0926530cea7f522bc882442cce14a2bec0d827996d3d23b" gracePeriod=30
Sep 10 17:49:37.018940 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:37.018911    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-624b5c35-7dddf7598-vk9vm"]
Sep 10 17:49:37.019158 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:37.019129    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-624b5c35-7dddf7598-vk9vm" podUID="a40c0ae9-962b-42da-b407-336d90b2987a" containerName="payload-pre-processing" containerID="cri-o://74bbac718ac63f2692351671fd8a5bea39e045389452256242eb4d2598fe1730" gracePeriod=30
Sep 10 17:49:37.294705 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:37.294678    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-624b5c35-5657b6d4cd-6fcv6"
Sep 10 17:49:37.310814 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:37.310791    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-624b5c35-7dddf7598-vk9vm"
Sep 10 17:49:37.353304 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:37.353259    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a25eadc9-13ff-4713-8f1a-ba4aab03721e-plugins-config-volume\") pod \"a25eadc9-13ff-4713-8f1a-ba4aab03721e\" (UID: \"a25eadc9-13ff-4713-8f1a-ba4aab03721e\") "
Sep 10 17:49:37.353465 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:37.353350    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7m792\" (UniqueName: \"kubernetes.io/projected/a25eadc9-13ff-4713-8f1a-ba4aab03721e-kube-api-access-7m792\") pod \"a25eadc9-13ff-4713-8f1a-ba4aab03721e\" (UID: \"a25eadc9-13ff-4713-8f1a-ba4aab03721e\") "
Sep 10 17:49:37.353849 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:37.353549    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a25eadc9-13ff-4713-8f1a-ba4aab03721e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a25eadc9-13ff-4713-8f1a-ba4aab03721e" (UID: "a25eadc9-13ff-4713-8f1a-ba4aab03721e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 17:49:37.353849 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:37.353708    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a25eadc9-13ff-4713-8f1a-ba4aab03721e-plugins-config-volume\") on node \"ip-10-0-129-122.ec2.internal\" DevicePath \"\""
Sep 10 17:49:37.355987 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:37.355953    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a25eadc9-13ff-4713-8f1a-ba4aab03721e-kube-api-access-7m792" (OuterVolumeSpecName: "kube-api-access-7m792") pod "a25eadc9-13ff-4713-8f1a-ba4aab03721e" (UID: "a25eadc9-13ff-4713-8f1a-ba4aab03721e"). InnerVolumeSpecName "kube-api-access-7m792". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:49:37.382413 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:37.382381    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-624b5c35-gw-openshift-default-ccdbc8bfb-s8wkj"]
Sep 10 17:49:37.382673 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:37.382646    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-624b5c35-gw-openshift-default-ccdbc8bfb-s8wkj" podUID="93e67e8a-cf2c-483a-8c4a-3ca7db13856f" containerName="istio-proxy" containerID="cri-o://d0230d30b92790296334cf4a1effcdd9fd49d2bd3d9b933068b15302ccfdd94c" gracePeriod=30
Sep 10 17:49:37.454859 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:37.454831    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a40c0ae9-962b-42da-b407-336d90b2987a-plugins-config-volume\") pod \"a40c0ae9-962b-42da-b407-336d90b2987a\" (UID: \"a40c0ae9-962b-42da-b407-336d90b2987a\") "
Sep 10 17:49:37.454985 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:37.454872    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7nwzd\" (UniqueName: \"kubernetes.io/projected/a40c0ae9-962b-42da-b407-336d90b2987a-kube-api-access-7nwzd\") pod \"a40c0ae9-962b-42da-b407-336d90b2987a\" (UID: \"a40c0ae9-962b-42da-b407-336d90b2987a\") "
Sep 10 17:49:37.455083 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:37.455071    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7m792\" (UniqueName: \"kubernetes.io/projected/a25eadc9-13ff-4713-8f1a-ba4aab03721e-kube-api-access-7m792\") on node \"ip-10-0-129-122.ec2.internal\" DevicePath \"\""
Sep 10 17:49:37.455174 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:37.455152    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a40c0ae9-962b-42da-b407-336d90b2987a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a40c0ae9-962b-42da-b407-336d90b2987a" (UID: "a40c0ae9-962b-42da-b407-336d90b2987a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 17:49:37.457122 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:37.457099    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a40c0ae9-962b-42da-b407-336d90b2987a-kube-api-access-7nwzd" (OuterVolumeSpecName: "kube-api-access-7nwzd") pod "a40c0ae9-962b-42da-b407-336d90b2987a" (UID: "a40c0ae9-962b-42da-b407-336d90b2987a"). InnerVolumeSpecName "kube-api-access-7nwzd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:49:37.556013 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:37.555932    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a40c0ae9-962b-42da-b407-336d90b2987a-plugins-config-volume\") on node \"ip-10-0-129-122.ec2.internal\" DevicePath \"\""
Sep 10 17:49:37.556013 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:37.555968    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7nwzd\" (UniqueName: \"kubernetes.io/projected/a40c0ae9-962b-42da-b407-336d90b2987a-kube-api-access-7nwzd\") on node \"ip-10-0-129-122.ec2.internal\" DevicePath \"\""
Sep 10 17:49:37.621333 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:37.621298    2579 generic.go:356] "Generic (PLEG): container finished" podID="a40c0ae9-962b-42da-b407-336d90b2987a" containerID="74bbac718ac63f2692351671fd8a5bea39e045389452256242eb4d2598fe1730" exitCode=0
Sep 10 17:49:37.621462 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:37.621342    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-624b5c35-7dddf7598-vk9vm"
Sep 10 17:49:37.621462 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:37.621344    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-624b5c35-7dddf7598-vk9vm" event={"ID":"a40c0ae9-962b-42da-b407-336d90b2987a","Type":"ContainerDied","Data":"74bbac718ac63f2692351671fd8a5bea39e045389452256242eb4d2598fe1730"}
Sep 10 17:49:37.621462 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:37.621404    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-624b5c35-7dddf7598-vk9vm" event={"ID":"a40c0ae9-962b-42da-b407-336d90b2987a","Type":"ContainerDied","Data":"85912e95ea0d924e3b1db207a63e70edf5bc4190d38c6f28dd1ecb3226e611d4"}
Sep 10 17:49:37.621462 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:37.621430    2579 scope.go:117] "RemoveContainer" containerID="74bbac718ac63f2692351671fd8a5bea39e045389452256242eb4d2598fe1730"
Sep 10 17:49:37.623043 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:37.623008    2579 generic.go:356] "Generic (PLEG): container finished" podID="a25eadc9-13ff-4713-8f1a-ba4aab03721e" containerID="d06e63194c5517e6e0926530cea7f522bc882442cce14a2bec0d827996d3d23b" exitCode=0
Sep 10 17:49:37.623326 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:37.623061    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-624b5c35-5657b6d4cd-6fcv6" event={"ID":"a25eadc9-13ff-4713-8f1a-ba4aab03721e","Type":"ContainerDied","Data":"d06e63194c5517e6e0926530cea7f522bc882442cce14a2bec0d827996d3d23b"}
Sep 10 17:49:37.623424 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:37.623400    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-624b5c35-5657b6d4cd-6fcv6"
Sep 10 17:49:37.623477 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:37.623442    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-624b5c35-5657b6d4cd-6fcv6" event={"ID":"a25eadc9-13ff-4713-8f1a-ba4aab03721e","Type":"ContainerDied","Data":"c2313813e72c2a342a68bcb8ad5043e295acf31f380d9e0cae9c223975b0a489"}
Sep 10 17:49:37.634465 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:37.634444    2579 scope.go:117] "RemoveContainer" containerID="74bbac718ac63f2692351671fd8a5bea39e045389452256242eb4d2598fe1730"
Sep 10 17:49:37.634781 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:49:37.634762    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74bbac718ac63f2692351671fd8a5bea39e045389452256242eb4d2598fe1730\": container with ID starting with 74bbac718ac63f2692351671fd8a5bea39e045389452256242eb4d2598fe1730 not found: ID does not exist" containerID="74bbac718ac63f2692351671fd8a5bea39e045389452256242eb4d2598fe1730"
Sep 10 17:49:37.634880 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:37.634792    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74bbac718ac63f2692351671fd8a5bea39e045389452256242eb4d2598fe1730"} err="failed to get container status \"74bbac718ac63f2692351671fd8a5bea39e045389452256242eb4d2598fe1730\": rpc error: code = NotFound desc = could not find container \"74bbac718ac63f2692351671fd8a5bea39e045389452256242eb4d2598fe1730\": container with ID starting with 74bbac718ac63f2692351671fd8a5bea39e045389452256242eb4d2598fe1730 not found: ID does not exist"
Sep 10 17:49:37.634880 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:37.634815    2579 scope.go:117] "RemoveContainer" containerID="d06e63194c5517e6e0926530cea7f522bc882442cce14a2bec0d827996d3d23b"
Sep 10 17:49:37.642606 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:37.642582    2579 scope.go:117] "RemoveContainer" containerID="d06e63194c5517e6e0926530cea7f522bc882442cce14a2bec0d827996d3d23b"
Sep 10 17:49:37.642863 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:49:37.642844    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d06e63194c5517e6e0926530cea7f522bc882442cce14a2bec0d827996d3d23b\": container with ID starting with d06e63194c5517e6e0926530cea7f522bc882442cce14a2bec0d827996d3d23b not found: ID does not exist" containerID="d06e63194c5517e6e0926530cea7f522bc882442cce14a2bec0d827996d3d23b"
Sep 10 17:49:37.642928 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:37.642868    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d06e63194c5517e6e0926530cea7f522bc882442cce14a2bec0d827996d3d23b"} err="failed to get container status \"d06e63194c5517e6e0926530cea7f522bc882442cce14a2bec0d827996d3d23b\": rpc error: code = NotFound desc = could not find container \"d06e63194c5517e6e0926530cea7f522bc882442cce14a2bec0d827996d3d23b\": container with ID starting with d06e63194c5517e6e0926530cea7f522bc882442cce14a2bec0d827996d3d23b not found: ID does not exist"
Sep 10 17:49:37.650130 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:37.650109    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-624b5c35-7dddf7598-vk9vm"]
Sep 10 17:49:37.653065 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:37.653046    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-624b5c35-7dddf7598-vk9vm"]
Sep 10 17:49:37.663898 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:37.663875    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-624b5c35-5657b6d4cd-6fcv6"]
Sep 10 17:49:37.667237 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:37.667219    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-624b5c35-5657b6d4cd-6fcv6"]
Sep 10 17:49:38.148770 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:38.148738    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-0ae95d09-openshift-default-97ff7856b-6pj6l"]
Sep 10 17:49:38.149360 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:38.149343    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a40c0ae9-962b-42da-b407-336d90b2987a" containerName="payload-pre-processing"
Sep 10 17:49:38.149403 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:38.149363    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="a40c0ae9-962b-42da-b407-336d90b2987a" containerName="payload-pre-processing"
Sep 10 17:49:38.149403 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:38.149388    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7774ff67-4869-4805-95f2-b6aad0e5303a" containerName="istio-proxy"
Sep 10 17:49:38.149403 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:38.149396    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="7774ff67-4869-4805-95f2-b6aad0e5303a" containerName="istio-proxy"
Sep 10 17:49:38.149487 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:38.149409    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7774ff67-4869-4805-95f2-b6aad0e5303a" containerName="istio-proxy"
Sep 10 17:49:38.149487 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:38.149417    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="7774ff67-4869-4805-95f2-b6aad0e5303a" containerName="istio-proxy"
Sep 10 17:49:38.149487 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:38.149425    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a25eadc9-13ff-4713-8f1a-ba4aab03721e" containerName="payload-processing"
Sep 10 17:49:38.149487 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:38.149433    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="a25eadc9-13ff-4713-8f1a-ba4aab03721e" containerName="payload-processing"
Sep 10 17:49:38.149635 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:38.149516    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="a25eadc9-13ff-4713-8f1a-ba4aab03721e" containerName="payload-processing"
Sep 10 17:49:38.149635 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:38.149528    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="7774ff67-4869-4805-95f2-b6aad0e5303a" containerName="istio-proxy"
Sep 10 17:49:38.149635 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:38.149538    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="7774ff67-4869-4805-95f2-b6aad0e5303a" containerName="istio-proxy"
Sep 10 17:49:38.149635 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:38.149548    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="a40c0ae9-962b-42da-b407-336d90b2987a" containerName="payload-pre-processing"
Sep 10 17:49:38.154140 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:38.154122    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-0ae95d09-openshift-default-97ff7856b-6pj6l"
Sep 10 17:49:38.156836 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:38.156816    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-0ae95d09-openshift-default-dockercfg-t6cs4\""
Sep 10 17:49:38.167691 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:38.167666    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-0ae95d09-openshift-default-97ff7856b-6pj6l"]
Sep 10 17:49:38.261772 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:38.261744    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea7e1957-8fba-4a60-95ce-3f241a43f2df-istio-token\") pod \"e2e-ait-0ae95d09-openshift-default-97ff7856b-6pj6l\" (UID: \"ea7e1957-8fba-4a60-95ce-3f241a43f2df\") " pod="openshift-ingress/e2e-ait-0ae95d09-openshift-default-97ff7856b-6pj6l"
Sep 10 17:49:38.261940 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:38.261782    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea7e1957-8fba-4a60-95ce-3f241a43f2df-istio-envoy\") pod \"e2e-ait-0ae95d09-openshift-default-97ff7856b-6pj6l\" (UID: \"ea7e1957-8fba-4a60-95ce-3f241a43f2df\") " pod="openshift-ingress/e2e-ait-0ae95d09-openshift-default-97ff7856b-6pj6l"
Sep 10 17:49:38.261940 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:38.261800    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea7e1957-8fba-4a60-95ce-3f241a43f2df-istio-podinfo\") pod \"e2e-ait-0ae95d09-openshift-default-97ff7856b-6pj6l\" (UID: \"ea7e1957-8fba-4a60-95ce-3f241a43f2df\") " pod="openshift-ingress/e2e-ait-0ae95d09-openshift-default-97ff7856b-6pj6l"
Sep 10 17:49:38.261940 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:38.261827    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea7e1957-8fba-4a60-95ce-3f241a43f2df-credential-socket\") pod \"e2e-ait-0ae95d09-openshift-default-97ff7856b-6pj6l\" (UID: \"ea7e1957-8fba-4a60-95ce-3f241a43f2df\") " pod="openshift-ingress/e2e-ait-0ae95d09-openshift-default-97ff7856b-6pj6l"
Sep 10 17:49:38.262072 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:38.261947    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea7e1957-8fba-4a60-95ce-3f241a43f2df-istiod-ca-cert\") pod \"e2e-ait-0ae95d09-openshift-default-97ff7856b-6pj6l\" (UID: \"ea7e1957-8fba-4a60-95ce-3f241a43f2df\") " pod="openshift-ingress/e2e-ait-0ae95d09-openshift-default-97ff7856b-6pj6l"
Sep 10 17:49:38.262072 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:38.261992    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea7e1957-8fba-4a60-95ce-3f241a43f2df-workload-socket\") pod \"e2e-ait-0ae95d09-openshift-default-97ff7856b-6pj6l\" (UID: \"ea7e1957-8fba-4a60-95ce-3f241a43f2df\") " pod="openshift-ingress/e2e-ait-0ae95d09-openshift-default-97ff7856b-6pj6l"
Sep 10 17:49:38.262072 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:38.262018    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea7e1957-8fba-4a60-95ce-3f241a43f2df-istio-data\") pod \"e2e-ait-0ae95d09-openshift-default-97ff7856b-6pj6l\" (UID: \"ea7e1957-8fba-4a60-95ce-3f241a43f2df\") " pod="openshift-ingress/e2e-ait-0ae95d09-openshift-default-97ff7856b-6pj6l"
Sep 10 17:49:38.262072 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:38.262033    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8tfst\" (UniqueName: \"kubernetes.io/projected/ea7e1957-8fba-4a60-95ce-3f241a43f2df-kube-api-access-8tfst\") pod \"e2e-ait-0ae95d09-openshift-default-97ff7856b-6pj6l\" (UID: \"ea7e1957-8fba-4a60-95ce-3f241a43f2df\") " pod="openshift-ingress/e2e-ait-0ae95d09-openshift-default-97ff7856b-6pj6l"
Sep 10 17:49:38.262072 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:38.262057    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea7e1957-8fba-4a60-95ce-3f241a43f2df-workload-certs\") pod \"e2e-ait-0ae95d09-openshift-default-97ff7856b-6pj6l\" (UID: \"ea7e1957-8fba-4a60-95ce-3f241a43f2df\") " pod="openshift-ingress/e2e-ait-0ae95d09-openshift-default-97ff7856b-6pj6l"
Sep 10 17:49:38.363064 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:38.362989    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea7e1957-8fba-4a60-95ce-3f241a43f2df-istio-token\") pod \"e2e-ait-0ae95d09-openshift-default-97ff7856b-6pj6l\" (UID: \"ea7e1957-8fba-4a60-95ce-3f241a43f2df\") " pod="openshift-ingress/e2e-ait-0ae95d09-openshift-default-97ff7856b-6pj6l"
Sep 10 17:49:38.363064 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:38.363026    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea7e1957-8fba-4a60-95ce-3f241a43f2df-istio-envoy\") pod \"e2e-ait-0ae95d09-openshift-default-97ff7856b-6pj6l\" (UID: \"ea7e1957-8fba-4a60-95ce-3f241a43f2df\") " pod="openshift-ingress/e2e-ait-0ae95d09-openshift-default-97ff7856b-6pj6l"
Sep 10 17:49:38.363064 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:38.363044    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea7e1957-8fba-4a60-95ce-3f241a43f2df-istio-podinfo\") pod \"e2e-ait-0ae95d09-openshift-default-97ff7856b-6pj6l\" (UID: \"ea7e1957-8fba-4a60-95ce-3f241a43f2df\") " pod="openshift-ingress/e2e-ait-0ae95d09-openshift-default-97ff7856b-6pj6l"
Sep 10 17:49:38.363340 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:38.363070    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea7e1957-8fba-4a60-95ce-3f241a43f2df-credential-socket\") pod \"e2e-ait-0ae95d09-openshift-default-97ff7856b-6pj6l\" (UID: \"ea7e1957-8fba-4a60-95ce-3f241a43f2df\") " pod="openshift-ingress/e2e-ait-0ae95d09-openshift-default-97ff7856b-6pj6l"
Sep 10 17:49:38.363340 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:38.363102    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea7e1957-8fba-4a60-95ce-3f241a43f2df-istiod-ca-cert\") pod \"e2e-ait-0ae95d09-openshift-default-97ff7856b-6pj6l\" (UID: \"ea7e1957-8fba-4a60-95ce-3f241a43f2df\") " pod="openshift-ingress/e2e-ait-0ae95d09-openshift-default-97ff7856b-6pj6l"
Sep 10 17:49:38.363340 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:38.363120    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea7e1957-8fba-4a60-95ce-3f241a43f2df-workload-socket\") pod \"e2e-ait-0ae95d09-openshift-default-97ff7856b-6pj6l\" (UID: \"ea7e1957-8fba-4a60-95ce-3f241a43f2df\") " pod="openshift-ingress/e2e-ait-0ae95d09-openshift-default-97ff7856b-6pj6l"
Sep 10 17:49:38.363340 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:38.363154    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea7e1957-8fba-4a60-95ce-3f241a43f2df-istio-data\") pod \"e2e-ait-0ae95d09-openshift-default-97ff7856b-6pj6l\" (UID: \"ea7e1957-8fba-4a60-95ce-3f241a43f2df\") " pod="openshift-ingress/e2e-ait-0ae95d09-openshift-default-97ff7856b-6pj6l"
Sep 10 17:49:38.363340 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:38.363170    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8tfst\" (UniqueName: \"kubernetes.io/projected/ea7e1957-8fba-4a60-95ce-3f241a43f2df-kube-api-access-8tfst\") pod \"e2e-ait-0ae95d09-openshift-default-97ff7856b-6pj6l\" (UID: \"ea7e1957-8fba-4a60-95ce-3f241a43f2df\") " pod="openshift-ingress/e2e-ait-0ae95d09-openshift-default-97ff7856b-6pj6l"
Sep 10 17:49:38.363340 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:38.363189    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea7e1957-8fba-4a60-95ce-3f241a43f2df-workload-certs\") pod \"e2e-ait-0ae95d09-openshift-default-97ff7856b-6pj6l\" (UID: \"ea7e1957-8fba-4a60-95ce-3f241a43f2df\") " pod="openshift-ingress/e2e-ait-0ae95d09-openshift-default-97ff7856b-6pj6l"
Sep 10 17:49:38.363673 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:38.363531    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea7e1957-8fba-4a60-95ce-3f241a43f2df-credential-socket\") pod \"e2e-ait-0ae95d09-openshift-default-97ff7856b-6pj6l\" (UID: \"ea7e1957-8fba-4a60-95ce-3f241a43f2df\") " pod="openshift-ingress/e2e-ait-0ae95d09-openshift-default-97ff7856b-6pj6l"
Sep 10 17:49:38.363897 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:38.363691    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea7e1957-8fba-4a60-95ce-3f241a43f2df-workload-certs\") pod \"e2e-ait-0ae95d09-openshift-default-97ff7856b-6pj6l\" (UID: \"ea7e1957-8fba-4a60-95ce-3f241a43f2df\") " pod="openshift-ingress/e2e-ait-0ae95d09-openshift-default-97ff7856b-6pj6l"
Sep 10 17:49:38.363897 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:38.363722    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea7e1957-8fba-4a60-95ce-3f241a43f2df-istio-data\") pod \"e2e-ait-0ae95d09-openshift-default-97ff7856b-6pj6l\" (UID: \"ea7e1957-8fba-4a60-95ce-3f241a43f2df\") " pod="openshift-ingress/e2e-ait-0ae95d09-openshift-default-97ff7856b-6pj6l"
Sep 10 17:49:38.363897 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:38.363759    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea7e1957-8fba-4a60-95ce-3f241a43f2df-workload-socket\") pod \"e2e-ait-0ae95d09-openshift-default-97ff7856b-6pj6l\" (UID: \"ea7e1957-8fba-4a60-95ce-3f241a43f2df\") " pod="openshift-ingress/e2e-ait-0ae95d09-openshift-default-97ff7856b-6pj6l"
Sep 10 17:49:38.363897 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:38.363894    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea7e1957-8fba-4a60-95ce-3f241a43f2df-istiod-ca-cert\") pod \"e2e-ait-0ae95d09-openshift-default-97ff7856b-6pj6l\" (UID: \"ea7e1957-8fba-4a60-95ce-3f241a43f2df\") " pod="openshift-ingress/e2e-ait-0ae95d09-openshift-default-97ff7856b-6pj6l"
Sep 10 17:49:38.365379 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:38.365358    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea7e1957-8fba-4a60-95ce-3f241a43f2df-istio-envoy\") pod \"e2e-ait-0ae95d09-openshift-default-97ff7856b-6pj6l\" (UID: \"ea7e1957-8fba-4a60-95ce-3f241a43f2df\") " pod="openshift-ingress/e2e-ait-0ae95d09-openshift-default-97ff7856b-6pj6l"
Sep 10 17:49:38.365702 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:38.365685    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea7e1957-8fba-4a60-95ce-3f241a43f2df-istio-podinfo\") pod \"e2e-ait-0ae95d09-openshift-default-97ff7856b-6pj6l\" (UID: \"ea7e1957-8fba-4a60-95ce-3f241a43f2df\") " pod="openshift-ingress/e2e-ait-0ae95d09-openshift-default-97ff7856b-6pj6l"
Sep 10 17:49:38.371601 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:38.371559    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea7e1957-8fba-4a60-95ce-3f241a43f2df-istio-token\") pod \"e2e-ait-0ae95d09-openshift-default-97ff7856b-6pj6l\" (UID: \"ea7e1957-8fba-4a60-95ce-3f241a43f2df\") " pod="openshift-ingress/e2e-ait-0ae95d09-openshift-default-97ff7856b-6pj6l"
Sep 10 17:49:38.371837 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:38.371818    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8tfst\" (UniqueName: \"kubernetes.io/projected/ea7e1957-8fba-4a60-95ce-3f241a43f2df-kube-api-access-8tfst\") pod \"e2e-ait-0ae95d09-openshift-default-97ff7856b-6pj6l\" (UID: \"ea7e1957-8fba-4a60-95ce-3f241a43f2df\") " pod="openshift-ingress/e2e-ait-0ae95d09-openshift-default-97ff7856b-6pj6l"
Sep 10 17:49:38.464236 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:38.464210    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-0ae95d09-openshift-default-97ff7856b-6pj6l"
Sep 10 17:49:38.476378 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:38.476349    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a25eadc9-13ff-4713-8f1a-ba4aab03721e" path="/var/lib/kubelet/pods/a25eadc9-13ff-4713-8f1a-ba4aab03721e/volumes"
Sep 10 17:49:38.476852 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:38.476832    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a40c0ae9-962b-42da-b407-336d90b2987a" path="/var/lib/kubelet/pods/a40c0ae9-962b-42da-b407-336d90b2987a/volumes"
Sep 10 17:49:38.742240 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:38.742217    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-0ae95d09-openshift-default-97ff7856b-6pj6l"]
Sep 10 17:49:38.744405 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:49:38.744376    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea7e1957_8fba_4a60_95ce_3f241a43f2df.slice/crio-d6e11bdd604bfe3e995cd0a3ebc09cd7ab9449774dd00069aa37cdcdb6d696c2 WatchSource:0}: Error finding container d6e11bdd604bfe3e995cd0a3ebc09cd7ab9449774dd00069aa37cdcdb6d696c2: Status 404 returned error can't find the container with id d6e11bdd604bfe3e995cd0a3ebc09cd7ab9449774dd00069aa37cdcdb6d696c2
Sep 10 17:49:38.746501 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:38.746462    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 10 17:49:38.746609 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:38.746522    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 10 17:49:38.746609 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:38.746548    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 10 17:49:39.307093 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:39.307052    2579 patch_prober.go:28] interesting pod/e2e-migrate-624b5c35-gw-openshift-default-ccdbc8bfb-s8wkj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused
Sep 10 17:49:39.307619 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:39.307155    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-624b5c35-gw-openshift-default-ccdbc8bfb-s8wkj" podUID="93e67e8a-cf2c-483a-8c4a-3ca7db13856f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 10 17:49:39.636381 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:39.636346    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0ae95d09-openshift-default-97ff7856b-6pj6l" event={"ID":"ea7e1957-8fba-4a60-95ce-3f241a43f2df","Type":"ContainerStarted","Data":"40d777077449038396f83b11cabd34756f7c474eb2005f77abe5ffa3437a57c9"}
Sep 10 17:49:39.636550 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:39.636410    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0ae95d09-openshift-default-97ff7856b-6pj6l" event={"ID":"ea7e1957-8fba-4a60-95ce-3f241a43f2df","Type":"ContainerStarted","Data":"d6e11bdd604bfe3e995cd0a3ebc09cd7ab9449774dd00069aa37cdcdb6d696c2"}
Sep 10 17:49:39.664499 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:39.664449    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-0ae95d09-openshift-default-97ff7856b-6pj6l" podStartSLOduration=1.664435942 podStartE2EDuration="1.664435942s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:49:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:49:39.663400967 +0000 UTC m=+823.663231037" watchObservedRunningTime="2026-09-10 17:49:39.664435942 +0000 UTC m=+823.664266007"
Sep 10 17:49:40.464643 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:40.464604    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-0ae95d09-openshift-default-97ff7856b-6pj6l"
Sep 10 17:49:40.476182 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:40.476158    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-0ae95d09-openshift-default-97ff7856b-6pj6l"
Sep 10 17:49:40.641729 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:40.641702    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-0ae95d09-openshift-default-97ff7856b-6pj6l"
Sep 10 17:49:40.642759 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:40.642728    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-0ae95d09-openshift-default-97ff7856b-6pj6l"
Sep 10 17:49:42.214344 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:42.214312    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0ae95d09-7d89ffd848-2522f"]
Sep 10 17:49:42.217963 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:42.217939    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0ae95d09-7d89ffd848-2522f"
Sep 10 17:49:42.221537 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:42.221508    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-0ae95d09-serving-cert\""
Sep 10 17:49:42.248173 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:42.248146    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0ae95d09-7d89ffd848-2522f"]
Sep 10 17:49:42.299436 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:42.299403    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/092af79b-03fe-43c8-857b-4922eea42355-openshift-service-ca\") pod \"maas-api-e2e-ait-0ae95d09-7d89ffd848-2522f\" (UID: \"092af79b-03fe-43c8-857b-4922eea42355\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0ae95d09-7d89ffd848-2522f"
Sep 10 17:49:42.299618 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:42.299487    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6klf\" (UniqueName: \"kubernetes.io/projected/092af79b-03fe-43c8-857b-4922eea42355-kube-api-access-r6klf\") pod \"maas-api-e2e-ait-0ae95d09-7d89ffd848-2522f\" (UID: \"092af79b-03fe-43c8-857b-4922eea42355\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0ae95d09-7d89ffd848-2522f"
Sep 10 17:49:42.299618 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:42.299544    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/092af79b-03fe-43c8-857b-4922eea42355-maas-api-tls\") pod \"maas-api-e2e-ait-0ae95d09-7d89ffd848-2522f\" (UID: \"092af79b-03fe-43c8-857b-4922eea42355\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0ae95d09-7d89ffd848-2522f"
Sep 10 17:49:42.299618 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:42.299559    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/092af79b-03fe-43c8-857b-4922eea42355-metrics-certs\") pod \"maas-api-e2e-ait-0ae95d09-7d89ffd848-2522f\" (UID: \"092af79b-03fe-43c8-857b-4922eea42355\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0ae95d09-7d89ffd848-2522f"
Sep 10 17:49:42.400986 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:42.400898    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/092af79b-03fe-43c8-857b-4922eea42355-openshift-service-ca\") pod \"maas-api-e2e-ait-0ae95d09-7d89ffd848-2522f\" (UID: \"092af79b-03fe-43c8-857b-4922eea42355\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0ae95d09-7d89ffd848-2522f"
Sep 10 17:49:42.401071 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:42.401016    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6klf\" (UniqueName: \"kubernetes.io/projected/092af79b-03fe-43c8-857b-4922eea42355-kube-api-access-r6klf\") pod \"maas-api-e2e-ait-0ae95d09-7d89ffd848-2522f\" (UID: \"092af79b-03fe-43c8-857b-4922eea42355\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0ae95d09-7d89ffd848-2522f"
Sep 10 17:49:42.401130 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:42.401107    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/092af79b-03fe-43c8-857b-4922eea42355-maas-api-tls\") pod \"maas-api-e2e-ait-0ae95d09-7d89ffd848-2522f\" (UID: \"092af79b-03fe-43c8-857b-4922eea42355\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0ae95d09-7d89ffd848-2522f"
Sep 10 17:49:42.401184 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:42.401133    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/092af79b-03fe-43c8-857b-4922eea42355-metrics-certs\") pod \"maas-api-e2e-ait-0ae95d09-7d89ffd848-2522f\" (UID: \"092af79b-03fe-43c8-857b-4922eea42355\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0ae95d09-7d89ffd848-2522f"
Sep 10 17:49:42.401746 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:42.401723    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/092af79b-03fe-43c8-857b-4922eea42355-openshift-service-ca\") pod \"maas-api-e2e-ait-0ae95d09-7d89ffd848-2522f\" (UID: \"092af79b-03fe-43c8-857b-4922eea42355\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0ae95d09-7d89ffd848-2522f"
Sep 10 17:49:42.403759 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:42.403739    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/092af79b-03fe-43c8-857b-4922eea42355-maas-api-tls\") pod \"maas-api-e2e-ait-0ae95d09-7d89ffd848-2522f\" (UID: \"092af79b-03fe-43c8-857b-4922eea42355\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0ae95d09-7d89ffd848-2522f"
Sep 10 17:49:42.403901 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:42.403884    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/092af79b-03fe-43c8-857b-4922eea42355-metrics-certs\") pod \"maas-api-e2e-ait-0ae95d09-7d89ffd848-2522f\" (UID: \"092af79b-03fe-43c8-857b-4922eea42355\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0ae95d09-7d89ffd848-2522f"
Sep 10 17:49:42.418923 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:42.418900    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6klf\" (UniqueName: \"kubernetes.io/projected/092af79b-03fe-43c8-857b-4922eea42355-kube-api-access-r6klf\") pod \"maas-api-e2e-ait-0ae95d09-7d89ffd848-2522f\" (UID: \"092af79b-03fe-43c8-857b-4922eea42355\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0ae95d09-7d89ffd848-2522f"
Sep 10 17:49:42.529954 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:42.529923    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0ae95d09-7d89ffd848-2522f"
Sep 10 17:49:42.653373 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:42.653347    2579 generic.go:356] "Generic (PLEG): container finished" podID="93e67e8a-cf2c-483a-8c4a-3ca7db13856f" containerID="d0230d30b92790296334cf4a1effcdd9fd49d2bd3d9b933068b15302ccfdd94c" exitCode=0
Sep 10 17:49:42.653488 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:42.653413    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-624b5c35-gw-openshift-default-ccdbc8bfb-s8wkj" event={"ID":"93e67e8a-cf2c-483a-8c4a-3ca7db13856f","Type":"ContainerDied","Data":"d0230d30b92790296334cf4a1effcdd9fd49d2bd3d9b933068b15302ccfdd94c"}
Sep 10 17:49:42.653488 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:42.653461    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-624b5c35-gw-openshift-default-ccdbc8bfb-s8wkj" event={"ID":"93e67e8a-cf2c-483a-8c4a-3ca7db13856f","Type":"ContainerDied","Data":"80dce0acd99276c461800828e7400fd6535476335fffc728f1bcf09d81aff167"}
Sep 10 17:49:42.653488 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:42.653477    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="80dce0acd99276c461800828e7400fd6535476335fffc728f1bcf09d81aff167"
Sep 10 17:49:42.653611 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:42.653490    2579 scope.go:117] "RemoveContainer" containerID="2b7d2e8dfc8388d05ab76f2e01a7a6c0b447e7b3df7cbb7befc59e7cd47ca125"
Sep 10 17:49:42.655602 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:42.655583    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-624b5c35-gw-openshift-default-ccdbc8bfb-s8wkj"
Sep 10 17:49:42.804427 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:42.804353    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93e67e8a-cf2c-483a-8c4a-3ca7db13856f-istio-data\") pod \"93e67e8a-cf2c-483a-8c4a-3ca7db13856f\" (UID: \"93e67e8a-cf2c-483a-8c4a-3ca7db13856f\") "
Sep 10 17:49:42.804427 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:42.804405    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93e67e8a-cf2c-483a-8c4a-3ca7db13856f-istio-podinfo\") pod \"93e67e8a-cf2c-483a-8c4a-3ca7db13856f\" (UID: \"93e67e8a-cf2c-483a-8c4a-3ca7db13856f\") "
Sep 10 17:49:42.804427 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:42.804431    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93e67e8a-cf2c-483a-8c4a-3ca7db13856f-workload-socket\") pod \"93e67e8a-cf2c-483a-8c4a-3ca7db13856f\" (UID: \"93e67e8a-cf2c-483a-8c4a-3ca7db13856f\") "
Sep 10 17:49:42.804729 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:42.804461    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93e67e8a-cf2c-483a-8c4a-3ca7db13856f-credential-socket\") pod \"93e67e8a-cf2c-483a-8c4a-3ca7db13856f\" (UID: \"93e67e8a-cf2c-483a-8c4a-3ca7db13856f\") "
Sep 10 17:49:42.804729 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:42.804493    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-94mmr\" (UniqueName: \"kubernetes.io/projected/93e67e8a-cf2c-483a-8c4a-3ca7db13856f-kube-api-access-94mmr\") pod \"93e67e8a-cf2c-483a-8c4a-3ca7db13856f\" (UID: \"93e67e8a-cf2c-483a-8c4a-3ca7db13856f\") "
Sep 10 17:49:42.804729 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:42.804627    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93e67e8a-cf2c-483a-8c4a-3ca7db13856f-istiod-ca-cert\") pod \"93e67e8a-cf2c-483a-8c4a-3ca7db13856f\" (UID: \"93e67e8a-cf2c-483a-8c4a-3ca7db13856f\") "
Sep 10 17:49:42.804729 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:42.804635    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93e67e8a-cf2c-483a-8c4a-3ca7db13856f-istio-data" (OuterVolumeSpecName: "istio-data") pod "93e67e8a-cf2c-483a-8c4a-3ca7db13856f" (UID: "93e67e8a-cf2c-483a-8c4a-3ca7db13856f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:49:42.805285 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:42.804658    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93e67e8a-cf2c-483a-8c4a-3ca7db13856f-istio-token\") pod \"93e67e8a-cf2c-483a-8c4a-3ca7db13856f\" (UID: \"93e67e8a-cf2c-483a-8c4a-3ca7db13856f\") "
Sep 10 17:49:42.805285 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:42.805214    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93e67e8a-cf2c-483a-8c4a-3ca7db13856f-workload-certs\") pod \"93e67e8a-cf2c-483a-8c4a-3ca7db13856f\" (UID: \"93e67e8a-cf2c-483a-8c4a-3ca7db13856f\") "
Sep 10 17:49:42.805285 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:42.805247    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93e67e8a-cf2c-483a-8c4a-3ca7db13856f-istio-envoy\") pod \"93e67e8a-cf2c-483a-8c4a-3ca7db13856f\" (UID: \"93e67e8a-cf2c-483a-8c4a-3ca7db13856f\") "
Sep 10 17:49:42.806008 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:42.805645    2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93e67e8a-cf2c-483a-8c4a-3ca7db13856f-istio-data\") on node \"ip-10-0-129-122.ec2.internal\" DevicePath \"\""
Sep 10 17:49:42.806008 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:42.804728    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93e67e8a-cf2c-483a-8c4a-3ca7db13856f-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "93e67e8a-cf2c-483a-8c4a-3ca7db13856f" (UID: "93e67e8a-cf2c-483a-8c4a-3ca7db13856f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:49:42.806008 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:42.804953    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93e67e8a-cf2c-483a-8c4a-3ca7db13856f-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "93e67e8a-cf2c-483a-8c4a-3ca7db13856f" (UID: "93e67e8a-cf2c-483a-8c4a-3ca7db13856f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 17:49:42.806008 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:42.805122    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93e67e8a-cf2c-483a-8c4a-3ca7db13856f-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "93e67e8a-cf2c-483a-8c4a-3ca7db13856f" (UID: "93e67e8a-cf2c-483a-8c4a-3ca7db13856f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:49:42.806793 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:42.806768    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93e67e8a-cf2c-483a-8c4a-3ca7db13856f-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "93e67e8a-cf2c-483a-8c4a-3ca7db13856f" (UID: "93e67e8a-cf2c-483a-8c4a-3ca7db13856f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:49:42.808700 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:42.808663    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93e67e8a-cf2c-483a-8c4a-3ca7db13856f-istio-token" (OuterVolumeSpecName: "istio-token") pod "93e67e8a-cf2c-483a-8c4a-3ca7db13856f" (UID: "93e67e8a-cf2c-483a-8c4a-3ca7db13856f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:49:42.809046 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:42.809022    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/93e67e8a-cf2c-483a-8c4a-3ca7db13856f-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "93e67e8a-cf2c-483a-8c4a-3ca7db13856f" (UID: "93e67e8a-cf2c-483a-8c4a-3ca7db13856f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 10 17:49:42.809175 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:42.809143    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93e67e8a-cf2c-483a-8c4a-3ca7db13856f-kube-api-access-94mmr" (OuterVolumeSpecName: "kube-api-access-94mmr") pod "93e67e8a-cf2c-483a-8c4a-3ca7db13856f" (UID: "93e67e8a-cf2c-483a-8c4a-3ca7db13856f"). InnerVolumeSpecName "kube-api-access-94mmr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:49:42.809668 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:42.809634    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93e67e8a-cf2c-483a-8c4a-3ca7db13856f-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "93e67e8a-cf2c-483a-8c4a-3ca7db13856f" (UID: "93e67e8a-cf2c-483a-8c4a-3ca7db13856f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:49:42.820161 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:42.820122    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0ae95d09-7d89ffd848-2522f"]
Sep 10 17:49:42.822086 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:49:42.822061    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod092af79b_03fe_43c8_857b_4922eea42355.slice/crio-ff253a4007fcc3ff0ae85d9afd4a7794fa77fe641bc5f109afffd53c75de3156 WatchSource:0}: Error finding container ff253a4007fcc3ff0ae85d9afd4a7794fa77fe641bc5f109afffd53c75de3156: Status 404 returned error can't find the container with id ff253a4007fcc3ff0ae85d9afd4a7794fa77fe641bc5f109afffd53c75de3156
Sep 10 17:49:42.906779 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:42.906751    2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93e67e8a-cf2c-483a-8c4a-3ca7db13856f-istiod-ca-cert\") on node \"ip-10-0-129-122.ec2.internal\" DevicePath \"\""
Sep 10 17:49:42.906779 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:42.906775    2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93e67e8a-cf2c-483a-8c4a-3ca7db13856f-istio-token\") on node \"ip-10-0-129-122.ec2.internal\" DevicePath \"\""
Sep 10 17:49:42.906779 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:42.906783    2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93e67e8a-cf2c-483a-8c4a-3ca7db13856f-workload-certs\") on node \"ip-10-0-129-122.ec2.internal\" DevicePath \"\""
Sep 10 17:49:42.906974 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:42.906790    2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93e67e8a-cf2c-483a-8c4a-3ca7db13856f-istio-envoy\") on node \"ip-10-0-129-122.ec2.internal\" DevicePath \"\""
Sep 10 17:49:42.906974 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:42.906797    2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93e67e8a-cf2c-483a-8c4a-3ca7db13856f-istio-podinfo\") on node \"ip-10-0-129-122.ec2.internal\" DevicePath \"\""
Sep 10 17:49:42.906974 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:42.906805    2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93e67e8a-cf2c-483a-8c4a-3ca7db13856f-workload-socket\") on node \"ip-10-0-129-122.ec2.internal\" DevicePath \"\""
Sep 10 17:49:42.906974 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:42.906819    2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93e67e8a-cf2c-483a-8c4a-3ca7db13856f-credential-socket\") on node \"ip-10-0-129-122.ec2.internal\" DevicePath \"\""
Sep 10 17:49:42.906974 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:42.906829    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-94mmr\" (UniqueName: \"kubernetes.io/projected/93e67e8a-cf2c-483a-8c4a-3ca7db13856f-kube-api-access-94mmr\") on node \"ip-10-0-129-122.ec2.internal\" DevicePath \"\""
Sep 10 17:49:42.968264 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:42.968230    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0ae95d09-75df89545d-kqk49"]
Sep 10 17:49:42.968648 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:42.968634    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="93e67e8a-cf2c-483a-8c4a-3ca7db13856f" containerName="istio-proxy"
Sep 10 17:49:42.968705 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:42.968650    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="93e67e8a-cf2c-483a-8c4a-3ca7db13856f" containerName="istio-proxy"
Sep 10 17:49:42.968705 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:42.968659    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="93e67e8a-cf2c-483a-8c4a-3ca7db13856f" containerName="istio-proxy"
Sep 10 17:49:42.968705 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:42.968664    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="93e67e8a-cf2c-483a-8c4a-3ca7db13856f" containerName="istio-proxy"
Sep 10 17:49:42.968809 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:42.968732    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="93e67e8a-cf2c-483a-8c4a-3ca7db13856f" containerName="istio-proxy"
Sep 10 17:49:42.968809 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:42.968739    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="93e67e8a-cf2c-483a-8c4a-3ca7db13856f" containerName="istio-proxy"
Sep 10 17:49:42.973597 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:42.973579    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-0ae95d09-75df89545d-kqk49"
Sep 10 17:49:42.976577 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:42.976546    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-0ae95d09\""
Sep 10 17:49:42.984392 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:42.984366    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0ae95d09-75df89545d-kqk49"]
Sep 10 17:49:43.107994 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:43.107918    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3c4208e-f973-4eec-8274-9ff1b7640b24-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-0ae95d09-75df89545d-kqk49\" (UID: \"f3c4208e-f973-4eec-8274-9ff1b7640b24\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0ae95d09-75df89545d-kqk49"
Sep 10 17:49:43.107994 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:43.107977    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2mhn\" (UniqueName: \"kubernetes.io/projected/f3c4208e-f973-4eec-8274-9ff1b7640b24-kube-api-access-q2mhn\") pod \"payload-pre-processing-e2e-ait-0ae95d09-75df89545d-kqk49\" (UID: \"f3c4208e-f973-4eec-8274-9ff1b7640b24\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0ae95d09-75df89545d-kqk49"
Sep 10 17:49:43.208931 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:43.208894    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3c4208e-f973-4eec-8274-9ff1b7640b24-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-0ae95d09-75df89545d-kqk49\" (UID: \"f3c4208e-f973-4eec-8274-9ff1b7640b24\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0ae95d09-75df89545d-kqk49"
Sep 10 17:49:43.209068 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:43.208941    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2mhn\" (UniqueName: \"kubernetes.io/projected/f3c4208e-f973-4eec-8274-9ff1b7640b24-kube-api-access-q2mhn\") pod \"payload-pre-processing-e2e-ait-0ae95d09-75df89545d-kqk49\" (UID: \"f3c4208e-f973-4eec-8274-9ff1b7640b24\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0ae95d09-75df89545d-kqk49"
Sep 10 17:49:43.209523 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:43.209504    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3c4208e-f973-4eec-8274-9ff1b7640b24-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-0ae95d09-75df89545d-kqk49\" (UID: \"f3c4208e-f973-4eec-8274-9ff1b7640b24\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0ae95d09-75df89545d-kqk49"
Sep 10 17:49:43.219286 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:43.219263    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2mhn\" (UniqueName: \"kubernetes.io/projected/f3c4208e-f973-4eec-8274-9ff1b7640b24-kube-api-access-q2mhn\") pod \"payload-pre-processing-e2e-ait-0ae95d09-75df89545d-kqk49\" (UID: \"f3c4208e-f973-4eec-8274-9ff1b7640b24\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0ae95d09-75df89545d-kqk49"
Sep 10 17:49:43.286283 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:43.286263    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-0ae95d09-75df89545d-kqk49"
Sep 10 17:49:43.565543 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:43.565517    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0ae95d09-75df89545d-kqk49"]
Sep 10 17:49:43.567087 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:49:43.567060    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3c4208e_f973_4eec_8274_9ff1b7640b24.slice/crio-a30cc81cfe005e7b866a8965e27e3b197c4fd8c3ba0a7299b937eddc9b4166cf WatchSource:0}: Error finding container a30cc81cfe005e7b866a8965e27e3b197c4fd8c3ba0a7299b937eddc9b4166cf: Status 404 returned error can't find the container with id a30cc81cfe005e7b866a8965e27e3b197c4fd8c3ba0a7299b937eddc9b4166cf
Sep 10 17:49:43.661484 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:43.661415    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-0ae95d09-75df89545d-kqk49" event={"ID":"f3c4208e-f973-4eec-8274-9ff1b7640b24","Type":"ContainerStarted","Data":"5aed558ed04cd7a1ceb9c42686ef8fdb4312dd8aa8fd0fbdcab4ff1032761b7f"}
Sep 10 17:49:43.661484 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:43.661473    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-0ae95d09-75df89545d-kqk49" event={"ID":"f3c4208e-f973-4eec-8274-9ff1b7640b24","Type":"ContainerStarted","Data":"a30cc81cfe005e7b866a8965e27e3b197c4fd8c3ba0a7299b937eddc9b4166cf"}
Sep 10 17:49:43.661693 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:43.661504    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-0ae95d09-75df89545d-kqk49"
Sep 10 17:49:43.663117 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:43.663092    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0ae95d09-7d89ffd848-2522f" event={"ID":"092af79b-03fe-43c8-857b-4922eea42355","Type":"ContainerStarted","Data":"71ab4ef05b20c1377cf8f593186da7b8cfbea1032b10e2c86a3c0ed95c163964"}
Sep 10 17:49:43.663243 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:43.663131    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0ae95d09-7d89ffd848-2522f" event={"ID":"092af79b-03fe-43c8-857b-4922eea42355","Type":"ContainerStarted","Data":"ff253a4007fcc3ff0ae85d9afd4a7794fa77fe641bc5f109afffd53c75de3156"}
Sep 10 17:49:43.663243 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:43.663173    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0ae95d09-7d89ffd848-2522f"
Sep 10 17:49:43.664516 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:43.664499    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-624b5c35-gw-openshift-default-ccdbc8bfb-s8wkj"
Sep 10 17:49:43.679969 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:43.679923    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-0ae95d09-75df89545d-kqk49" podStartSLOduration=1.679907625 podStartE2EDuration="1.679907625s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:49:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:49:43.677906998 +0000 UTC m=+827.677737064" watchObservedRunningTime="2026-09-10 17:49:43.679907625 +0000 UTC m=+827.679737690"
Sep 10 17:49:43.698759 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:43.698720    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0ae95d09-7d89ffd848-2522f" podStartSLOduration=1.2724997949999999 podStartE2EDuration="1.698706033s" totalImagesPullingTime="426.206238ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:49:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:49:43.697555536 +0000 UTC m=+827.697385604" watchObservedRunningTime="2026-09-10 17:49:43.698706033 +0000 UTC m=+827.698536098"
Sep 10 17:49:43.736673 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:43.736641    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-624b5c35-gw-openshift-default-ccdbc8bfb-s8wkj"]
Sep 10 17:49:43.745089 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:43.745065    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-624b5c35-gw-openshift-default-ccdbc8bfb-s8wkj"]
Sep 10 17:49:44.475014 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:44.474981    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="93e67e8a-cf2c-483a-8c4a-3ca7db13856f" path="/var/lib/kubelet/pods/93e67e8a-cf2c-483a-8c4a-3ca7db13856f/volumes"
Sep 10 17:49:49.682901 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:49.682861    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0ae95d09-7d89ffd848-2522f"
Sep 10 17:49:54.672780 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:54.672754    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-0ae95d09-75df89545d-kqk49"
Sep 10 17:49:56.682761 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:56.682721    2579 patch_prober.go:28] interesting pod/e2e-ait-0ae95d09-openshift-default-97ff7856b-6pj6l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 10 17:49:56.683111 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:56.682817    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-0ae95d09-openshift-default-97ff7856b-6pj6l" podUID="ea7e1957-8fba-4a60-95ce-3f241a43f2df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 10 17:49:57.749776 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:57.749743    2579 generic.go:356] "Generic (PLEG): container finished" podID="ea7e1957-8fba-4a60-95ce-3f241a43f2df" containerID="40d777077449038396f83b11cabd34756f7c474eb2005f77abe5ffa3437a57c9" exitCode=0
Sep 10 17:49:57.750279 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:57.749825    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0ae95d09-openshift-default-97ff7856b-6pj6l" event={"ID":"ea7e1957-8fba-4a60-95ce-3f241a43f2df","Type":"ContainerDied","Data":"40d777077449038396f83b11cabd34756f7c474eb2005f77abe5ffa3437a57c9"}
Sep 10 17:49:57.750279 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:57.750178    2579 scope.go:117] "RemoveContainer" containerID="40d777077449038396f83b11cabd34756f7c474eb2005f77abe5ffa3437a57c9"
Sep 10 17:49:57.751195 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:57.751168    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 10 17:49:57.751254 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:57.751234    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 10 17:49:57.751289 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:57.751265    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 10 17:49:58.756770 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:58.756735    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0ae95d09-openshift-default-97ff7856b-6pj6l" event={"ID":"ea7e1957-8fba-4a60-95ce-3f241a43f2df","Type":"ContainerStarted","Data":"d97a36313fad7cb3b251c968333a0f460bad6ef94b37760dde82ba369b2cf3b0"}
Sep 10 17:49:59.464644 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:59.464602    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-0ae95d09-openshift-default-97ff7856b-6pj6l"
Sep 10 17:49:59.464894 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:59.464871    2579 patch_prober.go:28] interesting pod/e2e-ait-0ae95d09-openshift-default-97ff7856b-6pj6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body=
Sep 10 17:49:59.464994 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:49:59.464923    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-0ae95d09-openshift-default-97ff7856b-6pj6l" podUID="ea7e1957-8fba-4a60-95ce-3f241a43f2df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused"
Sep 10 17:50:00.465147 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:00.465112    2579 patch_prober.go:28] interesting pod/e2e-ait-0ae95d09-openshift-default-97ff7856b-6pj6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body=
Sep 10 17:50:00.465484 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:00.465171    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-0ae95d09-openshift-default-97ff7856b-6pj6l" podUID="ea7e1957-8fba-4a60-95ce-3f241a43f2df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused"
Sep 10 17:50:01.465364 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:01.465329    2579 patch_prober.go:28] interesting pod/e2e-ait-0ae95d09-openshift-default-97ff7856b-6pj6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body=
Sep 10 17:50:01.465780 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:01.465385    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-0ae95d09-openshift-default-97ff7856b-6pj6l" podUID="ea7e1957-8fba-4a60-95ce-3f241a43f2df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused"
Sep 10 17:50:02.465495 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:02.465452    2579 patch_prober.go:28] interesting pod/e2e-ait-0ae95d09-openshift-default-97ff7856b-6pj6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body=
Sep 10 17:50:02.465910 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:02.465514    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-0ae95d09-openshift-default-97ff7856b-6pj6l" podUID="ea7e1957-8fba-4a60-95ce-3f241a43f2df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused"
Sep 10 17:50:03.464557 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:03.464523    2579 patch_prober.go:28] interesting pod/e2e-ait-0ae95d09-openshift-default-97ff7856b-6pj6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body=
Sep 10 17:50:03.464752 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:03.464602    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-0ae95d09-openshift-default-97ff7856b-6pj6l" podUID="ea7e1957-8fba-4a60-95ce-3f241a43f2df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused"
Sep 10 17:50:04.468144 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:04.468113    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-0ae95d09-openshift-default-97ff7856b-6pj6l"
Sep 10 17:50:04.468539 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:04.468381    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-0ae95d09-openshift-default-97ff7856b-6pj6l"
Sep 10 17:50:04.475768 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:04.475747    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-0ae95d09-openshift-default-97ff7856b-6pj6l"
Sep 10 17:50:05.448766 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:05.448730    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-51ed8d-6964d796d5-qjg6w"]
Sep 10 17:50:05.448975 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:05.448952    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-autores-51ed8d-6964d796d5-qjg6w" podUID="faae58be-d518-4d27-96bb-7545e8de34ad" containerName="maas-api" containerID="cri-o://0ddff592df0ee023a7cdf1e0b498b78d1dca2cee6fd0034e0f34462ae732159b" gracePeriod=30
Sep 10 17:50:05.552625 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:05.552591    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-51ed8d-79cf8c5678-6rwc4"]
Sep 10 17:50:05.553840 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:05.553785    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-autores-51ed8d-79cf8c5678-6rwc4" podUID="82f61180-278f-47de-a4ab-a5225f7e76a4" containerName="payload-pre-processing" containerID="cri-o://8c3826913dae7223b3c25dfec9fc0c687617e50a56d1b76e1b605cf0a37eda4c" gracePeriod=30
Sep 10 17:50:05.730812 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:05.730377    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-51ed8d-6964d796d5-qjg6w"
Sep 10 17:50:05.800746 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:05.800717    2579 generic.go:356] "Generic (PLEG): container finished" podID="82f61180-278f-47de-a4ab-a5225f7e76a4" containerID="8c3826913dae7223b3c25dfec9fc0c687617e50a56d1b76e1b605cf0a37eda4c" exitCode=0
Sep 10 17:50:05.800864 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:05.800791    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-51ed8d-79cf8c5678-6rwc4" event={"ID":"82f61180-278f-47de-a4ab-a5225f7e76a4","Type":"ContainerDied","Data":"8c3826913dae7223b3c25dfec9fc0c687617e50a56d1b76e1b605cf0a37eda4c"}
Sep 10 17:50:05.800924 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:05.800864    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-51ed8d-79cf8c5678-6rwc4" event={"ID":"82f61180-278f-47de-a4ab-a5225f7e76a4","Type":"ContainerDied","Data":"ef4fe80f4755da9250cc11f80274e189df6f66e2dc1331e3e926fad61e04dfc8"}
Sep 10 17:50:05.800924 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:05.800887    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ef4fe80f4755da9250cc11f80274e189df6f66e2dc1331e3e926fad61e04dfc8"
Sep 10 17:50:05.802274 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:05.802253    2579 generic.go:356] "Generic (PLEG): container finished" podID="faae58be-d518-4d27-96bb-7545e8de34ad" containerID="0ddff592df0ee023a7cdf1e0b498b78d1dca2cee6fd0034e0f34462ae732159b" exitCode=0
Sep 10 17:50:05.802383 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:05.802341    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-51ed8d-6964d796d5-qjg6w" event={"ID":"faae58be-d518-4d27-96bb-7545e8de34ad","Type":"ContainerDied","Data":"0ddff592df0ee023a7cdf1e0b498b78d1dca2cee6fd0034e0f34462ae732159b"}
Sep 10 17:50:05.802454 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:05.802384    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-51ed8d-6964d796d5-qjg6w" event={"ID":"faae58be-d518-4d27-96bb-7545e8de34ad","Type":"ContainerDied","Data":"5d8bbe1e0105e1351031a92f33b2abc597e10ff7d6d388cd22ec94d24d5fd500"}
Sep 10 17:50:05.802454 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:05.802406    2579 scope.go:117] "RemoveContainer" containerID="0ddff592df0ee023a7cdf1e0b498b78d1dca2cee6fd0034e0f34462ae732159b"
Sep 10 17:50:05.802454 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:05.802345    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-51ed8d-6964d796d5-qjg6w"
Sep 10 17:50:05.812419 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:05.812376    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-51ed8d-79cf8c5678-6rwc4"
Sep 10 17:50:05.812476 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:05.812435    2579 scope.go:117] "RemoveContainer" containerID="0ddff592df0ee023a7cdf1e0b498b78d1dca2cee6fd0034e0f34462ae732159b"
Sep 10 17:50:05.812743 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:50:05.812721    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ddff592df0ee023a7cdf1e0b498b78d1dca2cee6fd0034e0f34462ae732159b\": container with ID starting with 0ddff592df0ee023a7cdf1e0b498b78d1dca2cee6fd0034e0f34462ae732159b not found: ID does not exist" containerID="0ddff592df0ee023a7cdf1e0b498b78d1dca2cee6fd0034e0f34462ae732159b"
Sep 10 17:50:05.812829 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:05.812748    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ddff592df0ee023a7cdf1e0b498b78d1dca2cee6fd0034e0f34462ae732159b"} err="failed to get container status \"0ddff592df0ee023a7cdf1e0b498b78d1dca2cee6fd0034e0f34462ae732159b\": rpc error: code = NotFound desc = could not find container \"0ddff592df0ee023a7cdf1e0b498b78d1dca2cee6fd0034e0f34462ae732159b\": container with ID starting with 0ddff592df0ee023a7cdf1e0b498b78d1dca2cee6fd0034e0f34462ae732159b not found: ID does not exist"
Sep 10 17:50:05.816037 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:05.816017    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/faae58be-d518-4d27-96bb-7545e8de34ad-openshift-service-ca\") pod \"faae58be-d518-4d27-96bb-7545e8de34ad\" (UID: \"faae58be-d518-4d27-96bb-7545e8de34ad\") "
Sep 10 17:50:05.816105 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:05.816052    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/faae58be-d518-4d27-96bb-7545e8de34ad-maas-api-tls\") pod \"faae58be-d518-4d27-96bb-7545e8de34ad\" (UID: \"faae58be-d518-4d27-96bb-7545e8de34ad\") "
Sep 10 17:50:05.816105 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:05.816093    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/faae58be-d518-4d27-96bb-7545e8de34ad-metrics-certs\") pod \"faae58be-d518-4d27-96bb-7545e8de34ad\" (UID: \"faae58be-d518-4d27-96bb-7545e8de34ad\") "
Sep 10 17:50:05.816191 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:05.816145    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k8zpl\" (UniqueName: \"kubernetes.io/projected/faae58be-d518-4d27-96bb-7545e8de34ad-kube-api-access-k8zpl\") pod \"faae58be-d518-4d27-96bb-7545e8de34ad\" (UID: \"faae58be-d518-4d27-96bb-7545e8de34ad\") "
Sep 10 17:50:05.816406 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:05.816378    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/faae58be-d518-4d27-96bb-7545e8de34ad-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "faae58be-d518-4d27-96bb-7545e8de34ad" (UID: "faae58be-d518-4d27-96bb-7545e8de34ad"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 17:50:05.816545 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:05.816524    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/faae58be-d518-4d27-96bb-7545e8de34ad-openshift-service-ca\") on node \"ip-10-0-129-122.ec2.internal\" DevicePath \"\""
Sep 10 17:50:05.818392 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:05.818346    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/faae58be-d518-4d27-96bb-7545e8de34ad-kube-api-access-k8zpl" (OuterVolumeSpecName: "kube-api-access-k8zpl") pod "faae58be-d518-4d27-96bb-7545e8de34ad" (UID: "faae58be-d518-4d27-96bb-7545e8de34ad"). InnerVolumeSpecName "kube-api-access-k8zpl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:50:05.818472 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:05.818357    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/faae58be-d518-4d27-96bb-7545e8de34ad-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "faae58be-d518-4d27-96bb-7545e8de34ad" (UID: "faae58be-d518-4d27-96bb-7545e8de34ad"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 17:50:05.818519 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:05.818479    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/faae58be-d518-4d27-96bb-7545e8de34ad-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "faae58be-d518-4d27-96bb-7545e8de34ad" (UID: "faae58be-d518-4d27-96bb-7545e8de34ad"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 17:50:05.917163 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:05.917127    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rwhd7\" (UniqueName: \"kubernetes.io/projected/82f61180-278f-47de-a4ab-a5225f7e76a4-kube-api-access-rwhd7\") pod \"82f61180-278f-47de-a4ab-a5225f7e76a4\" (UID: \"82f61180-278f-47de-a4ab-a5225f7e76a4\") "
Sep 10 17:50:05.917163 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:05.917165    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82f61180-278f-47de-a4ab-a5225f7e76a4-plugins-config-volume\") pod \"82f61180-278f-47de-a4ab-a5225f7e76a4\" (UID: \"82f61180-278f-47de-a4ab-a5225f7e76a4\") "
Sep 10 17:50:05.917422 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:05.917406    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/faae58be-d518-4d27-96bb-7545e8de34ad-maas-api-tls\") on node \"ip-10-0-129-122.ec2.internal\" DevicePath \"\""
Sep 10 17:50:05.917469 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:05.917429    2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/faae58be-d518-4d27-96bb-7545e8de34ad-metrics-certs\") on node \"ip-10-0-129-122.ec2.internal\" DevicePath \"\""
Sep 10 17:50:05.917469 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:05.917442    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k8zpl\" (UniqueName: \"kubernetes.io/projected/faae58be-d518-4d27-96bb-7545e8de34ad-kube-api-access-k8zpl\") on node \"ip-10-0-129-122.ec2.internal\" DevicePath \"\""
Sep 10 17:50:05.917609 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:05.917551    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82f61180-278f-47de-a4ab-a5225f7e76a4-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "82f61180-278f-47de-a4ab-a5225f7e76a4" (UID: "82f61180-278f-47de-a4ab-a5225f7e76a4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 17:50:05.919307 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:05.919288    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82f61180-278f-47de-a4ab-a5225f7e76a4-kube-api-access-rwhd7" (OuterVolumeSpecName: "kube-api-access-rwhd7") pod "82f61180-278f-47de-a4ab-a5225f7e76a4" (UID: "82f61180-278f-47de-a4ab-a5225f7e76a4"). InnerVolumeSpecName "kube-api-access-rwhd7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:50:06.018247 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:06.018158    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rwhd7\" (UniqueName: \"kubernetes.io/projected/82f61180-278f-47de-a4ab-a5225f7e76a4-kube-api-access-rwhd7\") on node \"ip-10-0-129-122.ec2.internal\" DevicePath \"\""
Sep 10 17:50:06.018247 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:06.018188    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82f61180-278f-47de-a4ab-a5225f7e76a4-plugins-config-volume\") on node \"ip-10-0-129-122.ec2.internal\" DevicePath \"\""
Sep 10 17:50:06.137021 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:06.136993    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-51ed8d-6964d796d5-qjg6w"]
Sep 10 17:50:06.142999 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:06.142975    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-51ed8d-6964d796d5-qjg6w"]
Sep 10 17:50:06.476650 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:06.476625    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="faae58be-d518-4d27-96bb-7545e8de34ad" path="/var/lib/kubelet/pods/faae58be-d518-4d27-96bb-7545e8de34ad/volumes"
Sep 10 17:50:06.808218 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:06.808135    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-51ed8d-79cf8c5678-6rwc4"
Sep 10 17:50:06.826767 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:06.826730    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-51ed8d-79cf8c5678-6rwc4"]
Sep 10 17:50:06.831481 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:06.831454    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-51ed8d-79cf8c5678-6rwc4"]
Sep 10 17:50:08.475725 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:08.475694    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="82f61180-278f-47de-a4ab-a5225f7e76a4" path="/var/lib/kubelet/pods/82f61180-278f-47de-a4ab-a5225f7e76a4/volumes"
Sep 10 17:50:12.919332 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:12.919293    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0ae95d09-7d89ffd848-2522f"]
Sep 10 17:50:12.919865 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:12.919540    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0ae95d09-7d89ffd848-2522f" podUID="092af79b-03fe-43c8-857b-4922eea42355" containerName="maas-api" containerID="cri-o://71ab4ef05b20c1377cf8f593186da7b8cfbea1032b10e2c86a3c0ed95c163964" gracePeriod=30
Sep 10 17:50:12.986928 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:12.986894    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0ae95d09-75df89545d-kqk49"]
Sep 10 17:50:12.987389 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:12.987358    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-0ae95d09-75df89545d-kqk49" podUID="f3c4208e-f973-4eec-8274-9ff1b7640b24" containerName="payload-pre-processing" containerID="cri-o://5aed558ed04cd7a1ceb9c42686ef8fdb4312dd8aa8fd0fbdcab4ff1032761b7f" gracePeriod=30
Sep 10 17:50:13.239056 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:13.239031    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0ae95d09-7d89ffd848-2522f"
Sep 10 17:50:13.268413 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:13.268385    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-0ae95d09-75df89545d-kqk49"
Sep 10 17:50:13.279216 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:13.279194    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r6klf\" (UniqueName: \"kubernetes.io/projected/092af79b-03fe-43c8-857b-4922eea42355-kube-api-access-r6klf\") pod \"092af79b-03fe-43c8-857b-4922eea42355\" (UID: \"092af79b-03fe-43c8-857b-4922eea42355\") "
Sep 10 17:50:13.279346 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:13.279240    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/092af79b-03fe-43c8-857b-4922eea42355-metrics-certs\") pod \"092af79b-03fe-43c8-857b-4922eea42355\" (UID: \"092af79b-03fe-43c8-857b-4922eea42355\") "
Sep 10 17:50:13.279408 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:13.279346    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/092af79b-03fe-43c8-857b-4922eea42355-openshift-service-ca\") pod \"092af79b-03fe-43c8-857b-4922eea42355\" (UID: \"092af79b-03fe-43c8-857b-4922eea42355\") "
Sep 10 17:50:13.279408 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:13.279374    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/092af79b-03fe-43c8-857b-4922eea42355-maas-api-tls\") pod \"092af79b-03fe-43c8-857b-4922eea42355\" (UID: \"092af79b-03fe-43c8-857b-4922eea42355\") "
Sep 10 17:50:13.280586 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:13.280502    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/092af79b-03fe-43c8-857b-4922eea42355-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "092af79b-03fe-43c8-857b-4922eea42355" (UID: "092af79b-03fe-43c8-857b-4922eea42355"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 17:50:13.282135 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:13.282079    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/092af79b-03fe-43c8-857b-4922eea42355-kube-api-access-r6klf" (OuterVolumeSpecName: "kube-api-access-r6klf") pod "092af79b-03fe-43c8-857b-4922eea42355" (UID: "092af79b-03fe-43c8-857b-4922eea42355"). InnerVolumeSpecName "kube-api-access-r6klf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:50:13.282227 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:13.282195    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/092af79b-03fe-43c8-857b-4922eea42355-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "092af79b-03fe-43c8-857b-4922eea42355" (UID: "092af79b-03fe-43c8-857b-4922eea42355"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 17:50:13.282372 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:13.282352    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/092af79b-03fe-43c8-857b-4922eea42355-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "092af79b-03fe-43c8-857b-4922eea42355" (UID: "092af79b-03fe-43c8-857b-4922eea42355"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 17:50:13.380862 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:13.380830    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q2mhn\" (UniqueName: \"kubernetes.io/projected/f3c4208e-f973-4eec-8274-9ff1b7640b24-kube-api-access-q2mhn\") pod \"f3c4208e-f973-4eec-8274-9ff1b7640b24\" (UID: \"f3c4208e-f973-4eec-8274-9ff1b7640b24\") "
Sep 10 17:50:13.381034 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:13.380948    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3c4208e-f973-4eec-8274-9ff1b7640b24-plugins-config-volume\") pod \"f3c4208e-f973-4eec-8274-9ff1b7640b24\" (UID: \"f3c4208e-f973-4eec-8274-9ff1b7640b24\") "
Sep 10 17:50:13.381102 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:13.381088    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r6klf\" (UniqueName: \"kubernetes.io/projected/092af79b-03fe-43c8-857b-4922eea42355-kube-api-access-r6klf\") on node \"ip-10-0-129-122.ec2.internal\" DevicePath \"\""
Sep 10 17:50:13.381102 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:13.381100    2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/092af79b-03fe-43c8-857b-4922eea42355-metrics-certs\") on node \"ip-10-0-129-122.ec2.internal\" DevicePath \"\""
Sep 10 17:50:13.381204 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:13.381110    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/092af79b-03fe-43c8-857b-4922eea42355-openshift-service-ca\") on node \"ip-10-0-129-122.ec2.internal\" DevicePath \"\""
Sep 10 17:50:13.381204 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:13.381118    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/092af79b-03fe-43c8-857b-4922eea42355-maas-api-tls\") on node \"ip-10-0-129-122.ec2.internal\" DevicePath \"\""
Sep 10 17:50:13.381293 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:13.381261    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3c4208e-f973-4eec-8274-9ff1b7640b24-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f3c4208e-f973-4eec-8274-9ff1b7640b24" (UID: "f3c4208e-f973-4eec-8274-9ff1b7640b24"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 17:50:13.383059 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:13.383035    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3c4208e-f973-4eec-8274-9ff1b7640b24-kube-api-access-q2mhn" (OuterVolumeSpecName: "kube-api-access-q2mhn") pod "f3c4208e-f973-4eec-8274-9ff1b7640b24" (UID: "f3c4208e-f973-4eec-8274-9ff1b7640b24"). InnerVolumeSpecName "kube-api-access-q2mhn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:50:13.481842 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:13.481765    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3c4208e-f973-4eec-8274-9ff1b7640b24-plugins-config-volume\") on node \"ip-10-0-129-122.ec2.internal\" DevicePath \"\""
Sep 10 17:50:13.481842 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:13.481798    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q2mhn\" (UniqueName: \"kubernetes.io/projected/f3c4208e-f973-4eec-8274-9ff1b7640b24-kube-api-access-q2mhn\") on node \"ip-10-0-129-122.ec2.internal\" DevicePath \"\""
Sep 10 17:50:13.843520 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:13.843440    2579 generic.go:356] "Generic (PLEG): container finished" podID="f3c4208e-f973-4eec-8274-9ff1b7640b24" containerID="5aed558ed04cd7a1ceb9c42686ef8fdb4312dd8aa8fd0fbdcab4ff1032761b7f" exitCode=0
Sep 10 17:50:13.843520 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:13.843486    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-0ae95d09-75df89545d-kqk49"
Sep 10 17:50:13.843752 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:13.843524    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-0ae95d09-75df89545d-kqk49" event={"ID":"f3c4208e-f973-4eec-8274-9ff1b7640b24","Type":"ContainerDied","Data":"5aed558ed04cd7a1ceb9c42686ef8fdb4312dd8aa8fd0fbdcab4ff1032761b7f"}
Sep 10 17:50:13.843752 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:13.843603    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-0ae95d09-75df89545d-kqk49" event={"ID":"f3c4208e-f973-4eec-8274-9ff1b7640b24","Type":"ContainerDied","Data":"a30cc81cfe005e7b866a8965e27e3b197c4fd8c3ba0a7299b937eddc9b4166cf"}
Sep 10 17:50:13.843752 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:13.843631    2579 scope.go:117] "RemoveContainer" containerID="5aed558ed04cd7a1ceb9c42686ef8fdb4312dd8aa8fd0fbdcab4ff1032761b7f"
Sep 10 17:50:13.844867 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:13.844847    2579 generic.go:356] "Generic (PLEG): container finished" podID="092af79b-03fe-43c8-857b-4922eea42355" containerID="71ab4ef05b20c1377cf8f593186da7b8cfbea1032b10e2c86a3c0ed95c163964" exitCode=0
Sep 10 17:50:13.844989 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:13.844886    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0ae95d09-7d89ffd848-2522f"
Sep 10 17:50:13.844989 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:13.844930    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0ae95d09-7d89ffd848-2522f" event={"ID":"092af79b-03fe-43c8-857b-4922eea42355","Type":"ContainerDied","Data":"71ab4ef05b20c1377cf8f593186da7b8cfbea1032b10e2c86a3c0ed95c163964"}
Sep 10 17:50:13.844989 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:13.844977    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0ae95d09-7d89ffd848-2522f" event={"ID":"092af79b-03fe-43c8-857b-4922eea42355","Type":"ContainerDied","Data":"ff253a4007fcc3ff0ae85d9afd4a7794fa77fe641bc5f109afffd53c75de3156"}
Sep 10 17:50:13.853413 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:13.853393    2579 scope.go:117] "RemoveContainer" containerID="5aed558ed04cd7a1ceb9c42686ef8fdb4312dd8aa8fd0fbdcab4ff1032761b7f"
Sep 10 17:50:13.853697 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:50:13.853677    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5aed558ed04cd7a1ceb9c42686ef8fdb4312dd8aa8fd0fbdcab4ff1032761b7f\": container with ID starting with 5aed558ed04cd7a1ceb9c42686ef8fdb4312dd8aa8fd0fbdcab4ff1032761b7f not found: ID does not exist" containerID="5aed558ed04cd7a1ceb9c42686ef8fdb4312dd8aa8fd0fbdcab4ff1032761b7f"
Sep 10 17:50:13.853781 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:13.853708    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5aed558ed04cd7a1ceb9c42686ef8fdb4312dd8aa8fd0fbdcab4ff1032761b7f"} err="failed to get container status \"5aed558ed04cd7a1ceb9c42686ef8fdb4312dd8aa8fd0fbdcab4ff1032761b7f\": rpc error: code = NotFound desc = could not find container \"5aed558ed04cd7a1ceb9c42686ef8fdb4312dd8aa8fd0fbdcab4ff1032761b7f\": container with ID starting with 5aed558ed04cd7a1ceb9c42686ef8fdb4312dd8aa8fd0fbdcab4ff1032761b7f not found: ID does not exist"
Sep 10 17:50:13.853781 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:13.853729    2579 scope.go:117] "RemoveContainer" containerID="71ab4ef05b20c1377cf8f593186da7b8cfbea1032b10e2c86a3c0ed95c163964"
Sep 10 17:50:13.861389 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:13.861372    2579 scope.go:117] "RemoveContainer" containerID="71ab4ef05b20c1377cf8f593186da7b8cfbea1032b10e2c86a3c0ed95c163964"
Sep 10 17:50:13.861668 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:50:13.861645    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71ab4ef05b20c1377cf8f593186da7b8cfbea1032b10e2c86a3c0ed95c163964\": container with ID starting with 71ab4ef05b20c1377cf8f593186da7b8cfbea1032b10e2c86a3c0ed95c163964 not found: ID does not exist" containerID="71ab4ef05b20c1377cf8f593186da7b8cfbea1032b10e2c86a3c0ed95c163964"
Sep 10 17:50:13.861782 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:13.861671    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71ab4ef05b20c1377cf8f593186da7b8cfbea1032b10e2c86a3c0ed95c163964"} err="failed to get container status \"71ab4ef05b20c1377cf8f593186da7b8cfbea1032b10e2c86a3c0ed95c163964\": rpc error: code = NotFound desc = could not find container \"71ab4ef05b20c1377cf8f593186da7b8cfbea1032b10e2c86a3c0ed95c163964\": container with ID starting with 71ab4ef05b20c1377cf8f593186da7b8cfbea1032b10e2c86a3c0ed95c163964 not found: ID does not exist"
Sep 10 17:50:13.866310 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:13.866292    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0ae95d09-75df89545d-kqk49"]
Sep 10 17:50:13.870368 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:13.870348    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0ae95d09-75df89545d-kqk49"]
Sep 10 17:50:13.882609 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:13.882589    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0ae95d09-7d89ffd848-2522f"]
Sep 10 17:50:13.888766 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:13.888747    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0ae95d09-7d89ffd848-2522f"]
Sep 10 17:50:14.476073 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:14.476035    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="092af79b-03fe-43c8-857b-4922eea42355" path="/var/lib/kubelet/pods/092af79b-03fe-43c8-857b-4922eea42355/volumes"
Sep 10 17:50:14.476698 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:14.476677    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f3c4208e-f973-4eec-8274-9ff1b7640b24" path="/var/lib/kubelet/pods/f3c4208e-f973-4eec-8274-9ff1b7640b24/volumes"
Sep 10 17:50:14.525625 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:14.525594    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-0ae95d09-openshift-default-97ff7856b-6pj6l"]
Sep 10 17:50:14.525860 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:14.525837    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-0ae95d09-openshift-default-97ff7856b-6pj6l" podUID="ea7e1957-8fba-4a60-95ce-3f241a43f2df" containerName="istio-proxy" containerID="cri-o://d97a36313fad7cb3b251c968333a0f460bad6ef94b37760dde82ba369b2cf3b0" gracePeriod=30
Sep 10 17:50:15.325156 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:15.325124    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-d2a8a5f7-openshift-default-8bbc6c576-d8dw4"]
Sep 10 17:50:15.325604 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:15.325582    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f3c4208e-f973-4eec-8274-9ff1b7640b24" containerName="payload-pre-processing"
Sep 10 17:50:15.325604 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:15.325604    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="f3c4208e-f973-4eec-8274-9ff1b7640b24" containerName="payload-pre-processing"
Sep 10 17:50:15.325755 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:15.325626    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="faae58be-d518-4d27-96bb-7545e8de34ad" containerName="maas-api"
Sep 10 17:50:15.325755 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:15.325634    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="faae58be-d518-4d27-96bb-7545e8de34ad" containerName="maas-api"
Sep 10 17:50:15.325755 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:15.325649    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82f61180-278f-47de-a4ab-a5225f7e76a4" containerName="payload-pre-processing"
Sep 10 17:50:15.325755 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:15.325658    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="82f61180-278f-47de-a4ab-a5225f7e76a4" containerName="payload-pre-processing"
Sep 10 17:50:15.325755 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:15.325665    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="092af79b-03fe-43c8-857b-4922eea42355" containerName="maas-api"
Sep 10 17:50:15.325755 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:15.325673    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="092af79b-03fe-43c8-857b-4922eea42355" containerName="maas-api"
Sep 10 17:50:15.326039 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:15.325765    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="faae58be-d518-4d27-96bb-7545e8de34ad" containerName="maas-api"
Sep 10 17:50:15.326039 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:15.325787    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="f3c4208e-f973-4eec-8274-9ff1b7640b24" containerName="payload-pre-processing"
Sep 10 17:50:15.326039 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:15.325797    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="82f61180-278f-47de-a4ab-a5225f7e76a4" containerName="payload-pre-processing"
Sep 10 17:50:15.326039 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:15.325805    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="092af79b-03fe-43c8-857b-4922eea42355" containerName="maas-api"
Sep 10 17:50:15.330327 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:15.330307    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-d2a8a5f7-openshift-default-8bbc6c576-d8dw4"
Sep 10 17:50:15.333098 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:15.333080    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-d2a8a5f7-openshift-default-dockercfg-x7glb\""
Sep 10 17:50:15.340797 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:15.340775    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-d2a8a5f7-openshift-default-8bbc6c576-d8dw4"]
Sep 10 17:50:15.402361 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:15.402330    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1b2e0c7-18df-4262-9436-afef8fcc7095-istio-token\") pod \"e2e-derive-d2a8a5f7-openshift-default-8bbc6c576-d8dw4\" (UID: \"a1b2e0c7-18df-4262-9436-afef8fcc7095\") " pod="openshift-ingress/e2e-derive-d2a8a5f7-openshift-default-8bbc6c576-d8dw4"
Sep 10 17:50:15.402496 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:15.402384    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1b2e0c7-18df-4262-9436-afef8fcc7095-credential-socket\") pod \"e2e-derive-d2a8a5f7-openshift-default-8bbc6c576-d8dw4\" (UID: \"a1b2e0c7-18df-4262-9436-afef8fcc7095\") " pod="openshift-ingress/e2e-derive-d2a8a5f7-openshift-default-8bbc6c576-d8dw4"
Sep 10 17:50:15.402496 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:15.402411    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1b2e0c7-18df-4262-9436-afef8fcc7095-istiod-ca-cert\") pod \"e2e-derive-d2a8a5f7-openshift-default-8bbc6c576-d8dw4\" (UID: \"a1b2e0c7-18df-4262-9436-afef8fcc7095\") " pod="openshift-ingress/e2e-derive-d2a8a5f7-openshift-default-8bbc6c576-d8dw4"
Sep 10 17:50:15.402496 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:15.402452    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1b2e0c7-18df-4262-9436-afef8fcc7095-istio-data\") pod \"e2e-derive-d2a8a5f7-openshift-default-8bbc6c576-d8dw4\" (UID: \"a1b2e0c7-18df-4262-9436-afef8fcc7095\") " pod="openshift-ingress/e2e-derive-d2a8a5f7-openshift-default-8bbc6c576-d8dw4"
Sep 10 17:50:15.402689 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:15.402545    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1b2e0c7-18df-4262-9436-afef8fcc7095-workload-socket\") pod \"e2e-derive-d2a8a5f7-openshift-default-8bbc6c576-d8dw4\" (UID: \"a1b2e0c7-18df-4262-9436-afef8fcc7095\") " pod="openshift-ingress/e2e-derive-d2a8a5f7-openshift-default-8bbc6c576-d8dw4"
Sep 10 17:50:15.402689 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:15.402614    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1b2e0c7-18df-4262-9436-afef8fcc7095-istio-envoy\") pod \"e2e-derive-d2a8a5f7-openshift-default-8bbc6c576-d8dw4\" (UID: \"a1b2e0c7-18df-4262-9436-afef8fcc7095\") " pod="openshift-ingress/e2e-derive-d2a8a5f7-openshift-default-8bbc6c576-d8dw4"
Sep 10 17:50:15.402689 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:15.402675    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1b2e0c7-18df-4262-9436-afef8fcc7095-workload-certs\") pod \"e2e-derive-d2a8a5f7-openshift-default-8bbc6c576-d8dw4\" (UID: \"a1b2e0c7-18df-4262-9436-afef8fcc7095\") " pod="openshift-ingress/e2e-derive-d2a8a5f7-openshift-default-8bbc6c576-d8dw4"
Sep 10 17:50:15.402792 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:15.402719    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1b2e0c7-18df-4262-9436-afef8fcc7095-istio-podinfo\") pod \"e2e-derive-d2a8a5f7-openshift-default-8bbc6c576-d8dw4\" (UID: \"a1b2e0c7-18df-4262-9436-afef8fcc7095\") " pod="openshift-ingress/e2e-derive-d2a8a5f7-openshift-default-8bbc6c576-d8dw4"
Sep 10 17:50:15.402792 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:15.402761    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2gj8k\" (UniqueName: \"kubernetes.io/projected/a1b2e0c7-18df-4262-9436-afef8fcc7095-kube-api-access-2gj8k\") pod \"e2e-derive-d2a8a5f7-openshift-default-8bbc6c576-d8dw4\" (UID: \"a1b2e0c7-18df-4262-9436-afef8fcc7095\") " pod="openshift-ingress/e2e-derive-d2a8a5f7-openshift-default-8bbc6c576-d8dw4"
Sep 10 17:50:15.503977 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:15.503935    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1b2e0c7-18df-4262-9436-afef8fcc7095-workload-certs\") pod \"e2e-derive-d2a8a5f7-openshift-default-8bbc6c576-d8dw4\" (UID: \"a1b2e0c7-18df-4262-9436-afef8fcc7095\") " pod="openshift-ingress/e2e-derive-d2a8a5f7-openshift-default-8bbc6c576-d8dw4"
Sep 10 17:50:15.503977 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:15.503982    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1b2e0c7-18df-4262-9436-afef8fcc7095-istio-podinfo\") pod \"e2e-derive-d2a8a5f7-openshift-default-8bbc6c576-d8dw4\" (UID: \"a1b2e0c7-18df-4262-9436-afef8fcc7095\") " pod="openshift-ingress/e2e-derive-d2a8a5f7-openshift-default-8bbc6c576-d8dw4"
Sep 10 17:50:15.504475 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:15.504005    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gj8k\" (UniqueName: \"kubernetes.io/projected/a1b2e0c7-18df-4262-9436-afef8fcc7095-kube-api-access-2gj8k\") pod \"e2e-derive-d2a8a5f7-openshift-default-8bbc6c576-d8dw4\" (UID: \"a1b2e0c7-18df-4262-9436-afef8fcc7095\") " pod="openshift-ingress/e2e-derive-d2a8a5f7-openshift-default-8bbc6c576-d8dw4"
Sep 10 17:50:15.504475 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:15.504021    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1b2e0c7-18df-4262-9436-afef8fcc7095-istio-token\") pod \"e2e-derive-d2a8a5f7-openshift-default-8bbc6c576-d8dw4\" (UID: \"a1b2e0c7-18df-4262-9436-afef8fcc7095\") " pod="openshift-ingress/e2e-derive-d2a8a5f7-openshift-default-8bbc6c576-d8dw4"
Sep 10 17:50:15.504475 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:15.504047    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1b2e0c7-18df-4262-9436-afef8fcc7095-credential-socket\") pod \"e2e-derive-d2a8a5f7-openshift-default-8bbc6c576-d8dw4\" (UID: \"a1b2e0c7-18df-4262-9436-afef8fcc7095\") " pod="openshift-ingress/e2e-derive-d2a8a5f7-openshift-default-8bbc6c576-d8dw4"
Sep 10 17:50:15.504475 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:15.504063    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1b2e0c7-18df-4262-9436-afef8fcc7095-istiod-ca-cert\") pod \"e2e-derive-d2a8a5f7-openshift-default-8bbc6c576-d8dw4\" (UID: \"a1b2e0c7-18df-4262-9436-afef8fcc7095\") " pod="openshift-ingress/e2e-derive-d2a8a5f7-openshift-default-8bbc6c576-d8dw4"
Sep 10 17:50:15.504475 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:15.504090    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1b2e0c7-18df-4262-9436-afef8fcc7095-istio-data\") pod \"e2e-derive-d2a8a5f7-openshift-default-8bbc6c576-d8dw4\" (UID: \"a1b2e0c7-18df-4262-9436-afef8fcc7095\") " pod="openshift-ingress/e2e-derive-d2a8a5f7-openshift-default-8bbc6c576-d8dw4"
Sep 10 17:50:15.504475 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:15.504114    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1b2e0c7-18df-4262-9436-afef8fcc7095-workload-socket\") pod \"e2e-derive-d2a8a5f7-openshift-default-8bbc6c576-d8dw4\" (UID: \"a1b2e0c7-18df-4262-9436-afef8fcc7095\") " pod="openshift-ingress/e2e-derive-d2a8a5f7-openshift-default-8bbc6c576-d8dw4"
Sep 10 17:50:15.504475 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:15.504155    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1b2e0c7-18df-4262-9436-afef8fcc7095-istio-envoy\") pod \"e2e-derive-d2a8a5f7-openshift-default-8bbc6c576-d8dw4\" (UID: \"a1b2e0c7-18df-4262-9436-afef8fcc7095\") " pod="openshift-ingress/e2e-derive-d2a8a5f7-openshift-default-8bbc6c576-d8dw4"
Sep 10 17:50:15.504475 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:15.504388    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1b2e0c7-18df-4262-9436-afef8fcc7095-workload-certs\") pod \"e2e-derive-d2a8a5f7-openshift-default-8bbc6c576-d8dw4\" (UID: \"a1b2e0c7-18df-4262-9436-afef8fcc7095\") " pod="openshift-ingress/e2e-derive-d2a8a5f7-openshift-default-8bbc6c576-d8dw4"
Sep 10 17:50:15.504475 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:15.504437    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1b2e0c7-18df-4262-9436-afef8fcc7095-credential-socket\") pod \"e2e-derive-d2a8a5f7-openshift-default-8bbc6c576-d8dw4\" (UID: \"a1b2e0c7-18df-4262-9436-afef8fcc7095\") " pod="openshift-ingress/e2e-derive-d2a8a5f7-openshift-default-8bbc6c576-d8dw4"
Sep 10 17:50:15.504881 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:15.504503    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1b2e0c7-18df-4262-9436-afef8fcc7095-workload-socket\") pod \"e2e-derive-d2a8a5f7-openshift-default-8bbc6c576-d8dw4\" (UID: \"a1b2e0c7-18df-4262-9436-afef8fcc7095\") " pod="openshift-ingress/e2e-derive-d2a8a5f7-openshift-default-8bbc6c576-d8dw4"
Sep 10 17:50:15.504881 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:15.504595    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1b2e0c7-18df-4262-9436-afef8fcc7095-istio-data\") pod \"e2e-derive-d2a8a5f7-openshift-default-8bbc6c576-d8dw4\" (UID: \"a1b2e0c7-18df-4262-9436-afef8fcc7095\") " pod="openshift-ingress/e2e-derive-d2a8a5f7-openshift-default-8bbc6c576-d8dw4"
Sep 10 17:50:15.505143 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:15.504964    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1b2e0c7-18df-4262-9436-afef8fcc7095-istiod-ca-cert\") pod \"e2e-derive-d2a8a5f7-openshift-default-8bbc6c576-d8dw4\" (UID: \"a1b2e0c7-18df-4262-9436-afef8fcc7095\") " pod="openshift-ingress/e2e-derive-d2a8a5f7-openshift-default-8bbc6c576-d8dw4"
Sep 10 17:50:15.506388 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:15.506366    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1b2e0c7-18df-4262-9436-afef8fcc7095-istio-envoy\") pod \"e2e-derive-d2a8a5f7-openshift-default-8bbc6c576-d8dw4\" (UID: \"a1b2e0c7-18df-4262-9436-afef8fcc7095\") " pod="openshift-ingress/e2e-derive-d2a8a5f7-openshift-default-8bbc6c576-d8dw4"
Sep 10 17:50:15.506880 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:15.506862    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1b2e0c7-18df-4262-9436-afef8fcc7095-istio-podinfo\") pod \"e2e-derive-d2a8a5f7-openshift-default-8bbc6c576-d8dw4\" (UID: \"a1b2e0c7-18df-4262-9436-afef8fcc7095\") " pod="openshift-ingress/e2e-derive-d2a8a5f7-openshift-default-8bbc6c576-d8dw4"
Sep 10 17:50:15.518502 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:15.518472    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1b2e0c7-18df-4262-9436-afef8fcc7095-istio-token\") pod \"e2e-derive-d2a8a5f7-openshift-default-8bbc6c576-d8dw4\" (UID: \"a1b2e0c7-18df-4262-9436-afef8fcc7095\") " pod="openshift-ingress/e2e-derive-d2a8a5f7-openshift-default-8bbc6c576-d8dw4"
Sep 10 17:50:15.518712 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:15.518693    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2gj8k\" (UniqueName: \"kubernetes.io/projected/a1b2e0c7-18df-4262-9436-afef8fcc7095-kube-api-access-2gj8k\") pod \"e2e-derive-d2a8a5f7-openshift-default-8bbc6c576-d8dw4\" (UID: \"a1b2e0c7-18df-4262-9436-afef8fcc7095\") " pod="openshift-ingress/e2e-derive-d2a8a5f7-openshift-default-8bbc6c576-d8dw4"
Sep 10 17:50:15.641519 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:15.641500    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-d2a8a5f7-openshift-default-8bbc6c576-d8dw4"
Sep 10 17:50:15.922489 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:15.922463    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-d2a8a5f7-openshift-default-8bbc6c576-d8dw4"]
Sep 10 17:50:15.923409 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:50:15.923382    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1b2e0c7_18df_4262_9436_afef8fcc7095.slice/crio-0c9079ce8ba9cb2108303846afdd22f13946f524b8a405d4989e8b2ead43ace0 WatchSource:0}: Error finding container 0c9079ce8ba9cb2108303846afdd22f13946f524b8a405d4989e8b2ead43ace0: Status 404 returned error can't find the container with id 0c9079ce8ba9cb2108303846afdd22f13946f524b8a405d4989e8b2ead43ace0
Sep 10 17:50:15.925463 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:15.925433    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 10 17:50:15.925521 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:15.925496    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 10 17:50:15.925554 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:15.925525    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 10 17:50:16.861493 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:16.861464    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-d2a8a5f7-openshift-default-8bbc6c576-d8dw4" event={"ID":"a1b2e0c7-18df-4262-9436-afef8fcc7095","Type":"ContainerStarted","Data":"08a338d273e1fbc1db02e38ec807e8afa058e5094c62007e0972359c1393d179"}
Sep 10 17:50:16.861870 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:16.861516    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-d2a8a5f7-openshift-default-8bbc6c576-d8dw4" event={"ID":"a1b2e0c7-18df-4262-9436-afef8fcc7095","Type":"ContainerStarted","Data":"0c9079ce8ba9cb2108303846afdd22f13946f524b8a405d4989e8b2ead43ace0"}
Sep 10 17:50:16.885322 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:16.885264    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-d2a8a5f7-openshift-default-8bbc6c576-d8dw4" podStartSLOduration=1.8852481939999999 podStartE2EDuration="1.885248194s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:50:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:50:16.884528954 +0000 UTC m=+860.884359020" watchObservedRunningTime="2026-09-10 17:50:16.885248194 +0000 UTC m=+860.885078259"
Sep 10 17:50:17.642435 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:17.642400    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-d2a8a5f7-openshift-default-8bbc6c576-d8dw4"
Sep 10 17:50:17.647611 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:17.647559    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-d2a8a5f7-openshift-default-8bbc6c576-d8dw4"
Sep 10 17:50:17.866580 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:17.866536    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-d2a8a5f7-openshift-default-8bbc6c576-d8dw4"
Sep 10 17:50:17.867581 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:17.867541    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-d2a8a5f7-openshift-default-8bbc6c576-d8dw4"
Sep 10 17:50:19.470586 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:19.470535    2579 patch_prober.go:28] interesting pod/e2e-ait-0ae95d09-openshift-default-97ff7856b-6pj6l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused
Sep 10 17:50:19.470944 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:19.470607    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-0ae95d09-openshift-default-97ff7856b-6pj6l" podUID="ea7e1957-8fba-4a60-95ce-3f241a43f2df" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 10 17:50:19.557054 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:19.557011    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-d2a8a5f7-7977bddcb6-q57v5"]
Sep 10 17:50:19.563821 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:19.563470    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-d2a8a5f7-7977bddcb6-q57v5"
Sep 10 17:50:19.566818 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:19.566771    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-d2a8a5f7-serving-cert\""
Sep 10 17:50:19.578067 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:19.578026    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-d2a8a5f7-7977bddcb6-q57v5"]
Sep 10 17:50:19.640866 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:19.640836    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f7356a3-18ba-4f2d-85aa-fe6001286875-openshift-service-ca\") pod \"maas-api-e2e-derive-d2a8a5f7-7977bddcb6-q57v5\" (UID: \"4f7356a3-18ba-4f2d-85aa-fe6001286875\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-d2a8a5f7-7977bddcb6-q57v5"
Sep 10 17:50:19.640983 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:19.640902    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f7356a3-18ba-4f2d-85aa-fe6001286875-metrics-certs\") pod \"maas-api-e2e-derive-d2a8a5f7-7977bddcb6-q57v5\" (UID: \"4f7356a3-18ba-4f2d-85aa-fe6001286875\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-d2a8a5f7-7977bddcb6-q57v5"
Sep 10 17:50:19.641063 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:19.641039    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4f7356a3-18ba-4f2d-85aa-fe6001286875-maas-api-tls\") pod \"maas-api-e2e-derive-d2a8a5f7-7977bddcb6-q57v5\" (UID: \"4f7356a3-18ba-4f2d-85aa-fe6001286875\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-d2a8a5f7-7977bddcb6-q57v5"
Sep 10 17:50:19.641103 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:19.641079    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmpdd\" (UniqueName: \"kubernetes.io/projected/4f7356a3-18ba-4f2d-85aa-fe6001286875-kube-api-access-qmpdd\") pod \"maas-api-e2e-derive-d2a8a5f7-7977bddcb6-q57v5\" (UID: \"4f7356a3-18ba-4f2d-85aa-fe6001286875\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-d2a8a5f7-7977bddcb6-q57v5"
Sep 10 17:50:19.742200 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:19.742126    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4f7356a3-18ba-4f2d-85aa-fe6001286875-maas-api-tls\") pod \"maas-api-e2e-derive-d2a8a5f7-7977bddcb6-q57v5\" (UID: \"4f7356a3-18ba-4f2d-85aa-fe6001286875\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-d2a8a5f7-7977bddcb6-q57v5"
Sep 10 17:50:19.742200 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:19.742161    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmpdd\" (UniqueName: \"kubernetes.io/projected/4f7356a3-18ba-4f2d-85aa-fe6001286875-kube-api-access-qmpdd\") pod \"maas-api-e2e-derive-d2a8a5f7-7977bddcb6-q57v5\" (UID: \"4f7356a3-18ba-4f2d-85aa-fe6001286875\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-d2a8a5f7-7977bddcb6-q57v5"
Sep 10 17:50:19.742200 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:19.742195    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f7356a3-18ba-4f2d-85aa-fe6001286875-openshift-service-ca\") pod \"maas-api-e2e-derive-d2a8a5f7-7977bddcb6-q57v5\" (UID: \"4f7356a3-18ba-4f2d-85aa-fe6001286875\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-d2a8a5f7-7977bddcb6-q57v5"
Sep 10 17:50:19.742453 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:19.742253    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f7356a3-18ba-4f2d-85aa-fe6001286875-metrics-certs\") pod \"maas-api-e2e-derive-d2a8a5f7-7977bddcb6-q57v5\" (UID: \"4f7356a3-18ba-4f2d-85aa-fe6001286875\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-d2a8a5f7-7977bddcb6-q57v5"
Sep 10 17:50:19.742453 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:50:19.742316    2579 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-d2a8a5f7-serving-cert: secret "maas-api-e2e-derive-d2a8a5f7-serving-cert" not found
Sep 10 17:50:19.742453 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:50:19.742410    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4f7356a3-18ba-4f2d-85aa-fe6001286875-maas-api-tls podName:4f7356a3-18ba-4f2d-85aa-fe6001286875 nodeName:}" failed. No retries permitted until 2026-09-10 17:50:20.242386768 +0000 UTC m=+864.242216826 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4f7356a3-18ba-4f2d-85aa-fe6001286875-maas-api-tls") pod "maas-api-e2e-derive-d2a8a5f7-7977bddcb6-q57v5" (UID: "4f7356a3-18ba-4f2d-85aa-fe6001286875") : secret "maas-api-e2e-derive-d2a8a5f7-serving-cert" not found
Sep 10 17:50:19.742919 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:19.742897    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f7356a3-18ba-4f2d-85aa-fe6001286875-openshift-service-ca\") pod \"maas-api-e2e-derive-d2a8a5f7-7977bddcb6-q57v5\" (UID: \"4f7356a3-18ba-4f2d-85aa-fe6001286875\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-d2a8a5f7-7977bddcb6-q57v5"
Sep 10 17:50:19.744920 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:19.744896    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f7356a3-18ba-4f2d-85aa-fe6001286875-metrics-certs\") pod \"maas-api-e2e-derive-d2a8a5f7-7977bddcb6-q57v5\" (UID: \"4f7356a3-18ba-4f2d-85aa-fe6001286875\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-d2a8a5f7-7977bddcb6-q57v5"
Sep 10 17:50:19.761449 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:19.761425    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmpdd\" (UniqueName: \"kubernetes.io/projected/4f7356a3-18ba-4f2d-85aa-fe6001286875-kube-api-access-qmpdd\") pod \"maas-api-e2e-derive-d2a8a5f7-7977bddcb6-q57v5\" (UID: \"4f7356a3-18ba-4f2d-85aa-fe6001286875\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-d2a8a5f7-7977bddcb6-q57v5"
Sep 10 17:50:19.785303 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:19.785285    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-0ae95d09-openshift-default-97ff7856b-6pj6l"
Sep 10 17:50:19.843143 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:19.843114    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea7e1957-8fba-4a60-95ce-3f241a43f2df-credential-socket\") pod \"ea7e1957-8fba-4a60-95ce-3f241a43f2df\" (UID: \"ea7e1957-8fba-4a60-95ce-3f241a43f2df\") "
Sep 10 17:50:19.843255 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:19.843165    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea7e1957-8fba-4a60-95ce-3f241a43f2df-istiod-ca-cert\") pod \"ea7e1957-8fba-4a60-95ce-3f241a43f2df\" (UID: \"ea7e1957-8fba-4a60-95ce-3f241a43f2df\") "
Sep 10 17:50:19.843255 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:19.843211    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea7e1957-8fba-4a60-95ce-3f241a43f2df-workload-certs\") pod \"ea7e1957-8fba-4a60-95ce-3f241a43f2df\" (UID: \"ea7e1957-8fba-4a60-95ce-3f241a43f2df\") "
Sep 10 17:50:19.843255 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:19.843244    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea7e1957-8fba-4a60-95ce-3f241a43f2df-istio-data\") pod \"ea7e1957-8fba-4a60-95ce-3f241a43f2df\" (UID: \"ea7e1957-8fba-4a60-95ce-3f241a43f2df\") "
Sep 10 17:50:19.843414 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:19.843306    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea7e1957-8fba-4a60-95ce-3f241a43f2df-workload-socket\") pod \"ea7e1957-8fba-4a60-95ce-3f241a43f2df\" (UID: \"ea7e1957-8fba-4a60-95ce-3f241a43f2df\") "
Sep 10 17:50:19.843414 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:19.843358    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea7e1957-8fba-4a60-95ce-3f241a43f2df-istio-podinfo\") pod \"ea7e1957-8fba-4a60-95ce-3f241a43f2df\" (UID: \"ea7e1957-8fba-4a60-95ce-3f241a43f2df\") "
Sep 10 17:50:19.843414 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:19.843387    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea7e1957-8fba-4a60-95ce-3f241a43f2df-istio-token\") pod \"ea7e1957-8fba-4a60-95ce-3f241a43f2df\" (UID: \"ea7e1957-8fba-4a60-95ce-3f241a43f2df\") "
Sep 10 17:50:19.843414 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:19.843385    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea7e1957-8fba-4a60-95ce-3f241a43f2df-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "ea7e1957-8fba-4a60-95ce-3f241a43f2df" (UID: "ea7e1957-8fba-4a60-95ce-3f241a43f2df"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:50:19.843630 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:19.843417    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8tfst\" (UniqueName: \"kubernetes.io/projected/ea7e1957-8fba-4a60-95ce-3f241a43f2df-kube-api-access-8tfst\") pod \"ea7e1957-8fba-4a60-95ce-3f241a43f2df\" (UID: \"ea7e1957-8fba-4a60-95ce-3f241a43f2df\") "
Sep 10 17:50:19.843630 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:19.843477    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea7e1957-8fba-4a60-95ce-3f241a43f2df-istio-envoy\") pod \"ea7e1957-8fba-4a60-95ce-3f241a43f2df\" (UID: \"ea7e1957-8fba-4a60-95ce-3f241a43f2df\") "
Sep 10 17:50:19.843630 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:19.843520    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea7e1957-8fba-4a60-95ce-3f241a43f2df-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "ea7e1957-8fba-4a60-95ce-3f241a43f2df" (UID: "ea7e1957-8fba-4a60-95ce-3f241a43f2df"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:50:19.843630 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:19.843535    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea7e1957-8fba-4a60-95ce-3f241a43f2df-istio-data" (OuterVolumeSpecName: "istio-data") pod "ea7e1957-8fba-4a60-95ce-3f241a43f2df" (UID: "ea7e1957-8fba-4a60-95ce-3f241a43f2df"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:50:19.843822 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:19.843668    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea7e1957-8fba-4a60-95ce-3f241a43f2df-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "ea7e1957-8fba-4a60-95ce-3f241a43f2df" (UID: "ea7e1957-8fba-4a60-95ce-3f241a43f2df"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 17:50:19.843921 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:19.843900    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea7e1957-8fba-4a60-95ce-3f241a43f2df-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "ea7e1957-8fba-4a60-95ce-3f241a43f2df" (UID: "ea7e1957-8fba-4a60-95ce-3f241a43f2df"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:50:19.843986 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:19.843917    2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea7e1957-8fba-4a60-95ce-3f241a43f2df-workload-socket\") on node \"ip-10-0-129-122.ec2.internal\" DevicePath \"\""
Sep 10 17:50:19.843986 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:19.843934    2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea7e1957-8fba-4a60-95ce-3f241a43f2df-credential-socket\") on node \"ip-10-0-129-122.ec2.internal\" DevicePath \"\""
Sep 10 17:50:19.843986 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:19.843944    2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea7e1957-8fba-4a60-95ce-3f241a43f2df-istiod-ca-cert\") on node \"ip-10-0-129-122.ec2.internal\" DevicePath \"\""
Sep 10 17:50:19.843986 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:19.843951    2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea7e1957-8fba-4a60-95ce-3f241a43f2df-istio-data\") on node \"ip-10-0-129-122.ec2.internal\" DevicePath \"\""
Sep 10 17:50:19.845655 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:19.845629    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea7e1957-8fba-4a60-95ce-3f241a43f2df-istio-token" (OuterVolumeSpecName: "istio-token") pod "ea7e1957-8fba-4a60-95ce-3f241a43f2df" (UID: "ea7e1957-8fba-4a60-95ce-3f241a43f2df"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:50:19.846072 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:19.846054    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea7e1957-8fba-4a60-95ce-3f241a43f2df-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "ea7e1957-8fba-4a60-95ce-3f241a43f2df" (UID: "ea7e1957-8fba-4a60-95ce-3f241a43f2df"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:50:19.846136 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:19.846118    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea7e1957-8fba-4a60-95ce-3f241a43f2df-kube-api-access-8tfst" (OuterVolumeSpecName: "kube-api-access-8tfst") pod "ea7e1957-8fba-4a60-95ce-3f241a43f2df" (UID: "ea7e1957-8fba-4a60-95ce-3f241a43f2df"). InnerVolumeSpecName "kube-api-access-8tfst". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:50:19.846136 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:19.846118    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ea7e1957-8fba-4a60-95ce-3f241a43f2df-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "ea7e1957-8fba-4a60-95ce-3f241a43f2df" (UID: "ea7e1957-8fba-4a60-95ce-3f241a43f2df"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 10 17:50:19.877304 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:19.877274    2579 generic.go:356] "Generic (PLEG): container finished" podID="ea7e1957-8fba-4a60-95ce-3f241a43f2df" containerID="d97a36313fad7cb3b251c968333a0f460bad6ef94b37760dde82ba369b2cf3b0" exitCode=0
Sep 10 17:50:19.877404 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:19.877317    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-0ae95d09-openshift-default-97ff7856b-6pj6l"
Sep 10 17:50:19.877404 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:19.877360    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0ae95d09-openshift-default-97ff7856b-6pj6l" event={"ID":"ea7e1957-8fba-4a60-95ce-3f241a43f2df","Type":"ContainerDied","Data":"d97a36313fad7cb3b251c968333a0f460bad6ef94b37760dde82ba369b2cf3b0"}
Sep 10 17:50:19.877494 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:19.877408    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0ae95d09-openshift-default-97ff7856b-6pj6l" event={"ID":"ea7e1957-8fba-4a60-95ce-3f241a43f2df","Type":"ContainerDied","Data":"d6e11bdd604bfe3e995cd0a3ebc09cd7ab9449774dd00069aa37cdcdb6d696c2"}
Sep 10 17:50:19.877494 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:19.877430    2579 scope.go:117] "RemoveContainer" containerID="d97a36313fad7cb3b251c968333a0f460bad6ef94b37760dde82ba369b2cf3b0"
Sep 10 17:50:19.886326 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:19.886312    2579 scope.go:117] "RemoveContainer" containerID="40d777077449038396f83b11cabd34756f7c474eb2005f77abe5ffa3437a57c9"
Sep 10 17:50:19.893838 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:19.893819    2579 scope.go:117] "RemoveContainer" containerID="d97a36313fad7cb3b251c968333a0f460bad6ef94b37760dde82ba369b2cf3b0"
Sep 10 17:50:19.894082 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:50:19.894064    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d97a36313fad7cb3b251c968333a0f460bad6ef94b37760dde82ba369b2cf3b0\": container with ID starting with d97a36313fad7cb3b251c968333a0f460bad6ef94b37760dde82ba369b2cf3b0 not found: ID does not exist" containerID="d97a36313fad7cb3b251c968333a0f460bad6ef94b37760dde82ba369b2cf3b0"
Sep 10 17:50:19.894154 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:19.894094    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d97a36313fad7cb3b251c968333a0f460bad6ef94b37760dde82ba369b2cf3b0"} err="failed to get container status \"d97a36313fad7cb3b251c968333a0f460bad6ef94b37760dde82ba369b2cf3b0\": rpc error: code = NotFound desc = could not find container \"d97a36313fad7cb3b251c968333a0f460bad6ef94b37760dde82ba369b2cf3b0\": container with ID starting with d97a36313fad7cb3b251c968333a0f460bad6ef94b37760dde82ba369b2cf3b0 not found: ID does not exist"
Sep 10 17:50:19.894154 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:19.894116    2579 scope.go:117] "RemoveContainer" containerID="40d777077449038396f83b11cabd34756f7c474eb2005f77abe5ffa3437a57c9"
Sep 10 17:50:19.894368 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:50:19.894350    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40d777077449038396f83b11cabd34756f7c474eb2005f77abe5ffa3437a57c9\": container with ID starting with 40d777077449038396f83b11cabd34756f7c474eb2005f77abe5ffa3437a57c9 not found: ID does not exist" containerID="40d777077449038396f83b11cabd34756f7c474eb2005f77abe5ffa3437a57c9"
Sep 10 17:50:19.894412 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:19.894374    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40d777077449038396f83b11cabd34756f7c474eb2005f77abe5ffa3437a57c9"} err="failed to get container status \"40d777077449038396f83b11cabd34756f7c474eb2005f77abe5ffa3437a57c9\": rpc error: code = NotFound desc = could not find container \"40d777077449038396f83b11cabd34756f7c474eb2005f77abe5ffa3437a57c9\": container with ID starting with 40d777077449038396f83b11cabd34756f7c474eb2005f77abe5ffa3437a57c9 not found: ID does not exist"
Sep 10 17:50:19.911666 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:19.911620    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-0ae95d09-openshift-default-97ff7856b-6pj6l"]
Sep 10 17:50:19.918764 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:19.918742    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-0ae95d09-openshift-default-97ff7856b-6pj6l"]
Sep 10 17:50:19.944606 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:19.944554    2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea7e1957-8fba-4a60-95ce-3f241a43f2df-workload-certs\") on node \"ip-10-0-129-122.ec2.internal\" DevicePath \"\""
Sep 10 17:50:19.944606 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:19.944600    2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea7e1957-8fba-4a60-95ce-3f241a43f2df-istio-podinfo\") on node \"ip-10-0-129-122.ec2.internal\" DevicePath \"\""
Sep 10 17:50:19.944783 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:19.944612    2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea7e1957-8fba-4a60-95ce-3f241a43f2df-istio-token\") on node \"ip-10-0-129-122.ec2.internal\" DevicePath \"\""
Sep 10 17:50:19.944783 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:19.944627    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8tfst\" (UniqueName: \"kubernetes.io/projected/ea7e1957-8fba-4a60-95ce-3f241a43f2df-kube-api-access-8tfst\") on node \"ip-10-0-129-122.ec2.internal\" DevicePath \"\""
Sep 10 17:50:19.944783 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:19.944639    2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea7e1957-8fba-4a60-95ce-3f241a43f2df-istio-envoy\") on node \"ip-10-0-129-122.ec2.internal\" DevicePath \"\""
Sep 10 17:50:20.248073 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:20.248042    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4f7356a3-18ba-4f2d-85aa-fe6001286875-maas-api-tls\") pod \"maas-api-e2e-derive-d2a8a5f7-7977bddcb6-q57v5\" (UID: \"4f7356a3-18ba-4f2d-85aa-fe6001286875\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-d2a8a5f7-7977bddcb6-q57v5"
Sep 10 17:50:20.250601 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:20.250547    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4f7356a3-18ba-4f2d-85aa-fe6001286875-maas-api-tls\") pod \"maas-api-e2e-derive-d2a8a5f7-7977bddcb6-q57v5\" (UID: \"4f7356a3-18ba-4f2d-85aa-fe6001286875\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-d2a8a5f7-7977bddcb6-q57v5"
Sep 10 17:50:20.345409 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:20.345382    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-d2a8a5f7-6c985b7f7-dk4ql"]
Sep 10 17:50:20.345779 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:20.345761    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea7e1957-8fba-4a60-95ce-3f241a43f2df" containerName="istio-proxy"
Sep 10 17:50:20.345832 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:20.345784    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea7e1957-8fba-4a60-95ce-3f241a43f2df" containerName="istio-proxy"
Sep 10 17:50:20.345832 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:20.345792    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea7e1957-8fba-4a60-95ce-3f241a43f2df" containerName="istio-proxy"
Sep 10 17:50:20.345832 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:20.345797    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea7e1957-8fba-4a60-95ce-3f241a43f2df" containerName="istio-proxy"
Sep 10 17:50:20.345918 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:20.345860    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="ea7e1957-8fba-4a60-95ce-3f241a43f2df" containerName="istio-proxy"
Sep 10 17:50:20.345918 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:20.345869    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="ea7e1957-8fba-4a60-95ce-3f241a43f2df" containerName="istio-proxy"
Sep 10 17:50:20.350316 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:20.350295    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-d2a8a5f7-6c985b7f7-dk4ql"
Sep 10 17:50:20.353023 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:20.353002    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-d2a8a5f7-dockercfg-l6qtw\""
Sep 10 17:50:20.353266 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:20.353252    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-d2a8a5f7\""
Sep 10 17:50:20.359090 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:20.359068    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-d2a8a5f7-6c985b7f7-dk4ql"]
Sep 10 17:50:20.449726 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:20.449693    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xft5r\" (UniqueName: \"kubernetes.io/projected/84aa4afe-5cf4-4116-9a68-b69d7fd24567-kube-api-access-xft5r\") pod \"payload-processing-e2e-derive-d2a8a5f7-6c985b7f7-dk4ql\" (UID: \"84aa4afe-5cf4-4116-9a68-b69d7fd24567\") " pod="openshift-ingress/payload-processing-e2e-derive-d2a8a5f7-6c985b7f7-dk4ql"
Sep 10 17:50:20.449888 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:20.449746    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84aa4afe-5cf4-4116-9a68-b69d7fd24567-plugins-config-volume\") pod \"payload-processing-e2e-derive-d2a8a5f7-6c985b7f7-dk4ql\" (UID: \"84aa4afe-5cf4-4116-9a68-b69d7fd24567\") " pod="openshift-ingress/payload-processing-e2e-derive-d2a8a5f7-6c985b7f7-dk4ql"
Sep 10 17:50:20.475679 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:20.475648    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ea7e1957-8fba-4a60-95ce-3f241a43f2df" path="/var/lib/kubelet/pods/ea7e1957-8fba-4a60-95ce-3f241a43f2df/volumes"
Sep 10 17:50:20.495728 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:20.495705    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-d2a8a5f7-7977bddcb6-q57v5"
Sep 10 17:50:20.552146 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:20.552102    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84aa4afe-5cf4-4116-9a68-b69d7fd24567-plugins-config-volume\") pod \"payload-processing-e2e-derive-d2a8a5f7-6c985b7f7-dk4ql\" (UID: \"84aa4afe-5cf4-4116-9a68-b69d7fd24567\") " pod="openshift-ingress/payload-processing-e2e-derive-d2a8a5f7-6c985b7f7-dk4ql"
Sep 10 17:50:20.552512 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:20.552239    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xft5r\" (UniqueName: \"kubernetes.io/projected/84aa4afe-5cf4-4116-9a68-b69d7fd24567-kube-api-access-xft5r\") pod \"payload-processing-e2e-derive-d2a8a5f7-6c985b7f7-dk4ql\" (UID: \"84aa4afe-5cf4-4116-9a68-b69d7fd24567\") " pod="openshift-ingress/payload-processing-e2e-derive-d2a8a5f7-6c985b7f7-dk4ql"
Sep 10 17:50:20.553450 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:20.553420    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84aa4afe-5cf4-4116-9a68-b69d7fd24567-plugins-config-volume\") pod \"payload-processing-e2e-derive-d2a8a5f7-6c985b7f7-dk4ql\" (UID: \"84aa4afe-5cf4-4116-9a68-b69d7fd24567\") " pod="openshift-ingress/payload-processing-e2e-derive-d2a8a5f7-6c985b7f7-dk4ql"
Sep 10 17:50:20.561548 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:20.561526    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xft5r\" (UniqueName: \"kubernetes.io/projected/84aa4afe-5cf4-4116-9a68-b69d7fd24567-kube-api-access-xft5r\") pod \"payload-processing-e2e-derive-d2a8a5f7-6c985b7f7-dk4ql\" (UID: \"84aa4afe-5cf4-4116-9a68-b69d7fd24567\") " pod="openshift-ingress/payload-processing-e2e-derive-d2a8a5f7-6c985b7f7-dk4ql"
Sep 10 17:50:20.660786 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:20.660754    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-d2a8a5f7-6c985b7f7-dk4ql"
Sep 10 17:50:20.765369 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:20.765344    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-d2a8a5f7-7977bddcb6-q57v5"]
Sep 10 17:50:20.767419 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:50:20.767393    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f7356a3_18ba_4f2d_85aa_fe6001286875.slice/crio-f25240fcec3a6514a739d1fad725dc7c24335c95949c392379a7897887ecece6 WatchSource:0}: Error finding container f25240fcec3a6514a739d1fad725dc7c24335c95949c392379a7897887ecece6: Status 404 returned error can't find the container with id f25240fcec3a6514a739d1fad725dc7c24335c95949c392379a7897887ecece6
Sep 10 17:50:20.883856 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:20.883831    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-d2a8a5f7-7977bddcb6-q57v5" event={"ID":"4f7356a3-18ba-4f2d-85aa-fe6001286875","Type":"ContainerStarted","Data":"f25240fcec3a6514a739d1fad725dc7c24335c95949c392379a7897887ecece6"}
Sep 10 17:50:20.936599 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:20.936552    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-d2a8a5f7-6c985b7f7-dk4ql"]
Sep 10 17:50:20.938330 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:50:20.938296    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84aa4afe_5cf4_4116_9a68_b69d7fd24567.slice/crio-d3a86ee53d0fff0ecdbb0071b399bef0d934e3052b2dd2cf1edf0c9526e3757c WatchSource:0}: Error finding container d3a86ee53d0fff0ecdbb0071b399bef0d934e3052b2dd2cf1edf0c9526e3757c: Status 404 returned error can't find the container with id d3a86ee53d0fff0ecdbb0071b399bef0d934e3052b2dd2cf1edf0c9526e3757c
Sep 10 17:50:21.890320 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:21.890287    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-d2a8a5f7-7977bddcb6-q57v5" event={"ID":"4f7356a3-18ba-4f2d-85aa-fe6001286875","Type":"ContainerStarted","Data":"d2a54bd9b13a277c516b38894db86f98efb160196c9719136bbac47c1f503168"}
Sep 10 17:50:21.890774 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:21.890346    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-d2a8a5f7-7977bddcb6-q57v5"
Sep 10 17:50:21.891852 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:21.891831    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-d2a8a5f7-6c985b7f7-dk4ql" event={"ID":"84aa4afe-5cf4-4116-9a68-b69d7fd24567","Type":"ContainerStarted","Data":"49998fe54bf99f6c847a21c3899d5d1b010642c8c22ac7f86537e1caeff3bfdb"}
Sep 10 17:50:21.891921 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:21.891876    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-d2a8a5f7-6c985b7f7-dk4ql"
Sep 10 17:50:21.891921 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:21.891896    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-d2a8a5f7-6c985b7f7-dk4ql" event={"ID":"84aa4afe-5cf4-4116-9a68-b69d7fd24567","Type":"ContainerStarted","Data":"d3a86ee53d0fff0ecdbb0071b399bef0d934e3052b2dd2cf1edf0c9526e3757c"}
Sep 10 17:50:21.911188 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:21.911150    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-d2a8a5f7-7977bddcb6-q57v5" podStartSLOduration=2.5399843239999997 podStartE2EDuration="2.91113765s" totalImagesPullingTime="371.153326ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:50:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:50:21.910371908 +0000 UTC m=+865.910201973" watchObservedRunningTime="2026-09-10 17:50:21.91113765 +0000 UTC m=+865.910967716"
Sep 10 17:50:21.930356 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:21.930311    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-d2a8a5f7-6c985b7f7-dk4ql" podStartSLOduration=1.9302969380000001 podStartE2EDuration="1.930296938s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:50:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:50:21.928380898 +0000 UTC m=+865.928210966" watchObservedRunningTime="2026-09-10 17:50:21.930296938 +0000 UTC m=+865.930127040"
Sep 10 17:50:27.902695 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:27.902662    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-d2a8a5f7-7977bddcb6-q57v5"
Sep 10 17:50:32.899164 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:32.899131    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-d2a8a5f7-6c985b7f7-dk4ql"
Sep 10 17:50:37.978512 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:37.978471    2579 generic.go:356] "Generic (PLEG): container finished" podID="a1b2e0c7-18df-4262-9436-afef8fcc7095" containerID="08a338d273e1fbc1db02e38ec807e8afa058e5094c62007e0972359c1393d179" exitCode=0
Sep 10 17:50:37.978930 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:37.978544    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-d2a8a5f7-openshift-default-8bbc6c576-d8dw4" event={"ID":"a1b2e0c7-18df-4262-9436-afef8fcc7095","Type":"ContainerDied","Data":"08a338d273e1fbc1db02e38ec807e8afa058e5094c62007e0972359c1393d179"}
Sep 10 17:50:37.978973 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:37.978929    2579 scope.go:117] "RemoveContainer" containerID="08a338d273e1fbc1db02e38ec807e8afa058e5094c62007e0972359c1393d179"
Sep 10 17:50:37.979937 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:37.979910    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 10 17:50:37.980041 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:37.979975    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 10 17:50:37.980041 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:37.980006    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 10 17:50:38.994935 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:38.994060    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-d2a8a5f7-openshift-default-8bbc6c576-d8dw4" event={"ID":"a1b2e0c7-18df-4262-9436-afef8fcc7095","Type":"ContainerStarted","Data":"4768610b6552325d6de3ab5f51a16e9e0715faae30a83e4edd24a30d6f57404d"}
Sep 10 17:50:39.642675 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:39.642638    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-d2a8a5f7-openshift-default-8bbc6c576-d8dw4"
Sep 10 17:50:39.642905 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:39.642883    2579 patch_prober.go:28] interesting pod/e2e-derive-d2a8a5f7-openshift-default-8bbc6c576-d8dw4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body=
Sep 10 17:50:39.642986 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:39.642938    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-d2a8a5f7-openshift-default-8bbc6c576-d8dw4" podUID="a1b2e0c7-18df-4262-9436-afef8fcc7095" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused"
Sep 10 17:50:40.642822 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:40.642785    2579 patch_prober.go:28] interesting pod/e2e-derive-d2a8a5f7-openshift-default-8bbc6c576-d8dw4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body=
Sep 10 17:50:40.643198 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:40.642836    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-d2a8a5f7-openshift-default-8bbc6c576-d8dw4" podUID="a1b2e0c7-18df-4262-9436-afef8fcc7095" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused"
Sep 10 17:50:41.642024 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:41.641990    2579 patch_prober.go:28] interesting pod/e2e-derive-d2a8a5f7-openshift-default-8bbc6c576-d8dw4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body=
Sep 10 17:50:41.642196 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:41.642044    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-d2a8a5f7-openshift-default-8bbc6c576-d8dw4" podUID="a1b2e0c7-18df-4262-9436-afef8fcc7095" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused"
Sep 10 17:50:42.642286 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:42.642250    2579 patch_prober.go:28] interesting pod/e2e-derive-d2a8a5f7-openshift-default-8bbc6c576-d8dw4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body=
Sep 10 17:50:42.642685 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:42.642305    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-d2a8a5f7-openshift-default-8bbc6c576-d8dw4" podUID="a1b2e0c7-18df-4262-9436-afef8fcc7095" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused"
Sep 10 17:50:43.642451 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:43.642408    2579 patch_prober.go:28] interesting pod/e2e-derive-d2a8a5f7-openshift-default-8bbc6c576-d8dw4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body=
Sep 10 17:50:43.642957 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:43.642474    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-d2a8a5f7-openshift-default-8bbc6c576-d8dw4" podUID="a1b2e0c7-18df-4262-9436-afef8fcc7095" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused"
Sep 10 17:50:44.646401 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:44.646371    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-d2a8a5f7-openshift-default-8bbc6c576-d8dw4"
Sep 10 17:50:44.646828 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:44.646612    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-d2a8a5f7-openshift-default-8bbc6c576-d8dw4"
Sep 10 17:50:44.647409 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:44.647394    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-d2a8a5f7-openshift-default-8bbc6c576-d8dw4"
Sep 10 17:50:50.096553 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:50.096472    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-d2a8a5f7-7977bddcb6-q57v5"]
Sep 10 17:50:50.096963 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:50.096800    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-d2a8a5f7-7977bddcb6-q57v5" podUID="4f7356a3-18ba-4f2d-85aa-fe6001286875" containerName="maas-api" containerID="cri-o://d2a54bd9b13a277c516b38894db86f98efb160196c9719136bbac47c1f503168" gracePeriod=30
Sep 10 17:50:50.254851 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:50.253134    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-d2a8a5f7-6c985b7f7-dk4ql"]
Sep 10 17:50:50.254851 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:50.253379    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-d2a8a5f7-6c985b7f7-dk4ql" podUID="84aa4afe-5cf4-4116-9a68-b69d7fd24567" containerName="payload-processing" containerID="cri-o://49998fe54bf99f6c847a21c3899d5d1b010642c8c22ac7f86537e1caeff3bfdb" gracePeriod=30
Sep 10 17:50:50.380531 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:50.380501    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-d2a8a5f7-7977bddcb6-q57v5"
Sep 10 17:50:50.417796 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:50.417756    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f7356a3-18ba-4f2d-85aa-fe6001286875-openshift-service-ca\") pod \"4f7356a3-18ba-4f2d-85aa-fe6001286875\" (UID: \"4f7356a3-18ba-4f2d-85aa-fe6001286875\") "
Sep 10 17:50:50.417987 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:50.417833    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qmpdd\" (UniqueName: \"kubernetes.io/projected/4f7356a3-18ba-4f2d-85aa-fe6001286875-kube-api-access-qmpdd\") pod \"4f7356a3-18ba-4f2d-85aa-fe6001286875\" (UID: \"4f7356a3-18ba-4f2d-85aa-fe6001286875\") "
Sep 10 17:50:50.417987 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:50.417870    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4f7356a3-18ba-4f2d-85aa-fe6001286875-maas-api-tls\") pod \"4f7356a3-18ba-4f2d-85aa-fe6001286875\" (UID: \"4f7356a3-18ba-4f2d-85aa-fe6001286875\") "
Sep 10 17:50:50.417987 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:50.417899    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f7356a3-18ba-4f2d-85aa-fe6001286875-metrics-certs\") pod \"4f7356a3-18ba-4f2d-85aa-fe6001286875\" (UID: \"4f7356a3-18ba-4f2d-85aa-fe6001286875\") "
Sep 10 17:50:50.418994 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:50.418839    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f7356a3-18ba-4f2d-85aa-fe6001286875-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "4f7356a3-18ba-4f2d-85aa-fe6001286875" (UID: "4f7356a3-18ba-4f2d-85aa-fe6001286875"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 17:50:50.420425 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:50.420392    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f7356a3-18ba-4f2d-85aa-fe6001286875-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "4f7356a3-18ba-4f2d-85aa-fe6001286875" (UID: "4f7356a3-18ba-4f2d-85aa-fe6001286875"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 17:50:50.420654 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:50.420592    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f7356a3-18ba-4f2d-85aa-fe6001286875-kube-api-access-qmpdd" (OuterVolumeSpecName: "kube-api-access-qmpdd") pod "4f7356a3-18ba-4f2d-85aa-fe6001286875" (UID: "4f7356a3-18ba-4f2d-85aa-fe6001286875"). InnerVolumeSpecName "kube-api-access-qmpdd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:50:50.421540 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:50.421514    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f7356a3-18ba-4f2d-85aa-fe6001286875-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "4f7356a3-18ba-4f2d-85aa-fe6001286875" (UID: "4f7356a3-18ba-4f2d-85aa-fe6001286875"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 17:50:50.513011 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:50.512986    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-d2a8a5f7-6c985b7f7-dk4ql"
Sep 10 17:50:50.519152 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:50.519128    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f7356a3-18ba-4f2d-85aa-fe6001286875-openshift-service-ca\") on node \"ip-10-0-129-122.ec2.internal\" DevicePath \"\""
Sep 10 17:50:50.519248 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:50.519157    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qmpdd\" (UniqueName: \"kubernetes.io/projected/4f7356a3-18ba-4f2d-85aa-fe6001286875-kube-api-access-qmpdd\") on node \"ip-10-0-129-122.ec2.internal\" DevicePath \"\""
Sep 10 17:50:50.519248 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:50.519172    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4f7356a3-18ba-4f2d-85aa-fe6001286875-maas-api-tls\") on node \"ip-10-0-129-122.ec2.internal\" DevicePath \"\""
Sep 10 17:50:50.519248 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:50.519186    2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f7356a3-18ba-4f2d-85aa-fe6001286875-metrics-certs\") on node \"ip-10-0-129-122.ec2.internal\" DevicePath \"\""
Sep 10 17:50:50.619965 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:50.619886    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xft5r\" (UniqueName: \"kubernetes.io/projected/84aa4afe-5cf4-4116-9a68-b69d7fd24567-kube-api-access-xft5r\") pod \"84aa4afe-5cf4-4116-9a68-b69d7fd24567\" (UID: \"84aa4afe-5cf4-4116-9a68-b69d7fd24567\") "
Sep 10 17:50:50.619965 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:50.619955    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84aa4afe-5cf4-4116-9a68-b69d7fd24567-plugins-config-volume\") pod \"84aa4afe-5cf4-4116-9a68-b69d7fd24567\" (UID: \"84aa4afe-5cf4-4116-9a68-b69d7fd24567\") "
Sep 10 17:50:50.620296 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:50.620269    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84aa4afe-5cf4-4116-9a68-b69d7fd24567-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "84aa4afe-5cf4-4116-9a68-b69d7fd24567" (UID: "84aa4afe-5cf4-4116-9a68-b69d7fd24567"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 17:50:50.622141 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:50.622120    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84aa4afe-5cf4-4116-9a68-b69d7fd24567-kube-api-access-xft5r" (OuterVolumeSpecName: "kube-api-access-xft5r") pod "84aa4afe-5cf4-4116-9a68-b69d7fd24567" (UID: "84aa4afe-5cf4-4116-9a68-b69d7fd24567"). InnerVolumeSpecName "kube-api-access-xft5r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:50:50.686757 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:50.686719    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-d2a8a5f7-openshift-default-8bbc6c576-d8dw4"]
Sep 10 17:50:50.686960 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:50.686940    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-d2a8a5f7-openshift-default-8bbc6c576-d8dw4" podUID="a1b2e0c7-18df-4262-9436-afef8fcc7095" containerName="istio-proxy" containerID="cri-o://4768610b6552325d6de3ab5f51a16e9e0715faae30a83e4edd24a30d6f57404d" gracePeriod=30
Sep 10 17:50:50.721324 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:50.721299    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84aa4afe-5cf4-4116-9a68-b69d7fd24567-plugins-config-volume\") on node \"ip-10-0-129-122.ec2.internal\" DevicePath \"\""
Sep 10 17:50:50.721324 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:50.721320    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xft5r\" (UniqueName: \"kubernetes.io/projected/84aa4afe-5cf4-4116-9a68-b69d7fd24567-kube-api-access-xft5r\") on node \"ip-10-0-129-122.ec2.internal\" DevicePath \"\""
Sep 10 17:50:51.057395 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:51.057356    2579 generic.go:356] "Generic (PLEG): container finished" podID="4f7356a3-18ba-4f2d-85aa-fe6001286875" containerID="d2a54bd9b13a277c516b38894db86f98efb160196c9719136bbac47c1f503168" exitCode=0
Sep 10 17:50:51.057597 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:51.057416    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-d2a8a5f7-7977bddcb6-q57v5"
Sep 10 17:50:51.057597 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:51.057451    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-d2a8a5f7-7977bddcb6-q57v5" event={"ID":"4f7356a3-18ba-4f2d-85aa-fe6001286875","Type":"ContainerDied","Data":"d2a54bd9b13a277c516b38894db86f98efb160196c9719136bbac47c1f503168"}
Sep 10 17:50:51.057597 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:51.057505    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-d2a8a5f7-7977bddcb6-q57v5" event={"ID":"4f7356a3-18ba-4f2d-85aa-fe6001286875","Type":"ContainerDied","Data":"f25240fcec3a6514a739d1fad725dc7c24335c95949c392379a7897887ecece6"}
Sep 10 17:50:51.057597 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:51.057523    2579 scope.go:117] "RemoveContainer" containerID="d2a54bd9b13a277c516b38894db86f98efb160196c9719136bbac47c1f503168"
Sep 10 17:50:51.059063 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:51.059041    2579 generic.go:356] "Generic (PLEG): container finished" podID="84aa4afe-5cf4-4116-9a68-b69d7fd24567" containerID="49998fe54bf99f6c847a21c3899d5d1b010642c8c22ac7f86537e1caeff3bfdb" exitCode=0
Sep 10 17:50:51.059199 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:51.059089    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-d2a8a5f7-6c985b7f7-dk4ql"
Sep 10 17:50:51.059317 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:51.059101    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-d2a8a5f7-6c985b7f7-dk4ql" event={"ID":"84aa4afe-5cf4-4116-9a68-b69d7fd24567","Type":"ContainerDied","Data":"49998fe54bf99f6c847a21c3899d5d1b010642c8c22ac7f86537e1caeff3bfdb"}
Sep 10 17:50:51.059363 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:51.059340    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-d2a8a5f7-6c985b7f7-dk4ql" event={"ID":"84aa4afe-5cf4-4116-9a68-b69d7fd24567","Type":"ContainerDied","Data":"d3a86ee53d0fff0ecdbb0071b399bef0d934e3052b2dd2cf1edf0c9526e3757c"}
Sep 10 17:50:51.067030 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:51.067011    2579 scope.go:117] "RemoveContainer" containerID="d2a54bd9b13a277c516b38894db86f98efb160196c9719136bbac47c1f503168"
Sep 10 17:50:51.067281 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:50:51.067265    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2a54bd9b13a277c516b38894db86f98efb160196c9719136bbac47c1f503168\": container with ID starting with d2a54bd9b13a277c516b38894db86f98efb160196c9719136bbac47c1f503168 not found: ID does not exist" containerID="d2a54bd9b13a277c516b38894db86f98efb160196c9719136bbac47c1f503168"
Sep 10 17:50:51.067333 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:51.067289    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2a54bd9b13a277c516b38894db86f98efb160196c9719136bbac47c1f503168"} err="failed to get container status \"d2a54bd9b13a277c516b38894db86f98efb160196c9719136bbac47c1f503168\": rpc error: code = NotFound desc = could not find container \"d2a54bd9b13a277c516b38894db86f98efb160196c9719136bbac47c1f503168\": container with ID starting with d2a54bd9b13a277c516b38894db86f98efb160196c9719136bbac47c1f503168 not found: ID does not exist"
Sep 10 17:50:51.067333 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:51.067304    2579 scope.go:117] "RemoveContainer" containerID="49998fe54bf99f6c847a21c3899d5d1b010642c8c22ac7f86537e1caeff3bfdb"
Sep 10 17:50:51.076008 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:51.075992    2579 scope.go:117] "RemoveContainer" containerID="49998fe54bf99f6c847a21c3899d5d1b010642c8c22ac7f86537e1caeff3bfdb"
Sep 10 17:50:51.076237 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:50:51.076217    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49998fe54bf99f6c847a21c3899d5d1b010642c8c22ac7f86537e1caeff3bfdb\": container with ID starting with 49998fe54bf99f6c847a21c3899d5d1b010642c8c22ac7f86537e1caeff3bfdb not found: ID does not exist" containerID="49998fe54bf99f6c847a21c3899d5d1b010642c8c22ac7f86537e1caeff3bfdb"
Sep 10 17:50:51.076279 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:51.076244    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49998fe54bf99f6c847a21c3899d5d1b010642c8c22ac7f86537e1caeff3bfdb"} err="failed to get container status \"49998fe54bf99f6c847a21c3899d5d1b010642c8c22ac7f86537e1caeff3bfdb\": rpc error: code = NotFound desc = could not find container \"49998fe54bf99f6c847a21c3899d5d1b010642c8c22ac7f86537e1caeff3bfdb\": container with ID starting with 49998fe54bf99f6c847a21c3899d5d1b010642c8c22ac7f86537e1caeff3bfdb not found: ID does not exist"
Sep 10 17:50:51.081005 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:51.080984    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-d2a8a5f7-7977bddcb6-q57v5"]
Sep 10 17:50:51.085908 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:51.085885    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-d2a8a5f7-7977bddcb6-q57v5"]
Sep 10 17:50:51.099348 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:51.099324    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-d2a8a5f7-6c985b7f7-dk4ql"]
Sep 10 17:50:51.101909 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:51.101881    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-d2a8a5f7-6c985b7f7-dk4ql"]
Sep 10 17:50:52.474773 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:52.474739    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4f7356a3-18ba-4f2d-85aa-fe6001286875" path="/var/lib/kubelet/pods/4f7356a3-18ba-4f2d-85aa-fe6001286875/volumes"
Sep 10 17:50:52.475153 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:52.475114    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="84aa4afe-5cf4-4116-9a68-b69d7fd24567" path="/var/lib/kubelet/pods/84aa4afe-5cf4-4116-9a68-b69d7fd24567/volumes"
Sep 10 17:50:55.109114 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:55.109083    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-c7995b-57c8d55974-bm5wh"]
Sep 10 17:50:55.109631 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:55.109613    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="84aa4afe-5cf4-4116-9a68-b69d7fd24567" containerName="payload-processing"
Sep 10 17:50:55.109720 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:55.109633    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="84aa4afe-5cf4-4116-9a68-b69d7fd24567" containerName="payload-processing"
Sep 10 17:50:55.109720 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:55.109671    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f7356a3-18ba-4f2d-85aa-fe6001286875" containerName="maas-api"
Sep 10 17:50:55.109720 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:55.109680    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f7356a3-18ba-4f2d-85aa-fe6001286875" containerName="maas-api"
Sep 10 17:50:55.109905 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:55.109767    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="4f7356a3-18ba-4f2d-85aa-fe6001286875" containerName="maas-api"
Sep 10 17:50:55.109905 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:55.109792    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="84aa4afe-5cf4-4116-9a68-b69d7fd24567" containerName="payload-processing"
Sep 10 17:50:55.111984 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:55.111965    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c7995b-57c8d55974-bm5wh"
Sep 10 17:50:55.114622 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:55.114602    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-explicit-c7995b-serving-cert\""
Sep 10 17:50:55.128703 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:55.128682    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-c7995b-57c8d55974-bm5wh"]
Sep 10 17:50:55.163307 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:55.163277    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cde456ae-00a2-4c52-88b6-aea1d1d650e2-metrics-certs\") pod \"maas-api-e2e-explicit-c7995b-57c8d55974-bm5wh\" (UID: \"cde456ae-00a2-4c52-88b6-aea1d1d650e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c7995b-57c8d55974-bm5wh"
Sep 10 17:50:55.163458 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:55.163395    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cde456ae-00a2-4c52-88b6-aea1d1d650e2-maas-api-tls\") pod \"maas-api-e2e-explicit-c7995b-57c8d55974-bm5wh\" (UID: \"cde456ae-00a2-4c52-88b6-aea1d1d650e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c7995b-57c8d55974-bm5wh"
Sep 10 17:50:55.163519 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:55.163497    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6d8gr\" (UniqueName: \"kubernetes.io/projected/cde456ae-00a2-4c52-88b6-aea1d1d650e2-kube-api-access-6d8gr\") pod \"maas-api-e2e-explicit-c7995b-57c8d55974-bm5wh\" (UID: \"cde456ae-00a2-4c52-88b6-aea1d1d650e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c7995b-57c8d55974-bm5wh"
Sep 10 17:50:55.163615 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:55.163544    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cde456ae-00a2-4c52-88b6-aea1d1d650e2-openshift-service-ca\") pod \"maas-api-e2e-explicit-c7995b-57c8d55974-bm5wh\" (UID: \"cde456ae-00a2-4c52-88b6-aea1d1d650e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c7995b-57c8d55974-bm5wh"
Sep 10 17:50:55.264638 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:55.264600    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cde456ae-00a2-4c52-88b6-aea1d1d650e2-metrics-certs\") pod \"maas-api-e2e-explicit-c7995b-57c8d55974-bm5wh\" (UID: \"cde456ae-00a2-4c52-88b6-aea1d1d650e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c7995b-57c8d55974-bm5wh"
Sep 10 17:50:55.264823 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:55.264653    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cde456ae-00a2-4c52-88b6-aea1d1d650e2-maas-api-tls\") pod \"maas-api-e2e-explicit-c7995b-57c8d55974-bm5wh\" (UID: \"cde456ae-00a2-4c52-88b6-aea1d1d650e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c7995b-57c8d55974-bm5wh"
Sep 10 17:50:55.264823 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:55.264729    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6d8gr\" (UniqueName: \"kubernetes.io/projected/cde456ae-00a2-4c52-88b6-aea1d1d650e2-kube-api-access-6d8gr\") pod \"maas-api-e2e-explicit-c7995b-57c8d55974-bm5wh\" (UID: \"cde456ae-00a2-4c52-88b6-aea1d1d650e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c7995b-57c8d55974-bm5wh"
Sep 10 17:50:55.264823 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:55.264767    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cde456ae-00a2-4c52-88b6-aea1d1d650e2-openshift-service-ca\") pod \"maas-api-e2e-explicit-c7995b-57c8d55974-bm5wh\" (UID: \"cde456ae-00a2-4c52-88b6-aea1d1d650e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c7995b-57c8d55974-bm5wh"
Sep 10 17:50:55.264823 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:50:55.264791    2579 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-explicit-c7995b-serving-cert: secret "maas-api-e2e-explicit-c7995b-serving-cert" not found
Sep 10 17:50:55.265044 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:50:55.264874    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cde456ae-00a2-4c52-88b6-aea1d1d650e2-maas-api-tls podName:cde456ae-00a2-4c52-88b6-aea1d1d650e2 nodeName:}" failed. No retries permitted until 2026-09-10 17:50:55.764851798 +0000 UTC m=+899.764681851 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/cde456ae-00a2-4c52-88b6-aea1d1d650e2-maas-api-tls") pod "maas-api-e2e-explicit-c7995b-57c8d55974-bm5wh" (UID: "cde456ae-00a2-4c52-88b6-aea1d1d650e2") : secret "maas-api-e2e-explicit-c7995b-serving-cert" not found
Sep 10 17:50:55.265443 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:55.265420    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cde456ae-00a2-4c52-88b6-aea1d1d650e2-openshift-service-ca\") pod \"maas-api-e2e-explicit-c7995b-57c8d55974-bm5wh\" (UID: \"cde456ae-00a2-4c52-88b6-aea1d1d650e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c7995b-57c8d55974-bm5wh"
Sep 10 17:50:55.267341 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:55.267321    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cde456ae-00a2-4c52-88b6-aea1d1d650e2-metrics-certs\") pod \"maas-api-e2e-explicit-c7995b-57c8d55974-bm5wh\" (UID: \"cde456ae-00a2-4c52-88b6-aea1d1d650e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c7995b-57c8d55974-bm5wh"
Sep 10 17:50:55.274809 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:55.274783    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6d8gr\" (UniqueName: \"kubernetes.io/projected/cde456ae-00a2-4c52-88b6-aea1d1d650e2-kube-api-access-6d8gr\") pod \"maas-api-e2e-explicit-c7995b-57c8d55974-bm5wh\" (UID: \"cde456ae-00a2-4c52-88b6-aea1d1d650e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c7995b-57c8d55974-bm5wh"
Sep 10 17:50:55.748206 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:55.748171    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-c7995b-549947fbd6-9br8r"]
Sep 10 17:50:55.750755 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:55.750737    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-c7995b-549947fbd6-9br8r"
Sep 10 17:50:55.753412 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:55.753394    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-c7995b-dockercfg-rz5zm\""
Sep 10 17:50:55.753490 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:55.753395    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-c7995b\""
Sep 10 17:50:55.764926 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:55.764906    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-c7995b-549947fbd6-9br8r"]
Sep 10 17:50:55.770436 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:55.770407    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cde456ae-00a2-4c52-88b6-aea1d1d650e2-maas-api-tls\") pod \"maas-api-e2e-explicit-c7995b-57c8d55974-bm5wh\" (UID: \"cde456ae-00a2-4c52-88b6-aea1d1d650e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c7995b-57c8d55974-bm5wh"
Sep 10 17:50:55.772930 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:55.772911    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cde456ae-00a2-4c52-88b6-aea1d1d650e2-maas-api-tls\") pod \"maas-api-e2e-explicit-c7995b-57c8d55974-bm5wh\" (UID: \"cde456ae-00a2-4c52-88b6-aea1d1d650e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c7995b-57c8d55974-bm5wh"
Sep 10 17:50:55.871283 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:55.871247    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7cb5b3f6-2d76-4bfc-8641-9bfee57c0354-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-c7995b-549947fbd6-9br8r\" (UID: \"7cb5b3f6-2d76-4bfc-8641-9bfee57c0354\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-c7995b-549947fbd6-9br8r"
Sep 10 17:50:55.871446 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:55.871331    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6cx6\" (UniqueName: \"kubernetes.io/projected/7cb5b3f6-2d76-4bfc-8641-9bfee57c0354-kube-api-access-q6cx6\") pod \"payload-pre-processing-e2e-explicit-c7995b-549947fbd6-9br8r\" (UID: \"7cb5b3f6-2d76-4bfc-8641-9bfee57c0354\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-c7995b-549947fbd6-9br8r"
Sep 10 17:50:55.932922 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:55.932898    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-d2a8a5f7-openshift-default-8bbc6c576-d8dw4"
Sep 10 17:50:55.972575 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:55.972493    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1b2e0c7-18df-4262-9436-afef8fcc7095-istio-token\") pod \"a1b2e0c7-18df-4262-9436-afef8fcc7095\" (UID: \"a1b2e0c7-18df-4262-9436-afef8fcc7095\") "
Sep 10 17:50:55.972575 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:55.972528    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1b2e0c7-18df-4262-9436-afef8fcc7095-istio-envoy\") pod \"a1b2e0c7-18df-4262-9436-afef8fcc7095\" (UID: \"a1b2e0c7-18df-4262-9436-afef8fcc7095\") "
Sep 10 17:50:55.972780 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:55.972600    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1b2e0c7-18df-4262-9436-afef8fcc7095-istiod-ca-cert\") pod \"a1b2e0c7-18df-4262-9436-afef8fcc7095\" (UID: \"a1b2e0c7-18df-4262-9436-afef8fcc7095\") "
Sep 10 17:50:55.972780 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:55.972632    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1b2e0c7-18df-4262-9436-afef8fcc7095-istio-podinfo\") pod \"a1b2e0c7-18df-4262-9436-afef8fcc7095\" (UID: \"a1b2e0c7-18df-4262-9436-afef8fcc7095\") "
Sep 10 17:50:55.972780 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:55.972739    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1b2e0c7-18df-4262-9436-afef8fcc7095-credential-socket\") pod \"a1b2e0c7-18df-4262-9436-afef8fcc7095\" (UID: \"a1b2e0c7-18df-4262-9436-afef8fcc7095\") "
Sep 10 17:50:55.972780 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:55.972775    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1b2e0c7-18df-4262-9436-afef8fcc7095-istio-data\") pod \"a1b2e0c7-18df-4262-9436-afef8fcc7095\" (UID: \"a1b2e0c7-18df-4262-9436-afef8fcc7095\") "
Sep 10 17:50:55.972974 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:55.972831    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2gj8k\" (UniqueName: \"kubernetes.io/projected/a1b2e0c7-18df-4262-9436-afef8fcc7095-kube-api-access-2gj8k\") pod \"a1b2e0c7-18df-4262-9436-afef8fcc7095\" (UID: \"a1b2e0c7-18df-4262-9436-afef8fcc7095\") "
Sep 10 17:50:55.972974 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:55.972870    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1b2e0c7-18df-4262-9436-afef8fcc7095-workload-certs\") pod \"a1b2e0c7-18df-4262-9436-afef8fcc7095\" (UID: \"a1b2e0c7-18df-4262-9436-afef8fcc7095\") "
Sep 10 17:50:55.972974 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:55.972921    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1b2e0c7-18df-4262-9436-afef8fcc7095-workload-socket\") pod \"a1b2e0c7-18df-4262-9436-afef8fcc7095\" (UID: \"a1b2e0c7-18df-4262-9436-afef8fcc7095\") "
Sep 10 17:50:55.973104 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:55.973061    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1b2e0c7-18df-4262-9436-afef8fcc7095-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "a1b2e0c7-18df-4262-9436-afef8fcc7095" (UID: "a1b2e0c7-18df-4262-9436-afef8fcc7095"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 17:50:55.973104 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:55.973092    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7cb5b3f6-2d76-4bfc-8641-9bfee57c0354-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-c7995b-549947fbd6-9br8r\" (UID: \"7cb5b3f6-2d76-4bfc-8641-9bfee57c0354\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-c7995b-549947fbd6-9br8r"
Sep 10 17:50:55.973202 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:55.973175    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6cx6\" (UniqueName: \"kubernetes.io/projected/7cb5b3f6-2d76-4bfc-8641-9bfee57c0354-kube-api-access-q6cx6\") pod \"payload-pre-processing-e2e-explicit-c7995b-549947fbd6-9br8r\" (UID: \"7cb5b3f6-2d76-4bfc-8641-9bfee57c0354\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-c7995b-549947fbd6-9br8r"
Sep 10 17:50:55.973425 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:55.973285    2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1b2e0c7-18df-4262-9436-afef8fcc7095-istiod-ca-cert\") on node \"ip-10-0-129-122.ec2.internal\" DevicePath \"\""
Sep 10 17:50:55.973695 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:55.973555    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1b2e0c7-18df-4262-9436-afef8fcc7095-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "a1b2e0c7-18df-4262-9436-afef8fcc7095" (UID: "a1b2e0c7-18df-4262-9436-afef8fcc7095"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:50:55.973805 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:55.973760    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1b2e0c7-18df-4262-9436-afef8fcc7095-istio-data" (OuterVolumeSpecName: "istio-data") pod "a1b2e0c7-18df-4262-9436-afef8fcc7095" (UID: "a1b2e0c7-18df-4262-9436-afef8fcc7095"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:50:55.974056 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:55.974022    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1b2e0c7-18df-4262-9436-afef8fcc7095-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "a1b2e0c7-18df-4262-9436-afef8fcc7095" (UID: "a1b2e0c7-18df-4262-9436-afef8fcc7095"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:50:55.974553 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:55.974501    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7cb5b3f6-2d76-4bfc-8641-9bfee57c0354-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-c7995b-549947fbd6-9br8r\" (UID: \"7cb5b3f6-2d76-4bfc-8641-9bfee57c0354\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-c7995b-549947fbd6-9br8r"
Sep 10 17:50:55.974979 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:55.974948    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1b2e0c7-18df-4262-9436-afef8fcc7095-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "a1b2e0c7-18df-4262-9436-afef8fcc7095" (UID: "a1b2e0c7-18df-4262-9436-afef8fcc7095"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:50:55.975374 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:55.975353    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1b2e0c7-18df-4262-9436-afef8fcc7095-istio-token" (OuterVolumeSpecName: "istio-token") pod "a1b2e0c7-18df-4262-9436-afef8fcc7095" (UID: "a1b2e0c7-18df-4262-9436-afef8fcc7095"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:50:55.975460 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:55.975369    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a1b2e0c7-18df-4262-9436-afef8fcc7095-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "a1b2e0c7-18df-4262-9436-afef8fcc7095" (UID: "a1b2e0c7-18df-4262-9436-afef8fcc7095"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 10 17:50:55.976088 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:55.976068    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1b2e0c7-18df-4262-9436-afef8fcc7095-kube-api-access-2gj8k" (OuterVolumeSpecName: "kube-api-access-2gj8k") pod "a1b2e0c7-18df-4262-9436-afef8fcc7095" (UID: "a1b2e0c7-18df-4262-9436-afef8fcc7095"). InnerVolumeSpecName "kube-api-access-2gj8k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:50:55.976357 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:55.976335    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1b2e0c7-18df-4262-9436-afef8fcc7095-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "a1b2e0c7-18df-4262-9436-afef8fcc7095" (UID: "a1b2e0c7-18df-4262-9436-afef8fcc7095"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:50:55.982259 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:55.982241    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6cx6\" (UniqueName: \"kubernetes.io/projected/7cb5b3f6-2d76-4bfc-8641-9bfee57c0354-kube-api-access-q6cx6\") pod \"payload-pre-processing-e2e-explicit-c7995b-549947fbd6-9br8r\" (UID: \"7cb5b3f6-2d76-4bfc-8641-9bfee57c0354\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-c7995b-549947fbd6-9br8r"
Sep 10 17:50:56.022710 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:56.022689    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c7995b-57c8d55974-bm5wh"
Sep 10 17:50:56.074815 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:56.074663    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2gj8k\" (UniqueName: \"kubernetes.io/projected/a1b2e0c7-18df-4262-9436-afef8fcc7095-kube-api-access-2gj8k\") on node \"ip-10-0-129-122.ec2.internal\" DevicePath \"\""
Sep 10 17:50:56.074815 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:56.074698    2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1b2e0c7-18df-4262-9436-afef8fcc7095-workload-certs\") on node \"ip-10-0-129-122.ec2.internal\" DevicePath \"\""
Sep 10 17:50:56.074815 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:56.074713    2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1b2e0c7-18df-4262-9436-afef8fcc7095-workload-socket\") on node \"ip-10-0-129-122.ec2.internal\" DevicePath \"\""
Sep 10 17:50:56.074815 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:56.074730    2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1b2e0c7-18df-4262-9436-afef8fcc7095-istio-token\") on node \"ip-10-0-129-122.ec2.internal\" DevicePath \"\""
Sep 10 17:50:56.074815 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:56.074743    2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1b2e0c7-18df-4262-9436-afef8fcc7095-istio-envoy\") on node \"ip-10-0-129-122.ec2.internal\" DevicePath \"\""
Sep 10 17:50:56.074815 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:56.074756    2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1b2e0c7-18df-4262-9436-afef8fcc7095-istio-podinfo\") on node \"ip-10-0-129-122.ec2.internal\" DevicePath \"\""
Sep 10 17:50:56.074815 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:56.074770    2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1b2e0c7-18df-4262-9436-afef8fcc7095-credential-socket\") on node \"ip-10-0-129-122.ec2.internal\" DevicePath \"\""
Sep 10 17:50:56.074815 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:56.074784    2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1b2e0c7-18df-4262-9436-afef8fcc7095-istio-data\") on node \"ip-10-0-129-122.ec2.internal\" DevicePath \"\""
Sep 10 17:50:56.092421 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:56.092390    2579 generic.go:356] "Generic (PLEG): container finished" podID="a1b2e0c7-18df-4262-9436-afef8fcc7095" containerID="4768610b6552325d6de3ab5f51a16e9e0715faae30a83e4edd24a30d6f57404d" exitCode=0
Sep 10 17:50:56.092549 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:56.092446    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-d2a8a5f7-openshift-default-8bbc6c576-d8dw4"
Sep 10 17:50:56.092549 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:56.092452    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-d2a8a5f7-openshift-default-8bbc6c576-d8dw4" event={"ID":"a1b2e0c7-18df-4262-9436-afef8fcc7095","Type":"ContainerDied","Data":"4768610b6552325d6de3ab5f51a16e9e0715faae30a83e4edd24a30d6f57404d"}
Sep 10 17:50:56.092694 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:56.092640    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-d2a8a5f7-openshift-default-8bbc6c576-d8dw4" event={"ID":"a1b2e0c7-18df-4262-9436-afef8fcc7095","Type":"ContainerDied","Data":"0c9079ce8ba9cb2108303846afdd22f13946f524b8a405d4989e8b2ead43ace0"}
Sep 10 17:50:56.092694 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:56.092683    2579 scope.go:117] "RemoveContainer" containerID="4768610b6552325d6de3ab5f51a16e9e0715faae30a83e4edd24a30d6f57404d"
Sep 10 17:50:56.092791 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:56.092760    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-c7995b-549947fbd6-9br8r"
Sep 10 17:50:56.105186 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:56.105157    2579 scope.go:117] "RemoveContainer" containerID="08a338d273e1fbc1db02e38ec807e8afa058e5094c62007e0972359c1393d179"
Sep 10 17:50:56.115911 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:56.115891    2579 scope.go:117] "RemoveContainer" containerID="4768610b6552325d6de3ab5f51a16e9e0715faae30a83e4edd24a30d6f57404d"
Sep 10 17:50:56.116235 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:50:56.116215    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4768610b6552325d6de3ab5f51a16e9e0715faae30a83e4edd24a30d6f57404d\": container with ID starting with 4768610b6552325d6de3ab5f51a16e9e0715faae30a83e4edd24a30d6f57404d not found: ID does not exist" containerID="4768610b6552325d6de3ab5f51a16e9e0715faae30a83e4edd24a30d6f57404d"
Sep 10 17:50:56.116284 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:56.116243    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4768610b6552325d6de3ab5f51a16e9e0715faae30a83e4edd24a30d6f57404d"} err="failed to get container status \"4768610b6552325d6de3ab5f51a16e9e0715faae30a83e4edd24a30d6f57404d\": rpc error: code = NotFound desc = could not find container \"4768610b6552325d6de3ab5f51a16e9e0715faae30a83e4edd24a30d6f57404d\": container with ID starting with 4768610b6552325d6de3ab5f51a16e9e0715faae30a83e4edd24a30d6f57404d not found: ID does not exist"
Sep 10 17:50:56.116284 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:56.116259    2579 scope.go:117] "RemoveContainer" containerID="08a338d273e1fbc1db02e38ec807e8afa058e5094c62007e0972359c1393d179"
Sep 10 17:50:56.116664 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:50:56.116604    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08a338d273e1fbc1db02e38ec807e8afa058e5094c62007e0972359c1393d179\": container with ID starting with 08a338d273e1fbc1db02e38ec807e8afa058e5094c62007e0972359c1393d179 not found: ID does not exist" containerID="08a338d273e1fbc1db02e38ec807e8afa058e5094c62007e0972359c1393d179"
Sep 10 17:50:56.116664 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:56.116639    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08a338d273e1fbc1db02e38ec807e8afa058e5094c62007e0972359c1393d179"} err="failed to get container status \"08a338d273e1fbc1db02e38ec807e8afa058e5094c62007e0972359c1393d179\": rpc error: code = NotFound desc = could not find container \"08a338d273e1fbc1db02e38ec807e8afa058e5094c62007e0972359c1393d179\": container with ID starting with 08a338d273e1fbc1db02e38ec807e8afa058e5094c62007e0972359c1393d179 not found: ID does not exist"
Sep 10 17:50:56.120266 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:56.120243    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-d2a8a5f7-openshift-default-8bbc6c576-d8dw4"]
Sep 10 17:50:56.127309 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:56.127287    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-d2a8a5f7-openshift-default-8bbc6c576-d8dw4"]
Sep 10 17:50:56.295240 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:56.295217    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-c7995b-57c8d55974-bm5wh"]
Sep 10 17:50:56.297051 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:50:56.297026    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcde456ae_00a2_4c52_88b6_aea1d1d650e2.slice/crio-e7f1ca20a8c5b2d9ef9e2b1cab58fd419c0d441d64f30940673e6f641081fb5b WatchSource:0}: Error finding container e7f1ca20a8c5b2d9ef9e2b1cab58fd419c0d441d64f30940673e6f641081fb5b: Status 404 returned error can't find the container with id e7f1ca20a8c5b2d9ef9e2b1cab58fd419c0d441d64f30940673e6f641081fb5b
Sep 10 17:50:56.375181 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:56.375155    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-c7995b-549947fbd6-9br8r"]
Sep 10 17:50:56.377307 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:50:56.377276    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cb5b3f6_2d76_4bfc_8641_9bfee57c0354.slice/crio-c958bb559c2d7234ebf3f3a4d0dba8f4ab09313d5292405268ef317669a082b3 WatchSource:0}: Error finding container c958bb559c2d7234ebf3f3a4d0dba8f4ab09313d5292405268ef317669a082b3: Status 404 returned error can't find the container with id c958bb559c2d7234ebf3f3a4d0dba8f4ab09313d5292405268ef317669a082b3
Sep 10 17:50:56.453067 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:56.453043    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-rb854_97172b14-b84a-476e-89f1-05d3e70df416/console-operator/1.log"
Sep 10 17:50:56.453397 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:56.453375    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-rb854_97172b14-b84a-476e-89f1-05d3e70df416/console-operator/1.log"
Sep 10 17:50:56.477496 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:56.477459    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a1b2e0c7-18df-4262-9436-afef8fcc7095" path="/var/lib/kubelet/pods/a1b2e0c7-18df-4262-9436-afef8fcc7095/volumes"
Sep 10 17:50:57.099998 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:57.099962    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c7995b-57c8d55974-bm5wh" event={"ID":"cde456ae-00a2-4c52-88b6-aea1d1d650e2","Type":"ContainerStarted","Data":"cc5abf184fe2ffb4ceb74e191342bab3dd091681a4dfd45bc5c1d12c556af94a"}
Sep 10 17:50:57.100184 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:57.100015    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c7995b-57c8d55974-bm5wh" event={"ID":"cde456ae-00a2-4c52-88b6-aea1d1d650e2","Type":"ContainerStarted","Data":"e7f1ca20a8c5b2d9ef9e2b1cab58fd419c0d441d64f30940673e6f641081fb5b"}
Sep 10 17:50:57.100184 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:57.100072    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c7995b-57c8d55974-bm5wh"
Sep 10 17:50:57.101792 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:57.101755    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-c7995b-549947fbd6-9br8r" event={"ID":"7cb5b3f6-2d76-4bfc-8641-9bfee57c0354","Type":"ContainerStarted","Data":"5eb405712ff6a176b5ff5f4aff15ed81c477abcea52035645cd12982f58cece3"}
Sep 10 17:50:57.101924 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:57.101807    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-c7995b-549947fbd6-9br8r" event={"ID":"7cb5b3f6-2d76-4bfc-8641-9bfee57c0354","Type":"ContainerStarted","Data":"c958bb559c2d7234ebf3f3a4d0dba8f4ab09313d5292405268ef317669a082b3"}
Sep 10 17:50:57.101924 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:57.101834    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-c7995b-549947fbd6-9br8r"
Sep 10 17:50:57.120803 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:57.120760    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c7995b-57c8d55974-bm5wh" podStartSLOduration=1.701657876 podStartE2EDuration="2.120747199s" totalImagesPullingTime="419.089323ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:50:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:50:57.119003332 +0000 UTC m=+901.118833412" watchObservedRunningTime="2026-09-10 17:50:57.120747199 +0000 UTC m=+901.120577300"
Sep 10 17:50:57.137760 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:50:57.137718    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-explicit-c7995b-549947fbd6-9br8r" podStartSLOduration=2.137705424 podStartE2EDuration="2.137705424s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:50:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:50:57.136384004 +0000 UTC m=+901.136214072" watchObservedRunningTime="2026-09-10 17:50:57.137705424 +0000 UTC m=+901.137535490"
Sep 10 17:51:03.112078 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:51:03.112043    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c7995b-57c8d55974-bm5wh"
Sep 10 17:51:08.109204 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:51:08.109173    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-c7995b-549947fbd6-9br8r"
Sep 10 17:51:09.659281 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:51:09.659235    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-783a5e/test-model-783a5e-kserve-b6dbdcf95-4gwk4"]
Sep 10 17:51:09.659699 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:51:09.659685    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a1b2e0c7-18df-4262-9436-afef8fcc7095" containerName="istio-proxy"
Sep 10 17:51:09.659742 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:51:09.659701    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="a1b2e0c7-18df-4262-9436-afef8fcc7095" containerName="istio-proxy"
Sep 10 17:51:09.659742 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:51:09.659723    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a1b2e0c7-18df-4262-9436-afef8fcc7095" containerName="istio-proxy"
Sep 10 17:51:09.659742 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:51:09.659728    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="a1b2e0c7-18df-4262-9436-afef8fcc7095" containerName="istio-proxy"
Sep 10 17:51:09.659881 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:51:09.659787    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="a1b2e0c7-18df-4262-9436-afef8fcc7095" containerName="istio-proxy"
Sep 10 17:51:09.659881 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:51:09.659795    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="a1b2e0c7-18df-4262-9436-afef8fcc7095" containerName="istio-proxy"
Sep 10 17:51:09.662281 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:51:09.662265    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-783a5e/test-model-783a5e-kserve-b6dbdcf95-4gwk4"
Sep 10 17:51:09.665071 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:51:09.665043    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-783a5e\"/\"test-model-783a5e-kserve-self-signed-certs\""
Sep 10 17:51:09.666263 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:51:09.666240    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-783a5e\"/\"openshift-service-ca.crt\""
Sep 10 17:51:09.666397 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:51:09.666266    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-783a5e\"/\"kube-root-ca.crt\""
Sep 10 17:51:09.666397 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:51:09.666272    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-783a5e\"/\"default-dockercfg-fpbkl\""
Sep 10 17:51:09.676009 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:51:09.675983    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-783a5e/test-model-783a5e-kserve-b6dbdcf95-4gwk4"]
Sep 10 17:51:09.807531 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:51:09.807489    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12a5b682-06e8-454a-b25a-522a129abcbc-tmp-dir\") pod \"test-model-783a5e-kserve-b6dbdcf95-4gwk4\" (UID: \"12a5b682-06e8-454a-b25a-522a129abcbc\") " pod="ai-tenant-e2e-inf-b-783a5e/test-model-783a5e-kserve-b6dbdcf95-4gwk4"
Sep 10 17:51:09.807724 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:51:09.807535    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7d2vz\" (UniqueName: \"kubernetes.io/projected/12a5b682-06e8-454a-b25a-522a129abcbc-kube-api-access-7d2vz\") pod \"test-model-783a5e-kserve-b6dbdcf95-4gwk4\" (UID: \"12a5b682-06e8-454a-b25a-522a129abcbc\") " pod="ai-tenant-e2e-inf-b-783a5e/test-model-783a5e-kserve-b6dbdcf95-4gwk4"
Sep 10 17:51:09.807724 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:51:09.807691    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/12a5b682-06e8-454a-b25a-522a129abcbc-home\") pod \"test-model-783a5e-kserve-b6dbdcf95-4gwk4\" (UID: \"12a5b682-06e8-454a-b25a-522a129abcbc\") " pod="ai-tenant-e2e-inf-b-783a5e/test-model-783a5e-kserve-b6dbdcf95-4gwk4"
Sep 10 17:51:09.807724 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:51:09.807717    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/12a5b682-06e8-454a-b25a-522a129abcbc-dshm\") pod \"test-model-783a5e-kserve-b6dbdcf95-4gwk4\" (UID: \"12a5b682-06e8-454a-b25a-522a129abcbc\") " pod="ai-tenant-e2e-inf-b-783a5e/test-model-783a5e-kserve-b6dbdcf95-4gwk4"
Sep 10 17:51:09.807881 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:51:09.807750    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/12a5b682-06e8-454a-b25a-522a129abcbc-model-cache\") pod \"test-model-783a5e-kserve-b6dbdcf95-4gwk4\" (UID: \"12a5b682-06e8-454a-b25a-522a129abcbc\") " pod="ai-tenant-e2e-inf-b-783a5e/test-model-783a5e-kserve-b6dbdcf95-4gwk4"
Sep 10 17:51:09.807881 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:51:09.807816    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/12a5b682-06e8-454a-b25a-522a129abcbc-tls-certs\") pod \"test-model-783a5e-kserve-b6dbdcf95-4gwk4\" (UID: \"12a5b682-06e8-454a-b25a-522a129abcbc\") " pod="ai-tenant-e2e-inf-b-783a5e/test-model-783a5e-kserve-b6dbdcf95-4gwk4"
Sep 10 17:51:09.908901 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:51:09.908855    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/12a5b682-06e8-454a-b25a-522a129abcbc-home\") pod \"test-model-783a5e-kserve-b6dbdcf95-4gwk4\" (UID: \"12a5b682-06e8-454a-b25a-522a129abcbc\") " pod="ai-tenant-e2e-inf-b-783a5e/test-model-783a5e-kserve-b6dbdcf95-4gwk4"
Sep 10 17:51:09.908901 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:51:09.908901    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/12a5b682-06e8-454a-b25a-522a129abcbc-dshm\") pod \"test-model-783a5e-kserve-b6dbdcf95-4gwk4\" (UID: \"12a5b682-06e8-454a-b25a-522a129abcbc\") " pod="ai-tenant-e2e-inf-b-783a5e/test-model-783a5e-kserve-b6dbdcf95-4gwk4"
Sep 10 17:51:09.909111 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:51:09.908940    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/12a5b682-06e8-454a-b25a-522a129abcbc-model-cache\") pod \"test-model-783a5e-kserve-b6dbdcf95-4gwk4\" (UID: \"12a5b682-06e8-454a-b25a-522a129abcbc\") " pod="ai-tenant-e2e-inf-b-783a5e/test-model-783a5e-kserve-b6dbdcf95-4gwk4"
Sep 10 17:51:09.909111 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:51:09.908970    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/12a5b682-06e8-454a-b25a-522a129abcbc-tls-certs\") pod \"test-model-783a5e-kserve-b6dbdcf95-4gwk4\" (UID: \"12a5b682-06e8-454a-b25a-522a129abcbc\") " pod="ai-tenant-e2e-inf-b-783a5e/test-model-783a5e-kserve-b6dbdcf95-4gwk4"
Sep 10 17:51:09.909111 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:51:09.909035    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12a5b682-06e8-454a-b25a-522a129abcbc-tmp-dir\") pod \"test-model-783a5e-kserve-b6dbdcf95-4gwk4\" (UID: \"12a5b682-06e8-454a-b25a-522a129abcbc\") " pod="ai-tenant-e2e-inf-b-783a5e/test-model-783a5e-kserve-b6dbdcf95-4gwk4"
Sep 10 17:51:09.909111 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:51:09.909056    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7d2vz\" (UniqueName: \"kubernetes.io/projected/12a5b682-06e8-454a-b25a-522a129abcbc-kube-api-access-7d2vz\") pod \"test-model-783a5e-kserve-b6dbdcf95-4gwk4\" (UID: \"12a5b682-06e8-454a-b25a-522a129abcbc\") " pod="ai-tenant-e2e-inf-b-783a5e/test-model-783a5e-kserve-b6dbdcf95-4gwk4"
Sep 10 17:51:09.909386 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:51:09.909330    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/12a5b682-06e8-454a-b25a-522a129abcbc-model-cache\") pod \"test-model-783a5e-kserve-b6dbdcf95-4gwk4\" (UID: \"12a5b682-06e8-454a-b25a-522a129abcbc\") " pod="ai-tenant-e2e-inf-b-783a5e/test-model-783a5e-kserve-b6dbdcf95-4gwk4"
Sep 10 17:51:09.909386 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:51:09.909363    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/12a5b682-06e8-454a-b25a-522a129abcbc-home\") pod \"test-model-783a5e-kserve-b6dbdcf95-4gwk4\" (UID: \"12a5b682-06e8-454a-b25a-522a129abcbc\") " pod="ai-tenant-e2e-inf-b-783a5e/test-model-783a5e-kserve-b6dbdcf95-4gwk4"
Sep 10 17:51:09.909511 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:51:09.909447    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12a5b682-06e8-454a-b25a-522a129abcbc-tmp-dir\") pod \"test-model-783a5e-kserve-b6dbdcf95-4gwk4\" (UID: \"12a5b682-06e8-454a-b25a-522a129abcbc\") " pod="ai-tenant-e2e-inf-b-783a5e/test-model-783a5e-kserve-b6dbdcf95-4gwk4"
Sep 10 17:51:09.911545 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:51:09.911525    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/12a5b682-06e8-454a-b25a-522a129abcbc-dshm\") pod \"test-model-783a5e-kserve-b6dbdcf95-4gwk4\" (UID: \"12a5b682-06e8-454a-b25a-522a129abcbc\") " pod="ai-tenant-e2e-inf-b-783a5e/test-model-783a5e-kserve-b6dbdcf95-4gwk4"
Sep 10 17:51:09.911753 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:51:09.911736    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/12a5b682-06e8-454a-b25a-522a129abcbc-tls-certs\") pod \"test-model-783a5e-kserve-b6dbdcf95-4gwk4\" (UID: \"12a5b682-06e8-454a-b25a-522a129abcbc\") " pod="ai-tenant-e2e-inf-b-783a5e/test-model-783a5e-kserve-b6dbdcf95-4gwk4"
Sep 10 17:51:09.918382 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:51:09.918356    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7d2vz\" (UniqueName: \"kubernetes.io/projected/12a5b682-06e8-454a-b25a-522a129abcbc-kube-api-access-7d2vz\") pod \"test-model-783a5e-kserve-b6dbdcf95-4gwk4\" (UID: \"12a5b682-06e8-454a-b25a-522a129abcbc\") " pod="ai-tenant-e2e-inf-b-783a5e/test-model-783a5e-kserve-b6dbdcf95-4gwk4"
Sep 10 17:51:09.972777 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:51:09.972733    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-783a5e/test-model-783a5e-kserve-b6dbdcf95-4gwk4"
Sep 10 17:51:10.274484 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:51:10.274458    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-783a5e/test-model-783a5e-kserve-b6dbdcf95-4gwk4"]
Sep 10 17:51:10.276951 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:51:10.276920    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod12a5b682_06e8_454a_b25a_522a129abcbc.slice/crio-314d810843c17e2dff4bff511d10ddace27744723fa53beaac24e6b7ae4e1be0 WatchSource:0}: Error finding container 314d810843c17e2dff4bff511d10ddace27744723fa53beaac24e6b7ae4e1be0: Status 404 returned error can't find the container with id 314d810843c17e2dff4bff511d10ddace27744723fa53beaac24e6b7ae4e1be0
Sep 10 17:51:11.176342 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:51:11.176303    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-783a5e/test-model-783a5e-kserve-b6dbdcf95-4gwk4" event={"ID":"12a5b682-06e8-454a-b25a-522a129abcbc","Type":"ContainerStarted","Data":"071950a170562f2970a05ed32a0a0de08ca12d51550bab31933b8dc53af143e8"}
Sep 10 17:51:11.176342 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:51:11.176346    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-783a5e/test-model-783a5e-kserve-b6dbdcf95-4gwk4" event={"ID":"12a5b682-06e8-454a-b25a-522a129abcbc","Type":"ContainerStarted","Data":"314d810843c17e2dff4bff511d10ddace27744723fa53beaac24e6b7ae4e1be0"}
Sep 10 17:51:11.195649 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:51:11.195595    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-783a5e/test-model-783a5e-kserve-b6dbdcf95-4gwk4" podStartSLOduration=2.195558578 podStartE2EDuration="2.195558578s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:51:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:51:11.193986992 +0000 UTC m=+915.193817061" watchObservedRunningTime="2026-09-10 17:51:11.195558578 +0000 UTC m=+915.195388645"
Sep 10 17:51:19.973619 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:51:19.973558    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-783a5e/test-model-783a5e-kserve-b6dbdcf95-4gwk4"
Sep 10 17:51:19.974130 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:51:19.973639    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-783a5e/test-model-783a5e-kserve-b6dbdcf95-4gwk4"
Sep 10 17:51:19.987551 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:51:19.987523    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-783a5e/test-model-783a5e-kserve-b6dbdcf95-4gwk4"
Sep 10 17:51:20.237732 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:51:20.237647    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-783a5e/test-model-783a5e-kserve-b6dbdcf95-4gwk4"
Sep 10 17:52:13.606664 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:52:13.606631    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-c7995b-57c8d55974-bm5wh"]
Sep 10 17:52:13.607168 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:52:13.606841    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c7995b-57c8d55974-bm5wh" podUID="cde456ae-00a2-4c52-88b6-aea1d1d650e2" containerName="maas-api" containerID="cri-o://cc5abf184fe2ffb4ceb74e191342bab3dd091681a4dfd45bc5c1d12c556af94a" gracePeriod=30
Sep 10 17:52:13.775215 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:52:13.774414    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-c7995b-549947fbd6-9br8r"]
Sep 10 17:52:13.775215 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:52:13.774679    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-explicit-c7995b-549947fbd6-9br8r" podUID="7cb5b3f6-2d76-4bfc-8641-9bfee57c0354" containerName="payload-pre-processing" containerID="cri-o://5eb405712ff6a176b5ff5f4aff15ed81c477abcea52035645cd12982f58cece3" gracePeriod=30
Sep 10 17:52:13.901660 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:52:13.901635    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c7995b-57c8d55974-bm5wh"
Sep 10 17:52:14.002377 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:52:14.002342    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cde456ae-00a2-4c52-88b6-aea1d1d650e2-maas-api-tls\") pod \"cde456ae-00a2-4c52-88b6-aea1d1d650e2\" (UID: \"cde456ae-00a2-4c52-88b6-aea1d1d650e2\") "
Sep 10 17:52:14.002377 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:52:14.002383    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6d8gr\" (UniqueName: \"kubernetes.io/projected/cde456ae-00a2-4c52-88b6-aea1d1d650e2-kube-api-access-6d8gr\") pod \"cde456ae-00a2-4c52-88b6-aea1d1d650e2\" (UID: \"cde456ae-00a2-4c52-88b6-aea1d1d650e2\") "
Sep 10 17:52:14.002679 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:52:14.002405    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cde456ae-00a2-4c52-88b6-aea1d1d650e2-openshift-service-ca\") pod \"cde456ae-00a2-4c52-88b6-aea1d1d650e2\" (UID: \"cde456ae-00a2-4c52-88b6-aea1d1d650e2\") "
Sep 10 17:52:14.002679 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:52:14.002504    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cde456ae-00a2-4c52-88b6-aea1d1d650e2-metrics-certs\") pod \"cde456ae-00a2-4c52-88b6-aea1d1d650e2\" (UID: \"cde456ae-00a2-4c52-88b6-aea1d1d650e2\") "
Sep 10 17:52:14.002856 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:52:14.002833    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cde456ae-00a2-4c52-88b6-aea1d1d650e2-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "cde456ae-00a2-4c52-88b6-aea1d1d650e2" (UID: "cde456ae-00a2-4c52-88b6-aea1d1d650e2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 17:52:14.004810 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:52:14.004777    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cde456ae-00a2-4c52-88b6-aea1d1d650e2-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "cde456ae-00a2-4c52-88b6-aea1d1d650e2" (UID: "cde456ae-00a2-4c52-88b6-aea1d1d650e2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 17:52:14.004926 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:52:14.004868    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cde456ae-00a2-4c52-88b6-aea1d1d650e2-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "cde456ae-00a2-4c52-88b6-aea1d1d650e2" (UID: "cde456ae-00a2-4c52-88b6-aea1d1d650e2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 17:52:14.005058 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:52:14.005041    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cde456ae-00a2-4c52-88b6-aea1d1d650e2-kube-api-access-6d8gr" (OuterVolumeSpecName: "kube-api-access-6d8gr") pod "cde456ae-00a2-4c52-88b6-aea1d1d650e2" (UID: "cde456ae-00a2-4c52-88b6-aea1d1d650e2"). InnerVolumeSpecName "kube-api-access-6d8gr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:52:14.023199 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:52:14.023174    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-c7995b-549947fbd6-9br8r"
Sep 10 17:52:14.103313 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:52:14.103267    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q6cx6\" (UniqueName: \"kubernetes.io/projected/7cb5b3f6-2d76-4bfc-8641-9bfee57c0354-kube-api-access-q6cx6\") pod \"7cb5b3f6-2d76-4bfc-8641-9bfee57c0354\" (UID: \"7cb5b3f6-2d76-4bfc-8641-9bfee57c0354\") "
Sep 10 17:52:14.103313 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:52:14.103306    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7cb5b3f6-2d76-4bfc-8641-9bfee57c0354-plugins-config-volume\") pod \"7cb5b3f6-2d76-4bfc-8641-9bfee57c0354\" (UID: \"7cb5b3f6-2d76-4bfc-8641-9bfee57c0354\") "
Sep 10 17:52:14.103661 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:52:14.103641    2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cde456ae-00a2-4c52-88b6-aea1d1d650e2-metrics-certs\") on node \"ip-10-0-129-122.ec2.internal\" DevicePath \"\""
Sep 10 17:52:14.103762 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:52:14.103665    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cde456ae-00a2-4c52-88b6-aea1d1d650e2-maas-api-tls\") on node \"ip-10-0-129-122.ec2.internal\" DevicePath \"\""
Sep 10 17:52:14.103762 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:52:14.103680    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6d8gr\" (UniqueName: \"kubernetes.io/projected/cde456ae-00a2-4c52-88b6-aea1d1d650e2-kube-api-access-6d8gr\") on node \"ip-10-0-129-122.ec2.internal\" DevicePath \"\""
Sep 10 17:52:14.103762 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:52:14.103694    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cde456ae-00a2-4c52-88b6-aea1d1d650e2-openshift-service-ca\") on node \"ip-10-0-129-122.ec2.internal\" DevicePath \"\""
Sep 10 17:52:14.103762 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:52:14.103720    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7cb5b3f6-2d76-4bfc-8641-9bfee57c0354-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7cb5b3f6-2d76-4bfc-8641-9bfee57c0354" (UID: "7cb5b3f6-2d76-4bfc-8641-9bfee57c0354"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 17:52:14.105600 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:52:14.105553    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7cb5b3f6-2d76-4bfc-8641-9bfee57c0354-kube-api-access-q6cx6" (OuterVolumeSpecName: "kube-api-access-q6cx6") pod "7cb5b3f6-2d76-4bfc-8641-9bfee57c0354" (UID: "7cb5b3f6-2d76-4bfc-8641-9bfee57c0354"). InnerVolumeSpecName "kube-api-access-q6cx6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:52:14.204879 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:52:14.204842    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q6cx6\" (UniqueName: \"kubernetes.io/projected/7cb5b3f6-2d76-4bfc-8641-9bfee57c0354-kube-api-access-q6cx6\") on node \"ip-10-0-129-122.ec2.internal\" DevicePath \"\""
Sep 10 17:52:14.204879 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:52:14.204873    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7cb5b3f6-2d76-4bfc-8641-9bfee57c0354-plugins-config-volume\") on node \"ip-10-0-129-122.ec2.internal\" DevicePath \"\""
Sep 10 17:52:14.502354 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:52:14.502323    2579 generic.go:356] "Generic (PLEG): container finished" podID="cde456ae-00a2-4c52-88b6-aea1d1d650e2" containerID="cc5abf184fe2ffb4ceb74e191342bab3dd091681a4dfd45bc5c1d12c556af94a" exitCode=0
Sep 10 17:52:14.502499 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:52:14.502385    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c7995b-57c8d55974-bm5wh"
Sep 10 17:52:14.502499 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:52:14.502406    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c7995b-57c8d55974-bm5wh" event={"ID":"cde456ae-00a2-4c52-88b6-aea1d1d650e2","Type":"ContainerDied","Data":"cc5abf184fe2ffb4ceb74e191342bab3dd091681a4dfd45bc5c1d12c556af94a"}
Sep 10 17:52:14.502499 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:52:14.502459    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c7995b-57c8d55974-bm5wh" event={"ID":"cde456ae-00a2-4c52-88b6-aea1d1d650e2","Type":"ContainerDied","Data":"e7f1ca20a8c5b2d9ef9e2b1cab58fd419c0d441d64f30940673e6f641081fb5b"}
Sep 10 17:52:14.502499 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:52:14.502489    2579 scope.go:117] "RemoveContainer" containerID="cc5abf184fe2ffb4ceb74e191342bab3dd091681a4dfd45bc5c1d12c556af94a"
Sep 10 17:52:14.503968 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:52:14.503943    2579 generic.go:356] "Generic (PLEG): container finished" podID="7cb5b3f6-2d76-4bfc-8641-9bfee57c0354" containerID="5eb405712ff6a176b5ff5f4aff15ed81c477abcea52035645cd12982f58cece3" exitCode=0
Sep 10 17:52:14.504080 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:52:14.503991    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-c7995b-549947fbd6-9br8r"
Sep 10 17:52:14.504080 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:52:14.504012    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-c7995b-549947fbd6-9br8r" event={"ID":"7cb5b3f6-2d76-4bfc-8641-9bfee57c0354","Type":"ContainerDied","Data":"5eb405712ff6a176b5ff5f4aff15ed81c477abcea52035645cd12982f58cece3"}
Sep 10 17:52:14.504080 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:52:14.504047    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-c7995b-549947fbd6-9br8r" event={"ID":"7cb5b3f6-2d76-4bfc-8641-9bfee57c0354","Type":"ContainerDied","Data":"c958bb559c2d7234ebf3f3a4d0dba8f4ab09313d5292405268ef317669a082b3"}
Sep 10 17:52:14.511776 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:52:14.511752    2579 scope.go:117] "RemoveContainer" containerID="cc5abf184fe2ffb4ceb74e191342bab3dd091681a4dfd45bc5c1d12c556af94a"
Sep 10 17:52:14.512061 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:52:14.512045    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc5abf184fe2ffb4ceb74e191342bab3dd091681a4dfd45bc5c1d12c556af94a\": container with ID starting with cc5abf184fe2ffb4ceb74e191342bab3dd091681a4dfd45bc5c1d12c556af94a not found: ID does not exist" containerID="cc5abf184fe2ffb4ceb74e191342bab3dd091681a4dfd45bc5c1d12c556af94a"
Sep 10 17:52:14.512107 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:52:14.512068    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc5abf184fe2ffb4ceb74e191342bab3dd091681a4dfd45bc5c1d12c556af94a"} err="failed to get container status \"cc5abf184fe2ffb4ceb74e191342bab3dd091681a4dfd45bc5c1d12c556af94a\": rpc error: code = NotFound desc = could not find container \"cc5abf184fe2ffb4ceb74e191342bab3dd091681a4dfd45bc5c1d12c556af94a\": container with ID starting with cc5abf184fe2ffb4ceb74e191342bab3dd091681a4dfd45bc5c1d12c556af94a not found: ID does not exist"
Sep 10 17:52:14.512107 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:52:14.512083    2579 scope.go:117] "RemoveContainer" containerID="5eb405712ff6a176b5ff5f4aff15ed81c477abcea52035645cd12982f58cece3"
Sep 10 17:52:14.520302 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:52:14.520286    2579 scope.go:117] "RemoveContainer" containerID="5eb405712ff6a176b5ff5f4aff15ed81c477abcea52035645cd12982f58cece3"
Sep 10 17:52:14.520664 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:52:14.520556    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5eb405712ff6a176b5ff5f4aff15ed81c477abcea52035645cd12982f58cece3\": container with ID starting with 5eb405712ff6a176b5ff5f4aff15ed81c477abcea52035645cd12982f58cece3 not found: ID does not exist" containerID="5eb405712ff6a176b5ff5f4aff15ed81c477abcea52035645cd12982f58cece3"
Sep 10 17:52:14.520730 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:52:14.520674    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5eb405712ff6a176b5ff5f4aff15ed81c477abcea52035645cd12982f58cece3"} err="failed to get container status \"5eb405712ff6a176b5ff5f4aff15ed81c477abcea52035645cd12982f58cece3\": rpc error: code = NotFound desc = could not find container \"5eb405712ff6a176b5ff5f4aff15ed81c477abcea52035645cd12982f58cece3\": container with ID starting with 5eb405712ff6a176b5ff5f4aff15ed81c477abcea52035645cd12982f58cece3 not found: ID does not exist"
Sep 10 17:52:14.546699 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:52:14.546671    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-c7995b-549947fbd6-9br8r"]
Sep 10 17:52:14.554484 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:52:14.554466    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-c7995b-549947fbd6-9br8r"]
Sep 10 17:52:14.599675 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:52:14.599649    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-c7995b-57c8d55974-bm5wh"]
Sep 10 17:52:14.604309 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:52:14.604290    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-c7995b-57c8d55974-bm5wh"]
Sep 10 17:52:16.476537 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:52:16.476495    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7cb5b3f6-2d76-4bfc-8641-9bfee57c0354" path="/var/lib/kubelet/pods/7cb5b3f6-2d76-4bfc-8641-9bfee57c0354/volumes"
Sep 10 17:52:16.477038 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:52:16.476909    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cde456ae-00a2-4c52-88b6-aea1d1d650e2" path="/var/lib/kubelet/pods/cde456ae-00a2-4c52-88b6-aea1d1d650e2/volumes"
Sep 10 17:52:21.978418 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:52:21.978330    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-noten-8aeafa-openshift-default-b46f9b5b4-d5m5q"]
Sep 10 17:52:21.978920 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:52:21.978900    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7cb5b3f6-2d76-4bfc-8641-9bfee57c0354" containerName="payload-pre-processing"
Sep 10 17:52:21.978989 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:52:21.978923    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="7cb5b3f6-2d76-4bfc-8641-9bfee57c0354" containerName="payload-pre-processing"
Sep 10 17:52:21.978989 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:52:21.978953    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cde456ae-00a2-4c52-88b6-aea1d1d650e2" containerName="maas-api"
Sep 10 17:52:21.978989 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:52:21.978963    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="cde456ae-00a2-4c52-88b6-aea1d1d650e2" containerName="maas-api"
Sep 10 17:52:21.979142 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:52:21.979056    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="cde456ae-00a2-4c52-88b6-aea1d1d650e2" containerName="maas-api"
Sep 10 17:52:21.979142 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:52:21.979075    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="7cb5b3f6-2d76-4bfc-8641-9bfee57c0354" containerName="payload-pre-processing"
Sep 10 17:52:21.982180 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:52:21.982158    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-8aeafa-openshift-default-b46f9b5b4-d5m5q"
Sep 10 17:52:21.985223 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:52:21.985205    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 10 17:52:21.985324 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:52:21.985249    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-noten-8aeafa-openshift-default-dockercfg-k4df5\""
Sep 10 17:52:21.996036 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:52:21.996012    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-8aeafa-openshift-default-b46f9b5b4-d5m5q"]
Sep 10 17:52:22.071705 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:52:22.071668    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2562421b-1343-44cb-98fa-ee27d02c7e38-workload-socket\") pod \"e2e-noten-8aeafa-openshift-default-b46f9b5b4-d5m5q\" (UID: \"2562421b-1343-44cb-98fa-ee27d02c7e38\") " pod="openshift-ingress/e2e-noten-8aeafa-openshift-default-b46f9b5b4-d5m5q"
Sep 10 17:52:22.071869 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:52:22.071709    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2562421b-1343-44cb-98fa-ee27d02c7e38-workload-certs\") pod \"e2e-noten-8aeafa-openshift-default-b46f9b5b4-d5m5q\" (UID: \"2562421b-1343-44cb-98fa-ee27d02c7e38\") " pod="openshift-ingress/e2e-noten-8aeafa-openshift-default-b46f9b5b4-d5m5q"
Sep 10 17:52:22.071869 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:52:22.071750    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2562421b-1343-44cb-98fa-ee27d02c7e38-istio-token\") pod \"e2e-noten-8aeafa-openshift-default-b46f9b5b4-d5m5q\" (UID: \"2562421b-1343-44cb-98fa-ee27d02c7e38\") " pod="openshift-ingress/e2e-noten-8aeafa-openshift-default-b46f9b5b4-d5m5q"
Sep 10 17:52:22.071869 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:52:22.071800    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2562421b-1343-44cb-98fa-ee27d02c7e38-credential-socket\") pod \"e2e-noten-8aeafa-openshift-default-b46f9b5b4-d5m5q\" (UID: \"2562421b-1343-44cb-98fa-ee27d02c7e38\") " pod="openshift-ingress/e2e-noten-8aeafa-openshift-default-b46f9b5b4-d5m5q"
Sep 10 17:52:22.071869 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:52:22.071845    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2562421b-1343-44cb-98fa-ee27d02c7e38-istio-data\") pod \"e2e-noten-8aeafa-openshift-default-b46f9b5b4-d5m5q\" (UID: \"2562421b-1343-44cb-98fa-ee27d02c7e38\") " pod="openshift-ingress/e2e-noten-8aeafa-openshift-default-b46f9b5b4-d5m5q"
Sep 10 17:52:22.072075 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:52:22.071885    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2562421b-1343-44cb-98fa-ee27d02c7e38-istiod-ca-cert\") pod \"e2e-noten-8aeafa-openshift-default-b46f9b5b4-d5m5q\" (UID: \"2562421b-1343-44cb-98fa-ee27d02c7e38\") " pod="openshift-ingress/e2e-noten-8aeafa-openshift-default-b46f9b5b4-d5m5q"
Sep 10 17:52:22.072075 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:52:22.071945    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2562421b-1343-44cb-98fa-ee27d02c7e38-istio-envoy\") pod \"e2e-noten-8aeafa-openshift-default-b46f9b5b4-d5m5q\" (UID: \"2562421b-1343-44cb-98fa-ee27d02c7e38\") " pod="openshift-ingress/e2e-noten-8aeafa-openshift-default-b46f9b5b4-d5m5q"
Sep 10 17:52:22.072075 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:52:22.071961    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l89bl\" (UniqueName: \"kubernetes.io/projected/2562421b-1343-44cb-98fa-ee27d02c7e38-kube-api-access-l89bl\") pod \"e2e-noten-8aeafa-openshift-default-b46f9b5b4-d5m5q\" (UID: \"2562421b-1343-44cb-98fa-ee27d02c7e38\") " pod="openshift-ingress/e2e-noten-8aeafa-openshift-default-b46f9b5b4-d5m5q"
Sep 10 17:52:22.072075 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:52:22.072019    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2562421b-1343-44cb-98fa-ee27d02c7e38-istio-podinfo\") pod \"e2e-noten-8aeafa-openshift-default-b46f9b5b4-d5m5q\" (UID: \"2562421b-1343-44cb-98fa-ee27d02c7e38\") " pod="openshift-ingress/e2e-noten-8aeafa-openshift-default-b46f9b5b4-d5m5q"
Sep 10 17:52:22.172610 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:52:22.172553    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2562421b-1343-44cb-98fa-ee27d02c7e38-istio-envoy\") pod \"e2e-noten-8aeafa-openshift-default-b46f9b5b4-d5m5q\" (UID: \"2562421b-1343-44cb-98fa-ee27d02c7e38\") " pod="openshift-ingress/e2e-noten-8aeafa-openshift-default-b46f9b5b4-d5m5q"
Sep 10 17:52:22.172610 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:52:22.172609    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l89bl\" (UniqueName: \"kubernetes.io/projected/2562421b-1343-44cb-98fa-ee27d02c7e38-kube-api-access-l89bl\") pod \"e2e-noten-8aeafa-openshift-default-b46f9b5b4-d5m5q\" (UID: \"2562421b-1343-44cb-98fa-ee27d02c7e38\") " pod="openshift-ingress/e2e-noten-8aeafa-openshift-default-b46f9b5b4-d5m5q"
Sep 10 17:52:22.172841 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:52:22.172642    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2562421b-1343-44cb-98fa-ee27d02c7e38-istio-podinfo\") pod \"e2e-noten-8aeafa-openshift-default-b46f9b5b4-d5m5q\" (UID: \"2562421b-1343-44cb-98fa-ee27d02c7e38\") " pod="openshift-ingress/e2e-noten-8aeafa-openshift-default-b46f9b5b4-d5m5q"
Sep 10 17:52:22.172841 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:52:22.172668    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2562421b-1343-44cb-98fa-ee27d02c7e38-workload-socket\") pod \"e2e-noten-8aeafa-openshift-default-b46f9b5b4-d5m5q\" (UID: \"2562421b-1343-44cb-98fa-ee27d02c7e38\") " pod="openshift-ingress/e2e-noten-8aeafa-openshift-default-b46f9b5b4-d5m5q"
Sep 10 17:52:22.172841 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:52:22.172693    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2562421b-1343-44cb-98fa-ee27d02c7e38-workload-certs\") pod \"e2e-noten-8aeafa-openshift-default-b46f9b5b4-d5m5q\" (UID: \"2562421b-1343-44cb-98fa-ee27d02c7e38\") " pod="openshift-ingress/e2e-noten-8aeafa-openshift-default-b46f9b5b4-d5m5q"
Sep 10 17:52:22.172841 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:52:22.172730    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2562421b-1343-44cb-98fa-ee27d02c7e38-istio-token\") pod \"e2e-noten-8aeafa-openshift-default-b46f9b5b4-d5m5q\" (UID: \"2562421b-1343-44cb-98fa-ee27d02c7e38\") " pod="openshift-ingress/e2e-noten-8aeafa-openshift-default-b46f9b5b4-d5m5q"
Sep 10 17:52:22.172841 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:52:22.172781    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2562421b-1343-44cb-98fa-ee27d02c7e38-credential-socket\") pod \"e2e-noten-8aeafa-openshift-default-b46f9b5b4-d5m5q\" (UID: \"2562421b-1343-44cb-98fa-ee27d02c7e38\") " pod="openshift-ingress/e2e-noten-8aeafa-openshift-default-b46f9b5b4-d5m5q"
Sep 10 17:52:22.173106 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:52:22.173040    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2562421b-1343-44cb-98fa-ee27d02c7e38-istio-data\") pod \"e2e-noten-8aeafa-openshift-default-b46f9b5b4-d5m5q\" (UID: \"2562421b-1343-44cb-98fa-ee27d02c7e38\") " pod="openshift-ingress/e2e-noten-8aeafa-openshift-default-b46f9b5b4-d5m5q"
Sep 10 17:52:22.173169 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:52:22.173113    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2562421b-1343-44cb-98fa-ee27d02c7e38-istiod-ca-cert\") pod \"e2e-noten-8aeafa-openshift-default-b46f9b5b4-d5m5q\" (UID: \"2562421b-1343-44cb-98fa-ee27d02c7e38\") " pod="openshift-ingress/e2e-noten-8aeafa-openshift-default-b46f9b5b4-d5m5q"
Sep 10 17:52:22.173169 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:52:22.173123    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2562421b-1343-44cb-98fa-ee27d02c7e38-workload-certs\") pod \"e2e-noten-8aeafa-openshift-default-b46f9b5b4-d5m5q\" (UID: \"2562421b-1343-44cb-98fa-ee27d02c7e38\") " pod="openshift-ingress/e2e-noten-8aeafa-openshift-default-b46f9b5b4-d5m5q"
Sep 10 17:52:22.173169 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:52:22.173149    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2562421b-1343-44cb-98fa-ee27d02c7e38-credential-socket\") pod \"e2e-noten-8aeafa-openshift-default-b46f9b5b4-d5m5q\" (UID: \"2562421b-1343-44cb-98fa-ee27d02c7e38\") " pod="openshift-ingress/e2e-noten-8aeafa-openshift-default-b46f9b5b4-d5m5q"
Sep 10 17:52:22.173356 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:52:22.173333    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2562421b-1343-44cb-98fa-ee27d02c7e38-workload-socket\") pod \"e2e-noten-8aeafa-openshift-default-b46f9b5b4-d5m5q\" (UID: \"2562421b-1343-44cb-98fa-ee27d02c7e38\") " pod="openshift-ingress/e2e-noten-8aeafa-openshift-default-b46f9b5b4-d5m5q"
Sep 10 17:52:22.173417 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:52:22.173375    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2562421b-1343-44cb-98fa-ee27d02c7e38-istio-data\") pod \"e2e-noten-8aeafa-openshift-default-b46f9b5b4-d5m5q\" (UID: \"2562421b-1343-44cb-98fa-ee27d02c7e38\") " pod="openshift-ingress/e2e-noten-8aeafa-openshift-default-b46f9b5b4-d5m5q"
Sep 10 17:52:22.174065 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:52:22.174042    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2562421b-1343-44cb-98fa-ee27d02c7e38-istiod-ca-cert\") pod \"e2e-noten-8aeafa-openshift-default-b46f9b5b4-d5m5q\" (UID: \"2562421b-1343-44cb-98fa-ee27d02c7e38\") " pod="openshift-ingress/e2e-noten-8aeafa-openshift-default-b46f9b5b4-d5m5q"
Sep 10 17:52:22.175164 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:52:22.175138    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2562421b-1343-44cb-98fa-ee27d02c7e38-istio-envoy\") pod \"e2e-noten-8aeafa-openshift-default-b46f9b5b4-d5m5q\" (UID: \"2562421b-1343-44cb-98fa-ee27d02c7e38\") " pod="openshift-ingress/e2e-noten-8aeafa-openshift-default-b46f9b5b4-d5m5q"
Sep 10 17:52:22.175400 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:52:22.175381    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2562421b-1343-44cb-98fa-ee27d02c7e38-istio-podinfo\") pod \"e2e-noten-8aeafa-openshift-default-b46f9b5b4-d5m5q\" (UID: \"2562421b-1343-44cb-98fa-ee27d02c7e38\") " pod="openshift-ingress/e2e-noten-8aeafa-openshift-default-b46f9b5b4-d5m5q"
Sep 10 17:52:22.190554 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:52:22.190533    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2562421b-1343-44cb-98fa-ee27d02c7e38-istio-token\") pod \"e2e-noten-8aeafa-openshift-default-b46f9b5b4-d5m5q\" (UID: \"2562421b-1343-44cb-98fa-ee27d02c7e38\") " pod="openshift-ingress/e2e-noten-8aeafa-openshift-default-b46f9b5b4-d5m5q"
Sep 10 17:52:22.202436 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:52:22.202411    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l89bl\" (UniqueName: \"kubernetes.io/projected/2562421b-1343-44cb-98fa-ee27d02c7e38-kube-api-access-l89bl\") pod \"e2e-noten-8aeafa-openshift-default-b46f9b5b4-d5m5q\" (UID: \"2562421b-1343-44cb-98fa-ee27d02c7e38\") " pod="openshift-ingress/e2e-noten-8aeafa-openshift-default-b46f9b5b4-d5m5q"
Sep 10 17:52:22.293426 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:52:22.293352    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-8aeafa-openshift-default-b46f9b5b4-d5m5q"
Sep 10 17:52:22.582195 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:52:22.582173    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-8aeafa-openshift-default-b46f9b5b4-d5m5q"]
Sep 10 17:52:22.583791 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:52:22.583757    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2562421b_1343_44cb_98fa_ee27d02c7e38.slice/crio-54b831a8ca3bd4b3432d1fff77d0c7b98f8aba5753b52bd017c455c34d429c4c WatchSource:0}: Error finding container 54b831a8ca3bd4b3432d1fff77d0c7b98f8aba5753b52bd017c455c34d429c4c: Status 404 returned error can't find the container with id 54b831a8ca3bd4b3432d1fff77d0c7b98f8aba5753b52bd017c455c34d429c4c
Sep 10 17:52:22.586425 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:52:22.586392    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 10 17:52:22.586535 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:52:22.586469    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 10 17:52:22.586535 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:52:22.586514    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 10 17:52:23.550946 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:52:23.550910    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-8aeafa-openshift-default-b46f9b5b4-d5m5q" event={"ID":"2562421b-1343-44cb-98fa-ee27d02c7e38","Type":"ContainerStarted","Data":"b7695b1ab73b358730d4865f8fea450ed17274de1ada9c680589cf5cfa306cce"}
Sep 10 17:52:23.551308 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:52:23.550967    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-8aeafa-openshift-default-b46f9b5b4-d5m5q" event={"ID":"2562421b-1343-44cb-98fa-ee27d02c7e38","Type":"ContainerStarted","Data":"54b831a8ca3bd4b3432d1fff77d0c7b98f8aba5753b52bd017c455c34d429c4c"}
Sep 10 17:52:23.573101 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:52:23.573048    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-noten-8aeafa-openshift-default-b46f9b5b4-d5m5q" podStartSLOduration=2.573034151 podStartE2EDuration="2.573034151s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:52:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:52:23.571546383 +0000 UTC m=+987.571376450" watchObservedRunningTime="2026-09-10 17:52:23.573034151 +0000 UTC m=+987.572864218"
Sep 10 17:52:24.293616 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:52:24.293577    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-noten-8aeafa-openshift-default-b46f9b5b4-d5m5q"
Sep 10 17:52:24.298559 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:52:24.298534    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-noten-8aeafa-openshift-default-b46f9b5b4-d5m5q"
Sep 10 17:52:24.555606 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:52:24.555503    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-noten-8aeafa-openshift-default-b46f9b5b4-d5m5q"
Sep 10 17:52:24.556537 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:52:24.556521    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-noten-8aeafa-openshift-default-b46f9b5b4-d5m5q"
Sep 10 17:52:32.372081 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:52:32.372039    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-8aeafa-openshift-default-b46f9b5b4-d5m5q"]
Sep 10 17:52:32.372537 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:52:32.372299    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-noten-8aeafa-openshift-default-b46f9b5b4-d5m5q" podUID="2562421b-1343-44cb-98fa-ee27d02c7e38" containerName="istio-proxy" containerID="cri-o://b7695b1ab73b358730d4865f8fea450ed17274de1ada9c680589cf5cfa306cce" gracePeriod=30
Sep 10 17:52:34.952379 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:52:34.952343    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c523e8a0-c59685b47-nh2v9"]
Sep 10 17:52:34.956300 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:52:34.956275    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c523e8a0-c59685b47-nh2v9"
Sep 10 17:52:34.960987 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:52:34.960964    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-c523e8a0-serving-cert\""
Sep 10 17:52:34.979183 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:52:34.979154    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c523e8a0-c59685b47-nh2v9"]
Sep 10 17:52:35.087972 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:52:35.087942    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/62eae552-879c-4b6e-bf09-ff51e9f44cf2-openshift-service-ca\") pod \"maas-api-e2e-mt-c523e8a0-c59685b47-nh2v9\" (UID: \"62eae552-879c-4b6e-bf09-ff51e9f44cf2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c523e8a0-c59685b47-nh2v9"
Sep 10 17:52:35.087972 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:52:35.087975    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2stcb\" (UniqueName: \"kubernetes.io/projected/62eae552-879c-4b6e-bf09-ff51e9f44cf2-kube-api-access-2stcb\") pod \"maas-api-e2e-mt-c523e8a0-c59685b47-nh2v9\" (UID: \"62eae552-879c-4b6e-bf09-ff51e9f44cf2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c523e8a0-c59685b47-nh2v9"
Sep 10 17:52:35.088165 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:52:35.088076    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/62eae552-879c-4b6e-bf09-ff51e9f44cf2-metrics-certs\") pod \"maas-api-e2e-mt-c523e8a0-c59685b47-nh2v9\" (UID: \"62eae552-879c-4b6e-bf09-ff51e9f44cf2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c523e8a0-c59685b47-nh2v9"
Sep 10 17:52:35.088165 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:52:35.088145    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/62eae552-879c-4b6e-bf09-ff51e9f44cf2-maas-api-tls\") pod \"maas-api-e2e-mt-c523e8a0-c59685b47-nh2v9\" (UID: \"62eae552-879c-4b6e-bf09-ff51e9f44cf2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c523e8a0-c59685b47-nh2v9"
Sep 10 17:52:35.189285 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:52:35.189245    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/62eae552-879c-4b6e-bf09-ff51e9f44cf2-openshift-service-ca\") pod \"maas-api-e2e-mt-c523e8a0-c59685b47-nh2v9\" (UID: \"62eae552-879c-4b6e-bf09-ff51e9f44cf2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c523e8a0-c59685b47-nh2v9"
Sep 10 17:52:35.189459 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:52:35.189290    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2stcb\" (UniqueName: \"kubernetes.io/projected/62eae552-879c-4b6e-bf09-ff51e9f44cf2-kube-api-access-2stcb\") pod \"maas-api-e2e-mt-c523e8a0-c59685b47-nh2v9\" (UID: \"62eae552-879c-4b6e-bf09-ff51e9f44cf2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c523e8a0-c59685b47-nh2v9"
Sep 10 17:52:35.189459 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:52:35.189362    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/62eae552-879c-4b6e-bf09-ff51e9f44cf2-metrics-certs\") pod \"maas-api-e2e-mt-c523e8a0-c59685b47-nh2v9\" (UID: \"62eae552-879c-4b6e-bf09-ff51e9f44cf2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c523e8a0-c59685b47-nh2v9"
Sep 10 17:52:35.189459 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:52:35.189402    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/62eae552-879c-4b6e-bf09-ff51e9f44cf2-maas-api-tls\") pod \"maas-api-e2e-mt-c523e8a0-c59685b47-nh2v9\" (UID: \"62eae552-879c-4b6e-bf09-ff51e9f44cf2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c523e8a0-c59685b47-nh2v9"
Sep 10 17:52:35.189661 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:52:35.189527    2579 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-c523e8a0-serving-cert: secret "maas-api-e2e-mt-c523e8a0-serving-cert" not found
Sep 10 17:52:35.189661 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:52:35.189616    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62eae552-879c-4b6e-bf09-ff51e9f44cf2-maas-api-tls podName:62eae552-879c-4b6e-bf09-ff51e9f44cf2 nodeName:}" failed. No retries permitted until 2026-09-10 17:52:35.689595358 +0000 UTC m=+999.689425403 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/62eae552-879c-4b6e-bf09-ff51e9f44cf2-maas-api-tls") pod "maas-api-e2e-mt-c523e8a0-c59685b47-nh2v9" (UID: "62eae552-879c-4b6e-bf09-ff51e9f44cf2") : secret "maas-api-e2e-mt-c523e8a0-serving-cert" not found
Sep 10 17:52:35.190005 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:52:35.189984    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/62eae552-879c-4b6e-bf09-ff51e9f44cf2-openshift-service-ca\") pod \"maas-api-e2e-mt-c523e8a0-c59685b47-nh2v9\" (UID: \"62eae552-879c-4b6e-bf09-ff51e9f44cf2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c523e8a0-c59685b47-nh2v9"
Sep 10 17:52:35.192092 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:52:35.192073    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/62eae552-879c-4b6e-bf09-ff51e9f44cf2-metrics-certs\") pod \"maas-api-e2e-mt-c523e8a0-c59685b47-nh2v9\" (UID: \"62eae552-879c-4b6e-bf09-ff51e9f44cf2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c523e8a0-c59685b47-nh2v9"
Sep 10 17:52:35.198364 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:52:35.198341    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2stcb\" (UniqueName: \"kubernetes.io/projected/62eae552-879c-4b6e-bf09-ff51e9f44cf2-kube-api-access-2stcb\") pod \"maas-api-e2e-mt-c523e8a0-c59685b47-nh2v9\" (UID: \"62eae552-879c-4b6e-bf09-ff51e9f44cf2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c523e8a0-c59685b47-nh2v9"
Sep 10 17:52:35.623725 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:52:35.623693    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-c523e8a0-5ff497dc74-bngcn"]
Sep 10 17:52:35.627179 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:52:35.627160    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-c523e8a0-5ff497dc74-bngcn"
Sep 10 17:52:35.629757 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:52:35.629737    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-c523e8a0\""
Sep 10 17:52:35.642326 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:52:35.642303    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-c523e8a0-5ff497dc74-bngcn"]
Sep 10 17:52:35.693256 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:52:35.693225    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/62eae552-879c-4b6e-bf09-ff51e9f44cf2-maas-api-tls\") pod \"maas-api-e2e-mt-c523e8a0-c59685b47-nh2v9\" (UID: \"62eae552-879c-4b6e-bf09-ff51e9f44cf2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c523e8a0-c59685b47-nh2v9"
Sep 10 17:52:35.693401 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:52:35.693276    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2nnk5\" (UniqueName: \"kubernetes.io/projected/c2378963-fe0f-48ae-894a-234a6f77660f-kube-api-access-2nnk5\") pod \"payload-pre-processing-e2e-mt-c523e8a0-5ff497dc74-bngcn\" (UID: \"c2378963-fe0f-48ae-894a-234a6f77660f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-c523e8a0-5ff497dc74-bngcn"
Sep 10 17:52:35.693401 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:52:35.693361    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2378963-fe0f-48ae-894a-234a6f77660f-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-c523e8a0-5ff497dc74-bngcn\" (UID: \"c2378963-fe0f-48ae-894a-234a6f77660f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-c523e8a0-5ff497dc74-bngcn"
Sep 10 17:52:35.696147 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:52:35.696094    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/62eae552-879c-4b6e-bf09-ff51e9f44cf2-maas-api-tls\") pod \"maas-api-e2e-mt-c523e8a0-c59685b47-nh2v9\" (UID: \"62eae552-879c-4b6e-bf09-ff51e9f44cf2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c523e8a0-c59685b47-nh2v9"
Sep 10 17:52:35.794544 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:52:35.794505    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2378963-fe0f-48ae-894a-234a6f77660f-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-c523e8a0-5ff497dc74-bngcn\" (UID: \"c2378963-fe0f-48ae-894a-234a6f77660f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-c523e8a0-5ff497dc74-bngcn"
Sep 10 17:52:35.794761 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:52:35.794620    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2nnk5\" (UniqueName: \"kubernetes.io/projected/c2378963-fe0f-48ae-894a-234a6f77660f-kube-api-access-2nnk5\") pod \"payload-pre-processing-e2e-mt-c523e8a0-5ff497dc74-bngcn\" (UID: \"c2378963-fe0f-48ae-894a-234a6f77660f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-c523e8a0-5ff497dc74-bngcn"
Sep 10 17:52:35.795308 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:52:35.795272    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2378963-fe0f-48ae-894a-234a6f77660f-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-c523e8a0-5ff497dc74-bngcn\" (UID: \"c2378963-fe0f-48ae-894a-234a6f77660f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-c523e8a0-5ff497dc74-bngcn"
Sep 10 17:52:35.803443 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:52:35.803419    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2nnk5\" (UniqueName: \"kubernetes.io/projected/c2378963-fe0f-48ae-894a-234a6f77660f-kube-api-access-2nnk5\") pod \"payload-pre-processing-e2e-mt-c523e8a0-5ff497dc74-bngcn\" (UID: \"c2378963-fe0f-48ae-894a-234a6f77660f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-c523e8a0-5ff497dc74-bngcn"
Sep 10 17:52:35.867334 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:52:35.867306    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c523e8a0-c59685b47-nh2v9"
Sep 10 17:52:35.938507 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:52:35.937902    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-c523e8a0-5ff497dc74-bngcn"
Sep 10 17:52:36.144005 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:52:36.143983    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c523e8a0-c59685b47-nh2v9"]
Sep 10 17:52:36.145997 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:52:36.145966    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62eae552_879c_4b6e_bf09_ff51e9f44cf2.slice/crio-b571d8719bdcb102eaae493a740ead48b8d3fa28aba9bf3fe0451a18a326338d WatchSource:0}: Error finding container b571d8719bdcb102eaae493a740ead48b8d3fa28aba9bf3fe0451a18a326338d: Status 404 returned error can't find the container with id b571d8719bdcb102eaae493a740ead48b8d3fa28aba9bf3fe0451a18a326338d
Sep 10 17:52:36.214744 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:52:36.214724    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-c523e8a0-5ff497dc74-bngcn"]
Sep 10 17:52:36.216946 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:52:36.216922    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2378963_fe0f_48ae_894a_234a6f77660f.slice/crio-74cace6301ecbf4485a913fe0bb3f6ca6fc656887636d3524a85a6bcfe53f6ee WatchSource:0}: Error finding container 74cace6301ecbf4485a913fe0bb3f6ca6fc656887636d3524a85a6bcfe53f6ee: Status 404 returned error can't find the container with id 74cace6301ecbf4485a913fe0bb3f6ca6fc656887636d3524a85a6bcfe53f6ee
Sep 10 17:52:36.616764 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:52:36.616728    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-c523e8a0-5ff497dc74-bngcn" event={"ID":"c2378963-fe0f-48ae-894a-234a6f77660f","Type":"ContainerStarted","Data":"f76b88baf9be12c9281c859aaffee5c61441e049cbe836d7bb82fcf49e008fd2"}
Sep 10 17:52:36.616946 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:52:36.616795    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-c523e8a0-5ff497dc74-bngcn" event={"ID":"c2378963-fe0f-48ae-894a-234a6f77660f","Type":"ContainerStarted","Data":"74cace6301ecbf4485a913fe0bb3f6ca6fc656887636d3524a85a6bcfe53f6ee"}
Sep 10 17:52:36.616946 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:52:36.616830    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-c523e8a0-5ff497dc74-bngcn"
Sep 10 17:52:36.618330 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:52:36.618294    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c523e8a0-c59685b47-nh2v9" event={"ID":"62eae552-879c-4b6e-bf09-ff51e9f44cf2","Type":"ContainerStarted","Data":"494ce94557c705bdbe983471baa88790801373eb5cb5eb0d498eb053cce97ad9"}
Sep 10 17:52:36.618445 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:52:36.618341    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c523e8a0-c59685b47-nh2v9" event={"ID":"62eae552-879c-4b6e-bf09-ff51e9f44cf2","Type":"ContainerStarted","Data":"b571d8719bdcb102eaae493a740ead48b8d3fa28aba9bf3fe0451a18a326338d"}
Sep 10 17:52:36.618505 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:52:36.618482    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c523e8a0-c59685b47-nh2v9"
Sep 10 17:52:36.635623 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:52:36.635582    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-c523e8a0-5ff497dc74-bngcn" podStartSLOduration=1.635548432 podStartE2EDuration="1.635548432s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:52:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:52:36.633162905 +0000 UTC m=+1000.632992973" watchObservedRunningTime="2026-09-10 17:52:36.635548432 +0000 UTC m=+1000.635378559"
Sep 10 17:52:36.651222 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:52:36.651181    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c523e8a0-c59685b47-nh2v9" podStartSLOduration=2.312602545 podStartE2EDuration="2.651171157s" totalImagesPullingTime="338.568612ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:52:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:52:36.650637758 +0000 UTC m=+1000.650467826" watchObservedRunningTime="2026-09-10 17:52:36.651171157 +0000 UTC m=+1000.651001224"
Sep 10 17:52:37.627944 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:52:37.627918    2579 generic.go:356] "Generic (PLEG): container finished" podID="2562421b-1343-44cb-98fa-ee27d02c7e38" containerID="b7695b1ab73b358730d4865f8fea450ed17274de1ada9c680589cf5cfa306cce" exitCode=0
Sep 10 17:52:37.628313 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:52:37.627990    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-8aeafa-openshift-default-b46f9b5b4-d5m5q" event={"ID":"2562421b-1343-44cb-98fa-ee27d02c7e38","Type":"ContainerDied","Data":"b7695b1ab73b358730d4865f8fea450ed17274de1ada9c680589cf5cfa306cce"}
Sep 10 17:52:37.628313 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:52:37.628046    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-8aeafa-openshift-default-b46f9b5b4-d5m5q" event={"ID":"2562421b-1343-44cb-98fa-ee27d02c7e38","Type":"ContainerDied","Data":"54b831a8ca3bd4b3432d1fff77d0c7b98f8aba5753b52bd017c455c34d429c4c"}
Sep 10 17:52:37.628313 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:52:37.628067    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="54b831a8ca3bd4b3432d1fff77d0c7b98f8aba5753b52bd017c455c34d429c4c"
Sep 10 17:52:37.629813 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:52:37.629796    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-8aeafa-openshift-default-b46f9b5b4-d5m5q"
Sep 10 17:52:37.715172 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:52:37.715109    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2562421b-1343-44cb-98fa-ee27d02c7e38-istio-data\") pod \"2562421b-1343-44cb-98fa-ee27d02c7e38\" (UID: \"2562421b-1343-44cb-98fa-ee27d02c7e38\") "
Sep 10 17:52:37.715172 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:52:37.715147    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2562421b-1343-44cb-98fa-ee27d02c7e38-istiod-ca-cert\") pod \"2562421b-1343-44cb-98fa-ee27d02c7e38\" (UID: \"2562421b-1343-44cb-98fa-ee27d02c7e38\") "
Sep 10 17:52:37.715399 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:52:37.715198    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2562421b-1343-44cb-98fa-ee27d02c7e38-istio-podinfo\") pod \"2562421b-1343-44cb-98fa-ee27d02c7e38\" (UID: \"2562421b-1343-44cb-98fa-ee27d02c7e38\") "
Sep 10 17:52:37.715399 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:52:37.715221    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2562421b-1343-44cb-98fa-ee27d02c7e38-istio-envoy\") pod \"2562421b-1343-44cb-98fa-ee27d02c7e38\" (UID: \"2562421b-1343-44cb-98fa-ee27d02c7e38\") "
Sep 10 17:52:37.715399 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:52:37.715238    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2562421b-1343-44cb-98fa-ee27d02c7e38-istio-token\") pod \"2562421b-1343-44cb-98fa-ee27d02c7e38\" (UID: \"2562421b-1343-44cb-98fa-ee27d02c7e38\") "
Sep 10 17:52:37.715399 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:52:37.715272    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2562421b-1343-44cb-98fa-ee27d02c7e38-workload-certs\") pod \"2562421b-1343-44cb-98fa-ee27d02c7e38\" (UID: \"2562421b-1343-44cb-98fa-ee27d02c7e38\") "
Sep 10 17:52:37.715399 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:52:37.715341    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2562421b-1343-44cb-98fa-ee27d02c7e38-istio-data" (OuterVolumeSpecName: "istio-data") pod "2562421b-1343-44cb-98fa-ee27d02c7e38" (UID: "2562421b-1343-44cb-98fa-ee27d02c7e38"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:52:37.715399 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:52:37.715394    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2562421b-1343-44cb-98fa-ee27d02c7e38-credential-socket\") pod \"2562421b-1343-44cb-98fa-ee27d02c7e38\" (UID: \"2562421b-1343-44cb-98fa-ee27d02c7e38\") "
Sep 10 17:52:37.715751 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:52:37.715456    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l89bl\" (UniqueName: \"kubernetes.io/projected/2562421b-1343-44cb-98fa-ee27d02c7e38-kube-api-access-l89bl\") pod \"2562421b-1343-44cb-98fa-ee27d02c7e38\" (UID: \"2562421b-1343-44cb-98fa-ee27d02c7e38\") "
Sep 10 17:52:37.715751 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:52:37.715521    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2562421b-1343-44cb-98fa-ee27d02c7e38-workload-socket\") pod \"2562421b-1343-44cb-98fa-ee27d02c7e38\" (UID: \"2562421b-1343-44cb-98fa-ee27d02c7e38\") "
Sep 10 17:52:37.715751 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:52:37.715555    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2562421b-1343-44cb-98fa-ee27d02c7e38-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "2562421b-1343-44cb-98fa-ee27d02c7e38" (UID: "2562421b-1343-44cb-98fa-ee27d02c7e38"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:52:37.715751 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:52:37.715660    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2562421b-1343-44cb-98fa-ee27d02c7e38-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "2562421b-1343-44cb-98fa-ee27d02c7e38" (UID: "2562421b-1343-44cb-98fa-ee27d02c7e38"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 17:52:37.715751 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:52:37.715718    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2562421b-1343-44cb-98fa-ee27d02c7e38-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "2562421b-1343-44cb-98fa-ee27d02c7e38" (UID: "2562421b-1343-44cb-98fa-ee27d02c7e38"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:52:37.716163 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:52:37.715908    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2562421b-1343-44cb-98fa-ee27d02c7e38-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "2562421b-1343-44cb-98fa-ee27d02c7e38" (UID: "2562421b-1343-44cb-98fa-ee27d02c7e38"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:52:37.716371 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:52:37.716351    2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2562421b-1343-44cb-98fa-ee27d02c7e38-workload-socket\") on node \"ip-10-0-129-122.ec2.internal\" DevicePath \"\""
Sep 10 17:52:37.716466 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:52:37.716373    2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2562421b-1343-44cb-98fa-ee27d02c7e38-istio-data\") on node \"ip-10-0-129-122.ec2.internal\" DevicePath \"\""
Sep 10 17:52:37.716466 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:52:37.716382    2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2562421b-1343-44cb-98fa-ee27d02c7e38-istiod-ca-cert\") on node \"ip-10-0-129-122.ec2.internal\" DevicePath \"\""
Sep 10 17:52:37.716466 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:52:37.716390    2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2562421b-1343-44cb-98fa-ee27d02c7e38-workload-certs\") on node \"ip-10-0-129-122.ec2.internal\" DevicePath \"\""
Sep 10 17:52:37.716466 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:52:37.716402    2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2562421b-1343-44cb-98fa-ee27d02c7e38-credential-socket\") on node \"ip-10-0-129-122.ec2.internal\" DevicePath \"\""
Sep 10 17:52:37.717698 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:52:37.717671    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2562421b-1343-44cb-98fa-ee27d02c7e38-istio-token" (OuterVolumeSpecName: "istio-token") pod "2562421b-1343-44cb-98fa-ee27d02c7e38" (UID: "2562421b-1343-44cb-98fa-ee27d02c7e38"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:52:37.718144 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:52:37.718125    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2562421b-1343-44cb-98fa-ee27d02c7e38-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "2562421b-1343-44cb-98fa-ee27d02c7e38" (UID: "2562421b-1343-44cb-98fa-ee27d02c7e38"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 10 17:52:37.718199 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:52:37.718140    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2562421b-1343-44cb-98fa-ee27d02c7e38-kube-api-access-l89bl" (OuterVolumeSpecName: "kube-api-access-l89bl") pod "2562421b-1343-44cb-98fa-ee27d02c7e38" (UID: "2562421b-1343-44cb-98fa-ee27d02c7e38"). InnerVolumeSpecName "kube-api-access-l89bl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:52:37.718318 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:52:37.718295    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2562421b-1343-44cb-98fa-ee27d02c7e38-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "2562421b-1343-44cb-98fa-ee27d02c7e38" (UID: "2562421b-1343-44cb-98fa-ee27d02c7e38"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:52:37.817440 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:52:37.817415    2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2562421b-1343-44cb-98fa-ee27d02c7e38-istio-envoy\") on node \"ip-10-0-129-122.ec2.internal\" DevicePath \"\""
Sep 10 17:52:37.817440 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:52:37.817439    2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2562421b-1343-44cb-98fa-ee27d02c7e38-istio-token\") on node \"ip-10-0-129-122.ec2.internal\" DevicePath \"\""
Sep 10 17:52:37.817606 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:52:37.817448    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l89bl\" (UniqueName: \"kubernetes.io/projected/2562421b-1343-44cb-98fa-ee27d02c7e38-kube-api-access-l89bl\") on node \"ip-10-0-129-122.ec2.internal\" DevicePath \"\""
Sep 10 17:52:37.817606 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:52:37.817457    2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2562421b-1343-44cb-98fa-ee27d02c7e38-istio-podinfo\") on node \"ip-10-0-129-122.ec2.internal\" DevicePath \"\""
Sep 10 17:52:38.633256 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:52:38.633228    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-8aeafa-openshift-default-b46f9b5b4-d5m5q"
Sep 10 17:52:38.653049 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:52:38.653023    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-8aeafa-openshift-default-b46f9b5b4-d5m5q"]
Sep 10 17:52:38.657160 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:52:38.657121    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-noten-8aeafa-openshift-default-b46f9b5b4-d5m5q"]
Sep 10 17:52:40.264233 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:52:40.264201    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-783a5e/test-model-783a5e-kserve-b6dbdcf95-4gwk4"]
Sep 10 17:52:40.475588 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:52:40.475535    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2562421b-1343-44cb-98fa-ee27d02c7e38" path="/var/lib/kubelet/pods/2562421b-1343-44cb-98fa-ee27d02c7e38/volumes"
Sep 10 17:52:42.634342 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:52:42.634315    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c523e8a0-c59685b47-nh2v9"
Sep 10 17:52:47.630246 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:52:47.630219    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-c523e8a0-5ff497dc74-bngcn"
Sep 10 17:52:50.896050 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:52:50.896012    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-48d00b-54fc5bb588-sntsg"]
Sep 10 17:52:50.896539 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:52:50.896276    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-48d00b-54fc5bb588-sntsg" podUID="b8dc46ba-3e39-4014-86ff-ea9e12e38c16" containerName="maas-api" containerID="cri-o://4f91330bcc42e66c0e26e3aa8e454a4f99e1ad6e733bc91361ebc36f12232996" gracePeriod=30
Sep 10 17:52:51.154406 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:52:51.154326    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-48d00b-54fc5bb588-sntsg"
Sep 10 17:52:51.340135 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:52:51.340104    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8dc46ba-3e39-4014-86ff-ea9e12e38c16-openshift-service-ca\") pod \"b8dc46ba-3e39-4014-86ff-ea9e12e38c16\" (UID: \"b8dc46ba-3e39-4014-86ff-ea9e12e38c16\") "
Sep 10 17:52:51.340135 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:52:51.340135    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b8dc46ba-3e39-4014-86ff-ea9e12e38c16-maas-api-tls\") pod \"b8dc46ba-3e39-4014-86ff-ea9e12e38c16\" (UID: \"b8dc46ba-3e39-4014-86ff-ea9e12e38c16\") "
Sep 10 17:52:51.340343 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:52:51.340217    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b8dc46ba-3e39-4014-86ff-ea9e12e38c16-metrics-certs\") pod \"b8dc46ba-3e39-4014-86ff-ea9e12e38c16\" (UID: \"b8dc46ba-3e39-4014-86ff-ea9e12e38c16\") "
Sep 10 17:52:51.340343 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:52:51.340244    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jhnbj\" (UniqueName: \"kubernetes.io/projected/b8dc46ba-3e39-4014-86ff-ea9e12e38c16-kube-api-access-jhnbj\") pod \"b8dc46ba-3e39-4014-86ff-ea9e12e38c16\" (UID: \"b8dc46ba-3e39-4014-86ff-ea9e12e38c16\") "
Sep 10 17:52:51.340512 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:52:51.340473    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8dc46ba-3e39-4014-86ff-ea9e12e38c16-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b8dc46ba-3e39-4014-86ff-ea9e12e38c16" (UID: "b8dc46ba-3e39-4014-86ff-ea9e12e38c16"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 17:52:51.342411 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:52:51.342389    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8dc46ba-3e39-4014-86ff-ea9e12e38c16-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "b8dc46ba-3e39-4014-86ff-ea9e12e38c16" (UID: "b8dc46ba-3e39-4014-86ff-ea9e12e38c16"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 17:52:51.342487 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:52:51.342440    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8dc46ba-3e39-4014-86ff-ea9e12e38c16-kube-api-access-jhnbj" (OuterVolumeSpecName: "kube-api-access-jhnbj") pod "b8dc46ba-3e39-4014-86ff-ea9e12e38c16" (UID: "b8dc46ba-3e39-4014-86ff-ea9e12e38c16"). InnerVolumeSpecName "kube-api-access-jhnbj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:52:51.342530 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:52:51.342509    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8dc46ba-3e39-4014-86ff-ea9e12e38c16-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b8dc46ba-3e39-4014-86ff-ea9e12e38c16" (UID: "b8dc46ba-3e39-4014-86ff-ea9e12e38c16"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 17:52:51.441609 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:52:51.441534    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jhnbj\" (UniqueName: \"kubernetes.io/projected/b8dc46ba-3e39-4014-86ff-ea9e12e38c16-kube-api-access-jhnbj\") on node \"ip-10-0-129-122.ec2.internal\" DevicePath \"\""
Sep 10 17:52:51.441609 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:52:51.441559    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8dc46ba-3e39-4014-86ff-ea9e12e38c16-openshift-service-ca\") on node \"ip-10-0-129-122.ec2.internal\" DevicePath \"\""
Sep 10 17:52:51.441609 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:52:51.441591    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b8dc46ba-3e39-4014-86ff-ea9e12e38c16-maas-api-tls\") on node \"ip-10-0-129-122.ec2.internal\" DevicePath \"\""
Sep 10 17:52:51.441609 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:52:51.441600    2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b8dc46ba-3e39-4014-86ff-ea9e12e38c16-metrics-certs\") on node \"ip-10-0-129-122.ec2.internal\" DevicePath \"\""
Sep 10 17:52:51.697995 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:52:51.697902    2579 generic.go:356] "Generic (PLEG): container finished" podID="b8dc46ba-3e39-4014-86ff-ea9e12e38c16" containerID="4f91330bcc42e66c0e26e3aa8e454a4f99e1ad6e733bc91361ebc36f12232996" exitCode=0
Sep 10 17:52:51.697995 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:52:51.697950    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-48d00b-54fc5bb588-sntsg"
Sep 10 17:52:51.698186 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:52:51.697998    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-48d00b-54fc5bb588-sntsg" event={"ID":"b8dc46ba-3e39-4014-86ff-ea9e12e38c16","Type":"ContainerDied","Data":"4f91330bcc42e66c0e26e3aa8e454a4f99e1ad6e733bc91361ebc36f12232996"}
Sep 10 17:52:51.698186 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:52:51.698046    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-48d00b-54fc5bb588-sntsg" event={"ID":"b8dc46ba-3e39-4014-86ff-ea9e12e38c16","Type":"ContainerDied","Data":"aa4914eced937d10789fb0b789698bc3e8430fccb7839867adc9785bf4f6e729"}
Sep 10 17:52:51.698186 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:52:51.698070    2579 scope.go:117] "RemoveContainer" containerID="4f91330bcc42e66c0e26e3aa8e454a4f99e1ad6e733bc91361ebc36f12232996"
Sep 10 17:52:51.707366 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:52:51.707349    2579 scope.go:117] "RemoveContainer" containerID="4f91330bcc42e66c0e26e3aa8e454a4f99e1ad6e733bc91361ebc36f12232996"
Sep 10 17:52:51.707668 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:52:51.707650    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f91330bcc42e66c0e26e3aa8e454a4f99e1ad6e733bc91361ebc36f12232996\": container with ID starting with 4f91330bcc42e66c0e26e3aa8e454a4f99e1ad6e733bc91361ebc36f12232996 not found: ID does not exist" containerID="4f91330bcc42e66c0e26e3aa8e454a4f99e1ad6e733bc91361ebc36f12232996"
Sep 10 17:52:51.707744 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:52:51.707676    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f91330bcc42e66c0e26e3aa8e454a4f99e1ad6e733bc91361ebc36f12232996"} err="failed to get container status \"4f91330bcc42e66c0e26e3aa8e454a4f99e1ad6e733bc91361ebc36f12232996\": rpc error: code = NotFound desc = could not find container \"4f91330bcc42e66c0e26e3aa8e454a4f99e1ad6e733bc91361ebc36f12232996\": container with ID starting with 4f91330bcc42e66c0e26e3aa8e454a4f99e1ad6e733bc91361ebc36f12232996 not found: ID does not exist"
Sep 10 17:52:51.722311 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:52:51.722282    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-48d00b-54fc5bb588-sntsg"]
Sep 10 17:52:51.725338 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:52:51.725316    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-48d00b-54fc5bb588-sntsg"]
Sep 10 17:52:52.475671 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:52:52.475632    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b8dc46ba-3e39-4014-86ff-ea9e12e38c16" path="/var/lib/kubelet/pods/b8dc46ba-3e39-4014-86ff-ea9e12e38c16/volumes"
Sep 10 17:52:55.285311 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:52:55.285271    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-783a5e/test-model-783a5e-kserve-b6dbdcf95-4gwk4" podUID="12a5b682-06e8-454a-b25a-522a129abcbc" containerName="main" containerID="cri-o://071950a170562f2970a05ed32a0a0de08ca12d51550bab31933b8dc53af143e8" gracePeriod=45
Sep 10 17:52:55.540510 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:52:55.540455    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-783a5e/test-model-783a5e-kserve-b6dbdcf95-4gwk4"
Sep 10 17:52:55.581131 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:52:55.581100    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/12a5b682-06e8-454a-b25a-522a129abcbc-home\") pod \"12a5b682-06e8-454a-b25a-522a129abcbc\" (UID: \"12a5b682-06e8-454a-b25a-522a129abcbc\") "
Sep 10 17:52:55.581272 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:52:55.581143    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/12a5b682-06e8-454a-b25a-522a129abcbc-model-cache\") pod \"12a5b682-06e8-454a-b25a-522a129abcbc\" (UID: \"12a5b682-06e8-454a-b25a-522a129abcbc\") "
Sep 10 17:52:55.581272 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:52:55.581192    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/12a5b682-06e8-454a-b25a-522a129abcbc-tls-certs\") pod \"12a5b682-06e8-454a-b25a-522a129abcbc\" (UID: \"12a5b682-06e8-454a-b25a-522a129abcbc\") "
Sep 10 17:52:55.581272 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:52:55.581220    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12a5b682-06e8-454a-b25a-522a129abcbc-tmp-dir\") pod \"12a5b682-06e8-454a-b25a-522a129abcbc\" (UID: \"12a5b682-06e8-454a-b25a-522a129abcbc\") "
Sep 10 17:52:55.581272 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:52:55.581254    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/12a5b682-06e8-454a-b25a-522a129abcbc-dshm\") pod \"12a5b682-06e8-454a-b25a-522a129abcbc\" (UID: \"12a5b682-06e8-454a-b25a-522a129abcbc\") "
Sep 10 17:52:55.581491 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:52:55.581297    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7d2vz\" (UniqueName: \"kubernetes.io/projected/12a5b682-06e8-454a-b25a-522a129abcbc-kube-api-access-7d2vz\") pod \"12a5b682-06e8-454a-b25a-522a129abcbc\" (UID: \"12a5b682-06e8-454a-b25a-522a129abcbc\") "
Sep 10 17:52:55.581491 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:52:55.581366    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12a5b682-06e8-454a-b25a-522a129abcbc-home" (OuterVolumeSpecName: "home") pod "12a5b682-06e8-454a-b25a-522a129abcbc" (UID: "12a5b682-06e8-454a-b25a-522a129abcbc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:52:55.581491 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:52:55.581452    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12a5b682-06e8-454a-b25a-522a129abcbc-model-cache" (OuterVolumeSpecName: "model-cache") pod "12a5b682-06e8-454a-b25a-522a129abcbc" (UID: "12a5b682-06e8-454a-b25a-522a129abcbc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:52:55.581700 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:52:55.581674    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/12a5b682-06e8-454a-b25a-522a129abcbc-home\") on node \"ip-10-0-129-122.ec2.internal\" DevicePath \"\""
Sep 10 17:52:55.581767 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:52:55.581697    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/12a5b682-06e8-454a-b25a-522a129abcbc-model-cache\") on node \"ip-10-0-129-122.ec2.internal\" DevicePath \"\""
Sep 10 17:52:55.581934 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:52:55.581893    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12a5b682-06e8-454a-b25a-522a129abcbc-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "12a5b682-06e8-454a-b25a-522a129abcbc" (UID: "12a5b682-06e8-454a-b25a-522a129abcbc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:52:55.584259 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:52:55.584227    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12a5b682-06e8-454a-b25a-522a129abcbc-dshm" (OuterVolumeSpecName: "dshm") pod "12a5b682-06e8-454a-b25a-522a129abcbc" (UID: "12a5b682-06e8-454a-b25a-522a129abcbc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:52:55.584337 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:52:55.584274    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12a5b682-06e8-454a-b25a-522a129abcbc-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "12a5b682-06e8-454a-b25a-522a129abcbc" (UID: "12a5b682-06e8-454a-b25a-522a129abcbc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 17:52:55.584337 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:52:55.584284    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12a5b682-06e8-454a-b25a-522a129abcbc-kube-api-access-7d2vz" (OuterVolumeSpecName: "kube-api-access-7d2vz") pod "12a5b682-06e8-454a-b25a-522a129abcbc" (UID: "12a5b682-06e8-454a-b25a-522a129abcbc"). InnerVolumeSpecName "kube-api-access-7d2vz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:52:55.682826 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:52:55.682796    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/12a5b682-06e8-454a-b25a-522a129abcbc-tls-certs\") on node \"ip-10-0-129-122.ec2.internal\" DevicePath \"\""
Sep 10 17:52:55.682826 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:52:55.682820    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12a5b682-06e8-454a-b25a-522a129abcbc-tmp-dir\") on node \"ip-10-0-129-122.ec2.internal\" DevicePath \"\""
Sep 10 17:52:55.682826 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:52:55.682828    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/12a5b682-06e8-454a-b25a-522a129abcbc-dshm\") on node \"ip-10-0-129-122.ec2.internal\" DevicePath \"\""
Sep 10 17:52:55.683002 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:52:55.682837    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7d2vz\" (UniqueName: \"kubernetes.io/projected/12a5b682-06e8-454a-b25a-522a129abcbc-kube-api-access-7d2vz\") on node \"ip-10-0-129-122.ec2.internal\" DevicePath \"\""
Sep 10 17:52:55.719878 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:52:55.719841    2579 generic.go:356] "Generic (PLEG): container finished" podID="12a5b682-06e8-454a-b25a-522a129abcbc" containerID="071950a170562f2970a05ed32a0a0de08ca12d51550bab31933b8dc53af143e8" exitCode=0
Sep 10 17:52:55.720030 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:52:55.719888    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-783a5e/test-model-783a5e-kserve-b6dbdcf95-4gwk4"
Sep 10 17:52:55.720030 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:52:55.719921    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-783a5e/test-model-783a5e-kserve-b6dbdcf95-4gwk4" event={"ID":"12a5b682-06e8-454a-b25a-522a129abcbc","Type":"ContainerDied","Data":"071950a170562f2970a05ed32a0a0de08ca12d51550bab31933b8dc53af143e8"}
Sep 10 17:52:55.720030 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:52:55.719965    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-783a5e/test-model-783a5e-kserve-b6dbdcf95-4gwk4" event={"ID":"12a5b682-06e8-454a-b25a-522a129abcbc","Type":"ContainerDied","Data":"314d810843c17e2dff4bff511d10ddace27744723fa53beaac24e6b7ae4e1be0"}
Sep 10 17:52:55.720030 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:52:55.719982    2579 scope.go:117] "RemoveContainer" containerID="071950a170562f2970a05ed32a0a0de08ca12d51550bab31933b8dc53af143e8"
Sep 10 17:52:55.729249 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:52:55.729226    2579 scope.go:117] "RemoveContainer" containerID="071950a170562f2970a05ed32a0a0de08ca12d51550bab31933b8dc53af143e8"
Sep 10 17:52:55.729519 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:52:55.729499    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"071950a170562f2970a05ed32a0a0de08ca12d51550bab31933b8dc53af143e8\": container with ID starting with 071950a170562f2970a05ed32a0a0de08ca12d51550bab31933b8dc53af143e8 not found: ID does not exist" containerID="071950a170562f2970a05ed32a0a0de08ca12d51550bab31933b8dc53af143e8"
Sep 10 17:52:55.729610 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:52:55.729527    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"071950a170562f2970a05ed32a0a0de08ca12d51550bab31933b8dc53af143e8"} err="failed to get container status \"071950a170562f2970a05ed32a0a0de08ca12d51550bab31933b8dc53af143e8\": rpc error: code = NotFound desc = could not find container \"071950a170562f2970a05ed32a0a0de08ca12d51550bab31933b8dc53af143e8\": container with ID starting with 071950a170562f2970a05ed32a0a0de08ca12d51550bab31933b8dc53af143e8 not found: ID does not exist"
Sep 10 17:52:55.742509 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:52:55.742481    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-783a5e/test-model-783a5e-kserve-b6dbdcf95-4gwk4"]
Sep 10 17:52:55.744119 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:52:55.744100    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-783a5e/test-model-783a5e-kserve-b6dbdcf95-4gwk4"]
Sep 10 17:52:56.474399 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:52:56.474368    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="12a5b682-06e8-454a-b25a-522a129abcbc" path="/var/lib/kubelet/pods/12a5b682-06e8-454a-b25a-522a129abcbc/volumes"
Sep 10 17:53:09.835296 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:53:09.835253    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-783a5e-77dd655589-szhsh"]
Sep 10 17:53:09.835797 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:53:09.835514    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-783a5e-77dd655589-szhsh" podUID="91e65bf9-acd8-45c9-9d04-16f8cf6f4286" containerName="payload-processing" containerID="cri-o://0141e1b4122a6a79b5fd728cc02e071cec4c8753a68384b43193f462018d5b9f" gracePeriod=30
Sep 10 17:53:10.106727 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:53:10.106695    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-783a5e-77dd655589-szhsh"
Sep 10 17:53:10.206479 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:53:10.206441    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91e65bf9-acd8-45c9-9d04-16f8cf6f4286-plugins-config-volume\") pod \"91e65bf9-acd8-45c9-9d04-16f8cf6f4286\" (UID: \"91e65bf9-acd8-45c9-9d04-16f8cf6f4286\") "
Sep 10 17:53:10.206707 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:53:10.206598    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-52tp9\" (UniqueName: \"kubernetes.io/projected/91e65bf9-acd8-45c9-9d04-16f8cf6f4286-kube-api-access-52tp9\") pod \"91e65bf9-acd8-45c9-9d04-16f8cf6f4286\" (UID: \"91e65bf9-acd8-45c9-9d04-16f8cf6f4286\") "
Sep 10 17:53:10.206930 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:53:10.206890    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91e65bf9-acd8-45c9-9d04-16f8cf6f4286-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "91e65bf9-acd8-45c9-9d04-16f8cf6f4286" (UID: "91e65bf9-acd8-45c9-9d04-16f8cf6f4286"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 17:53:10.208985 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:53:10.208953    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91e65bf9-acd8-45c9-9d04-16f8cf6f4286-kube-api-access-52tp9" (OuterVolumeSpecName: "kube-api-access-52tp9") pod "91e65bf9-acd8-45c9-9d04-16f8cf6f4286" (UID: "91e65bf9-acd8-45c9-9d04-16f8cf6f4286"). InnerVolumeSpecName "kube-api-access-52tp9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:53:10.307576 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:53:10.307539    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-52tp9\" (UniqueName: \"kubernetes.io/projected/91e65bf9-acd8-45c9-9d04-16f8cf6f4286-kube-api-access-52tp9\") on node \"ip-10-0-129-122.ec2.internal\" DevicePath \"\""
Sep 10 17:53:10.307761 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:53:10.307601    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91e65bf9-acd8-45c9-9d04-16f8cf6f4286-plugins-config-volume\") on node \"ip-10-0-129-122.ec2.internal\" DevicePath \"\""
Sep 10 17:53:10.797167 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:53:10.797069    2579 generic.go:356] "Generic (PLEG): container finished" podID="91e65bf9-acd8-45c9-9d04-16f8cf6f4286" containerID="0141e1b4122a6a79b5fd728cc02e071cec4c8753a68384b43193f462018d5b9f" exitCode=0
Sep 10 17:53:10.797167 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:53:10.797126    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-783a5e-77dd655589-szhsh"
Sep 10 17:53:10.797351 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:53:10.797160    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-783a5e-77dd655589-szhsh" event={"ID":"91e65bf9-acd8-45c9-9d04-16f8cf6f4286","Type":"ContainerDied","Data":"0141e1b4122a6a79b5fd728cc02e071cec4c8753a68384b43193f462018d5b9f"}
Sep 10 17:53:10.797351 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:53:10.797247    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-783a5e-77dd655589-szhsh" event={"ID":"91e65bf9-acd8-45c9-9d04-16f8cf6f4286","Type":"ContainerDied","Data":"36d3340bec1283cacd92e56c003c030ef4c1418fd69b0b13c064524d7f742705"}
Sep 10 17:53:10.797351 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:53:10.797274    2579 scope.go:117] "RemoveContainer" containerID="0141e1b4122a6a79b5fd728cc02e071cec4c8753a68384b43193f462018d5b9f"
Sep 10 17:53:10.806340 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:53:10.806315    2579 scope.go:117] "RemoveContainer" containerID="0141e1b4122a6a79b5fd728cc02e071cec4c8753a68384b43193f462018d5b9f"
Sep 10 17:53:10.806663 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:53:10.806642    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0141e1b4122a6a79b5fd728cc02e071cec4c8753a68384b43193f462018d5b9f\": container with ID starting with 0141e1b4122a6a79b5fd728cc02e071cec4c8753a68384b43193f462018d5b9f not found: ID does not exist" containerID="0141e1b4122a6a79b5fd728cc02e071cec4c8753a68384b43193f462018d5b9f"
Sep 10 17:53:10.806719 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:53:10.806672    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0141e1b4122a6a79b5fd728cc02e071cec4c8753a68384b43193f462018d5b9f"} err="failed to get container status \"0141e1b4122a6a79b5fd728cc02e071cec4c8753a68384b43193f462018d5b9f\": rpc error: code = NotFound desc = could not find container \"0141e1b4122a6a79b5fd728cc02e071cec4c8753a68384b43193f462018d5b9f\": container with ID starting with 0141e1b4122a6a79b5fd728cc02e071cec4c8753a68384b43193f462018d5b9f not found: ID does not exist"
Sep 10 17:53:10.820324 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:53:10.820294    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-783a5e-77dd655589-szhsh"]
Sep 10 17:53:10.822001 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:53:10.821981    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-783a5e-77dd655589-szhsh"]
Sep 10 17:53:12.476310 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:53:12.476276    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="91e65bf9-acd8-45c9-9d04-16f8cf6f4286" path="/var/lib/kubelet/pods/91e65bf9-acd8-45c9-9d04-16f8cf6f4286/volumes"
Sep 10 17:53:17.166641 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:53:17.166610    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c523e8a0-c59685b47-nh2v9"]
Sep 10 17:53:17.167019 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:53:17.166834    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c523e8a0-c59685b47-nh2v9" podUID="62eae552-879c-4b6e-bf09-ff51e9f44cf2" containerName="maas-api" containerID="cri-o://494ce94557c705bdbe983471baa88790801373eb5cb5eb0d498eb053cce97ad9" gracePeriod=30
Sep 10 17:53:17.288702 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:53:17.288677    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-c523e8a0-5ff497dc74-bngcn"]
Sep 10 17:53:17.288902 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:53:17.288881    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-c523e8a0-5ff497dc74-bngcn" podUID="c2378963-fe0f-48ae-894a-234a6f77660f" containerName="payload-pre-processing" containerID="cri-o://f76b88baf9be12c9281c859aaffee5c61441e049cbe836d7bb82fcf49e008fd2" gracePeriod=30
Sep 10 17:53:17.428503 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:53:17.428430    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c523e8a0-c59685b47-nh2v9"
Sep 10 17:53:17.540785 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:53:17.540756    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-c523e8a0-5ff497dc74-bngcn"
Sep 10 17:53:17.569911 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:53:17.569879    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/62eae552-879c-4b6e-bf09-ff51e9f44cf2-openshift-service-ca\") pod \"62eae552-879c-4b6e-bf09-ff51e9f44cf2\" (UID: \"62eae552-879c-4b6e-bf09-ff51e9f44cf2\") "
Sep 10 17:53:17.570077 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:53:17.569971    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/62eae552-879c-4b6e-bf09-ff51e9f44cf2-metrics-certs\") pod \"62eae552-879c-4b6e-bf09-ff51e9f44cf2\" (UID: \"62eae552-879c-4b6e-bf09-ff51e9f44cf2\") "
Sep 10 17:53:17.570077 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:53:17.570001    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/62eae552-879c-4b6e-bf09-ff51e9f44cf2-maas-api-tls\") pod \"62eae552-879c-4b6e-bf09-ff51e9f44cf2\" (UID: \"62eae552-879c-4b6e-bf09-ff51e9f44cf2\") "
Sep 10 17:53:17.570077 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:53:17.570032    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2stcb\" (UniqueName: \"kubernetes.io/projected/62eae552-879c-4b6e-bf09-ff51e9f44cf2-kube-api-access-2stcb\") pod \"62eae552-879c-4b6e-bf09-ff51e9f44cf2\" (UID: \"62eae552-879c-4b6e-bf09-ff51e9f44cf2\") "
Sep 10 17:53:17.570281 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:53:17.570257    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62eae552-879c-4b6e-bf09-ff51e9f44cf2-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "62eae552-879c-4b6e-bf09-ff51e9f44cf2" (UID: "62eae552-879c-4b6e-bf09-ff51e9f44cf2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 17:53:17.572474 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:53:17.572441    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62eae552-879c-4b6e-bf09-ff51e9f44cf2-kube-api-access-2stcb" (OuterVolumeSpecName: "kube-api-access-2stcb") pod "62eae552-879c-4b6e-bf09-ff51e9f44cf2" (UID: "62eae552-879c-4b6e-bf09-ff51e9f44cf2"). InnerVolumeSpecName "kube-api-access-2stcb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:53:17.572788 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:53:17.572757    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62eae552-879c-4b6e-bf09-ff51e9f44cf2-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "62eae552-879c-4b6e-bf09-ff51e9f44cf2" (UID: "62eae552-879c-4b6e-bf09-ff51e9f44cf2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 17:53:17.572901 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:53:17.572801    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62eae552-879c-4b6e-bf09-ff51e9f44cf2-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "62eae552-879c-4b6e-bf09-ff51e9f44cf2" (UID: "62eae552-879c-4b6e-bf09-ff51e9f44cf2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 17:53:17.670723 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:53:17.670690    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2378963-fe0f-48ae-894a-234a6f77660f-plugins-config-volume\") pod \"c2378963-fe0f-48ae-894a-234a6f77660f\" (UID: \"c2378963-fe0f-48ae-894a-234a6f77660f\") "
Sep 10 17:53:17.670891 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:53:17.670745    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2nnk5\" (UniqueName: \"kubernetes.io/projected/c2378963-fe0f-48ae-894a-234a6f77660f-kube-api-access-2nnk5\") pod \"c2378963-fe0f-48ae-894a-234a6f77660f\" (UID: \"c2378963-fe0f-48ae-894a-234a6f77660f\") "
Sep 10 17:53:17.670966 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:53:17.670926    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/62eae552-879c-4b6e-bf09-ff51e9f44cf2-openshift-service-ca\") on node \"ip-10-0-129-122.ec2.internal\" DevicePath \"\""
Sep 10 17:53:17.670966 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:53:17.670937    2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/62eae552-879c-4b6e-bf09-ff51e9f44cf2-metrics-certs\") on node \"ip-10-0-129-122.ec2.internal\" DevicePath \"\""
Sep 10 17:53:17.670966 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:53:17.670946    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/62eae552-879c-4b6e-bf09-ff51e9f44cf2-maas-api-tls\") on node \"ip-10-0-129-122.ec2.internal\" DevicePath \"\""
Sep 10 17:53:17.670966 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:53:17.670954    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2stcb\" (UniqueName: \"kubernetes.io/projected/62eae552-879c-4b6e-bf09-ff51e9f44cf2-kube-api-access-2stcb\") on node \"ip-10-0-129-122.ec2.internal\" DevicePath \"\""
Sep 10 17:53:17.671176 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:53:17.671011    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2378963-fe0f-48ae-894a-234a6f77660f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c2378963-fe0f-48ae-894a-234a6f77660f" (UID: "c2378963-fe0f-48ae-894a-234a6f77660f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 17:53:17.672906 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:53:17.672882    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2378963-fe0f-48ae-894a-234a6f77660f-kube-api-access-2nnk5" (OuterVolumeSpecName: "kube-api-access-2nnk5") pod "c2378963-fe0f-48ae-894a-234a6f77660f" (UID: "c2378963-fe0f-48ae-894a-234a6f77660f"). InnerVolumeSpecName "kube-api-access-2nnk5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:53:17.771596 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:53:17.771515    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2378963-fe0f-48ae-894a-234a6f77660f-plugins-config-volume\") on node \"ip-10-0-129-122.ec2.internal\" DevicePath \"\""
Sep 10 17:53:17.771596 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:53:17.771542    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2nnk5\" (UniqueName: \"kubernetes.io/projected/c2378963-fe0f-48ae-894a-234a6f77660f-kube-api-access-2nnk5\") on node \"ip-10-0-129-122.ec2.internal\" DevicePath \"\""
Sep 10 17:53:17.834945 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:53:17.834910    2579 generic.go:356] "Generic (PLEG): container finished" podID="c2378963-fe0f-48ae-894a-234a6f77660f" containerID="f76b88baf9be12c9281c859aaffee5c61441e049cbe836d7bb82fcf49e008fd2" exitCode=0
Sep 10 17:53:17.835112 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:53:17.834956    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-c523e8a0-5ff497dc74-bngcn"
Sep 10 17:53:17.835112 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:53:17.835005    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-c523e8a0-5ff497dc74-bngcn" event={"ID":"c2378963-fe0f-48ae-894a-234a6f77660f","Type":"ContainerDied","Data":"f76b88baf9be12c9281c859aaffee5c61441e049cbe836d7bb82fcf49e008fd2"}
Sep 10 17:53:17.835112 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:53:17.835059    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-c523e8a0-5ff497dc74-bngcn" event={"ID":"c2378963-fe0f-48ae-894a-234a6f77660f","Type":"ContainerDied","Data":"74cace6301ecbf4485a913fe0bb3f6ca6fc656887636d3524a85a6bcfe53f6ee"}
Sep 10 17:53:17.835112 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:53:17.835085    2579 scope.go:117] "RemoveContainer" containerID="f76b88baf9be12c9281c859aaffee5c61441e049cbe836d7bb82fcf49e008fd2"
Sep 10 17:53:17.836424 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:53:17.836399    2579 generic.go:356] "Generic (PLEG): container finished" podID="62eae552-879c-4b6e-bf09-ff51e9f44cf2" containerID="494ce94557c705bdbe983471baa88790801373eb5cb5eb0d498eb053cce97ad9" exitCode=0
Sep 10 17:53:17.836555 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:53:17.836438    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c523e8a0-c59685b47-nh2v9"
Sep 10 17:53:17.836555 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:53:17.836457    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c523e8a0-c59685b47-nh2v9" event={"ID":"62eae552-879c-4b6e-bf09-ff51e9f44cf2","Type":"ContainerDied","Data":"494ce94557c705bdbe983471baa88790801373eb5cb5eb0d498eb053cce97ad9"}
Sep 10 17:53:17.836705 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:53:17.836540    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c523e8a0-c59685b47-nh2v9" event={"ID":"62eae552-879c-4b6e-bf09-ff51e9f44cf2","Type":"ContainerDied","Data":"b571d8719bdcb102eaae493a740ead48b8d3fa28aba9bf3fe0451a18a326338d"}
Sep 10 17:53:17.843507 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:53:17.843490    2579 scope.go:117] "RemoveContainer" containerID="f76b88baf9be12c9281c859aaffee5c61441e049cbe836d7bb82fcf49e008fd2"
Sep 10 17:53:17.843807 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:53:17.843781    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f76b88baf9be12c9281c859aaffee5c61441e049cbe836d7bb82fcf49e008fd2\": container with ID starting with f76b88baf9be12c9281c859aaffee5c61441e049cbe836d7bb82fcf49e008fd2 not found: ID does not exist" containerID="f76b88baf9be12c9281c859aaffee5c61441e049cbe836d7bb82fcf49e008fd2"
Sep 10 17:53:17.843913 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:53:17.843814    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f76b88baf9be12c9281c859aaffee5c61441e049cbe836d7bb82fcf49e008fd2"} err="failed to get container status \"f76b88baf9be12c9281c859aaffee5c61441e049cbe836d7bb82fcf49e008fd2\": rpc error: code = NotFound desc = could not find container \"f76b88baf9be12c9281c859aaffee5c61441e049cbe836d7bb82fcf49e008fd2\": container with ID starting with f76b88baf9be12c9281c859aaffee5c61441e049cbe836d7bb82fcf49e008fd2 not found: ID does not exist"
Sep 10 17:53:17.843913 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:53:17.843831    2579 scope.go:117] "RemoveContainer" containerID="494ce94557c705bdbe983471baa88790801373eb5cb5eb0d498eb053cce97ad9"
Sep 10 17:53:17.852537 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:53:17.852521    2579 scope.go:117] "RemoveContainer" containerID="494ce94557c705bdbe983471baa88790801373eb5cb5eb0d498eb053cce97ad9"
Sep 10 17:53:17.852844 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:53:17.852828    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"494ce94557c705bdbe983471baa88790801373eb5cb5eb0d498eb053cce97ad9\": container with ID starting with 494ce94557c705bdbe983471baa88790801373eb5cb5eb0d498eb053cce97ad9 not found: ID does not exist" containerID="494ce94557c705bdbe983471baa88790801373eb5cb5eb0d498eb053cce97ad9"
Sep 10 17:53:17.852897 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:53:17.852849    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"494ce94557c705bdbe983471baa88790801373eb5cb5eb0d498eb053cce97ad9"} err="failed to get container status \"494ce94557c705bdbe983471baa88790801373eb5cb5eb0d498eb053cce97ad9\": rpc error: code = NotFound desc = could not find container \"494ce94557c705bdbe983471baa88790801373eb5cb5eb0d498eb053cce97ad9\": container with ID starting with 494ce94557c705bdbe983471baa88790801373eb5cb5eb0d498eb053cce97ad9 not found: ID does not exist"
Sep 10 17:53:17.882449 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:53:17.882429    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c523e8a0-c59685b47-nh2v9"]
Sep 10 17:53:17.895954 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:53:17.895930    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c523e8a0-c59685b47-nh2v9"]
Sep 10 17:53:17.944327 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:53:17.944304    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-c523e8a0-5ff497dc74-bngcn"]
Sep 10 17:53:17.952595 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:53:17.952550    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-c523e8a0-5ff497dc74-bngcn"]
Sep 10 17:53:18.475094 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:53:18.475019    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="62eae552-879c-4b6e-bf09-ff51e9f44cf2" path="/var/lib/kubelet/pods/62eae552-879c-4b6e-bf09-ff51e9f44cf2/volumes"
Sep 10 17:53:18.475429 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:53:18.475368    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c2378963-fe0f-48ae-894a-234a6f77660f" path="/var/lib/kubelet/pods/c2378963-fe0f-48ae-894a-234a6f77660f/volumes"
Sep 10 17:53:19.053379 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:53:19.053348    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-18ecb2-openshift-default-86c6bff59d-9zh8x"]
Sep 10 17:53:19.053713 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:53:19.053701    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="12a5b682-06e8-454a-b25a-522a129abcbc" containerName="main"
Sep 10 17:53:19.053777 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:53:19.053715    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="12a5b682-06e8-454a-b25a-522a129abcbc" containerName="main"
Sep 10 17:53:19.053777 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:53:19.053724    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="91e65bf9-acd8-45c9-9d04-16f8cf6f4286" containerName="payload-processing"
Sep 10 17:53:19.053777 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:53:19.053730    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="91e65bf9-acd8-45c9-9d04-16f8cf6f4286" containerName="payload-processing"
Sep 10 17:53:19.053777 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:53:19.053747    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8dc46ba-3e39-4014-86ff-ea9e12e38c16" containerName="maas-api"
Sep 10 17:53:19.053777 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:53:19.053752    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8dc46ba-3e39-4014-86ff-ea9e12e38c16" containerName="maas-api"
Sep 10 17:53:19.053777 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:53:19.053760    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c2378963-fe0f-48ae-894a-234a6f77660f" containerName="payload-pre-processing"
Sep 10 17:53:19.053777 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:53:19.053765    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="c2378963-fe0f-48ae-894a-234a6f77660f" containerName="payload-pre-processing"
Sep 10 17:53:19.053777 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:53:19.053772    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62eae552-879c-4b6e-bf09-ff51e9f44cf2" containerName="maas-api"
Sep 10 17:53:19.053777 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:53:19.053777    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="62eae552-879c-4b6e-bf09-ff51e9f44cf2" containerName="maas-api"
Sep 10 17:53:19.054105 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:53:19.053783    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2562421b-1343-44cb-98fa-ee27d02c7e38" containerName="istio-proxy"
Sep 10 17:53:19.054105 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:53:19.053788    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="2562421b-1343-44cb-98fa-ee27d02c7e38" containerName="istio-proxy"
Sep 10 17:53:19.054105 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:53:19.053842    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="91e65bf9-acd8-45c9-9d04-16f8cf6f4286" containerName="payload-processing"
Sep 10 17:53:19.054105 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:53:19.053850    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="2562421b-1343-44cb-98fa-ee27d02c7e38" containerName="istio-proxy"
Sep 10 17:53:19.054105 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:53:19.053857    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="c2378963-fe0f-48ae-894a-234a6f77660f" containerName="payload-pre-processing"
Sep 10 17:53:19.054105 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:53:19.053864    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="62eae552-879c-4b6e-bf09-ff51e9f44cf2" containerName="maas-api"
Sep 10 17:53:19.054105 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:53:19.053870    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="b8dc46ba-3e39-4014-86ff-ea9e12e38c16" containerName="maas-api"
Sep 10 17:53:19.054105 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:53:19.053876    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="12a5b682-06e8-454a-b25a-522a129abcbc" containerName="main"
Sep 10 17:53:19.058145 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:53:19.058124    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-18ecb2-openshift-default-86c6bff59d-9zh8x"
Sep 10 17:53:19.070873 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:53:19.070850    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 10 17:53:19.070994 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:53:19.070912    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-18ecb2-openshift-default-dockercfg-v8dg2\""
Sep 10 17:53:19.139109 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:53:19.139080    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-18ecb2-openshift-default-86c6bff59d-9zh8x"]
Sep 10 17:53:19.182237 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:53:19.182210    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5268e09-33ea-4f48-97cc-24afd8b14c63-istio-podinfo\") pod \"e2e-shared-a-w3-18ecb2-openshift-default-86c6bff59d-9zh8x\" (UID: \"d5268e09-33ea-4f48-97cc-24afd8b14c63\") " pod="openshift-ingress/e2e-shared-a-w3-18ecb2-openshift-default-86c6bff59d-9zh8x"
Sep 10 17:53:19.182393 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:53:19.182258    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5268e09-33ea-4f48-97cc-24afd8b14c63-credential-socket\") pod \"e2e-shared-a-w3-18ecb2-openshift-default-86c6bff59d-9zh8x\" (UID: \"d5268e09-33ea-4f48-97cc-24afd8b14c63\") " pod="openshift-ingress/e2e-shared-a-w3-18ecb2-openshift-default-86c6bff59d-9zh8x"
Sep 10 17:53:19.182393 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:53:19.182338    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5268e09-33ea-4f48-97cc-24afd8b14c63-istio-envoy\") pod \"e2e-shared-a-w3-18ecb2-openshift-default-86c6bff59d-9zh8x\" (UID: \"d5268e09-33ea-4f48-97cc-24afd8b14c63\") " pod="openshift-ingress/e2e-shared-a-w3-18ecb2-openshift-default-86c6bff59d-9zh8x"
Sep 10 17:53:19.182494 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:53:19.182442    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5268e09-33ea-4f48-97cc-24afd8b14c63-istio-data\") pod \"e2e-shared-a-w3-18ecb2-openshift-default-86c6bff59d-9zh8x\" (UID: \"d5268e09-33ea-4f48-97cc-24afd8b14c63\") " pod="openshift-ingress/e2e-shared-a-w3-18ecb2-openshift-default-86c6bff59d-9zh8x"
Sep 10 17:53:19.182494 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:53:19.182481    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5268e09-33ea-4f48-97cc-24afd8b14c63-istiod-ca-cert\") pod \"e2e-shared-a-w3-18ecb2-openshift-default-86c6bff59d-9zh8x\" (UID: \"d5268e09-33ea-4f48-97cc-24afd8b14c63\") " pod="openshift-ingress/e2e-shared-a-w3-18ecb2-openshift-default-86c6bff59d-9zh8x"
Sep 10 17:53:19.182673 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:53:19.182508    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68rm9\" (UniqueName: \"kubernetes.io/projected/d5268e09-33ea-4f48-97cc-24afd8b14c63-kube-api-access-68rm9\") pod \"e2e-shared-a-w3-18ecb2-openshift-default-86c6bff59d-9zh8x\" (UID: \"d5268e09-33ea-4f48-97cc-24afd8b14c63\") " pod="openshift-ingress/e2e-shared-a-w3-18ecb2-openshift-default-86c6bff59d-9zh8x"
Sep 10 17:53:19.182673 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:53:19.182591    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5268e09-33ea-4f48-97cc-24afd8b14c63-workload-socket\") pod \"e2e-shared-a-w3-18ecb2-openshift-default-86c6bff59d-9zh8x\" (UID: \"d5268e09-33ea-4f48-97cc-24afd8b14c63\") " pod="openshift-ingress/e2e-shared-a-w3-18ecb2-openshift-default-86c6bff59d-9zh8x"
Sep 10 17:53:19.182673 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:53:19.182623    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5268e09-33ea-4f48-97cc-24afd8b14c63-workload-certs\") pod \"e2e-shared-a-w3-18ecb2-openshift-default-86c6bff59d-9zh8x\" (UID: \"d5268e09-33ea-4f48-97cc-24afd8b14c63\") " pod="openshift-ingress/e2e-shared-a-w3-18ecb2-openshift-default-86c6bff59d-9zh8x"
Sep 10 17:53:19.182673 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:53:19.182646    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5268e09-33ea-4f48-97cc-24afd8b14c63-istio-token\") pod \"e2e-shared-a-w3-18ecb2-openshift-default-86c6bff59d-9zh8x\" (UID: \"d5268e09-33ea-4f48-97cc-24afd8b14c63\") " pod="openshift-ingress/e2e-shared-a-w3-18ecb2-openshift-default-86c6bff59d-9zh8x"
Sep 10 17:53:19.283953 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:53:19.283930    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5268e09-33ea-4f48-97cc-24afd8b14c63-credential-socket\") pod \"e2e-shared-a-w3-18ecb2-openshift-default-86c6bff59d-9zh8x\" (UID: \"d5268e09-33ea-4f48-97cc-24afd8b14c63\") " pod="openshift-ingress/e2e-shared-a-w3-18ecb2-openshift-default-86c6bff59d-9zh8x"
Sep 10 17:53:19.284091 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:53:19.283971    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5268e09-33ea-4f48-97cc-24afd8b14c63-istio-envoy\") pod \"e2e-shared-a-w3-18ecb2-openshift-default-86c6bff59d-9zh8x\" (UID: \"d5268e09-33ea-4f48-97cc-24afd8b14c63\") " pod="openshift-ingress/e2e-shared-a-w3-18ecb2-openshift-default-86c6bff59d-9zh8x"
Sep 10 17:53:19.284091 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:53:19.283999    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5268e09-33ea-4f48-97cc-24afd8b14c63-istio-data\") pod \"e2e-shared-a-w3-18ecb2-openshift-default-86c6bff59d-9zh8x\" (UID: \"d5268e09-33ea-4f48-97cc-24afd8b14c63\") " pod="openshift-ingress/e2e-shared-a-w3-18ecb2-openshift-default-86c6bff59d-9zh8x"
Sep 10 17:53:19.284091 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:53:19.284017    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5268e09-33ea-4f48-97cc-24afd8b14c63-istiod-ca-cert\") pod \"e2e-shared-a-w3-18ecb2-openshift-default-86c6bff59d-9zh8x\" (UID: \"d5268e09-33ea-4f48-97cc-24afd8b14c63\") " pod="openshift-ingress/e2e-shared-a-w3-18ecb2-openshift-default-86c6bff59d-9zh8x"
Sep 10 17:53:19.284255 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:53:19.284129    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68rm9\" (UniqueName: \"kubernetes.io/projected/d5268e09-33ea-4f48-97cc-24afd8b14c63-kube-api-access-68rm9\") pod \"e2e-shared-a-w3-18ecb2-openshift-default-86c6bff59d-9zh8x\" (UID: \"d5268e09-33ea-4f48-97cc-24afd8b14c63\") " pod="openshift-ingress/e2e-shared-a-w3-18ecb2-openshift-default-86c6bff59d-9zh8x"
Sep 10 17:53:19.284255 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:53:19.284163    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5268e09-33ea-4f48-97cc-24afd8b14c63-workload-socket\") pod \"e2e-shared-a-w3-18ecb2-openshift-default-86c6bff59d-9zh8x\" (UID: \"d5268e09-33ea-4f48-97cc-24afd8b14c63\") " pod="openshift-ingress/e2e-shared-a-w3-18ecb2-openshift-default-86c6bff59d-9zh8x"
Sep 10 17:53:19.284255 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:53:19.284183    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5268e09-33ea-4f48-97cc-24afd8b14c63-workload-certs\") pod \"e2e-shared-a-w3-18ecb2-openshift-default-86c6bff59d-9zh8x\" (UID: \"d5268e09-33ea-4f48-97cc-24afd8b14c63\") " pod="openshift-ingress/e2e-shared-a-w3-18ecb2-openshift-default-86c6bff59d-9zh8x"
Sep 10 17:53:19.284255 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:53:19.284204    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5268e09-33ea-4f48-97cc-24afd8b14c63-istio-token\") pod \"e2e-shared-a-w3-18ecb2-openshift-default-86c6bff59d-9zh8x\" (UID: \"d5268e09-33ea-4f48-97cc-24afd8b14c63\") " pod="openshift-ingress/e2e-shared-a-w3-18ecb2-openshift-default-86c6bff59d-9zh8x"
Sep 10 17:53:19.284255 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:53:19.284248    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5268e09-33ea-4f48-97cc-24afd8b14c63-istio-podinfo\") pod \"e2e-shared-a-w3-18ecb2-openshift-default-86c6bff59d-9zh8x\" (UID: \"d5268e09-33ea-4f48-97cc-24afd8b14c63\") " pod="openshift-ingress/e2e-shared-a-w3-18ecb2-openshift-default-86c6bff59d-9zh8x"
Sep 10 17:53:19.284518 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:53:19.284346    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5268e09-33ea-4f48-97cc-24afd8b14c63-credential-socket\") pod \"e2e-shared-a-w3-18ecb2-openshift-default-86c6bff59d-9zh8x\" (UID: \"d5268e09-33ea-4f48-97cc-24afd8b14c63\") " pod="openshift-ingress/e2e-shared-a-w3-18ecb2-openshift-default-86c6bff59d-9zh8x"
Sep 10 17:53:19.284518 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:53:19.284409    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5268e09-33ea-4f48-97cc-24afd8b14c63-istio-data\") pod \"e2e-shared-a-w3-18ecb2-openshift-default-86c6bff59d-9zh8x\" (UID: \"d5268e09-33ea-4f48-97cc-24afd8b14c63\") " pod="openshift-ingress/e2e-shared-a-w3-18ecb2-openshift-default-86c6bff59d-9zh8x"
Sep 10 17:53:19.284684 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:53:19.284638    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5268e09-33ea-4f48-97cc-24afd8b14c63-workload-certs\") pod \"e2e-shared-a-w3-18ecb2-openshift-default-86c6bff59d-9zh8x\" (UID: \"d5268e09-33ea-4f48-97cc-24afd8b14c63\") " pod="openshift-ingress/e2e-shared-a-w3-18ecb2-openshift-default-86c6bff59d-9zh8x"
Sep 10 17:53:19.284800 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:53:19.284780    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5268e09-33ea-4f48-97cc-24afd8b14c63-workload-socket\") pod \"e2e-shared-a-w3-18ecb2-openshift-default-86c6bff59d-9zh8x\" (UID: \"d5268e09-33ea-4f48-97cc-24afd8b14c63\") " pod="openshift-ingress/e2e-shared-a-w3-18ecb2-openshift-default-86c6bff59d-9zh8x"
Sep 10 17:53:19.284891 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:53:19.284873    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5268e09-33ea-4f48-97cc-24afd8b14c63-istiod-ca-cert\") pod \"e2e-shared-a-w3-18ecb2-openshift-default-86c6bff59d-9zh8x\" (UID: \"d5268e09-33ea-4f48-97cc-24afd8b14c63\") " pod="openshift-ingress/e2e-shared-a-w3-18ecb2-openshift-default-86c6bff59d-9zh8x"
Sep 10 17:53:19.286752 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:53:19.286733    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5268e09-33ea-4f48-97cc-24afd8b14c63-istio-podinfo\") pod \"e2e-shared-a-w3-18ecb2-openshift-default-86c6bff59d-9zh8x\" (UID: \"d5268e09-33ea-4f48-97cc-24afd8b14c63\") " pod="openshift-ingress/e2e-shared-a-w3-18ecb2-openshift-default-86c6bff59d-9zh8x"
Sep 10 17:53:19.286834 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:53:19.286733    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5268e09-33ea-4f48-97cc-24afd8b14c63-istio-envoy\") pod \"e2e-shared-a-w3-18ecb2-openshift-default-86c6bff59d-9zh8x\" (UID: \"d5268e09-33ea-4f48-97cc-24afd8b14c63\") " pod="openshift-ingress/e2e-shared-a-w3-18ecb2-openshift-default-86c6bff59d-9zh8x"
Sep 10 17:53:19.296865 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:53:19.296838    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5268e09-33ea-4f48-97cc-24afd8b14c63-istio-token\") pod \"e2e-shared-a-w3-18ecb2-openshift-default-86c6bff59d-9zh8x\" (UID: \"d5268e09-33ea-4f48-97cc-24afd8b14c63\") " pod="openshift-ingress/e2e-shared-a-w3-18ecb2-openshift-default-86c6bff59d-9zh8x"
Sep 10 17:53:19.297039 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:53:19.297018    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-68rm9\" (UniqueName: \"kubernetes.io/projected/d5268e09-33ea-4f48-97cc-24afd8b14c63-kube-api-access-68rm9\") pod \"e2e-shared-a-w3-18ecb2-openshift-default-86c6bff59d-9zh8x\" (UID: \"d5268e09-33ea-4f48-97cc-24afd8b14c63\") " pod="openshift-ingress/e2e-shared-a-w3-18ecb2-openshift-default-86c6bff59d-9zh8x"
Sep 10 17:53:19.368260 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:53:19.368202    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-18ecb2-openshift-default-86c6bff59d-9zh8x"
Sep 10 17:53:19.652153 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:53:19.652126    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-18ecb2-openshift-default-86c6bff59d-9zh8x"]
Sep 10 17:53:19.655161 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:53:19.655136    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5268e09_33ea_4f48_97cc_24afd8b14c63.slice/crio-f83d262084740b89096fa9e668b0992023b26c3fcdac9d5ff06c8655c24a4007 WatchSource:0}: Error finding container f83d262084740b89096fa9e668b0992023b26c3fcdac9d5ff06c8655c24a4007: Status 404 returned error can't find the container with id f83d262084740b89096fa9e668b0992023b26c3fcdac9d5ff06c8655c24a4007
Sep 10 17:53:19.657156 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:53:19.657128    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 10 17:53:19.657221 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:53:19.657208    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 10 17:53:19.657259 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:53:19.657237    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 10 17:53:19.850137 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:53:19.850106    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-18ecb2-openshift-default-86c6bff59d-9zh8x" event={"ID":"d5268e09-33ea-4f48-97cc-24afd8b14c63","Type":"ContainerStarted","Data":"cd4206ead03e557afa1f81fea8e64f913743ca377d6317d6902815a52d76f86b"}
Sep 10 17:53:19.850329 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:53:19.850156    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-18ecb2-openshift-default-86c6bff59d-9zh8x" event={"ID":"d5268e09-33ea-4f48-97cc-24afd8b14c63","Type":"ContainerStarted","Data":"f83d262084740b89096fa9e668b0992023b26c3fcdac9d5ff06c8655c24a4007"}
Sep 10 17:53:19.874063 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:53:19.874015    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-18ecb2-openshift-default-86c6bff59d-9zh8x" podStartSLOduration=1.874001187 podStartE2EDuration="1.874001187s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:53:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:53:19.872547743 +0000 UTC m=+1043.872377811" watchObservedRunningTime="2026-09-10 17:53:19.874001187 +0000 UTC m=+1043.873831253"
Sep 10 17:53:20.368453 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:53:20.368421    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-18ecb2-openshift-default-86c6bff59d-9zh8x"
Sep 10 17:53:20.373875 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:53:20.373850    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-18ecb2-openshift-default-86c6bff59d-9zh8x"
Sep 10 17:53:20.855483 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:53:20.855454    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-18ecb2-openshift-default-86c6bff59d-9zh8x"
Sep 10 17:53:20.856630 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:53:20.856606    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-18ecb2-openshift-default-86c6bff59d-9zh8x"
Sep 10 17:53:22.962428 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:53:22.962390    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-18ecb2-78c8777656-2lncg"]
Sep 10 17:53:22.966187 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:53:22.966155    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-18ecb2-78c8777656-2lncg"
Sep 10 17:53:22.969047 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:53:22.969022    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-18ecb2-serving-cert\""
Sep 10 17:53:22.969210 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:53:22.969157    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-tdclw\""
Sep 10 17:53:22.980103 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:53:22.980053    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-18ecb2-78c8777656-2lncg"]
Sep 10 17:53:23.117096 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:53:23.117054    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ce9649a-4505-48dc-84c0-e3ee22ca3e7e-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-18ecb2-78c8777656-2lncg\" (UID: \"0ce9649a-4505-48dc-84c0-e3ee22ca3e7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-18ecb2-78c8777656-2lncg"
Sep 10 17:53:23.117096 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:53:23.117099    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ffrsk\" (UniqueName: \"kubernetes.io/projected/0ce9649a-4505-48dc-84c0-e3ee22ca3e7e-kube-api-access-ffrsk\") pod \"maas-api-e2e-shared-a-w3-18ecb2-78c8777656-2lncg\" (UID: \"0ce9649a-4505-48dc-84c0-e3ee22ca3e7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-18ecb2-78c8777656-2lncg"
Sep 10 17:53:23.117346 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:53:23.117153    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ce9649a-4505-48dc-84c0-e3ee22ca3e7e-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-18ecb2-78c8777656-2lncg\" (UID: \"0ce9649a-4505-48dc-84c0-e3ee22ca3e7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-18ecb2-78c8777656-2lncg"
Sep 10 17:53:23.117346 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:53:23.117298    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ce9649a-4505-48dc-84c0-e3ee22ca3e7e-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-18ecb2-78c8777656-2lncg\" (UID: \"0ce9649a-4505-48dc-84c0-e3ee22ca3e7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-18ecb2-78c8777656-2lncg"
Sep 10 17:53:23.217994 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:53:23.217892    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ce9649a-4505-48dc-84c0-e3ee22ca3e7e-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-18ecb2-78c8777656-2lncg\" (UID: \"0ce9649a-4505-48dc-84c0-e3ee22ca3e7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-18ecb2-78c8777656-2lncg"
Sep 10 17:53:23.217994 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:53:23.217949    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffrsk\" (UniqueName: \"kubernetes.io/projected/0ce9649a-4505-48dc-84c0-e3ee22ca3e7e-kube-api-access-ffrsk\") pod \"maas-api-e2e-shared-a-w3-18ecb2-78c8777656-2lncg\" (UID: \"0ce9649a-4505-48dc-84c0-e3ee22ca3e7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-18ecb2-78c8777656-2lncg"
Sep 10 17:53:23.217994 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:53:23.217976    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ce9649a-4505-48dc-84c0-e3ee22ca3e7e-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-18ecb2-78c8777656-2lncg\" (UID: \"0ce9649a-4505-48dc-84c0-e3ee22ca3e7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-18ecb2-78c8777656-2lncg"
Sep 10 17:53:23.218212 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:53:23.218015    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ce9649a-4505-48dc-84c0-e3ee22ca3e7e-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-18ecb2-78c8777656-2lncg\" (UID: \"0ce9649a-4505-48dc-84c0-e3ee22ca3e7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-18ecb2-78c8777656-2lncg"
Sep 10 17:53:23.218212 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:53:23.218116    2579 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-18ecb2-serving-cert: secret "maas-api-e2e-shared-a-w3-18ecb2-serving-cert" not found
Sep 10 17:53:23.218212 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:53:23.218175    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0ce9649a-4505-48dc-84c0-e3ee22ca3e7e-maas-api-tls podName:0ce9649a-4505-48dc-84c0-e3ee22ca3e7e nodeName:}" failed. No retries permitted until 2026-09-10 17:53:23.718155454 +0000 UTC m=+1047.717985500 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0ce9649a-4505-48dc-84c0-e3ee22ca3e7e-maas-api-tls") pod "maas-api-e2e-shared-a-w3-18ecb2-78c8777656-2lncg" (UID: "0ce9649a-4505-48dc-84c0-e3ee22ca3e7e") : secret "maas-api-e2e-shared-a-w3-18ecb2-serving-cert" not found
Sep 10 17:53:23.218780 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:53:23.218752    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ce9649a-4505-48dc-84c0-e3ee22ca3e7e-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-18ecb2-78c8777656-2lncg\" (UID: \"0ce9649a-4505-48dc-84c0-e3ee22ca3e7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-18ecb2-78c8777656-2lncg"
Sep 10 17:53:23.220535 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:53:23.220516    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ce9649a-4505-48dc-84c0-e3ee22ca3e7e-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-18ecb2-78c8777656-2lncg\" (UID: \"0ce9649a-4505-48dc-84c0-e3ee22ca3e7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-18ecb2-78c8777656-2lncg"
Sep 10 17:53:23.232823 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:53:23.232795    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffrsk\" (UniqueName: \"kubernetes.io/projected/0ce9649a-4505-48dc-84c0-e3ee22ca3e7e-kube-api-access-ffrsk\") pod \"maas-api-e2e-shared-a-w3-18ecb2-78c8777656-2lncg\" (UID: \"0ce9649a-4505-48dc-84c0-e3ee22ca3e7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-18ecb2-78c8777656-2lncg"
Sep 10 17:53:23.723007 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:53:23.722973    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ce9649a-4505-48dc-84c0-e3ee22ca3e7e-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-18ecb2-78c8777656-2lncg\" (UID: \"0ce9649a-4505-48dc-84c0-e3ee22ca3e7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-18ecb2-78c8777656-2lncg"
Sep 10 17:53:23.725622 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:53:23.725602    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ce9649a-4505-48dc-84c0-e3ee22ca3e7e-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-18ecb2-78c8777656-2lncg\" (UID: \"0ce9649a-4505-48dc-84c0-e3ee22ca3e7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-18ecb2-78c8777656-2lncg"
Sep 10 17:53:23.880878 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:53:23.880848    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-18ecb2-78c8777656-2lncg"
Sep 10 17:53:24.173081 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:53:24.173052    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-18ecb2-78c8777656-2lncg"]
Sep 10 17:53:24.174414 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:53:24.174387    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ce9649a_4505_48dc_84c0_e3ee22ca3e7e.slice/crio-8fc209ecbb8c6ca007dc2ff5a38ef55c8d0faf1f109968001b8886fb114a2620 WatchSource:0}: Error finding container 8fc209ecbb8c6ca007dc2ff5a38ef55c8d0faf1f109968001b8886fb114a2620: Status 404 returned error can't find the container with id 8fc209ecbb8c6ca007dc2ff5a38ef55c8d0faf1f109968001b8886fb114a2620
Sep 10 17:53:24.175956 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:53:24.175938    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 17:53:24.880217 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:53:24.880118    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-18ecb2-78c8777656-2lncg" event={"ID":"0ce9649a-4505-48dc-84c0-e3ee22ca3e7e","Type":"ContainerStarted","Data":"e221ffb0e453abe05f8ef7c3d43ea008350ff30742b7a99c66305415145ff2b2"}
Sep 10 17:53:24.880217 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:53:24.880174    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-18ecb2-78c8777656-2lncg" event={"ID":"0ce9649a-4505-48dc-84c0-e3ee22ca3e7e","Type":"ContainerStarted","Data":"8fc209ecbb8c6ca007dc2ff5a38ef55c8d0faf1f109968001b8886fb114a2620"}
Sep 10 17:53:24.880432 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:53:24.880241    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-18ecb2-78c8777656-2lncg"
Sep 10 17:53:24.905276 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:53:24.905222    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-18ecb2-78c8777656-2lncg" podStartSLOduration=2.547857567 podStartE2EDuration="2.905204944s" totalImagesPullingTime="357.347377ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:53:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:53:24.90300342 +0000 UTC m=+1048.902833491" watchObservedRunningTime="2026-09-10 17:53:24.905204944 +0000 UTC m=+1048.905035011"
Sep 10 17:53:30.895409 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:53:30.895374    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-18ecb2-78c8777656-2lncg"
Sep 10 17:53:35.877638 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:53:35.877167    2579 patch_prober.go:28] interesting pod/e2e-shared-a-w3-18ecb2-openshift-default-86c6bff59d-9zh8x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": read tcp 10.132.0.2:54850->10.132.0.66:15021: read: connection reset by peer" start-of-body=
Sep 10 17:53:35.877638 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:53:35.877246    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-18ecb2-openshift-default-86c6bff59d-9zh8x" podUID="d5268e09-33ea-4f48-97cc-24afd8b14c63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": read tcp 10.132.0.2:54850->10.132.0.66:15021: read: connection reset by peer"
Sep 10 17:53:35.940025 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:53:35.939997    2579 generic.go:356] "Generic (PLEG): container finished" podID="d5268e09-33ea-4f48-97cc-24afd8b14c63" containerID="cd4206ead03e557afa1f81fea8e64f913743ca377d6317d6902815a52d76f86b" exitCode=0
Sep 10 17:53:35.940152 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:53:35.940050    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-18ecb2-openshift-default-86c6bff59d-9zh8x" event={"ID":"d5268e09-33ea-4f48-97cc-24afd8b14c63","Type":"ContainerDied","Data":"cd4206ead03e557afa1f81fea8e64f913743ca377d6317d6902815a52d76f86b"}
Sep 10 17:53:35.940394 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:53:35.940381    2579 scope.go:117] "RemoveContainer" containerID="cd4206ead03e557afa1f81fea8e64f913743ca377d6317d6902815a52d76f86b"
Sep 10 17:53:35.941326 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:53:35.941290    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 10 17:53:35.941398 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:53:35.941356    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 10 17:53:35.941398 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:53:35.941384    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 10 17:53:36.946386 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:53:36.946349    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-18ecb2-openshift-default-86c6bff59d-9zh8x" event={"ID":"d5268e09-33ea-4f48-97cc-24afd8b14c63","Type":"ContainerStarted","Data":"ebf45602388eca065559eca96fb0f0b02802def6553a3dfb38b4c419f735c86d"}
Sep 10 17:53:37.368764 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:53:37.368669    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-18ecb2-openshift-default-86c6bff59d-9zh8x"
Sep 10 17:53:37.369135 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:53:37.369109    2579 patch_prober.go:28] interesting pod/e2e-shared-a-w3-18ecb2-openshift-default-86c6bff59d-9zh8x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body=
Sep 10 17:53:37.369334 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:53:37.369310    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-18ecb2-openshift-default-86c6bff59d-9zh8x" podUID="d5268e09-33ea-4f48-97cc-24afd8b14c63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused"
Sep 10 17:53:38.369066 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:53:38.369021    2579 patch_prober.go:28] interesting pod/e2e-shared-a-w3-18ecb2-openshift-default-86c6bff59d-9zh8x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body=
Sep 10 17:53:38.369685 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:53:38.369651    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-18ecb2-openshift-default-86c6bff59d-9zh8x" podUID="d5268e09-33ea-4f48-97cc-24afd8b14c63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused"
Sep 10 17:53:39.368972 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:53:39.368933    2579 patch_prober.go:28] interesting pod/e2e-shared-a-w3-18ecb2-openshift-default-86c6bff59d-9zh8x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body=
Sep 10 17:53:39.369846 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:53:39.369797    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-18ecb2-openshift-default-86c6bff59d-9zh8x" podUID="d5268e09-33ea-4f48-97cc-24afd8b14c63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused"
Sep 10 17:53:40.369070 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:53:40.369026    2579 patch_prober.go:28] interesting pod/e2e-shared-a-w3-18ecb2-openshift-default-86c6bff59d-9zh8x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body=
Sep 10 17:53:40.369263 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:53:40.369129    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-18ecb2-openshift-default-86c6bff59d-9zh8x" podUID="d5268e09-33ea-4f48-97cc-24afd8b14c63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused"
Sep 10 17:53:41.369126 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:53:41.369088    2579 patch_prober.go:28] interesting pod/e2e-shared-a-w3-18ecb2-openshift-default-86c6bff59d-9zh8x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body=
Sep 10 17:53:41.369715 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:53:41.369685    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-18ecb2-openshift-default-86c6bff59d-9zh8x" podUID="d5268e09-33ea-4f48-97cc-24afd8b14c63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused"
Sep 10 17:53:42.376004 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:53:42.375968    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-18ecb2-openshift-default-86c6bff59d-9zh8x"
Sep 10 17:53:42.376462 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:53:42.376244    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-18ecb2-openshift-default-86c6bff59d-9zh8x"
Sep 10 17:53:42.377100 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:53:42.377080    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-18ecb2-openshift-default-86c6bff59d-9zh8x"
Sep 10 17:53:45.544452 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:53:45.544418    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-9391d2-78f89686b8-n24kz"]
Sep 10 17:53:45.548480 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:53:45.548455    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-9391d2-78f89686b8-n24kz"
Sep 10 17:53:45.551527 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:53:45.551499    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-9391d2-serving-cert\""
Sep 10 17:53:45.566889 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:53:45.566853    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-9391d2-78f89686b8-n24kz"]
Sep 10 17:53:45.636246 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:53:45.636210    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bpf4m\" (UniqueName: \"kubernetes.io/projected/15a7e05e-738a-4789-a718-e8f7c37b6487-kube-api-access-bpf4m\") pod \"maas-api-e2e-shared-a-w2-9391d2-78f89686b8-n24kz\" (UID: \"15a7e05e-738a-4789-a718-e8f7c37b6487\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-9391d2-78f89686b8-n24kz"
Sep 10 17:53:45.636460 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:53:45.636262    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/15a7e05e-738a-4789-a718-e8f7c37b6487-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-9391d2-78f89686b8-n24kz\" (UID: \"15a7e05e-738a-4789-a718-e8f7c37b6487\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-9391d2-78f89686b8-n24kz"
Sep 10 17:53:45.636460 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:53:45.636344    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/15a7e05e-738a-4789-a718-e8f7c37b6487-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-9391d2-78f89686b8-n24kz\" (UID: \"15a7e05e-738a-4789-a718-e8f7c37b6487\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-9391d2-78f89686b8-n24kz"
Sep 10 17:53:45.636460 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:53:45.636418    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/15a7e05e-738a-4789-a718-e8f7c37b6487-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-9391d2-78f89686b8-n24kz\" (UID: \"15a7e05e-738a-4789-a718-e8f7c37b6487\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-9391d2-78f89686b8-n24kz"
Sep 10 17:53:45.736988 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:53:45.736946    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/15a7e05e-738a-4789-a718-e8f7c37b6487-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-9391d2-78f89686b8-n24kz\" (UID: \"15a7e05e-738a-4789-a718-e8f7c37b6487\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-9391d2-78f89686b8-n24kz"
Sep 10 17:53:45.737205 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:53:45.737007    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/15a7e05e-738a-4789-a718-e8f7c37b6487-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-9391d2-78f89686b8-n24kz\" (UID: \"15a7e05e-738a-4789-a718-e8f7c37b6487\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-9391d2-78f89686b8-n24kz"
Sep 10 17:53:45.737205 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:53:45.737079    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bpf4m\" (UniqueName: \"kubernetes.io/projected/15a7e05e-738a-4789-a718-e8f7c37b6487-kube-api-access-bpf4m\") pod \"maas-api-e2e-shared-a-w2-9391d2-78f89686b8-n24kz\" (UID: \"15a7e05e-738a-4789-a718-e8f7c37b6487\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-9391d2-78f89686b8-n24kz"
Sep 10 17:53:45.737325 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:53:45.737214    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/15a7e05e-738a-4789-a718-e8f7c37b6487-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-9391d2-78f89686b8-n24kz\" (UID: \"15a7e05e-738a-4789-a718-e8f7c37b6487\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-9391d2-78f89686b8-n24kz"
Sep 10 17:53:45.737391 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:53:45.737358    2579 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-9391d2-serving-cert: secret "maas-api-e2e-shared-a-w2-9391d2-serving-cert" not found
Sep 10 17:53:45.737473 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:53:45.737462    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/15a7e05e-738a-4789-a718-e8f7c37b6487-maas-api-tls podName:15a7e05e-738a-4789-a718-e8f7c37b6487 nodeName:}" failed. No retries permitted until 2026-09-10 17:53:46.237442463 +0000 UTC m=+1070.237272514 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/15a7e05e-738a-4789-a718-e8f7c37b6487-maas-api-tls") pod "maas-api-e2e-shared-a-w2-9391d2-78f89686b8-n24kz" (UID: "15a7e05e-738a-4789-a718-e8f7c37b6487") : secret "maas-api-e2e-shared-a-w2-9391d2-serving-cert" not found
Sep 10 17:53:45.737883 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:53:45.737861    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/15a7e05e-738a-4789-a718-e8f7c37b6487-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-9391d2-78f89686b8-n24kz\" (UID: \"15a7e05e-738a-4789-a718-e8f7c37b6487\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-9391d2-78f89686b8-n24kz"
Sep 10 17:53:45.739614 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:53:45.739559    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/15a7e05e-738a-4789-a718-e8f7c37b6487-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-9391d2-78f89686b8-n24kz\" (UID: \"15a7e05e-738a-4789-a718-e8f7c37b6487\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-9391d2-78f89686b8-n24kz"
Sep 10 17:53:45.753389 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:53:45.753358    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bpf4m\" (UniqueName: \"kubernetes.io/projected/15a7e05e-738a-4789-a718-e8f7c37b6487-kube-api-access-bpf4m\") pod \"maas-api-e2e-shared-a-w2-9391d2-78f89686b8-n24kz\" (UID: \"15a7e05e-738a-4789-a718-e8f7c37b6487\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-9391d2-78f89686b8-n24kz"
Sep 10 17:53:46.240660 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:53:46.240609    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/15a7e05e-738a-4789-a718-e8f7c37b6487-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-9391d2-78f89686b8-n24kz\" (UID: \"15a7e05e-738a-4789-a718-e8f7c37b6487\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-9391d2-78f89686b8-n24kz"
Sep 10 17:53:46.243181 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:53:46.243149    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/15a7e05e-738a-4789-a718-e8f7c37b6487-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-9391d2-78f89686b8-n24kz\" (UID: \"15a7e05e-738a-4789-a718-e8f7c37b6487\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-9391d2-78f89686b8-n24kz"
Sep 10 17:53:46.459977 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:53:46.459936    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-9391d2-78f89686b8-n24kz"
Sep 10 17:53:46.735916 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:53:46.735885    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-9391d2-78f89686b8-n24kz"]
Sep 10 17:53:46.737751 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:53:46.737720    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15a7e05e_738a_4789_a718_e8f7c37b6487.slice/crio-95a482b6c1b585e6f382618162e39f07adb2dfb0f8eb10cb287272efe453b7e1 WatchSource:0}: Error finding container 95a482b6c1b585e6f382618162e39f07adb2dfb0f8eb10cb287272efe453b7e1: Status 404 returned error can't find the container with id 95a482b6c1b585e6f382618162e39f07adb2dfb0f8eb10cb287272efe453b7e1
Sep 10 17:53:47.001442 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:53:47.001403    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-9391d2-78f89686b8-n24kz" event={"ID":"15a7e05e-738a-4789-a718-e8f7c37b6487","Type":"ContainerStarted","Data":"95a482b6c1b585e6f382618162e39f07adb2dfb0f8eb10cb287272efe453b7e1"}
Sep 10 17:53:48.009094 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:53:48.009001    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-9391d2-78f89686b8-n24kz" event={"ID":"15a7e05e-738a-4789-a718-e8f7c37b6487","Type":"ContainerStarted","Data":"b61278981094fe776741fb1ee4a9b6f294fb0e2e39a9cf00533954dfb3977156"}
Sep 10 17:53:48.009608 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:53:48.009116    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-9391d2-78f89686b8-n24kz"
Sep 10 17:53:48.042709 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:53:48.042659    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-9391d2-78f89686b8-n24kz" podStartSLOduration=2.66517102 podStartE2EDuration="3.042643512s" totalImagesPullingTime="377.472492ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:53:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:53:48.040866853 +0000 UTC m=+1072.040696920" watchObservedRunningTime="2026-09-10 17:53:48.042643512 +0000 UTC m=+1072.042473579"
Sep 10 17:53:53.360237 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:53:53.360204    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-57474d-644fb5dd75-mrfnj"]
Sep 10 17:53:53.363765 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:53:53.363746    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-57474d-644fb5dd75-mrfnj"
Sep 10 17:53:53.367999 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:53:53.367969    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-57474d-dockercfg-hg6hb\""
Sep 10 17:53:53.368147 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:53:53.367969    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-57474d\""
Sep 10 17:53:53.378394 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:53:53.378361    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-57474d-644fb5dd75-mrfnj"]
Sep 10 17:53:53.407459 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:53:53.407423    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa4f0384-9c16-43c4-b52f-0e4f67a24d25-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-57474d-644fb5dd75-mrfnj\" (UID: \"aa4f0384-9c16-43c4-b52f-0e4f67a24d25\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-57474d-644fb5dd75-mrfnj"
Sep 10 17:53:53.407676 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:53:53.407470    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qlr8\" (UniqueName: \"kubernetes.io/projected/aa4f0384-9c16-43c4-b52f-0e4f67a24d25-kube-api-access-2qlr8\") pod \"payload-processing-e2e-shared-b-w3-57474d-644fb5dd75-mrfnj\" (UID: \"aa4f0384-9c16-43c4-b52f-0e4f67a24d25\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-57474d-644fb5dd75-mrfnj"
Sep 10 17:53:53.428406 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:53:53.428373    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-57474d-d7f965c5d-zgnx6"]
Sep 10 17:53:53.431828 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:53:53.431811    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-57474d-d7f965c5d-zgnx6"
Sep 10 17:53:53.452083 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:53:53.452052    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-57474d-d7f965c5d-zgnx6"]
Sep 10 17:53:53.508216 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:53:53.508181    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa4f0384-9c16-43c4-b52f-0e4f67a24d25-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-57474d-644fb5dd75-mrfnj\" (UID: \"aa4f0384-9c16-43c4-b52f-0e4f67a24d25\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-57474d-644fb5dd75-mrfnj"
Sep 10 17:53:53.508216 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:53:53.508226    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qlr8\" (UniqueName: \"kubernetes.io/projected/aa4f0384-9c16-43c4-b52f-0e4f67a24d25-kube-api-access-2qlr8\") pod \"payload-processing-e2e-shared-b-w3-57474d-644fb5dd75-mrfnj\" (UID: \"aa4f0384-9c16-43c4-b52f-0e4f67a24d25\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-57474d-644fb5dd75-mrfnj"
Sep 10 17:53:53.508465 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:53:53.508265    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mb782\" (UniqueName: \"kubernetes.io/projected/f879e029-a958-4467-b001-d5eebb380ceb-kube-api-access-mb782\") pod \"payload-pre-processing-e2e-shared-b-w3-57474d-d7f965c5d-zgnx6\" (UID: \"f879e029-a958-4467-b001-d5eebb380ceb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-57474d-d7f965c5d-zgnx6"
Sep 10 17:53:53.508465 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:53:53.508289    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f879e029-a958-4467-b001-d5eebb380ceb-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-57474d-d7f965c5d-zgnx6\" (UID: \"f879e029-a958-4467-b001-d5eebb380ceb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-57474d-d7f965c5d-zgnx6"
Sep 10 17:53:53.509022 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:53:53.509001    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa4f0384-9c16-43c4-b52f-0e4f67a24d25-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-57474d-644fb5dd75-mrfnj\" (UID: \"aa4f0384-9c16-43c4-b52f-0e4f67a24d25\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-57474d-644fb5dd75-mrfnj"
Sep 10 17:53:53.521290 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:53:53.521262    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qlr8\" (UniqueName: \"kubernetes.io/projected/aa4f0384-9c16-43c4-b52f-0e4f67a24d25-kube-api-access-2qlr8\") pod \"payload-processing-e2e-shared-b-w3-57474d-644fb5dd75-mrfnj\" (UID: \"aa4f0384-9c16-43c4-b52f-0e4f67a24d25\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-57474d-644fb5dd75-mrfnj"
Sep 10 17:53:53.609506 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:53:53.609468    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mb782\" (UniqueName: \"kubernetes.io/projected/f879e029-a958-4467-b001-d5eebb380ceb-kube-api-access-mb782\") pod \"payload-pre-processing-e2e-shared-b-w3-57474d-d7f965c5d-zgnx6\" (UID: \"f879e029-a958-4467-b001-d5eebb380ceb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-57474d-d7f965c5d-zgnx6"
Sep 10 17:53:53.609506 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:53:53.609509    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f879e029-a958-4467-b001-d5eebb380ceb-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-57474d-d7f965c5d-zgnx6\" (UID: \"f879e029-a958-4467-b001-d5eebb380ceb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-57474d-d7f965c5d-zgnx6"
Sep 10 17:53:53.610324 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:53:53.610259    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f879e029-a958-4467-b001-d5eebb380ceb-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-57474d-d7f965c5d-zgnx6\" (UID: \"f879e029-a958-4467-b001-d5eebb380ceb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-57474d-d7f965c5d-zgnx6"
Sep 10 17:53:53.638085 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:53:53.638053    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mb782\" (UniqueName: \"kubernetes.io/projected/f879e029-a958-4467-b001-d5eebb380ceb-kube-api-access-mb782\") pod \"payload-pre-processing-e2e-shared-b-w3-57474d-d7f965c5d-zgnx6\" (UID: \"f879e029-a958-4467-b001-d5eebb380ceb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-57474d-d7f965c5d-zgnx6"
Sep 10 17:53:53.675963 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:53:53.675912    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-57474d-644fb5dd75-mrfnj"
Sep 10 17:53:53.743042 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:53:53.743000    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-57474d-d7f965c5d-zgnx6"
Sep 10 17:53:53.951697 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:53:53.951673    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-57474d-644fb5dd75-mrfnj"]
Sep 10 17:53:53.953955 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:53:53.953917    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa4f0384_9c16_43c4_b52f_0e4f67a24d25.slice/crio-27c614edc8ce8c4950f7073828246f21451dbc7b3c2e8b7602f70826846c82e5 WatchSource:0}: Error finding container 27c614edc8ce8c4950f7073828246f21451dbc7b3c2e8b7602f70826846c82e5: Status 404 returned error can't find the container with id 27c614edc8ce8c4950f7073828246f21451dbc7b3c2e8b7602f70826846c82e5
Sep 10 17:53:54.020479 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:53:54.020454    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-9391d2-78f89686b8-n24kz"
Sep 10 17:53:54.024160 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:53:54.024138    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-57474d-d7f965c5d-zgnx6"]
Sep 10 17:53:54.026507 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:53:54.026485    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf879e029_a958_4467_b001_d5eebb380ceb.slice/crio-1e13ba4e27e7a5cf570ca9a74b8d59d718016d7a128b35bbbbd83b34617c46aa WatchSource:0}: Error finding container 1e13ba4e27e7a5cf570ca9a74b8d59d718016d7a128b35bbbbd83b34617c46aa: Status 404 returned error can't find the container with id 1e13ba4e27e7a5cf570ca9a74b8d59d718016d7a128b35bbbbd83b34617c46aa
Sep 10 17:53:54.042606 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:53:54.042559    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-57474d-644fb5dd75-mrfnj" event={"ID":"aa4f0384-9c16-43c4-b52f-0e4f67a24d25","Type":"ContainerStarted","Data":"1bbce74b004946cfa3140aeecaefa32f517251c86ac2e68421ad76d38161c78b"}
Sep 10 17:53:54.042741 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:53:54.042629    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-57474d-644fb5dd75-mrfnj" event={"ID":"aa4f0384-9c16-43c4-b52f-0e4f67a24d25","Type":"ContainerStarted","Data":"27c614edc8ce8c4950f7073828246f21451dbc7b3c2e8b7602f70826846c82e5"}
Sep 10 17:53:54.042741 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:53:54.042672    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-57474d-644fb5dd75-mrfnj"
Sep 10 17:53:54.044407 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:53:54.044380    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-57474d-d7f965c5d-zgnx6" event={"ID":"f879e029-a958-4467-b001-d5eebb380ceb","Type":"ContainerStarted","Data":"1e13ba4e27e7a5cf570ca9a74b8d59d718016d7a128b35bbbbd83b34617c46aa"}
Sep 10 17:53:54.066351 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:53:54.066303    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-57474d-644fb5dd75-mrfnj" podStartSLOduration=1.066287597 podStartE2EDuration="1.066287597s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:53:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:53:54.065080494 +0000 UTC m=+1078.064910561" watchObservedRunningTime="2026-09-10 17:53:54.066287597 +0000 UTC m=+1078.066117670"
Sep 10 17:53:55.051042 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:53:55.050997    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-57474d-d7f965c5d-zgnx6" event={"ID":"f879e029-a958-4467-b001-d5eebb380ceb","Type":"ContainerStarted","Data":"7ab2ff8d1fb104321b7d21ef948c275a27403ec84423e4dfbb032f3b5888c697"}
Sep 10 17:53:55.051549 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:53:55.051109    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-57474d-d7f965c5d-zgnx6"
Sep 10 17:53:55.072412 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:53:55.072347    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-57474d-d7f965c5d-zgnx6" podStartSLOduration=2.072330739 podStartE2EDuration="2.072330739s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:53:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:53:55.069787451 +0000 UTC m=+1079.069617519" watchObservedRunningTime="2026-09-10 17:53:55.072330739 +0000 UTC m=+1079.072160806"
Sep 10 17:54:05.053599 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:54:05.053545    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-57474d-644fb5dd75-mrfnj"
Sep 10 17:54:06.059705 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:54:06.059673    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-57474d-d7f965c5d-zgnx6"
Sep 10 17:54:15.485559 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:54:15.485524    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-bdf47c-d85f68558-r2bt5"]
Sep 10 17:54:15.489202 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:54:15.489173    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-bdf47c-d85f68558-r2bt5"
Sep 10 17:54:15.491944 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:54:15.491926    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-bdf47c\""
Sep 10 17:54:15.504289 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:54:15.504264    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-bdf47c-d85f68558-r2bt5"]
Sep 10 17:54:15.621518 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:54:15.621469    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e3058bf-33b7-4c50-81a3-b99147b0ba0f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-bdf47c-d85f68558-r2bt5\" (UID: \"6e3058bf-33b7-4c50-81a3-b99147b0ba0f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-bdf47c-d85f68558-r2bt5"
Sep 10 17:54:15.621719 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:54:15.621539    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bg95w\" (UniqueName: \"kubernetes.io/projected/6e3058bf-33b7-4c50-81a3-b99147b0ba0f-kube-api-access-bg95w\") pod \"payload-pre-processing-e2e-shared-b-w2-bdf47c-d85f68558-r2bt5\" (UID: \"6e3058bf-33b7-4c50-81a3-b99147b0ba0f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-bdf47c-d85f68558-r2bt5"
Sep 10 17:54:15.722288 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:54:15.722245    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e3058bf-33b7-4c50-81a3-b99147b0ba0f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-bdf47c-d85f68558-r2bt5\" (UID: \"6e3058bf-33b7-4c50-81a3-b99147b0ba0f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-bdf47c-d85f68558-r2bt5"
Sep 10 17:54:15.722471 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:54:15.722299    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bg95w\" (UniqueName: \"kubernetes.io/projected/6e3058bf-33b7-4c50-81a3-b99147b0ba0f-kube-api-access-bg95w\") pod \"payload-pre-processing-e2e-shared-b-w2-bdf47c-d85f68558-r2bt5\" (UID: \"6e3058bf-33b7-4c50-81a3-b99147b0ba0f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-bdf47c-d85f68558-r2bt5"
Sep 10 17:54:15.722989 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:54:15.722969    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e3058bf-33b7-4c50-81a3-b99147b0ba0f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-bdf47c-d85f68558-r2bt5\" (UID: \"6e3058bf-33b7-4c50-81a3-b99147b0ba0f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-bdf47c-d85f68558-r2bt5"
Sep 10 17:54:15.741638 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:54:15.741579    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bg95w\" (UniqueName: \"kubernetes.io/projected/6e3058bf-33b7-4c50-81a3-b99147b0ba0f-kube-api-access-bg95w\") pod \"payload-pre-processing-e2e-shared-b-w2-bdf47c-d85f68558-r2bt5\" (UID: \"6e3058bf-33b7-4c50-81a3-b99147b0ba0f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-bdf47c-d85f68558-r2bt5"
Sep 10 17:54:15.799130 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:54:15.799089    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-bdf47c-d85f68558-r2bt5"
Sep 10 17:54:16.105979 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:54:16.105955    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-bdf47c-d85f68558-r2bt5"]
Sep 10 17:54:16.108238 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:54:16.108209    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e3058bf_33b7_4c50_81a3_b99147b0ba0f.slice/crio-da52ae2c2d529747a292f5ce6e09afd40e4be59cca45f9b58efd95a9e42d153b WatchSource:0}: Error finding container da52ae2c2d529747a292f5ce6e09afd40e4be59cca45f9b58efd95a9e42d153b: Status 404 returned error can't find the container with id da52ae2c2d529747a292f5ce6e09afd40e4be59cca45f9b58efd95a9e42d153b
Sep 10 17:54:16.158641 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:54:16.158606    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-bdf47c-d85f68558-r2bt5" event={"ID":"6e3058bf-33b7-4c50-81a3-b99147b0ba0f","Type":"ContainerStarted","Data":"da52ae2c2d529747a292f5ce6e09afd40e4be59cca45f9b58efd95a9e42d153b"}
Sep 10 17:54:17.165395 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:54:17.165351    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-bdf47c-d85f68558-r2bt5" event={"ID":"6e3058bf-33b7-4c50-81a3-b99147b0ba0f","Type":"ContainerStarted","Data":"24c8d5ddb3058a9e682a9c493b3454753a45637320a1ecac12598210a0f1b131"}
Sep 10 17:54:17.165829 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:54:17.165478    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-bdf47c-d85f68558-r2bt5"
Sep 10 17:54:17.189036 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:54:17.188986    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-bdf47c-d85f68558-r2bt5" podStartSLOduration=2.1889690330000002 podStartE2EDuration="2.188969033s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:54:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:54:17.186816628 +0000 UTC m=+1101.186646695" watchObservedRunningTime="2026-09-10 17:54:17.188969033 +0000 UTC m=+1101.188799100"
Sep 10 17:54:28.173012 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:54:28.172980    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-bdf47c-d85f68558-r2bt5"
Sep 10 17:54:49.594448 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:54:49.594415    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-655f7f0b-openshift-default-86546b74d-wrvn7"]
Sep 10 17:54:49.599436 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:54:49.599418    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-655f7f0b-openshift-default-86546b74d-wrvn7"
Sep 10 17:54:49.602054 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:54:49.602030    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-655f7f0b-openshift-default-dockercfg-pvzgp\""
Sep 10 17:54:49.615975 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:54:49.615953    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-655f7f0b-openshift-default-86546b74d-wrvn7"]
Sep 10 17:54:49.725597 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:54:49.725539    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e05bf37b-74f1-445c-ad58-5464347cb2f3-workload-certs\") pod \"e2e-mt-655f7f0b-openshift-default-86546b74d-wrvn7\" (UID: \"e05bf37b-74f1-445c-ad58-5464347cb2f3\") " pod="openshift-ingress/e2e-mt-655f7f0b-openshift-default-86546b74d-wrvn7"
Sep 10 17:54:49.725775 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:54:49.725674    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85qxw\" (UniqueName: \"kubernetes.io/projected/e05bf37b-74f1-445c-ad58-5464347cb2f3-kube-api-access-85qxw\") pod \"e2e-mt-655f7f0b-openshift-default-86546b74d-wrvn7\" (UID: \"e05bf37b-74f1-445c-ad58-5464347cb2f3\") " pod="openshift-ingress/e2e-mt-655f7f0b-openshift-default-86546b74d-wrvn7"
Sep 10 17:54:49.725775 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:54:49.725729    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e05bf37b-74f1-445c-ad58-5464347cb2f3-istio-podinfo\") pod \"e2e-mt-655f7f0b-openshift-default-86546b74d-wrvn7\" (UID: \"e05bf37b-74f1-445c-ad58-5464347cb2f3\") " pod="openshift-ingress/e2e-mt-655f7f0b-openshift-default-86546b74d-wrvn7"
Sep 10 17:54:49.725874 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:54:49.725800    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e05bf37b-74f1-445c-ad58-5464347cb2f3-istiod-ca-cert\") pod \"e2e-mt-655f7f0b-openshift-default-86546b74d-wrvn7\" (UID: \"e05bf37b-74f1-445c-ad58-5464347cb2f3\") " pod="openshift-ingress/e2e-mt-655f7f0b-openshift-default-86546b74d-wrvn7"
Sep 10 17:54:49.725929 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:54:49.725860    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e05bf37b-74f1-445c-ad58-5464347cb2f3-istio-envoy\") pod \"e2e-mt-655f7f0b-openshift-default-86546b74d-wrvn7\" (UID: \"e05bf37b-74f1-445c-ad58-5464347cb2f3\") " pod="openshift-ingress/e2e-mt-655f7f0b-openshift-default-86546b74d-wrvn7"
Sep 10 17:54:49.725929 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:54:49.725917    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e05bf37b-74f1-445c-ad58-5464347cb2f3-istio-token\") pod \"e2e-mt-655f7f0b-openshift-default-86546b74d-wrvn7\" (UID: \"e05bf37b-74f1-445c-ad58-5464347cb2f3\") " pod="openshift-ingress/e2e-mt-655f7f0b-openshift-default-86546b74d-wrvn7"
Sep 10 17:54:49.726014 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:54:49.725976    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e05bf37b-74f1-445c-ad58-5464347cb2f3-workload-socket\") pod \"e2e-mt-655f7f0b-openshift-default-86546b74d-wrvn7\" (UID: \"e05bf37b-74f1-445c-ad58-5464347cb2f3\") " pod="openshift-ingress/e2e-mt-655f7f0b-openshift-default-86546b74d-wrvn7"
Sep 10 17:54:49.726072 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:54:49.726016    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e05bf37b-74f1-445c-ad58-5464347cb2f3-credential-socket\") pod \"e2e-mt-655f7f0b-openshift-default-86546b74d-wrvn7\" (UID: \"e05bf37b-74f1-445c-ad58-5464347cb2f3\") " pod="openshift-ingress/e2e-mt-655f7f0b-openshift-default-86546b74d-wrvn7"
Sep 10 17:54:49.726072 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:54:49.726038    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e05bf37b-74f1-445c-ad58-5464347cb2f3-istio-data\") pod \"e2e-mt-655f7f0b-openshift-default-86546b74d-wrvn7\" (UID: \"e05bf37b-74f1-445c-ad58-5464347cb2f3\") " pod="openshift-ingress/e2e-mt-655f7f0b-openshift-default-86546b74d-wrvn7"
Sep 10 17:54:49.827226 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:54:49.827188    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e05bf37b-74f1-445c-ad58-5464347cb2f3-workload-socket\") pod \"e2e-mt-655f7f0b-openshift-default-86546b74d-wrvn7\" (UID: \"e05bf37b-74f1-445c-ad58-5464347cb2f3\") " pod="openshift-ingress/e2e-mt-655f7f0b-openshift-default-86546b74d-wrvn7"
Sep 10 17:54:49.827226 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:54:49.827225    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e05bf37b-74f1-445c-ad58-5464347cb2f3-credential-socket\") pod \"e2e-mt-655f7f0b-openshift-default-86546b74d-wrvn7\" (UID: \"e05bf37b-74f1-445c-ad58-5464347cb2f3\") " pod="openshift-ingress/e2e-mt-655f7f0b-openshift-default-86546b74d-wrvn7"
Sep 10 17:54:49.827472 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:54:49.827243    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e05bf37b-74f1-445c-ad58-5464347cb2f3-istio-data\") pod \"e2e-mt-655f7f0b-openshift-default-86546b74d-wrvn7\" (UID: \"e05bf37b-74f1-445c-ad58-5464347cb2f3\") " pod="openshift-ingress/e2e-mt-655f7f0b-openshift-default-86546b74d-wrvn7"
Sep 10 17:54:49.827472 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:54:49.827367    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e05bf37b-74f1-445c-ad58-5464347cb2f3-workload-certs\") pod \"e2e-mt-655f7f0b-openshift-default-86546b74d-wrvn7\" (UID: \"e05bf37b-74f1-445c-ad58-5464347cb2f3\") " pod="openshift-ingress/e2e-mt-655f7f0b-openshift-default-86546b74d-wrvn7"
Sep 10 17:54:49.827472 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:54:49.827422    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85qxw\" (UniqueName: \"kubernetes.io/projected/e05bf37b-74f1-445c-ad58-5464347cb2f3-kube-api-access-85qxw\") pod \"e2e-mt-655f7f0b-openshift-default-86546b74d-wrvn7\" (UID: \"e05bf37b-74f1-445c-ad58-5464347cb2f3\") " pod="openshift-ingress/e2e-mt-655f7f0b-openshift-default-86546b74d-wrvn7"
Sep 10 17:54:49.827472 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:54:49.827455    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e05bf37b-74f1-445c-ad58-5464347cb2f3-istio-podinfo\") pod \"e2e-mt-655f7f0b-openshift-default-86546b74d-wrvn7\" (UID: \"e05bf37b-74f1-445c-ad58-5464347cb2f3\") " pod="openshift-ingress/e2e-mt-655f7f0b-openshift-default-86546b74d-wrvn7"
Sep 10 17:54:49.827713 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:54:49.827528    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e05bf37b-74f1-445c-ad58-5464347cb2f3-istiod-ca-cert\") pod \"e2e-mt-655f7f0b-openshift-default-86546b74d-wrvn7\" (UID: \"e05bf37b-74f1-445c-ad58-5464347cb2f3\") " pod="openshift-ingress/e2e-mt-655f7f0b-openshift-default-86546b74d-wrvn7"
Sep 10 17:54:49.827713 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:54:49.827555    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e05bf37b-74f1-445c-ad58-5464347cb2f3-istio-envoy\") pod \"e2e-mt-655f7f0b-openshift-default-86546b74d-wrvn7\" (UID: \"e05bf37b-74f1-445c-ad58-5464347cb2f3\") " pod="openshift-ingress/e2e-mt-655f7f0b-openshift-default-86546b74d-wrvn7"
Sep 10 17:54:49.827713 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:54:49.827614    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e05bf37b-74f1-445c-ad58-5464347cb2f3-istio-token\") pod \"e2e-mt-655f7f0b-openshift-default-86546b74d-wrvn7\" (UID: \"e05bf37b-74f1-445c-ad58-5464347cb2f3\") " pod="openshift-ingress/e2e-mt-655f7f0b-openshift-default-86546b74d-wrvn7"
Sep 10 17:54:49.828031 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:54:49.828004    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e05bf37b-74f1-445c-ad58-5464347cb2f3-istio-data\") pod \"e2e-mt-655f7f0b-openshift-default-86546b74d-wrvn7\" (UID: \"e05bf37b-74f1-445c-ad58-5464347cb2f3\") " pod="openshift-ingress/e2e-mt-655f7f0b-openshift-default-86546b74d-wrvn7"
Sep 10 17:54:49.828166 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:54:49.827632    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e05bf37b-74f1-445c-ad58-5464347cb2f3-workload-socket\") pod \"e2e-mt-655f7f0b-openshift-default-86546b74d-wrvn7\" (UID: \"e05bf37b-74f1-445c-ad58-5464347cb2f3\") " pod="openshift-ingress/e2e-mt-655f7f0b-openshift-default-86546b74d-wrvn7"
Sep 10 17:54:49.828239 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:54:49.828196    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e05bf37b-74f1-445c-ad58-5464347cb2f3-workload-certs\") pod \"e2e-mt-655f7f0b-openshift-default-86546b74d-wrvn7\" (UID: \"e05bf37b-74f1-445c-ad58-5464347cb2f3\") " pod="openshift-ingress/e2e-mt-655f7f0b-openshift-default-86546b74d-wrvn7"
Sep 10 17:54:49.828325 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:54:49.828294    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e05bf37b-74f1-445c-ad58-5464347cb2f3-credential-socket\") pod \"e2e-mt-655f7f0b-openshift-default-86546b74d-wrvn7\" (UID: \"e05bf37b-74f1-445c-ad58-5464347cb2f3\") " pod="openshift-ingress/e2e-mt-655f7f0b-openshift-default-86546b74d-wrvn7"
Sep 10 17:54:49.829342 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:54:49.829312    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e05bf37b-74f1-445c-ad58-5464347cb2f3-istiod-ca-cert\") pod \"e2e-mt-655f7f0b-openshift-default-86546b74d-wrvn7\" (UID: \"e05bf37b-74f1-445c-ad58-5464347cb2f3\") " pod="openshift-ingress/e2e-mt-655f7f0b-openshift-default-86546b74d-wrvn7"
Sep 10 17:54:49.833361 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:54:49.831150    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e05bf37b-74f1-445c-ad58-5464347cb2f3-istio-envoy\") pod \"e2e-mt-655f7f0b-openshift-default-86546b74d-wrvn7\" (UID: \"e05bf37b-74f1-445c-ad58-5464347cb2f3\") " pod="openshift-ingress/e2e-mt-655f7f0b-openshift-default-86546b74d-wrvn7"
Sep 10 17:54:49.833361 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:54:49.831492    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e05bf37b-74f1-445c-ad58-5464347cb2f3-istio-podinfo\") pod \"e2e-mt-655f7f0b-openshift-default-86546b74d-wrvn7\" (UID: \"e05bf37b-74f1-445c-ad58-5464347cb2f3\") " pod="openshift-ingress/e2e-mt-655f7f0b-openshift-default-86546b74d-wrvn7"
Sep 10 17:54:49.840651 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:54:49.840630    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-85qxw\" (UniqueName: \"kubernetes.io/projected/e05bf37b-74f1-445c-ad58-5464347cb2f3-kube-api-access-85qxw\") pod \"e2e-mt-655f7f0b-openshift-default-86546b74d-wrvn7\" (UID: \"e05bf37b-74f1-445c-ad58-5464347cb2f3\") " pod="openshift-ingress/e2e-mt-655f7f0b-openshift-default-86546b74d-wrvn7"
Sep 10 17:54:49.844143 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:54:49.844115    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e05bf37b-74f1-445c-ad58-5464347cb2f3-istio-token\") pod \"e2e-mt-655f7f0b-openshift-default-86546b74d-wrvn7\" (UID: \"e05bf37b-74f1-445c-ad58-5464347cb2f3\") " pod="openshift-ingress/e2e-mt-655f7f0b-openshift-default-86546b74d-wrvn7"
Sep 10 17:54:49.912044 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:54:49.912008    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-655f7f0b-openshift-default-86546b74d-wrvn7"
Sep 10 17:54:50.199612 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:54:50.199583    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-655f7f0b-openshift-default-86546b74d-wrvn7"]
Sep 10 17:54:50.200690 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:54:50.200664    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode05bf37b_74f1_445c_ad58_5464347cb2f3.slice/crio-1e51f9d33821cddea967b3dd8373363175707b52de4d33ff2c4671030611e990 WatchSource:0}: Error finding container 1e51f9d33821cddea967b3dd8373363175707b52de4d33ff2c4671030611e990: Status 404 returned error can't find the container with id 1e51f9d33821cddea967b3dd8373363175707b52de4d33ff2c4671030611e990
Sep 10 17:54:50.202873 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:54:50.202839    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 10 17:54:50.202956 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:54:50.202912    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 10 17:54:50.202956 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:54:50.202944    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 10 17:54:50.337261 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:54:50.337217    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-655f7f0b-openshift-default-86546b74d-wrvn7" event={"ID":"e05bf37b-74f1-445c-ad58-5464347cb2f3","Type":"ContainerStarted","Data":"7d99a27e60fe4b415004eac9cb4d7d40d4dc752dee6f6c7deeec583cedbb3899"}
Sep 10 17:54:50.337429 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:54:50.337286    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-655f7f0b-openshift-default-86546b74d-wrvn7" event={"ID":"e05bf37b-74f1-445c-ad58-5464347cb2f3","Type":"ContainerStarted","Data":"1e51f9d33821cddea967b3dd8373363175707b52de4d33ff2c4671030611e990"}
Sep 10 17:54:50.369672 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:54:50.369607    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-655f7f0b-openshift-default-86546b74d-wrvn7" podStartSLOduration=1.369589009 podStartE2EDuration="1.369589009s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:54:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:54:50.366359159 +0000 UTC m=+1134.366189225" watchObservedRunningTime="2026-09-10 17:54:50.369589009 +0000 UTC m=+1134.369419071"
Sep 10 17:54:50.912099 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:54:50.912061    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-655f7f0b-openshift-default-86546b74d-wrvn7"
Sep 10 17:54:51.917521 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:54:51.917488    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-655f7f0b-openshift-default-86546b74d-wrvn7"
Sep 10 17:54:52.347121 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:54:52.347046    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-655f7f0b-openshift-default-86546b74d-wrvn7"
Sep 10 17:54:52.347996 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:54:52.347977    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-655f7f0b-openshift-default-86546b74d-wrvn7"
Sep 10 17:54:53.474328 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:54:53.474285    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-655f7f0b-67f9896d7c-wswvl"]
Sep 10 17:54:53.478784 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:54:53.478760    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-655f7f0b-67f9896d7c-wswvl"
Sep 10 17:54:53.481931 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:54:53.481908    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-655f7f0b-serving-cert\""
Sep 10 17:54:53.491547 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:54:53.491523    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-655f7f0b-67f9896d7c-wswvl"]
Sep 10 17:54:53.563648 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:54:53.563616    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ddb013a-2517-4093-aab0-3ccbef086e0b-openshift-service-ca\") pod \"maas-api-e2e-mt-655f7f0b-67f9896d7c-wswvl\" (UID: \"0ddb013a-2517-4093-aab0-3ccbef086e0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-655f7f0b-67f9896d7c-wswvl"
Sep 10 17:54:53.563987 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:54:53.563719    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2pscb\" (UniqueName: \"kubernetes.io/projected/0ddb013a-2517-4093-aab0-3ccbef086e0b-kube-api-access-2pscb\") pod \"maas-api-e2e-mt-655f7f0b-67f9896d7c-wswvl\" (UID: \"0ddb013a-2517-4093-aab0-3ccbef086e0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-655f7f0b-67f9896d7c-wswvl"
Sep 10 17:54:53.563987 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:54:53.563744    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ddb013a-2517-4093-aab0-3ccbef086e0b-metrics-certs\") pod \"maas-api-e2e-mt-655f7f0b-67f9896d7c-wswvl\" (UID: \"0ddb013a-2517-4093-aab0-3ccbef086e0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-655f7f0b-67f9896d7c-wswvl"
Sep 10 17:54:53.563987 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:54:53.563761    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ddb013a-2517-4093-aab0-3ccbef086e0b-maas-api-tls\") pod \"maas-api-e2e-mt-655f7f0b-67f9896d7c-wswvl\" (UID: \"0ddb013a-2517-4093-aab0-3ccbef086e0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-655f7f0b-67f9896d7c-wswvl"
Sep 10 17:54:53.665001 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:54:53.664970    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pscb\" (UniqueName: \"kubernetes.io/projected/0ddb013a-2517-4093-aab0-3ccbef086e0b-kube-api-access-2pscb\") pod \"maas-api-e2e-mt-655f7f0b-67f9896d7c-wswvl\" (UID: \"0ddb013a-2517-4093-aab0-3ccbef086e0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-655f7f0b-67f9896d7c-wswvl"
Sep 10 17:54:53.665001 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:54:53.665005    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ddb013a-2517-4093-aab0-3ccbef086e0b-metrics-certs\") pod \"maas-api-e2e-mt-655f7f0b-67f9896d7c-wswvl\" (UID: \"0ddb013a-2517-4093-aab0-3ccbef086e0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-655f7f0b-67f9896d7c-wswvl"
Sep 10 17:54:53.665246 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:54:53.665025    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ddb013a-2517-4093-aab0-3ccbef086e0b-maas-api-tls\") pod \"maas-api-e2e-mt-655f7f0b-67f9896d7c-wswvl\" (UID: \"0ddb013a-2517-4093-aab0-3ccbef086e0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-655f7f0b-67f9896d7c-wswvl"
Sep 10 17:54:53.665246 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:54:53.665075    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ddb013a-2517-4093-aab0-3ccbef086e0b-openshift-service-ca\") pod \"maas-api-e2e-mt-655f7f0b-67f9896d7c-wswvl\" (UID: \"0ddb013a-2517-4093-aab0-3ccbef086e0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-655f7f0b-67f9896d7c-wswvl"
Sep 10 17:54:53.665246 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:54:53.665210    2579 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-655f7f0b-serving-cert: secret "maas-api-e2e-mt-655f7f0b-serving-cert" not found
Sep 10 17:54:53.665407 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:54:53.665307    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0ddb013a-2517-4093-aab0-3ccbef086e0b-maas-api-tls podName:0ddb013a-2517-4093-aab0-3ccbef086e0b nodeName:}" failed. No retries permitted until 2026-09-10 17:54:54.165286052 +0000 UTC m=+1138.165116100 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0ddb013a-2517-4093-aab0-3ccbef086e0b-maas-api-tls") pod "maas-api-e2e-mt-655f7f0b-67f9896d7c-wswvl" (UID: "0ddb013a-2517-4093-aab0-3ccbef086e0b") : secret "maas-api-e2e-mt-655f7f0b-serving-cert" not found
Sep 10 17:54:53.665801 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:54:53.665782    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ddb013a-2517-4093-aab0-3ccbef086e0b-openshift-service-ca\") pod \"maas-api-e2e-mt-655f7f0b-67f9896d7c-wswvl\" (UID: \"0ddb013a-2517-4093-aab0-3ccbef086e0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-655f7f0b-67f9896d7c-wswvl"
Sep 10 17:54:53.667712 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:54:53.667685    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ddb013a-2517-4093-aab0-3ccbef086e0b-metrics-certs\") pod \"maas-api-e2e-mt-655f7f0b-67f9896d7c-wswvl\" (UID: \"0ddb013a-2517-4093-aab0-3ccbef086e0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-655f7f0b-67f9896d7c-wswvl"
Sep 10 17:54:53.674237 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:54:53.674206    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2pscb\" (UniqueName: \"kubernetes.io/projected/0ddb013a-2517-4093-aab0-3ccbef086e0b-kube-api-access-2pscb\") pod \"maas-api-e2e-mt-655f7f0b-67f9896d7c-wswvl\" (UID: \"0ddb013a-2517-4093-aab0-3ccbef086e0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-655f7f0b-67f9896d7c-wswvl"
Sep 10 17:54:54.170755 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:54:54.170718    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ddb013a-2517-4093-aab0-3ccbef086e0b-maas-api-tls\") pod \"maas-api-e2e-mt-655f7f0b-67f9896d7c-wswvl\" (UID: \"0ddb013a-2517-4093-aab0-3ccbef086e0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-655f7f0b-67f9896d7c-wswvl"
Sep 10 17:54:54.170968 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:54:54.170883    2579 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-655f7f0b-serving-cert: secret "maas-api-e2e-mt-655f7f0b-serving-cert" not found
Sep 10 17:54:54.170968 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:54:54.170947    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0ddb013a-2517-4093-aab0-3ccbef086e0b-maas-api-tls podName:0ddb013a-2517-4093-aab0-3ccbef086e0b nodeName:}" failed. No retries permitted until 2026-09-10 17:54:55.170931163 +0000 UTC m=+1139.170761208 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0ddb013a-2517-4093-aab0-3ccbef086e0b-maas-api-tls") pod "maas-api-e2e-mt-655f7f0b-67f9896d7c-wswvl" (UID: "0ddb013a-2517-4093-aab0-3ccbef086e0b") : secret "maas-api-e2e-mt-655f7f0b-serving-cert" not found
Sep 10 17:54:54.401717 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:54:54.401664    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-655f7f0b-dc5fbb498-n2szr"]
Sep 10 17:54:54.407088 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:54:54.407062    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-655f7f0b-dc5fbb498-n2szr"
Sep 10 17:54:54.409888 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:54:54.409867    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-655f7f0b-dockercfg-5wr9d\""
Sep 10 17:54:54.410175 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:54:54.410157    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-655f7f0b\""
Sep 10 17:54:54.419254 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:54:54.419235    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-655f7f0b-dc5fbb498-n2szr"]
Sep 10 17:54:54.575697 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:54:54.575613    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5980f284-9fc2-42a5-9090-d12b1cec9baf-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-655f7f0b-dc5fbb498-n2szr\" (UID: \"5980f284-9fc2-42a5-9090-d12b1cec9baf\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-655f7f0b-dc5fbb498-n2szr"
Sep 10 17:54:54.576142 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:54:54.575691    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6g2j\" (UniqueName: \"kubernetes.io/projected/5980f284-9fc2-42a5-9090-d12b1cec9baf-kube-api-access-p6g2j\") pod \"payload-pre-processing-e2e-mt-655f7f0b-dc5fbb498-n2szr\" (UID: \"5980f284-9fc2-42a5-9090-d12b1cec9baf\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-655f7f0b-dc5fbb498-n2szr"
Sep 10 17:54:54.677111 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:54:54.677070    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5980f284-9fc2-42a5-9090-d12b1cec9baf-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-655f7f0b-dc5fbb498-n2szr\" (UID: \"5980f284-9fc2-42a5-9090-d12b1cec9baf\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-655f7f0b-dc5fbb498-n2szr"
Sep 10 17:54:54.677279 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:54:54.677132    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6g2j\" (UniqueName: \"kubernetes.io/projected/5980f284-9fc2-42a5-9090-d12b1cec9baf-kube-api-access-p6g2j\") pod \"payload-pre-processing-e2e-mt-655f7f0b-dc5fbb498-n2szr\" (UID: \"5980f284-9fc2-42a5-9090-d12b1cec9baf\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-655f7f0b-dc5fbb498-n2szr"
Sep 10 17:54:54.677838 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:54:54.677815    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5980f284-9fc2-42a5-9090-d12b1cec9baf-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-655f7f0b-dc5fbb498-n2szr\" (UID: \"5980f284-9fc2-42a5-9090-d12b1cec9baf\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-655f7f0b-dc5fbb498-n2szr"
Sep 10 17:54:54.688699 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:54:54.688671    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6g2j\" (UniqueName: \"kubernetes.io/projected/5980f284-9fc2-42a5-9090-d12b1cec9baf-kube-api-access-p6g2j\") pod \"payload-pre-processing-e2e-mt-655f7f0b-dc5fbb498-n2szr\" (UID: \"5980f284-9fc2-42a5-9090-d12b1cec9baf\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-655f7f0b-dc5fbb498-n2szr"
Sep 10 17:54:54.718201 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:54:54.718179    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-655f7f0b-dc5fbb498-n2szr"
Sep 10 17:54:54.992472 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:54:54.992440    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-655f7f0b-dc5fbb498-n2szr"]
Sep 10 17:54:54.993803 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:54:54.993775    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5980f284_9fc2_42a5_9090_d12b1cec9baf.slice/crio-2bcfd4a6bc662ef0392d4396cd3bf16c83bfe7ead2ce8da0ce33e4b4a17946d2 WatchSource:0}: Error finding container 2bcfd4a6bc662ef0392d4396cd3bf16c83bfe7ead2ce8da0ce33e4b4a17946d2: Status 404 returned error can't find the container with id 2bcfd4a6bc662ef0392d4396cd3bf16c83bfe7ead2ce8da0ce33e4b4a17946d2
Sep 10 17:54:55.182908 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:54:55.182875    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ddb013a-2517-4093-aab0-3ccbef086e0b-maas-api-tls\") pod \"maas-api-e2e-mt-655f7f0b-67f9896d7c-wswvl\" (UID: \"0ddb013a-2517-4093-aab0-3ccbef086e0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-655f7f0b-67f9896d7c-wswvl"
Sep 10 17:54:55.185403 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:54:55.185382    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ddb013a-2517-4093-aab0-3ccbef086e0b-maas-api-tls\") pod \"maas-api-e2e-mt-655f7f0b-67f9896d7c-wswvl\" (UID: \"0ddb013a-2517-4093-aab0-3ccbef086e0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-655f7f0b-67f9896d7c-wswvl"
Sep 10 17:54:55.292227 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:54:55.292158    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-655f7f0b-67f9896d7c-wswvl"
Sep 10 17:54:55.364830 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:54:55.364689    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-655f7f0b-dc5fbb498-n2szr" event={"ID":"5980f284-9fc2-42a5-9090-d12b1cec9baf","Type":"ContainerStarted","Data":"b4be9a30f3691943a2dd2d6479f92c334568142eec2eb10f87825678d0c34e92"}
Sep 10 17:54:55.364830 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:54:55.364741    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-655f7f0b-dc5fbb498-n2szr" event={"ID":"5980f284-9fc2-42a5-9090-d12b1cec9baf","Type":"ContainerStarted","Data":"2bcfd4a6bc662ef0392d4396cd3bf16c83bfe7ead2ce8da0ce33e4b4a17946d2"}
Sep 10 17:54:55.365040 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:54:55.364848    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-655f7f0b-dc5fbb498-n2szr"
Sep 10 17:54:55.388063 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:54:55.388004    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-655f7f0b-dc5fbb498-n2szr" podStartSLOduration=1.3879881570000001 podStartE2EDuration="1.387988157s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:54:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:54:55.385835298 +0000 UTC m=+1139.385665372" watchObservedRunningTime="2026-09-10 17:54:55.387988157 +0000 UTC m=+1139.387818244"
Sep 10 17:54:55.577521 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:54:55.577497    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-655f7f0b-67f9896d7c-wswvl"]
Sep 10 17:54:55.579433 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:54:55.579409    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ddb013a_2517_4093_aab0_3ccbef086e0b.slice/crio-69e508c36d4bd2875f62d666f5e9e33711b6c2da12b12f7c79e59f97c1dd50ab WatchSource:0}: Error finding container 69e508c36d4bd2875f62d666f5e9e33711b6c2da12b12f7c79e59f97c1dd50ab: Status 404 returned error can't find the container with id 69e508c36d4bd2875f62d666f5e9e33711b6c2da12b12f7c79e59f97c1dd50ab
Sep 10 17:54:56.374450 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:54:56.374349    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-655f7f0b-67f9896d7c-wswvl" event={"ID":"0ddb013a-2517-4093-aab0-3ccbef086e0b","Type":"ContainerStarted","Data":"ca02889435c79be71837febcd483670d89217986f33774109451e29731823d15"}
Sep 10 17:54:56.374450 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:54:56.374409    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-655f7f0b-67f9896d7c-wswvl" event={"ID":"0ddb013a-2517-4093-aab0-3ccbef086e0b","Type":"ContainerStarted","Data":"69e508c36d4bd2875f62d666f5e9e33711b6c2da12b12f7c79e59f97c1dd50ab"}
Sep 10 17:54:56.374721 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:54:56.374579    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-655f7f0b-67f9896d7c-wswvl"
Sep 10 17:54:56.401522 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:54:56.401466    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-655f7f0b-67f9896d7c-wswvl" podStartSLOduration=2.990017219 podStartE2EDuration="3.40144518s" totalImagesPullingTime="411.427961ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:54:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:54:56.40138717 +0000 UTC m=+1140.401217238" watchObservedRunningTime="2026-09-10 17:54:56.40144518 +0000 UTC m=+1140.401275249"
Sep 10 17:54:56.600019 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:54:56.599988    2579 scope.go:117] "RemoveContainer" containerID="8c3826913dae7223b3c25dfec9fc0c687617e50a56d1b76e1b605cf0a37eda4c"
Sep 10 17:55:02.386848 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:55:02.386816    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-655f7f0b-67f9896d7c-wswvl"
Sep 10 17:55:06.377811 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:55:06.377774    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-655f7f0b-dc5fbb498-n2szr"
Sep 10 17:55:12.467834 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:55:12.467763    2579 generic.go:356] "Generic (PLEG): container finished" podID="e05bf37b-74f1-445c-ad58-5464347cb2f3" containerID="7d99a27e60fe4b415004eac9cb4d7d40d4dc752dee6f6c7deeec583cedbb3899" exitCode=0
Sep 10 17:55:12.468242 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:55:12.467834    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-655f7f0b-openshift-default-86546b74d-wrvn7" event={"ID":"e05bf37b-74f1-445c-ad58-5464347cb2f3","Type":"ContainerDied","Data":"7d99a27e60fe4b415004eac9cb4d7d40d4dc752dee6f6c7deeec583cedbb3899"}
Sep 10 17:55:12.468309 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:55:12.468294    2579 scope.go:117] "RemoveContainer" containerID="7d99a27e60fe4b415004eac9cb4d7d40d4dc752dee6f6c7deeec583cedbb3899"
Sep 10 17:55:12.469672 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:55:12.469639    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 10 17:55:12.469748 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:55:12.469722    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 10 17:55:12.469787 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:55:12.469764    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 10 17:55:13.478924 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:55:13.478868    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-655f7f0b-openshift-default-86546b74d-wrvn7" event={"ID":"e05bf37b-74f1-445c-ad58-5464347cb2f3","Type":"ContainerStarted","Data":"0b9e021549db2169a6cfa053655478440c11a34c834db89e7dd79783b58c2164"}
Sep 10 17:55:13.912991 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:55:13.912957    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-655f7f0b-openshift-default-86546b74d-wrvn7"
Sep 10 17:55:13.913186 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:55:13.913144    2579 patch_prober.go:28] interesting pod/e2e-mt-655f7f0b-openshift-default-86546b74d-wrvn7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body=
Sep 10 17:55:13.913262 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:55:13.913196    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-655f7f0b-openshift-default-86546b74d-wrvn7" podUID="e05bf37b-74f1-445c-ad58-5464347cb2f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused"
Sep 10 17:55:14.913262 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:55:14.913225    2579 patch_prober.go:28] interesting pod/e2e-mt-655f7f0b-openshift-default-86546b74d-wrvn7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body=
Sep 10 17:55:14.913676 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:55:14.913283    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-655f7f0b-openshift-default-86546b74d-wrvn7" podUID="e05bf37b-74f1-445c-ad58-5464347cb2f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused"
Sep 10 17:55:15.912812 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:55:15.912778    2579 patch_prober.go:28] interesting pod/e2e-mt-655f7f0b-openshift-default-86546b74d-wrvn7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body=
Sep 10 17:55:15.912991 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:55:15.912834    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-655f7f0b-openshift-default-86546b74d-wrvn7" podUID="e05bf37b-74f1-445c-ad58-5464347cb2f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused"
Sep 10 17:55:16.913356 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:55:16.913316    2579 patch_prober.go:28] interesting pod/e2e-mt-655f7f0b-openshift-default-86546b74d-wrvn7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body=
Sep 10 17:55:16.913784 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:55:16.913381    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-655f7f0b-openshift-default-86546b74d-wrvn7" podUID="e05bf37b-74f1-445c-ad58-5464347cb2f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused"
Sep 10 17:55:17.912696 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:55:17.912620    2579 patch_prober.go:28] interesting pod/e2e-mt-655f7f0b-openshift-default-86546b74d-wrvn7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body=
Sep 10 17:55:17.912696 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:55:17.912672    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-655f7f0b-openshift-default-86546b74d-wrvn7" podUID="e05bf37b-74f1-445c-ad58-5464347cb2f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused"
Sep 10 17:55:18.917181 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:55:18.917148    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-655f7f0b-openshift-default-86546b74d-wrvn7"
Sep 10 17:55:18.917587 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:55:18.917400    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-655f7f0b-openshift-default-86546b74d-wrvn7"
Sep 10 17:55:18.918044 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:55:18.918029    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-655f7f0b-openshift-default-86546b74d-wrvn7"
Sep 10 17:55:56.496741 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:55:56.496713    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-rb854_97172b14-b84a-476e-89f1-05d3e70df416/console-operator/1.log"
Sep 10 17:55:56.499064 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:55:56.499039    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-rb854_97172b14-b84a-476e-89f1-05d3e70df416/console-operator/1.log"
Sep 10 17:55:56.619706 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:55:56.619680    2579 scope.go:117] "RemoveContainer" containerID="d0230d30b92790296334cf4a1effcdd9fd49d2bd3d9b933068b15302ccfdd94c"
Sep 10 17:56:07.155804 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:56:07.155772    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-655f7f0b-67f9896d7c-wswvl"]
Sep 10 17:56:07.156450 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:56:07.155978    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-655f7f0b-67f9896d7c-wswvl" podUID="0ddb013a-2517-4093-aab0-3ccbef086e0b" containerName="maas-api" containerID="cri-o://ca02889435c79be71837febcd483670d89217986f33774109451e29731823d15" gracePeriod=30
Sep 10 17:56:07.219166 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:56:07.219136    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-655f7f0b-dc5fbb498-n2szr"]
Sep 10 17:56:07.219400 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:56:07.219355    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-655f7f0b-dc5fbb498-n2szr" podUID="5980f284-9fc2-42a5-9090-d12b1cec9baf" containerName="payload-pre-processing" containerID="cri-o://b4be9a30f3691943a2dd2d6479f92c334568142eec2eb10f87825678d0c34e92" gracePeriod=30
Sep 10 17:56:07.381479 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:56:07.381437    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-mt-655f7f0b-67f9896d7c-wswvl" podUID="0ddb013a-2517-4093-aab0-3ccbef086e0b" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.73:8443/health\": dial tcp 10.132.0.73:8443: connect: connection refused"
Sep 10 17:56:07.513881 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:56:07.513853    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-655f7f0b-67f9896d7c-wswvl"
Sep 10 17:56:07.516892 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:56:07.516870    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-655f7f0b-dc5fbb498-n2szr"
Sep 10 17:56:07.556992 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:56:07.556966    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5980f284-9fc2-42a5-9090-d12b1cec9baf-plugins-config-volume\") pod \"5980f284-9fc2-42a5-9090-d12b1cec9baf\" (UID: \"5980f284-9fc2-42a5-9090-d12b1cec9baf\") "
Sep 10 17:56:07.557167 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:56:07.557008    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ddb013a-2517-4093-aab0-3ccbef086e0b-openshift-service-ca\") pod \"0ddb013a-2517-4093-aab0-3ccbef086e0b\" (UID: \"0ddb013a-2517-4093-aab0-3ccbef086e0b\") "
Sep 10 17:56:07.557167 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:56:07.557041    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p6g2j\" (UniqueName: \"kubernetes.io/projected/5980f284-9fc2-42a5-9090-d12b1cec9baf-kube-api-access-p6g2j\") pod \"5980f284-9fc2-42a5-9090-d12b1cec9baf\" (UID: \"5980f284-9fc2-42a5-9090-d12b1cec9baf\") "
Sep 10 17:56:07.557167 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:56:07.557066    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ddb013a-2517-4093-aab0-3ccbef086e0b-maas-api-tls\") pod \"0ddb013a-2517-4093-aab0-3ccbef086e0b\" (UID: \"0ddb013a-2517-4093-aab0-3ccbef086e0b\") "
Sep 10 17:56:07.557167 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:56:07.557127    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ddb013a-2517-4093-aab0-3ccbef086e0b-metrics-certs\") pod \"0ddb013a-2517-4093-aab0-3ccbef086e0b\" (UID: \"0ddb013a-2517-4093-aab0-3ccbef086e0b\") "
Sep 10 17:56:07.557389 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:56:07.557207    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2pscb\" (UniqueName: \"kubernetes.io/projected/0ddb013a-2517-4093-aab0-3ccbef086e0b-kube-api-access-2pscb\") pod \"0ddb013a-2517-4093-aab0-3ccbef086e0b\" (UID: \"0ddb013a-2517-4093-aab0-3ccbef086e0b\") "
Sep 10 17:56:07.557453 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:56:07.557389    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ddb013a-2517-4093-aab0-3ccbef086e0b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "0ddb013a-2517-4093-aab0-3ccbef086e0b" (UID: "0ddb013a-2517-4093-aab0-3ccbef086e0b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 17:56:07.557692 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:56:07.557669    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ddb013a-2517-4093-aab0-3ccbef086e0b-openshift-service-ca\") on node \"ip-10-0-129-122.ec2.internal\" DevicePath \"\""
Sep 10 17:56:07.557901 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:56:07.557864    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5980f284-9fc2-42a5-9090-d12b1cec9baf-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5980f284-9fc2-42a5-9090-d12b1cec9baf" (UID: "5980f284-9fc2-42a5-9090-d12b1cec9baf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 17:56:07.560121 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:56:07.560095    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ddb013a-2517-4093-aab0-3ccbef086e0b-kube-api-access-2pscb" (OuterVolumeSpecName: "kube-api-access-2pscb") pod "0ddb013a-2517-4093-aab0-3ccbef086e0b" (UID: "0ddb013a-2517-4093-aab0-3ccbef086e0b"). InnerVolumeSpecName "kube-api-access-2pscb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:56:07.560223 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:56:07.560132    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ddb013a-2517-4093-aab0-3ccbef086e0b-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "0ddb013a-2517-4093-aab0-3ccbef086e0b" (UID: "0ddb013a-2517-4093-aab0-3ccbef086e0b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 17:56:07.560223 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:56:07.560144    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ddb013a-2517-4093-aab0-3ccbef086e0b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "0ddb013a-2517-4093-aab0-3ccbef086e0b" (UID: "0ddb013a-2517-4093-aab0-3ccbef086e0b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 17:56:07.560223 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:56:07.560188    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5980f284-9fc2-42a5-9090-d12b1cec9baf-kube-api-access-p6g2j" (OuterVolumeSpecName: "kube-api-access-p6g2j") pod "5980f284-9fc2-42a5-9090-d12b1cec9baf" (UID: "5980f284-9fc2-42a5-9090-d12b1cec9baf"). InnerVolumeSpecName "kube-api-access-p6g2j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:56:07.658584 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:56:07.658524    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5980f284-9fc2-42a5-9090-d12b1cec9baf-plugins-config-volume\") on node \"ip-10-0-129-122.ec2.internal\" DevicePath \"\""
Sep 10 17:56:07.658584 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:56:07.658560    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p6g2j\" (UniqueName: \"kubernetes.io/projected/5980f284-9fc2-42a5-9090-d12b1cec9baf-kube-api-access-p6g2j\") on node \"ip-10-0-129-122.ec2.internal\" DevicePath \"\""
Sep 10 17:56:07.658755 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:56:07.658604    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ddb013a-2517-4093-aab0-3ccbef086e0b-maas-api-tls\") on node \"ip-10-0-129-122.ec2.internal\" DevicePath \"\""
Sep 10 17:56:07.658755 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:56:07.658618    2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ddb013a-2517-4093-aab0-3ccbef086e0b-metrics-certs\") on node \"ip-10-0-129-122.ec2.internal\" DevicePath \"\""
Sep 10 17:56:07.658755 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:56:07.658630    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2pscb\" (UniqueName: \"kubernetes.io/projected/0ddb013a-2517-4093-aab0-3ccbef086e0b-kube-api-access-2pscb\") on node \"ip-10-0-129-122.ec2.internal\" DevicePath \"\""
Sep 10 17:56:07.770420 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:56:07.770340    2579 generic.go:356] "Generic (PLEG): container finished" podID="5980f284-9fc2-42a5-9090-d12b1cec9baf" containerID="b4be9a30f3691943a2dd2d6479f92c334568142eec2eb10f87825678d0c34e92" exitCode=0
Sep 10 17:56:07.770420 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:56:07.770385    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-655f7f0b-dc5fbb498-n2szr"
Sep 10 17:56:07.770642 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:56:07.770422    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-655f7f0b-dc5fbb498-n2szr" event={"ID":"5980f284-9fc2-42a5-9090-d12b1cec9baf","Type":"ContainerDied","Data":"b4be9a30f3691943a2dd2d6479f92c334568142eec2eb10f87825678d0c34e92"}
Sep 10 17:56:07.770642 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:56:07.770470    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-655f7f0b-dc5fbb498-n2szr" event={"ID":"5980f284-9fc2-42a5-9090-d12b1cec9baf","Type":"ContainerDied","Data":"2bcfd4a6bc662ef0392d4396cd3bf16c83bfe7ead2ce8da0ce33e4b4a17946d2"}
Sep 10 17:56:07.770642 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:56:07.770496    2579 scope.go:117] "RemoveContainer" containerID="b4be9a30f3691943a2dd2d6479f92c334568142eec2eb10f87825678d0c34e92"
Sep 10 17:56:07.771934 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:56:07.771914    2579 generic.go:356] "Generic (PLEG): container finished" podID="0ddb013a-2517-4093-aab0-3ccbef086e0b" containerID="ca02889435c79be71837febcd483670d89217986f33774109451e29731823d15" exitCode=0
Sep 10 17:56:07.772049 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:56:07.771962    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-655f7f0b-67f9896d7c-wswvl" event={"ID":"0ddb013a-2517-4093-aab0-3ccbef086e0b","Type":"ContainerDied","Data":"ca02889435c79be71837febcd483670d89217986f33774109451e29731823d15"}
Sep 10 17:56:07.772123 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:56:07.772029    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-655f7f0b-67f9896d7c-wswvl" event={"ID":"0ddb013a-2517-4093-aab0-3ccbef086e0b","Type":"ContainerDied","Data":"69e508c36d4bd2875f62d666f5e9e33711b6c2da12b12f7c79e59f97c1dd50ab"}
Sep 10 17:56:07.772123 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:56:07.771974    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-655f7f0b-67f9896d7c-wswvl"
Sep 10 17:56:07.780469 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:56:07.780446    2579 scope.go:117] "RemoveContainer" containerID="b4be9a30f3691943a2dd2d6479f92c334568142eec2eb10f87825678d0c34e92"
Sep 10 17:56:07.780788 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:56:07.780761    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4be9a30f3691943a2dd2d6479f92c334568142eec2eb10f87825678d0c34e92\": container with ID starting with b4be9a30f3691943a2dd2d6479f92c334568142eec2eb10f87825678d0c34e92 not found: ID does not exist" containerID="b4be9a30f3691943a2dd2d6479f92c334568142eec2eb10f87825678d0c34e92"
Sep 10 17:56:07.780862 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:56:07.780793    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4be9a30f3691943a2dd2d6479f92c334568142eec2eb10f87825678d0c34e92"} err="failed to get container status \"b4be9a30f3691943a2dd2d6479f92c334568142eec2eb10f87825678d0c34e92\": rpc error: code = NotFound desc = could not find container \"b4be9a30f3691943a2dd2d6479f92c334568142eec2eb10f87825678d0c34e92\": container with ID starting with b4be9a30f3691943a2dd2d6479f92c334568142eec2eb10f87825678d0c34e92 not found: ID does not exist"
Sep 10 17:56:07.780862 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:56:07.780817    2579 scope.go:117] "RemoveContainer" containerID="ca02889435c79be71837febcd483670d89217986f33774109451e29731823d15"
Sep 10 17:56:07.791358 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:56:07.791344    2579 scope.go:117] "RemoveContainer" containerID="ca02889435c79be71837febcd483670d89217986f33774109451e29731823d15"
Sep 10 17:56:07.791604 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:56:07.791589    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca02889435c79be71837febcd483670d89217986f33774109451e29731823d15\": container with ID starting with ca02889435c79be71837febcd483670d89217986f33774109451e29731823d15 not found: ID does not exist" containerID="ca02889435c79be71837febcd483670d89217986f33774109451e29731823d15"
Sep 10 17:56:07.791655 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:56:07.791609    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca02889435c79be71837febcd483670d89217986f33774109451e29731823d15"} err="failed to get container status \"ca02889435c79be71837febcd483670d89217986f33774109451e29731823d15\": rpc error: code = NotFound desc = could not find container \"ca02889435c79be71837febcd483670d89217986f33774109451e29731823d15\": container with ID starting with ca02889435c79be71837febcd483670d89217986f33774109451e29731823d15 not found: ID does not exist"
Sep 10 17:56:07.800173 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:56:07.800151    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-655f7f0b-67f9896d7c-wswvl"]
Sep 10 17:56:07.804439 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:56:07.804417    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-655f7f0b-67f9896d7c-wswvl"]
Sep 10 17:56:07.815429 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:56:07.815406    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-655f7f0b-dc5fbb498-n2szr"]
Sep 10 17:56:07.823827 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:56:07.823808    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-655f7f0b-dc5fbb498-n2szr"]
Sep 10 17:56:08.476541 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:56:08.476509    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0ddb013a-2517-4093-aab0-3ccbef086e0b" path="/var/lib/kubelet/pods/0ddb013a-2517-4093-aab0-3ccbef086e0b/volumes"
Sep 10 17:56:08.477045 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:56:08.477027    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5980f284-9fc2-42a5-9090-d12b1cec9baf" path="/var/lib/kubelet/pods/5980f284-9fc2-42a5-9090-d12b1cec9baf/volumes"
Sep 10 17:56:59.263776 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:56:59.263742    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-655f7f0b-openshift-default-86546b74d-wrvn7"]
Sep 10 17:56:59.264277 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:56:59.263987    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-655f7f0b-openshift-default-86546b74d-wrvn7" podUID="e05bf37b-74f1-445c-ad58-5464347cb2f3" containerName="istio-proxy" containerID="cri-o://0b9e021549db2169a6cfa053655478440c11a34c834db89e7dd79783b58c2164" gracePeriod=30
Sep 10 17:57:03.920321 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:57:03.920281    2579 patch_prober.go:28] interesting pod/e2e-mt-655f7f0b-openshift-default-86546b74d-wrvn7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 10 17:57:03.920712 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:57:03.920343    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-655f7f0b-openshift-default-86546b74d-wrvn7" podUID="e05bf37b-74f1-445c-ad58-5464347cb2f3" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 10 17:57:04.522138 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:57:04.522109    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-655f7f0b-openshift-default-86546b74d-wrvn7"
Sep 10 17:57:04.674368 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:57:04.674329    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e05bf37b-74f1-445c-ad58-5464347cb2f3-istio-podinfo\") pod \"e05bf37b-74f1-445c-ad58-5464347cb2f3\" (UID: \"e05bf37b-74f1-445c-ad58-5464347cb2f3\") "
Sep 10 17:57:04.674596 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:57:04.674399    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e05bf37b-74f1-445c-ad58-5464347cb2f3-workload-socket\") pod \"e05bf37b-74f1-445c-ad58-5464347cb2f3\" (UID: \"e05bf37b-74f1-445c-ad58-5464347cb2f3\") "
Sep 10 17:57:04.674596 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:57:04.674437    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e05bf37b-74f1-445c-ad58-5464347cb2f3-workload-certs\") pod \"e05bf37b-74f1-445c-ad58-5464347cb2f3\" (UID: \"e05bf37b-74f1-445c-ad58-5464347cb2f3\") "
Sep 10 17:57:04.674596 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:57:04.674471    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-85qxw\" (UniqueName: \"kubernetes.io/projected/e05bf37b-74f1-445c-ad58-5464347cb2f3-kube-api-access-85qxw\") pod \"e05bf37b-74f1-445c-ad58-5464347cb2f3\" (UID: \"e05bf37b-74f1-445c-ad58-5464347cb2f3\") "
Sep 10 17:57:04.674596 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:57:04.674530    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e05bf37b-74f1-445c-ad58-5464347cb2f3-istio-envoy\") pod \"e05bf37b-74f1-445c-ad58-5464347cb2f3\" (UID: \"e05bf37b-74f1-445c-ad58-5464347cb2f3\") "
Sep 10 17:57:04.674864 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:57:04.674629    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e05bf37b-74f1-445c-ad58-5464347cb2f3-credential-socket\") pod \"e05bf37b-74f1-445c-ad58-5464347cb2f3\" (UID: \"e05bf37b-74f1-445c-ad58-5464347cb2f3\") "
Sep 10 17:57:04.674864 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:57:04.674680    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e05bf37b-74f1-445c-ad58-5464347cb2f3-istiod-ca-cert\") pod \"e05bf37b-74f1-445c-ad58-5464347cb2f3\" (UID: \"e05bf37b-74f1-445c-ad58-5464347cb2f3\") "
Sep 10 17:57:04.674864 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:57:04.674744    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e05bf37b-74f1-445c-ad58-5464347cb2f3-istio-token\") pod \"e05bf37b-74f1-445c-ad58-5464347cb2f3\" (UID: \"e05bf37b-74f1-445c-ad58-5464347cb2f3\") "
Sep 10 17:57:04.674864 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:57:04.674778    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e05bf37b-74f1-445c-ad58-5464347cb2f3-istio-data\") pod \"e05bf37b-74f1-445c-ad58-5464347cb2f3\" (UID: \"e05bf37b-74f1-445c-ad58-5464347cb2f3\") "
Sep 10 17:57:04.675079 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:57:04.675007    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e05bf37b-74f1-445c-ad58-5464347cb2f3-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e05bf37b-74f1-445c-ad58-5464347cb2f3" (UID: "e05bf37b-74f1-445c-ad58-5464347cb2f3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:57:04.675140 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:57:04.675108    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e05bf37b-74f1-445c-ad58-5464347cb2f3-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e05bf37b-74f1-445c-ad58-5464347cb2f3" (UID: "e05bf37b-74f1-445c-ad58-5464347cb2f3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:57:04.675245 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:57:04.675222    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e05bf37b-74f1-445c-ad58-5464347cb2f3-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e05bf37b-74f1-445c-ad58-5464347cb2f3" (UID: "e05bf37b-74f1-445c-ad58-5464347cb2f3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:57:04.675343 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:57:04.675307    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e05bf37b-74f1-445c-ad58-5464347cb2f3-istio-data" (OuterVolumeSpecName: "istio-data") pod "e05bf37b-74f1-445c-ad58-5464347cb2f3" (UID: "e05bf37b-74f1-445c-ad58-5464347cb2f3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:57:04.675409 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:57:04.675335    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e05bf37b-74f1-445c-ad58-5464347cb2f3-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e05bf37b-74f1-445c-ad58-5464347cb2f3" (UID: "e05bf37b-74f1-445c-ad58-5464347cb2f3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 17:57:04.677307 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:57:04.677270    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e05bf37b-74f1-445c-ad58-5464347cb2f3-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e05bf37b-74f1-445c-ad58-5464347cb2f3" (UID: "e05bf37b-74f1-445c-ad58-5464347cb2f3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 10 17:57:04.677601 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:57:04.677553    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e05bf37b-74f1-445c-ad58-5464347cb2f3-istio-token" (OuterVolumeSpecName: "istio-token") pod "e05bf37b-74f1-445c-ad58-5464347cb2f3" (UID: "e05bf37b-74f1-445c-ad58-5464347cb2f3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:57:04.677795 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:57:04.677775    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e05bf37b-74f1-445c-ad58-5464347cb2f3-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e05bf37b-74f1-445c-ad58-5464347cb2f3" (UID: "e05bf37b-74f1-445c-ad58-5464347cb2f3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:57:04.677919 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:57:04.677901    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e05bf37b-74f1-445c-ad58-5464347cb2f3-kube-api-access-85qxw" (OuterVolumeSpecName: "kube-api-access-85qxw") pod "e05bf37b-74f1-445c-ad58-5464347cb2f3" (UID: "e05bf37b-74f1-445c-ad58-5464347cb2f3"). InnerVolumeSpecName "kube-api-access-85qxw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:57:04.776370 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:57:04.776330    2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e05bf37b-74f1-445c-ad58-5464347cb2f3-istiod-ca-cert\") on node \"ip-10-0-129-122.ec2.internal\" DevicePath \"\""
Sep 10 17:57:04.776370 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:57:04.776362    2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e05bf37b-74f1-445c-ad58-5464347cb2f3-istio-token\") on node \"ip-10-0-129-122.ec2.internal\" DevicePath \"\""
Sep 10 17:57:04.776370 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:57:04.776370    2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e05bf37b-74f1-445c-ad58-5464347cb2f3-istio-data\") on node \"ip-10-0-129-122.ec2.internal\" DevicePath \"\""
Sep 10 17:57:04.776370 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:57:04.776378    2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e05bf37b-74f1-445c-ad58-5464347cb2f3-istio-podinfo\") on node \"ip-10-0-129-122.ec2.internal\" DevicePath \"\""
Sep 10 17:57:04.776689 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:57:04.776385    2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e05bf37b-74f1-445c-ad58-5464347cb2f3-workload-socket\") on node \"ip-10-0-129-122.ec2.internal\" DevicePath \"\""
Sep 10 17:57:04.776689 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:57:04.776394    2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e05bf37b-74f1-445c-ad58-5464347cb2f3-workload-certs\") on node \"ip-10-0-129-122.ec2.internal\" DevicePath \"\""
Sep 10 17:57:04.776689 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:57:04.776403    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-85qxw\" (UniqueName: \"kubernetes.io/projected/e05bf37b-74f1-445c-ad58-5464347cb2f3-kube-api-access-85qxw\") on node \"ip-10-0-129-122.ec2.internal\" DevicePath \"\""
Sep 10 17:57:04.776689 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:57:04.776411    2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e05bf37b-74f1-445c-ad58-5464347cb2f3-istio-envoy\") on node \"ip-10-0-129-122.ec2.internal\" DevicePath \"\""
Sep 10 17:57:04.776689 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:57:04.776419    2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e05bf37b-74f1-445c-ad58-5464347cb2f3-credential-socket\") on node \"ip-10-0-129-122.ec2.internal\" DevicePath \"\""
Sep 10 17:57:05.065514 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:57:05.065406    2579 generic.go:356] "Generic (PLEG): container finished" podID="e05bf37b-74f1-445c-ad58-5464347cb2f3" containerID="0b9e021549db2169a6cfa053655478440c11a34c834db89e7dd79783b58c2164" exitCode=0
Sep 10 17:57:05.065989 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:57:05.065528    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-655f7f0b-openshift-default-86546b74d-wrvn7" event={"ID":"e05bf37b-74f1-445c-ad58-5464347cb2f3","Type":"ContainerDied","Data":"0b9e021549db2169a6cfa053655478440c11a34c834db89e7dd79783b58c2164"}
Sep 10 17:57:05.065989 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:57:05.065647    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-655f7f0b-openshift-default-86546b74d-wrvn7" event={"ID":"e05bf37b-74f1-445c-ad58-5464347cb2f3","Type":"ContainerDied","Data":"1e51f9d33821cddea967b3dd8373363175707b52de4d33ff2c4671030611e990"}
Sep 10 17:57:05.065989 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:57:05.065688    2579 scope.go:117] "RemoveContainer" containerID="0b9e021549db2169a6cfa053655478440c11a34c834db89e7dd79783b58c2164"
Sep 10 17:57:05.065989 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:57:05.065897    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-655f7f0b-openshift-default-86546b74d-wrvn7"
Sep 10 17:57:05.079357 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:57:05.079329    2579 scope.go:117] "RemoveContainer" containerID="7d99a27e60fe4b415004eac9cb4d7d40d4dc752dee6f6c7deeec583cedbb3899"
Sep 10 17:57:05.088351 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:57:05.088331    2579 scope.go:117] "RemoveContainer" containerID="0b9e021549db2169a6cfa053655478440c11a34c834db89e7dd79783b58c2164"
Sep 10 17:57:05.088719 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:57:05.088699    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b9e021549db2169a6cfa053655478440c11a34c834db89e7dd79783b58c2164\": container with ID starting with 0b9e021549db2169a6cfa053655478440c11a34c834db89e7dd79783b58c2164 not found: ID does not exist" containerID="0b9e021549db2169a6cfa053655478440c11a34c834db89e7dd79783b58c2164"
Sep 10 17:57:05.088792 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:57:05.088728    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b9e021549db2169a6cfa053655478440c11a34c834db89e7dd79783b58c2164"} err="failed to get container status \"0b9e021549db2169a6cfa053655478440c11a34c834db89e7dd79783b58c2164\": rpc error: code = NotFound desc = could not find container \"0b9e021549db2169a6cfa053655478440c11a34c834db89e7dd79783b58c2164\": container with ID starting with 0b9e021549db2169a6cfa053655478440c11a34c834db89e7dd79783b58c2164 not found: ID does not exist"
Sep 10 17:57:05.088792 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:57:05.088745    2579 scope.go:117] "RemoveContainer" containerID="7d99a27e60fe4b415004eac9cb4d7d40d4dc752dee6f6c7deeec583cedbb3899"
Sep 10 17:57:05.089034 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:57:05.089019    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d99a27e60fe4b415004eac9cb4d7d40d4dc752dee6f6c7deeec583cedbb3899\": container with ID starting with 7d99a27e60fe4b415004eac9cb4d7d40d4dc752dee6f6c7deeec583cedbb3899 not found: ID does not exist" containerID="7d99a27e60fe4b415004eac9cb4d7d40d4dc752dee6f6c7deeec583cedbb3899"
Sep 10 17:57:05.089078 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:57:05.089050    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d99a27e60fe4b415004eac9cb4d7d40d4dc752dee6f6c7deeec583cedbb3899"} err="failed to get container status \"7d99a27e60fe4b415004eac9cb4d7d40d4dc752dee6f6c7deeec583cedbb3899\": rpc error: code = NotFound desc = could not find container \"7d99a27e60fe4b415004eac9cb4d7d40d4dc752dee6f6c7deeec583cedbb3899\": container with ID starting with 7d99a27e60fe4b415004eac9cb4d7d40d4dc752dee6f6c7deeec583cedbb3899 not found: ID does not exist"
Sep 10 17:57:05.094145 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:57:05.094117    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-655f7f0b-openshift-default-86546b74d-wrvn7"]
Sep 10 17:57:05.097820 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:57:05.097800    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-655f7f0b-openshift-default-86546b74d-wrvn7"]
Sep 10 17:57:06.477424 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:57:06.477388    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e05bf37b-74f1-445c-ad58-5464347cb2f3" path="/var/lib/kubelet/pods/e05bf37b-74f1-445c-ad58-5464347cb2f3/volumes"
Sep 10 17:57:54.384583 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:57:54.384533    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-c5c751-openshift-default-586c478496-llxnx"]
Sep 10 17:57:54.385181 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:57:54.385155    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5980f284-9fc2-42a5-9090-d12b1cec9baf" containerName="payload-pre-processing"
Sep 10 17:57:54.385292 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:57:54.385184    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="5980f284-9fc2-42a5-9090-d12b1cec9baf" containerName="payload-pre-processing"
Sep 10 17:57:54.385292 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:57:54.385200    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0ddb013a-2517-4093-aab0-3ccbef086e0b" containerName="maas-api"
Sep 10 17:57:54.385292 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:57:54.385208    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="0ddb013a-2517-4093-aab0-3ccbef086e0b" containerName="maas-api"
Sep 10 17:57:54.385292 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:57:54.385228    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e05bf37b-74f1-445c-ad58-5464347cb2f3" containerName="istio-proxy"
Sep 10 17:57:54.385292 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:57:54.385237    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="e05bf37b-74f1-445c-ad58-5464347cb2f3" containerName="istio-proxy"
Sep 10 17:57:54.385292 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:57:54.385258    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e05bf37b-74f1-445c-ad58-5464347cb2f3" containerName="istio-proxy"
Sep 10 17:57:54.385292 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:57:54.385267    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="e05bf37b-74f1-445c-ad58-5464347cb2f3" containerName="istio-proxy"
Sep 10 17:57:54.385546 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:57:54.385385    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="e05bf37b-74f1-445c-ad58-5464347cb2f3" containerName="istio-proxy"
Sep 10 17:57:54.385546 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:57:54.385405    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="5980f284-9fc2-42a5-9090-d12b1cec9baf" containerName="payload-pre-processing"
Sep 10 17:57:54.385546 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:57:54.385416    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="0ddb013a-2517-4093-aab0-3ccbef086e0b" containerName="maas-api"
Sep 10 17:57:54.388960 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:57:54.388936    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-c5c751-openshift-default-586c478496-llxnx"
Sep 10 17:57:54.391846 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:57:54.391826    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-c5c751-openshift-default-dockercfg-c7rfq\""
Sep 10 17:57:54.405433 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:57:54.405407    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-c5c751-openshift-default-586c478496-llxnx"]
Sep 10 17:57:54.539687 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:57:54.539645    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/013e342b-e2dc-4034-b58f-ea9bafb7b379-istio-podinfo\") pod \"e2e-api-a-c5c751-openshift-default-586c478496-llxnx\" (UID: \"013e342b-e2dc-4034-b58f-ea9bafb7b379\") " pod="openshift-ingress/e2e-api-a-c5c751-openshift-default-586c478496-llxnx"
Sep 10 17:57:54.539893 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:57:54.539702    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/013e342b-e2dc-4034-b58f-ea9bafb7b379-istio-envoy\") pod \"e2e-api-a-c5c751-openshift-default-586c478496-llxnx\" (UID: \"013e342b-e2dc-4034-b58f-ea9bafb7b379\") " pod="openshift-ingress/e2e-api-a-c5c751-openshift-default-586c478496-llxnx"
Sep 10 17:57:54.539893 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:57:54.539750    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2j5r\" (UniqueName: \"kubernetes.io/projected/013e342b-e2dc-4034-b58f-ea9bafb7b379-kube-api-access-s2j5r\") pod \"e2e-api-a-c5c751-openshift-default-586c478496-llxnx\" (UID: \"013e342b-e2dc-4034-b58f-ea9bafb7b379\") " pod="openshift-ingress/e2e-api-a-c5c751-openshift-default-586c478496-llxnx"
Sep 10 17:57:54.539893 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:57:54.539775    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/013e342b-e2dc-4034-b58f-ea9bafb7b379-credential-socket\") pod \"e2e-api-a-c5c751-openshift-default-586c478496-llxnx\" (UID: \"013e342b-e2dc-4034-b58f-ea9bafb7b379\") " pod="openshift-ingress/e2e-api-a-c5c751-openshift-default-586c478496-llxnx"
Sep 10 17:57:54.539893 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:57:54.539796    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/013e342b-e2dc-4034-b58f-ea9bafb7b379-istiod-ca-cert\") pod \"e2e-api-a-c5c751-openshift-default-586c478496-llxnx\" (UID: \"013e342b-e2dc-4034-b58f-ea9bafb7b379\") " pod="openshift-ingress/e2e-api-a-c5c751-openshift-default-586c478496-llxnx"
Sep 10 17:57:54.539893 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:57:54.539814    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/013e342b-e2dc-4034-b58f-ea9bafb7b379-workload-certs\") pod \"e2e-api-a-c5c751-openshift-default-586c478496-llxnx\" (UID: \"013e342b-e2dc-4034-b58f-ea9bafb7b379\") " pod="openshift-ingress/e2e-api-a-c5c751-openshift-default-586c478496-llxnx"
Sep 10 17:57:54.539893 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:57:54.539833    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/013e342b-e2dc-4034-b58f-ea9bafb7b379-istio-data\") pod \"e2e-api-a-c5c751-openshift-default-586c478496-llxnx\" (UID: \"013e342b-e2dc-4034-b58f-ea9bafb7b379\") " pod="openshift-ingress/e2e-api-a-c5c751-openshift-default-586c478496-llxnx"
Sep 10 17:57:54.539893 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:57:54.539865    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/013e342b-e2dc-4034-b58f-ea9bafb7b379-istio-token\") pod \"e2e-api-a-c5c751-openshift-default-586c478496-llxnx\" (UID: \"013e342b-e2dc-4034-b58f-ea9bafb7b379\") " pod="openshift-ingress/e2e-api-a-c5c751-openshift-default-586c478496-llxnx"
Sep 10 17:57:54.540253 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:57:54.539907    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/013e342b-e2dc-4034-b58f-ea9bafb7b379-workload-socket\") pod \"e2e-api-a-c5c751-openshift-default-586c478496-llxnx\" (UID: \"013e342b-e2dc-4034-b58f-ea9bafb7b379\") " pod="openshift-ingress/e2e-api-a-c5c751-openshift-default-586c478496-llxnx"
Sep 10 17:57:54.640863 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:57:54.640762    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2j5r\" (UniqueName: \"kubernetes.io/projected/013e342b-e2dc-4034-b58f-ea9bafb7b379-kube-api-access-s2j5r\") pod \"e2e-api-a-c5c751-openshift-default-586c478496-llxnx\" (UID: \"013e342b-e2dc-4034-b58f-ea9bafb7b379\") " pod="openshift-ingress/e2e-api-a-c5c751-openshift-default-586c478496-llxnx"
Sep 10 17:57:54.640863 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:57:54.640831    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/013e342b-e2dc-4034-b58f-ea9bafb7b379-credential-socket\") pod \"e2e-api-a-c5c751-openshift-default-586c478496-llxnx\" (UID: \"013e342b-e2dc-4034-b58f-ea9bafb7b379\") " pod="openshift-ingress/e2e-api-a-c5c751-openshift-default-586c478496-llxnx"
Sep 10 17:57:54.640863 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:57:54.640855    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/013e342b-e2dc-4034-b58f-ea9bafb7b379-istiod-ca-cert\") pod \"e2e-api-a-c5c751-openshift-default-586c478496-llxnx\" (UID: \"013e342b-e2dc-4034-b58f-ea9bafb7b379\") " pod="openshift-ingress/e2e-api-a-c5c751-openshift-default-586c478496-llxnx"
Sep 10 17:57:54.641199 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:57:54.640916    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/013e342b-e2dc-4034-b58f-ea9bafb7b379-workload-certs\") pod \"e2e-api-a-c5c751-openshift-default-586c478496-llxnx\" (UID: \"013e342b-e2dc-4034-b58f-ea9bafb7b379\") " pod="openshift-ingress/e2e-api-a-c5c751-openshift-default-586c478496-llxnx"
Sep 10 17:57:54.641258 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:57:54.641237    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/013e342b-e2dc-4034-b58f-ea9bafb7b379-credential-socket\") pod \"e2e-api-a-c5c751-openshift-default-586c478496-llxnx\" (UID: \"013e342b-e2dc-4034-b58f-ea9bafb7b379\") " pod="openshift-ingress/e2e-api-a-c5c751-openshift-default-586c478496-llxnx"
Sep 10 17:57:54.641312 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:57:54.641240    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/013e342b-e2dc-4034-b58f-ea9bafb7b379-istio-data\") pod \"e2e-api-a-c5c751-openshift-default-586c478496-llxnx\" (UID: \"013e342b-e2dc-4034-b58f-ea9bafb7b379\") " pod="openshift-ingress/e2e-api-a-c5c751-openshift-default-586c478496-llxnx"
Sep 10 17:57:54.641312 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:57:54.641298    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/013e342b-e2dc-4034-b58f-ea9bafb7b379-istio-token\") pod \"e2e-api-a-c5c751-openshift-default-586c478496-llxnx\" (UID: \"013e342b-e2dc-4034-b58f-ea9bafb7b379\") " pod="openshift-ingress/e2e-api-a-c5c751-openshift-default-586c478496-llxnx"
Sep 10 17:57:54.641415 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:57:54.641388    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/013e342b-e2dc-4034-b58f-ea9bafb7b379-workload-socket\") pod \"e2e-api-a-c5c751-openshift-default-586c478496-llxnx\" (UID: \"013e342b-e2dc-4034-b58f-ea9bafb7b379\") " pod="openshift-ingress/e2e-api-a-c5c751-openshift-default-586c478496-llxnx"
Sep 10 17:57:54.641480 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:57:54.641460    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/013e342b-e2dc-4034-b58f-ea9bafb7b379-workload-certs\") pod \"e2e-api-a-c5c751-openshift-default-586c478496-llxnx\" (UID: \"013e342b-e2dc-4034-b58f-ea9bafb7b379\") " pod="openshift-ingress/e2e-api-a-c5c751-openshift-default-586c478496-llxnx"
Sep 10 17:57:54.641533 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:57:54.641487    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/013e342b-e2dc-4034-b58f-ea9bafb7b379-istio-podinfo\") pod \"e2e-api-a-c5c751-openshift-default-586c478496-llxnx\" (UID: \"013e342b-e2dc-4034-b58f-ea9bafb7b379\") " pod="openshift-ingress/e2e-api-a-c5c751-openshift-default-586c478496-llxnx"
Sep 10 17:57:54.641533 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:57:54.641526    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/013e342b-e2dc-4034-b58f-ea9bafb7b379-istio-envoy\") pod \"e2e-api-a-c5c751-openshift-default-586c478496-llxnx\" (UID: \"013e342b-e2dc-4034-b58f-ea9bafb7b379\") " pod="openshift-ingress/e2e-api-a-c5c751-openshift-default-586c478496-llxnx"
Sep 10 17:57:54.641697 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:57:54.641680    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/013e342b-e2dc-4034-b58f-ea9bafb7b379-istio-data\") pod \"e2e-api-a-c5c751-openshift-default-586c478496-llxnx\" (UID: \"013e342b-e2dc-4034-b58f-ea9bafb7b379\") " pod="openshift-ingress/e2e-api-a-c5c751-openshift-default-586c478496-llxnx"
Sep 10 17:57:54.641765 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:57:54.641730    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/013e342b-e2dc-4034-b58f-ea9bafb7b379-workload-socket\") pod \"e2e-api-a-c5c751-openshift-default-586c478496-llxnx\" (UID: \"013e342b-e2dc-4034-b58f-ea9bafb7b379\") " pod="openshift-ingress/e2e-api-a-c5c751-openshift-default-586c478496-llxnx"
Sep 10 17:57:54.641817 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:57:54.641776    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/013e342b-e2dc-4034-b58f-ea9bafb7b379-istiod-ca-cert\") pod \"e2e-api-a-c5c751-openshift-default-586c478496-llxnx\" (UID: \"013e342b-e2dc-4034-b58f-ea9bafb7b379\") " pod="openshift-ingress/e2e-api-a-c5c751-openshift-default-586c478496-llxnx"
Sep 10 17:57:54.643986 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:57:54.643957    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/013e342b-e2dc-4034-b58f-ea9bafb7b379-istio-envoy\") pod \"e2e-api-a-c5c751-openshift-default-586c478496-llxnx\" (UID: \"013e342b-e2dc-4034-b58f-ea9bafb7b379\") " pod="openshift-ingress/e2e-api-a-c5c751-openshift-default-586c478496-llxnx"
Sep 10 17:57:54.644241 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:57:54.644223    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/013e342b-e2dc-4034-b58f-ea9bafb7b379-istio-podinfo\") pod \"e2e-api-a-c5c751-openshift-default-586c478496-llxnx\" (UID: \"013e342b-e2dc-4034-b58f-ea9bafb7b379\") " pod="openshift-ingress/e2e-api-a-c5c751-openshift-default-586c478496-llxnx"
Sep 10 17:57:54.652107 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:57:54.652088    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2j5r\" (UniqueName: \"kubernetes.io/projected/013e342b-e2dc-4034-b58f-ea9bafb7b379-kube-api-access-s2j5r\") pod \"e2e-api-a-c5c751-openshift-default-586c478496-llxnx\" (UID: \"013e342b-e2dc-4034-b58f-ea9bafb7b379\") " pod="openshift-ingress/e2e-api-a-c5c751-openshift-default-586c478496-llxnx"
Sep 10 17:57:54.655517 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:57:54.655497    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/013e342b-e2dc-4034-b58f-ea9bafb7b379-istio-token\") pod \"e2e-api-a-c5c751-openshift-default-586c478496-llxnx\" (UID: \"013e342b-e2dc-4034-b58f-ea9bafb7b379\") " pod="openshift-ingress/e2e-api-a-c5c751-openshift-default-586c478496-llxnx"
Sep 10 17:57:54.699491 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:57:54.699452    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-c5c751-openshift-default-586c478496-llxnx"
Sep 10 17:57:54.998321 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:57:54.998235    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-c5c751-openshift-default-586c478496-llxnx"]
Sep 10 17:57:55.005151 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:57:55.005109    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 10 17:57:55.005266 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:57:55.005200    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 10 17:57:55.005266 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:57:55.005244    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 10 17:57:55.321478 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:57:55.321394    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c5c751-openshift-default-586c478496-llxnx" event={"ID":"013e342b-e2dc-4034-b58f-ea9bafb7b379","Type":"ContainerStarted","Data":"e5013314d8a7dec634ead2722945328aedf17c68b85155aeb54ddbe4a0b950c8"}
Sep 10 17:57:55.321478 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:57:55.321461    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c5c751-openshift-default-586c478496-llxnx" event={"ID":"013e342b-e2dc-4034-b58f-ea9bafb7b379","Type":"ContainerStarted","Data":"b9cae73f1ba1d5cbfbbc113462998f006e9eb5d4adb933a2ab1c606eac19afc3"}
Sep 10 17:57:55.347005 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:57:55.346953    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-c5c751-openshift-default-586c478496-llxnx" podStartSLOduration=1.346935977 podStartE2EDuration="1.346935977s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:57:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:57:55.345216425 +0000 UTC m=+1319.345046505" watchObservedRunningTime="2026-09-10 17:57:55.346935977 +0000 UTC m=+1319.346766089"
Sep 10 17:57:55.700290 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:57:55.700254    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-c5c751-openshift-default-586c478496-llxnx"
Sep 10 17:57:56.706038 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:57:56.706005    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-c5c751-openshift-default-586c478496-llxnx"
Sep 10 17:57:57.331610 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:57:57.331577    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-c5c751-openshift-default-586c478496-llxnx"
Sep 10 17:57:57.332632 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:57:57.332613    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-c5c751-openshift-default-586c478496-llxnx"
Sep 10 17:57:58.440017 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:57:58.439981    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c5c751-7b9d859477-d4jb7"]
Sep 10 17:57:58.440510 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:57:58.440494    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="e05bf37b-74f1-445c-ad58-5464347cb2f3" containerName="istio-proxy"
Sep 10 17:57:58.445086 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:57:58.445063    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c5c751-7b9d859477-d4jb7"
Sep 10 17:57:58.447719 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:57:58.447696    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-c5c751-serving-cert\""
Sep 10 17:57:58.456382 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:57:58.456354    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c5c751-7b9d859477-d4jb7"]
Sep 10 17:57:58.480610 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:57:58.480555    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e82663d4-b257-42ee-a0b3-a28b8f3f06e0-openshift-service-ca\") pod \"maas-api-e2e-api-a-c5c751-7b9d859477-d4jb7\" (UID: \"e82663d4-b257-42ee-a0b3-a28b8f3f06e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c5c751-7b9d859477-d4jb7"
Sep 10 17:57:58.480780 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:57:58.480635    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6mng\" (UniqueName: \"kubernetes.io/projected/e82663d4-b257-42ee-a0b3-a28b8f3f06e0-kube-api-access-t6mng\") pod \"maas-api-e2e-api-a-c5c751-7b9d859477-d4jb7\" (UID: \"e82663d4-b257-42ee-a0b3-a28b8f3f06e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c5c751-7b9d859477-d4jb7"
Sep 10 17:57:58.480780 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:57:58.480657    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e82663d4-b257-42ee-a0b3-a28b8f3f06e0-maas-api-tls\") pod \"maas-api-e2e-api-a-c5c751-7b9d859477-d4jb7\" (UID: \"e82663d4-b257-42ee-a0b3-a28b8f3f06e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c5c751-7b9d859477-d4jb7"
Sep 10 17:57:58.480780 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:57:58.480700    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e82663d4-b257-42ee-a0b3-a28b8f3f06e0-metrics-certs\") pod \"maas-api-e2e-api-a-c5c751-7b9d859477-d4jb7\" (UID: \"e82663d4-b257-42ee-a0b3-a28b8f3f06e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c5c751-7b9d859477-d4jb7"
Sep 10 17:57:58.581789 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:57:58.581756    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e82663d4-b257-42ee-a0b3-a28b8f3f06e0-openshift-service-ca\") pod \"maas-api-e2e-api-a-c5c751-7b9d859477-d4jb7\" (UID: \"e82663d4-b257-42ee-a0b3-a28b8f3f06e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c5c751-7b9d859477-d4jb7"
Sep 10 17:57:58.581984 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:57:58.581799    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6mng\" (UniqueName: \"kubernetes.io/projected/e82663d4-b257-42ee-a0b3-a28b8f3f06e0-kube-api-access-t6mng\") pod \"maas-api-e2e-api-a-c5c751-7b9d859477-d4jb7\" (UID: \"e82663d4-b257-42ee-a0b3-a28b8f3f06e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c5c751-7b9d859477-d4jb7"
Sep 10 17:57:58.581984 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:57:58.581854    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e82663d4-b257-42ee-a0b3-a28b8f3f06e0-maas-api-tls\") pod \"maas-api-e2e-api-a-c5c751-7b9d859477-d4jb7\" (UID: \"e82663d4-b257-42ee-a0b3-a28b8f3f06e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c5c751-7b9d859477-d4jb7"
Sep 10 17:57:58.581984 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:57:58.581897    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e82663d4-b257-42ee-a0b3-a28b8f3f06e0-metrics-certs\") pod \"maas-api-e2e-api-a-c5c751-7b9d859477-d4jb7\" (UID: \"e82663d4-b257-42ee-a0b3-a28b8f3f06e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c5c751-7b9d859477-d4jb7"
Sep 10 17:57:58.582699 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:57:58.582679    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e82663d4-b257-42ee-a0b3-a28b8f3f06e0-openshift-service-ca\") pod \"maas-api-e2e-api-a-c5c751-7b9d859477-d4jb7\" (UID: \"e82663d4-b257-42ee-a0b3-a28b8f3f06e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c5c751-7b9d859477-d4jb7"
Sep 10 17:57:58.584451 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:57:58.584433    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e82663d4-b257-42ee-a0b3-a28b8f3f06e0-metrics-certs\") pod \"maas-api-e2e-api-a-c5c751-7b9d859477-d4jb7\" (UID: \"e82663d4-b257-42ee-a0b3-a28b8f3f06e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c5c751-7b9d859477-d4jb7"
Sep 10 17:57:58.584539 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:57:58.584526    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e82663d4-b257-42ee-a0b3-a28b8f3f06e0-maas-api-tls\") pod \"maas-api-e2e-api-a-c5c751-7b9d859477-d4jb7\" (UID: \"e82663d4-b257-42ee-a0b3-a28b8f3f06e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c5c751-7b9d859477-d4jb7"
Sep 10 17:57:58.595429 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:57:58.595402    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6mng\" (UniqueName: \"kubernetes.io/projected/e82663d4-b257-42ee-a0b3-a28b8f3f06e0-kube-api-access-t6mng\") pod \"maas-api-e2e-api-a-c5c751-7b9d859477-d4jb7\" (UID: \"e82663d4-b257-42ee-a0b3-a28b8f3f06e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c5c751-7b9d859477-d4jb7"
Sep 10 17:57:58.758636 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:57:58.758532    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c5c751-7b9d859477-d4jb7"
Sep 10 17:57:59.035646 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:57:59.035620    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c5c751-7b9d859477-d4jb7"]
Sep 10 17:57:59.036988 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:57:59.036964    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode82663d4_b257_42ee_a0b3_a28b8f3f06e0.slice/crio-397c9c67f5f0cb747bc18472bfdebc101fea17275bb604064fa6f7c2dc2b8aac WatchSource:0}: Error finding container 397c9c67f5f0cb747bc18472bfdebc101fea17275bb604064fa6f7c2dc2b8aac: Status 404 returned error can't find the container with id 397c9c67f5f0cb747bc18472bfdebc101fea17275bb604064fa6f7c2dc2b8aac
Sep 10 17:57:59.215302 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:57:59.215266    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-c5c751-75d6b7dfcd-5bhm6"]
Sep 10 17:57:59.219991 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:57:59.219969    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c5c751-75d6b7dfcd-5bhm6"
Sep 10 17:57:59.224045 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:57:59.224024    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-c5c751\""
Sep 10 17:57:59.224160 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:57:59.224080    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-c5c751-dockercfg-8gbn8\""
Sep 10 17:57:59.251299 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:57:59.251269    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-c5c751-75d6b7dfcd-5bhm6"]
Sep 10 17:57:59.288096 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:57:59.288002    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bh9bz\" (UniqueName: \"kubernetes.io/projected/87ba0e8f-9ab6-45be-b6ec-531f2c60b4a8-kube-api-access-bh9bz\") pod \"payload-pre-processing-e2e-api-a-c5c751-75d6b7dfcd-5bhm6\" (UID: \"87ba0e8f-9ab6-45be-b6ec-531f2c60b4a8\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-c5c751-75d6b7dfcd-5bhm6"
Sep 10 17:57:59.288096 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:57:59.288058    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87ba0e8f-9ab6-45be-b6ec-531f2c60b4a8-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-c5c751-75d6b7dfcd-5bhm6\" (UID: \"87ba0e8f-9ab6-45be-b6ec-531f2c60b4a8\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-c5c751-75d6b7dfcd-5bhm6"
Sep 10 17:57:59.344975 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:57:59.344939    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c5c751-7b9d859477-d4jb7" event={"ID":"e82663d4-b257-42ee-a0b3-a28b8f3f06e0","Type":"ContainerStarted","Data":"397c9c67f5f0cb747bc18472bfdebc101fea17275bb604064fa6f7c2dc2b8aac"}
Sep 10 17:57:59.389054 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:57:59.389017    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bh9bz\" (UniqueName: \"kubernetes.io/projected/87ba0e8f-9ab6-45be-b6ec-531f2c60b4a8-kube-api-access-bh9bz\") pod \"payload-pre-processing-e2e-api-a-c5c751-75d6b7dfcd-5bhm6\" (UID: \"87ba0e8f-9ab6-45be-b6ec-531f2c60b4a8\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-c5c751-75d6b7dfcd-5bhm6"
Sep 10 17:57:59.389276 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:57:59.389119    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87ba0e8f-9ab6-45be-b6ec-531f2c60b4a8-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-c5c751-75d6b7dfcd-5bhm6\" (UID: \"87ba0e8f-9ab6-45be-b6ec-531f2c60b4a8\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-c5c751-75d6b7dfcd-5bhm6"
Sep 10 17:57:59.389927 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:57:59.389860    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87ba0e8f-9ab6-45be-b6ec-531f2c60b4a8-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-c5c751-75d6b7dfcd-5bhm6\" (UID: \"87ba0e8f-9ab6-45be-b6ec-531f2c60b4a8\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-c5c751-75d6b7dfcd-5bhm6"
Sep 10 17:57:59.402914 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:57:59.402888    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bh9bz\" (UniqueName: \"kubernetes.io/projected/87ba0e8f-9ab6-45be-b6ec-531f2c60b4a8-kube-api-access-bh9bz\") pod \"payload-pre-processing-e2e-api-a-c5c751-75d6b7dfcd-5bhm6\" (UID: \"87ba0e8f-9ab6-45be-b6ec-531f2c60b4a8\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-c5c751-75d6b7dfcd-5bhm6"
Sep 10 17:57:59.530810 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:57:59.530782    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c5c751-75d6b7dfcd-5bhm6"
Sep 10 17:57:59.826132 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:57:59.826105    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-c5c751-75d6b7dfcd-5bhm6"]
Sep 10 17:57:59.828055 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:57:59.828025    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87ba0e8f_9ab6_45be_b6ec_531f2c60b4a8.slice/crio-da607699c41bc0af59c6d2276b3fc1995a8ffebc570b8ac97b775324be595ffe WatchSource:0}: Error finding container da607699c41bc0af59c6d2276b3fc1995a8ffebc570b8ac97b775324be595ffe: Status 404 returned error can't find the container with id da607699c41bc0af59c6d2276b3fc1995a8ffebc570b8ac97b775324be595ffe
Sep 10 17:58:00.351088 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:58:00.351054    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c5c751-7b9d859477-d4jb7" event={"ID":"e82663d4-b257-42ee-a0b3-a28b8f3f06e0","Type":"ContainerStarted","Data":"1983c2edd249e06d8a66c9336c862123a3415d80c0b2bc188fd1a94a0e3f9a03"}
Sep 10 17:58:00.351311 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:58:00.351166    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c5c751-7b9d859477-d4jb7"
Sep 10 17:58:00.352673 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:58:00.352650    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c5c751-75d6b7dfcd-5bhm6" event={"ID":"87ba0e8f-9ab6-45be-b6ec-531f2c60b4a8","Type":"ContainerStarted","Data":"17318ab7581920abacdb128e1c60e3986e2ce58463857c918e85a67244ab74d8"}
Sep 10 17:58:00.352777 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:58:00.352693    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c5c751-75d6b7dfcd-5bhm6"
Sep 10 17:58:00.352777 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:58:00.352709    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c5c751-75d6b7dfcd-5bhm6" event={"ID":"87ba0e8f-9ab6-45be-b6ec-531f2c60b4a8","Type":"ContainerStarted","Data":"da607699c41bc0af59c6d2276b3fc1995a8ffebc570b8ac97b775324be595ffe"}
Sep 10 17:58:00.379585 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:58:00.379527    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c5c751-7b9d859477-d4jb7" podStartSLOduration=1.956982387 podStartE2EDuration="2.379512335s" totalImagesPullingTime="422.529948ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:57:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:58:00.377477915 +0000 UTC m=+1324.377307982" watchObservedRunningTime="2026-09-10 17:58:00.379512335 +0000 UTC m=+1324.379342434"
Sep 10 17:58:00.402899 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:58:00.402844    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c5c751-75d6b7dfcd-5bhm6" podStartSLOduration=1.402828697 podStartE2EDuration="1.402828697s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:57:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:58:00.400864882 +0000 UTC m=+1324.400694950" watchObservedRunningTime="2026-09-10 17:58:00.402828697 +0000 UTC m=+1324.402658764"
Sep 10 17:58:06.364005 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:58:06.363975    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c5c751-7b9d859477-d4jb7"
Sep 10 17:58:11.361180 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:58:11.361151    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c5c751-75d6b7dfcd-5bhm6"
Sep 10 17:58:13.373775 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:58:13.373729    2579 patch_prober.go:28] interesting pod/e2e-api-a-c5c751-openshift-default-586c478496-llxnx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 10 17:58:13.374188 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:58:13.373812    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-c5c751-openshift-default-586c478496-llxnx" podUID="013e342b-e2dc-4034-b58f-ea9bafb7b379" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 10 17:58:15.445668 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:58:15.445633    2579 generic.go:356] "Generic (PLEG): container finished" podID="013e342b-e2dc-4034-b58f-ea9bafb7b379" containerID="e5013314d8a7dec634ead2722945328aedf17c68b85155aeb54ddbe4a0b950c8" exitCode=0
Sep 10 17:58:15.446077 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:58:15.445672    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c5c751-openshift-default-586c478496-llxnx" event={"ID":"013e342b-e2dc-4034-b58f-ea9bafb7b379","Type":"ContainerDied","Data":"e5013314d8a7dec634ead2722945328aedf17c68b85155aeb54ddbe4a0b950c8"}
Sep 10 17:58:15.446077 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:58:15.446016    2579 scope.go:117] "RemoveContainer" containerID="e5013314d8a7dec634ead2722945328aedf17c68b85155aeb54ddbe4a0b950c8"
Sep 10 17:58:15.447132 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:58:15.447105    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 10 17:58:15.447189 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:58:15.447175    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 10 17:58:15.447229 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:58:15.447211    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 10 17:58:16.453588 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:58:16.453542    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c5c751-openshift-default-586c478496-llxnx" event={"ID":"013e342b-e2dc-4034-b58f-ea9bafb7b379","Type":"ContainerStarted","Data":"c0f81fd56a99c0f9d2b8a1c9c8e46a92df86d9f269abfae60be5c58961833800"}
Sep 10 17:58:16.700643 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:58:16.700598    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-c5c751-openshift-default-586c478496-llxnx"
Sep 10 17:58:16.700842 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:58:16.700817    2579 patch_prober.go:28] interesting pod/e2e-api-a-c5c751-openshift-default-586c478496-llxnx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body=
Sep 10 17:58:16.700900 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:58:16.700881    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c5c751-openshift-default-586c478496-llxnx" podUID="013e342b-e2dc-4034-b58f-ea9bafb7b379" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused"
Sep 10 17:58:17.700489 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:58:17.700403    2579 patch_prober.go:28] interesting pod/e2e-api-a-c5c751-openshift-default-586c478496-llxnx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body=
Sep 10 17:58:17.700489 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:58:17.700467    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c5c751-openshift-default-586c478496-llxnx" podUID="013e342b-e2dc-4034-b58f-ea9bafb7b379" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused"
Sep 10 17:58:18.700582 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:58:18.700526    2579 patch_prober.go:28] interesting pod/e2e-api-a-c5c751-openshift-default-586c478496-llxnx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body=
Sep 10 17:58:18.700952 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:58:18.700614    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c5c751-openshift-default-586c478496-llxnx" podUID="013e342b-e2dc-4034-b58f-ea9bafb7b379" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused"
Sep 10 17:58:19.700212 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:58:19.700171    2579 patch_prober.go:28] interesting pod/e2e-api-a-c5c751-openshift-default-586c478496-llxnx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body=
Sep 10 17:58:19.700370 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:58:19.700244    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c5c751-openshift-default-586c478496-llxnx" podUID="013e342b-e2dc-4034-b58f-ea9bafb7b379" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused"
Sep 10 17:58:20.700301 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:58:20.700258    2579 patch_prober.go:28] interesting pod/e2e-api-a-c5c751-openshift-default-586c478496-llxnx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body=
Sep 10 17:58:20.700727 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:58:20.700316    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c5c751-openshift-default-586c478496-llxnx" podUID="013e342b-e2dc-4034-b58f-ea9bafb7b379" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused"
Sep 10 17:58:21.791038 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:58:21.791006    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-c5c751-openshift-default-586c478496-llxnx"
Sep 10 17:58:21.791502 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:58:21.791416    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-c5c751-openshift-default-586c478496-llxnx"
Sep 10 17:58:21.792218 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:58:21.792189    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-c5c751-openshift-default-586c478496-llxnx"
Sep 10 17:58:23.114201 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:58:23.114168    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-abad1a-openshift-default-7969487574-l69b4"]
Sep 10 17:58:23.117193 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:58:23.117172    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-abad1a-openshift-default-7969487574-l69b4"
Sep 10 17:58:23.120065 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:58:23.120046    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-abad1a-openshift-default-dockercfg-4n74w\""
Sep 10 17:58:23.140376 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:58:23.140353    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-abad1a-openshift-default-7969487574-l69b4"]
Sep 10 17:58:23.204892 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:58:23.204854    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b252299-8b3e-4fd7-986e-99f19feae0e3-istiod-ca-cert\") pod \"e2e-api-b-abad1a-openshift-default-7969487574-l69b4\" (UID: \"6b252299-8b3e-4fd7-986e-99f19feae0e3\") " pod="openshift-ingress/e2e-api-b-abad1a-openshift-default-7969487574-l69b4"
Sep 10 17:58:23.204892 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:58:23.204890    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b252299-8b3e-4fd7-986e-99f19feae0e3-istio-data\") pod \"e2e-api-b-abad1a-openshift-default-7969487574-l69b4\" (UID: \"6b252299-8b3e-4fd7-986e-99f19feae0e3\") " pod="openshift-ingress/e2e-api-b-abad1a-openshift-default-7969487574-l69b4"
Sep 10 17:58:23.205146 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:58:23.204991    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b252299-8b3e-4fd7-986e-99f19feae0e3-workload-socket\") pod \"e2e-api-b-abad1a-openshift-default-7969487574-l69b4\" (UID: \"6b252299-8b3e-4fd7-986e-99f19feae0e3\") " pod="openshift-ingress/e2e-api-b-abad1a-openshift-default-7969487574-l69b4"
Sep 10 17:58:23.205146 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:58:23.205022    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b252299-8b3e-4fd7-986e-99f19feae0e3-istio-token\") pod \"e2e-api-b-abad1a-openshift-default-7969487574-l69b4\" (UID: \"6b252299-8b3e-4fd7-986e-99f19feae0e3\") " pod="openshift-ingress/e2e-api-b-abad1a-openshift-default-7969487574-l69b4"
Sep 10 17:58:23.205146 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:58:23.205052    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b252299-8b3e-4fd7-986e-99f19feae0e3-istio-podinfo\") pod \"e2e-api-b-abad1a-openshift-default-7969487574-l69b4\" (UID: \"6b252299-8b3e-4fd7-986e-99f19feae0e3\") " pod="openshift-ingress/e2e-api-b-abad1a-openshift-default-7969487574-l69b4"
Sep 10 17:58:23.205146 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:58:23.205084    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b252299-8b3e-4fd7-986e-99f19feae0e3-workload-certs\") pod \"e2e-api-b-abad1a-openshift-default-7969487574-l69b4\" (UID: \"6b252299-8b3e-4fd7-986e-99f19feae0e3\") " pod="openshift-ingress/e2e-api-b-abad1a-openshift-default-7969487574-l69b4"
Sep 10 17:58:23.205146 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:58:23.205107    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhbft\" (UniqueName: \"kubernetes.io/projected/6b252299-8b3e-4fd7-986e-99f19feae0e3-kube-api-access-lhbft\") pod \"e2e-api-b-abad1a-openshift-default-7969487574-l69b4\" (UID: \"6b252299-8b3e-4fd7-986e-99f19feae0e3\") " pod="openshift-ingress/e2e-api-b-abad1a-openshift-default-7969487574-l69b4"
Sep 10 17:58:23.205146 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:58:23.205126    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b252299-8b3e-4fd7-986e-99f19feae0e3-istio-envoy\") pod \"e2e-api-b-abad1a-openshift-default-7969487574-l69b4\" (UID: \"6b252299-8b3e-4fd7-986e-99f19feae0e3\") " pod="openshift-ingress/e2e-api-b-abad1a-openshift-default-7969487574-l69b4"
Sep 10 17:58:23.205369 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:58:23.205190    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b252299-8b3e-4fd7-986e-99f19feae0e3-credential-socket\") pod \"e2e-api-b-abad1a-openshift-default-7969487574-l69b4\" (UID: \"6b252299-8b3e-4fd7-986e-99f19feae0e3\") " pod="openshift-ingress/e2e-api-b-abad1a-openshift-default-7969487574-l69b4"
Sep 10 17:58:23.306078 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:58:23.306034    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b252299-8b3e-4fd7-986e-99f19feae0e3-workload-socket\") pod \"e2e-api-b-abad1a-openshift-default-7969487574-l69b4\" (UID: \"6b252299-8b3e-4fd7-986e-99f19feae0e3\") " pod="openshift-ingress/e2e-api-b-abad1a-openshift-default-7969487574-l69b4"
Sep 10 17:58:23.306078 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:58:23.306068    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b252299-8b3e-4fd7-986e-99f19feae0e3-istio-token\") pod \"e2e-api-b-abad1a-openshift-default-7969487574-l69b4\" (UID: \"6b252299-8b3e-4fd7-986e-99f19feae0e3\") " pod="openshift-ingress/e2e-api-b-abad1a-openshift-default-7969487574-l69b4"
Sep 10 17:58:23.306332 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:58:23.306096    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b252299-8b3e-4fd7-986e-99f19feae0e3-istio-podinfo\") pod \"e2e-api-b-abad1a-openshift-default-7969487574-l69b4\" (UID: \"6b252299-8b3e-4fd7-986e-99f19feae0e3\") " pod="openshift-ingress/e2e-api-b-abad1a-openshift-default-7969487574-l69b4"
Sep 10 17:58:23.306332 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:58:23.306123    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b252299-8b3e-4fd7-986e-99f19feae0e3-workload-certs\") pod \"e2e-api-b-abad1a-openshift-default-7969487574-l69b4\" (UID: \"6b252299-8b3e-4fd7-986e-99f19feae0e3\") " pod="openshift-ingress/e2e-api-b-abad1a-openshift-default-7969487574-l69b4"
Sep 10 17:58:23.306332 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:58:23.306156    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhbft\" (UniqueName: \"kubernetes.io/projected/6b252299-8b3e-4fd7-986e-99f19feae0e3-kube-api-access-lhbft\") pod \"e2e-api-b-abad1a-openshift-default-7969487574-l69b4\" (UID: \"6b252299-8b3e-4fd7-986e-99f19feae0e3\") " pod="openshift-ingress/e2e-api-b-abad1a-openshift-default-7969487574-l69b4"
Sep 10 17:58:23.306332 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:58:23.306183    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b252299-8b3e-4fd7-986e-99f19feae0e3-istio-envoy\") pod \"e2e-api-b-abad1a-openshift-default-7969487574-l69b4\" (UID: \"6b252299-8b3e-4fd7-986e-99f19feae0e3\") " pod="openshift-ingress/e2e-api-b-abad1a-openshift-default-7969487574-l69b4"
Sep 10 17:58:23.306332 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:58:23.306214    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b252299-8b3e-4fd7-986e-99f19feae0e3-credential-socket\") pod \"e2e-api-b-abad1a-openshift-default-7969487574-l69b4\" (UID: \"6b252299-8b3e-4fd7-986e-99f19feae0e3\") " pod="openshift-ingress/e2e-api-b-abad1a-openshift-default-7969487574-l69b4"
Sep 10 17:58:23.306332 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:58:23.306299    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b252299-8b3e-4fd7-986e-99f19feae0e3-istiod-ca-cert\") pod \"e2e-api-b-abad1a-openshift-default-7969487574-l69b4\" (UID: \"6b252299-8b3e-4fd7-986e-99f19feae0e3\") " pod="openshift-ingress/e2e-api-b-abad1a-openshift-default-7969487574-l69b4"
Sep 10 17:58:23.306332 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:58:23.306325    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b252299-8b3e-4fd7-986e-99f19feae0e3-istio-data\") pod \"e2e-api-b-abad1a-openshift-default-7969487574-l69b4\" (UID: \"6b252299-8b3e-4fd7-986e-99f19feae0e3\") " pod="openshift-ingress/e2e-api-b-abad1a-openshift-default-7969487574-l69b4"
Sep 10 17:58:23.306758 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:58:23.306482    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b252299-8b3e-4fd7-986e-99f19feae0e3-workload-socket\") pod \"e2e-api-b-abad1a-openshift-default-7969487574-l69b4\" (UID: \"6b252299-8b3e-4fd7-986e-99f19feae0e3\") " pod="openshift-ingress/e2e-api-b-abad1a-openshift-default-7969487574-l69b4"
Sep 10 17:58:23.306758 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:58:23.306504    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b252299-8b3e-4fd7-986e-99f19feae0e3-workload-certs\") pod \"e2e-api-b-abad1a-openshift-default-7969487574-l69b4\" (UID: \"6b252299-8b3e-4fd7-986e-99f19feae0e3\") " pod="openshift-ingress/e2e-api-b-abad1a-openshift-default-7969487574-l69b4"
Sep 10 17:58:23.306758 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:58:23.306739    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b252299-8b3e-4fd7-986e-99f19feae0e3-credential-socket\") pod \"e2e-api-b-abad1a-openshift-default-7969487574-l69b4\" (UID: \"6b252299-8b3e-4fd7-986e-99f19feae0e3\") " pod="openshift-ingress/e2e-api-b-abad1a-openshift-default-7969487574-l69b4"
Sep 10 17:58:23.306758 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:58:23.306748    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b252299-8b3e-4fd7-986e-99f19feae0e3-istio-data\") pod \"e2e-api-b-abad1a-openshift-default-7969487574-l69b4\" (UID: \"6b252299-8b3e-4fd7-986e-99f19feae0e3\") " pod="openshift-ingress/e2e-api-b-abad1a-openshift-default-7969487574-l69b4"
Sep 10 17:58:23.307391 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:58:23.307368    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b252299-8b3e-4fd7-986e-99f19feae0e3-istiod-ca-cert\") pod \"e2e-api-b-abad1a-openshift-default-7969487574-l69b4\" (UID: \"6b252299-8b3e-4fd7-986e-99f19feae0e3\") " pod="openshift-ingress/e2e-api-b-abad1a-openshift-default-7969487574-l69b4"
Sep 10 17:58:23.308808 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:58:23.308786    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b252299-8b3e-4fd7-986e-99f19feae0e3-istio-envoy\") pod \"e2e-api-b-abad1a-openshift-default-7969487574-l69b4\" (UID: \"6b252299-8b3e-4fd7-986e-99f19feae0e3\") " pod="openshift-ingress/e2e-api-b-abad1a-openshift-default-7969487574-l69b4"
Sep 10 17:58:23.308914 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:58:23.308895    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b252299-8b3e-4fd7-986e-99f19feae0e3-istio-podinfo\") pod \"e2e-api-b-abad1a-openshift-default-7969487574-l69b4\" (UID: \"6b252299-8b3e-4fd7-986e-99f19feae0e3\") " pod="openshift-ingress/e2e-api-b-abad1a-openshift-default-7969487574-l69b4"
Sep 10 17:58:23.322763 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:58:23.322736    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b252299-8b3e-4fd7-986e-99f19feae0e3-istio-token\") pod \"e2e-api-b-abad1a-openshift-default-7969487574-l69b4\" (UID: \"6b252299-8b3e-4fd7-986e-99f19feae0e3\") " pod="openshift-ingress/e2e-api-b-abad1a-openshift-default-7969487574-l69b4"
Sep 10 17:58:23.323551 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:58:23.323529    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhbft\" (UniqueName: \"kubernetes.io/projected/6b252299-8b3e-4fd7-986e-99f19feae0e3-kube-api-access-lhbft\") pod \"e2e-api-b-abad1a-openshift-default-7969487574-l69b4\" (UID: \"6b252299-8b3e-4fd7-986e-99f19feae0e3\") " pod="openshift-ingress/e2e-api-b-abad1a-openshift-default-7969487574-l69b4"
Sep 10 17:58:23.427907 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:58:23.427879    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-abad1a-openshift-default-7969487574-l69b4"
Sep 10 17:58:23.713199 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:58:23.713107    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-abad1a-openshift-default-7969487574-l69b4"]
Sep 10 17:58:23.717607 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:58:23.717558    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b252299_8b3e_4fd7_986e_99f19feae0e3.slice/crio-2c074e93b961cccb55c870539b55712d7d33810652897d468c81249335aa1195 WatchSource:0}: Error finding container 2c074e93b961cccb55c870539b55712d7d33810652897d468c81249335aa1195: Status 404 returned error can't find the container with id 2c074e93b961cccb55c870539b55712d7d33810652897d468c81249335aa1195
Sep 10 17:58:23.719951 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:58:23.719915    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 10 17:58:23.720028 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:58:23.719981    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 10 17:58:23.720028 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:58:23.720008    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 10 17:58:24.516145 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:58:24.516110    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-abad1a-openshift-default-7969487574-l69b4" event={"ID":"6b252299-8b3e-4fd7-986e-99f19feae0e3","Type":"ContainerStarted","Data":"15dd723eed76fa41f52e17247186aaa28000d84d0d7aa4e023f0aff341aec39a"}
Sep 10 17:58:24.516500 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:58:24.516172    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-abad1a-openshift-default-7969487574-l69b4" event={"ID":"6b252299-8b3e-4fd7-986e-99f19feae0e3","Type":"ContainerStarted","Data":"2c074e93b961cccb55c870539b55712d7d33810652897d468c81249335aa1195"}
Sep 10 17:58:24.539640 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:58:24.539588    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-abad1a-openshift-default-7969487574-l69b4" podStartSLOduration=1.5395521639999998 podStartE2EDuration="1.539552164s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:58:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:58:24.537050958 +0000 UTC m=+1348.536881028" watchObservedRunningTime="2026-09-10 17:58:24.539552164 +0000 UTC m=+1348.539382231"
Sep 10 17:58:25.428682 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:58:25.428644    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-abad1a-openshift-default-7969487574-l69b4"
Sep 10 17:58:25.433306 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:58:25.433283    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-abad1a-openshift-default-7969487574-l69b4"
Sep 10 17:58:25.521459 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:58:25.521429    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-abad1a-openshift-default-7969487574-l69b4"
Sep 10 17:58:25.522411 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:58:25.522393    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-abad1a-openshift-default-7969487574-l69b4"
Sep 10 17:58:28.601537 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:58:28.601503    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-abad1a-6c745c8b5-5tjzp"]
Sep 10 17:58:28.604424 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:58:28.604404    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-abad1a-6c745c8b5-5tjzp"
Sep 10 17:58:28.608023 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:58:28.608002    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-abad1a-dockercfg-jwwcv\""
Sep 10 17:58:28.608124 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:58:28.608105    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-abad1a\""
Sep 10 17:58:28.624950 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:58:28.624928    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-abad1a-6c745c8b5-5tjzp"]
Sep 10 17:58:28.654961 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:58:28.654932    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2560bb66-7ee8-4285-8d26-913a26f10099-plugins-config-volume\") pod \"payload-processing-e2e-api-b-abad1a-6c745c8b5-5tjzp\" (UID: \"2560bb66-7ee8-4285-8d26-913a26f10099\") " pod="openshift-ingress/payload-processing-e2e-api-b-abad1a-6c745c8b5-5tjzp"
Sep 10 17:58:28.655079 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:58:28.655042    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jspfm\" (UniqueName: \"kubernetes.io/projected/2560bb66-7ee8-4285-8d26-913a26f10099-kube-api-access-jspfm\") pod \"payload-processing-e2e-api-b-abad1a-6c745c8b5-5tjzp\" (UID: \"2560bb66-7ee8-4285-8d26-913a26f10099\") " pod="openshift-ingress/payload-processing-e2e-api-b-abad1a-6c745c8b5-5tjzp"
Sep 10 17:58:28.756608 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:58:28.756551    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2560bb66-7ee8-4285-8d26-913a26f10099-plugins-config-volume\") pod \"payload-processing-e2e-api-b-abad1a-6c745c8b5-5tjzp\" (UID: \"2560bb66-7ee8-4285-8d26-913a26f10099\") " pod="openshift-ingress/payload-processing-e2e-api-b-abad1a-6c745c8b5-5tjzp"
Sep 10 17:58:28.756780 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:58:28.756645    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jspfm\" (UniqueName: \"kubernetes.io/projected/2560bb66-7ee8-4285-8d26-913a26f10099-kube-api-access-jspfm\") pod \"payload-processing-e2e-api-b-abad1a-6c745c8b5-5tjzp\" (UID: \"2560bb66-7ee8-4285-8d26-913a26f10099\") " pod="openshift-ingress/payload-processing-e2e-api-b-abad1a-6c745c8b5-5tjzp"
Sep 10 17:58:28.757256 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:58:28.757236    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2560bb66-7ee8-4285-8d26-913a26f10099-plugins-config-volume\") pod \"payload-processing-e2e-api-b-abad1a-6c745c8b5-5tjzp\" (UID: \"2560bb66-7ee8-4285-8d26-913a26f10099\") " pod="openshift-ingress/payload-processing-e2e-api-b-abad1a-6c745c8b5-5tjzp"
Sep 10 17:58:28.765464 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:58:28.765438    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jspfm\" (UniqueName: \"kubernetes.io/projected/2560bb66-7ee8-4285-8d26-913a26f10099-kube-api-access-jspfm\") pod \"payload-processing-e2e-api-b-abad1a-6c745c8b5-5tjzp\" (UID: \"2560bb66-7ee8-4285-8d26-913a26f10099\") " pod="openshift-ingress/payload-processing-e2e-api-b-abad1a-6c745c8b5-5tjzp"
Sep 10 17:58:28.915940 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:58:28.915909    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-abad1a-6c745c8b5-5tjzp"
Sep 10 17:58:29.187182 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:58:29.187156    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-abad1a-6c745c8b5-5tjzp"]
Sep 10 17:58:29.188681 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:58:29.188654    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2560bb66_7ee8_4285_8d26_913a26f10099.slice/crio-3293b925bc4d7e456ae3e15a597773d49e15a725b1f43228bd16df72be81466f WatchSource:0}: Error finding container 3293b925bc4d7e456ae3e15a597773d49e15a725b1f43228bd16df72be81466f: Status 404 returned error can't find the container with id 3293b925bc4d7e456ae3e15a597773d49e15a725b1f43228bd16df72be81466f
Sep 10 17:58:29.546885 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:58:29.546787    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-abad1a-6c745c8b5-5tjzp" event={"ID":"2560bb66-7ee8-4285-8d26-913a26f10099","Type":"ContainerStarted","Data":"b236d0de0b9be08ab08db6378ec3201364afb6181d7110a9d1737ce55ed2c696"}
Sep 10 17:58:29.547053 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:58:29.546954    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-abad1a-6c745c8b5-5tjzp"
Sep 10 17:58:29.547053 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:58:29.546979    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-abad1a-6c745c8b5-5tjzp" event={"ID":"2560bb66-7ee8-4285-8d26-913a26f10099","Type":"ContainerStarted","Data":"3293b925bc4d7e456ae3e15a597773d49e15a725b1f43228bd16df72be81466f"}
Sep 10 17:58:29.564634 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:58:29.564585    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-abad1a-6c745c8b5-5tjzp" podStartSLOduration=1.5645523909999999 podStartE2EDuration="1.564552391s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:58:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:58:29.562994269 +0000 UTC m=+1353.562824335" watchObservedRunningTime="2026-09-10 17:58:29.564552391 +0000 UTC m=+1353.564382458"
Sep 10 17:58:40.554153 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:58:40.554125    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-abad1a-6c745c8b5-5tjzp"
Sep 10 17:58:48.646819 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:58:48.646782    2579 generic.go:356] "Generic (PLEG): container finished" podID="6b252299-8b3e-4fd7-986e-99f19feae0e3" containerID="15dd723eed76fa41f52e17247186aaa28000d84d0d7aa4e023f0aff341aec39a" exitCode=0
Sep 10 17:58:48.647290 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:58:48.646852    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-abad1a-openshift-default-7969487574-l69b4" event={"ID":"6b252299-8b3e-4fd7-986e-99f19feae0e3","Type":"ContainerDied","Data":"15dd723eed76fa41f52e17247186aaa28000d84d0d7aa4e023f0aff341aec39a"}
Sep 10 17:58:48.647364 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:58:48.647293    2579 scope.go:117] "RemoveContainer" containerID="15dd723eed76fa41f52e17247186aaa28000d84d0d7aa4e023f0aff341aec39a"
Sep 10 17:58:48.648618 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:58:48.648583    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 10 17:58:48.648701 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:58:48.648676    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 10 17:58:48.648765 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:58:48.648747    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 10 17:58:49.653644 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:58:49.653602    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-abad1a-openshift-default-7969487574-l69b4" event={"ID":"6b252299-8b3e-4fd7-986e-99f19feae0e3","Type":"ContainerStarted","Data":"72f73be54d9b32da69de161708190152cb9d81380b2726cb4c51522a88a7b3a6"}
Sep 10 17:58:50.429071 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:58:50.429034    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-abad1a-openshift-default-7969487574-l69b4"
Sep 10 17:58:50.429246 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:58:50.429193    2579 patch_prober.go:28] interesting pod/e2e-api-b-abad1a-openshift-default-7969487574-l69b4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body=
Sep 10 17:58:50.429291 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:58:50.429238    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-abad1a-openshift-default-7969487574-l69b4" podUID="6b252299-8b3e-4fd7-986e-99f19feae0e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused"
Sep 10 17:58:51.429040 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:58:51.429003    2579 patch_prober.go:28] interesting pod/e2e-api-b-abad1a-openshift-default-7969487574-l69b4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body=
Sep 10 17:58:51.429423 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:58:51.429065    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-abad1a-openshift-default-7969487574-l69b4" podUID="6b252299-8b3e-4fd7-986e-99f19feae0e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused"
Sep 10 17:58:52.428914 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:58:52.428878    2579 patch_prober.go:28] interesting pod/e2e-api-b-abad1a-openshift-default-7969487574-l69b4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body=
Sep 10 17:58:52.429097 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:58:52.428930    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-abad1a-openshift-default-7969487574-l69b4" podUID="6b252299-8b3e-4fd7-986e-99f19feae0e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused"
Sep 10 17:58:53.428784 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:58:53.428753    2579 patch_prober.go:28] interesting pod/e2e-api-b-abad1a-openshift-default-7969487574-l69b4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body=
Sep 10 17:58:53.429027 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:58:53.428997    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-abad1a-openshift-default-7969487574-l69b4" podUID="6b252299-8b3e-4fd7-986e-99f19feae0e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused"
Sep 10 17:58:54.428634 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:58:54.428591    2579 patch_prober.go:28] interesting pod/e2e-api-b-abad1a-openshift-default-7969487574-l69b4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body=
Sep 10 17:58:54.429003 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:58:54.428654    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-abad1a-openshift-default-7969487574-l69b4" podUID="6b252299-8b3e-4fd7-986e-99f19feae0e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused"
Sep 10 17:58:55.433208 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:58:55.433177    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-abad1a-openshift-default-7969487574-l69b4"
Sep 10 17:58:55.433579 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:58:55.433493    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-abad1a-openshift-default-7969487574-l69b4"
Sep 10 17:58:55.434052 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:58:55.434037    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-abad1a-openshift-default-7969487574-l69b4"
Sep 10 17:58:56.667425 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:58:56.667397    2579 scope.go:117] "RemoveContainer" containerID="b7695b1ab73b358730d4865f8fea450ed17274de1ada9c680589cf5cfa306cce"
Sep 10 17:59:09.901343 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:09.901313    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c5c751-7b9d859477-d4jb7"]
Sep 10 17:59:09.901843 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:09.901514    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c5c751-7b9d859477-d4jb7" podUID="e82663d4-b257-42ee-a0b3-a28b8f3f06e0" containerName="maas-api" containerID="cri-o://1983c2edd249e06d8a66c9336c862123a3415d80c0b2bc188fd1a94a0e3f9a03" gracePeriod=30
Sep 10 17:59:09.992415 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:09.992379    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-c5c751-75d6b7dfcd-5bhm6"]
Sep 10 17:59:09.992643 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:09.992602    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c5c751-75d6b7dfcd-5bhm6" podUID="87ba0e8f-9ab6-45be-b6ec-531f2c60b4a8" containerName="payload-pre-processing" containerID="cri-o://17318ab7581920abacdb128e1c60e3986e2ce58463857c918e85a67244ab74d8" gracePeriod=30
Sep 10 17:59:10.242732 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:10.242707    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c5c751-7b9d859477-d4jb7"
Sep 10 17:59:10.271383 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:10.271357    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c5c751-75d6b7dfcd-5bhm6"
Sep 10 17:59:10.367978 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:10.367949    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t6mng\" (UniqueName: \"kubernetes.io/projected/e82663d4-b257-42ee-a0b3-a28b8f3f06e0-kube-api-access-t6mng\") pod \"e82663d4-b257-42ee-a0b3-a28b8f3f06e0\" (UID: \"e82663d4-b257-42ee-a0b3-a28b8f3f06e0\") "
Sep 10 17:59:10.368176 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:10.368008    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e82663d4-b257-42ee-a0b3-a28b8f3f06e0-maas-api-tls\") pod \"e82663d4-b257-42ee-a0b3-a28b8f3f06e0\" (UID: \"e82663d4-b257-42ee-a0b3-a28b8f3f06e0\") "
Sep 10 17:59:10.368176 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:10.368062    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e82663d4-b257-42ee-a0b3-a28b8f3f06e0-metrics-certs\") pod \"e82663d4-b257-42ee-a0b3-a28b8f3f06e0\" (UID: \"e82663d4-b257-42ee-a0b3-a28b8f3f06e0\") "
Sep 10 17:59:10.368176 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:10.368089    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bh9bz\" (UniqueName: \"kubernetes.io/projected/87ba0e8f-9ab6-45be-b6ec-531f2c60b4a8-kube-api-access-bh9bz\") pod \"87ba0e8f-9ab6-45be-b6ec-531f2c60b4a8\" (UID: \"87ba0e8f-9ab6-45be-b6ec-531f2c60b4a8\") "
Sep 10 17:59:10.368176 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:10.368134    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e82663d4-b257-42ee-a0b3-a28b8f3f06e0-openshift-service-ca\") pod \"e82663d4-b257-42ee-a0b3-a28b8f3f06e0\" (UID: \"e82663d4-b257-42ee-a0b3-a28b8f3f06e0\") "
Sep 10 17:59:10.368176 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:10.368158    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87ba0e8f-9ab6-45be-b6ec-531f2c60b4a8-plugins-config-volume\") pod \"87ba0e8f-9ab6-45be-b6ec-531f2c60b4a8\" (UID: \"87ba0e8f-9ab6-45be-b6ec-531f2c60b4a8\") "
Sep 10 17:59:10.368645 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:10.368559    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e82663d4-b257-42ee-a0b3-a28b8f3f06e0-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e82663d4-b257-42ee-a0b3-a28b8f3f06e0" (UID: "e82663d4-b257-42ee-a0b3-a28b8f3f06e0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 17:59:10.368779 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:10.368730    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87ba0e8f-9ab6-45be-b6ec-531f2c60b4a8-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "87ba0e8f-9ab6-45be-b6ec-531f2c60b4a8" (UID: "87ba0e8f-9ab6-45be-b6ec-531f2c60b4a8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 17:59:10.368997 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:10.368909    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e82663d4-b257-42ee-a0b3-a28b8f3f06e0-openshift-service-ca\") on node \"ip-10-0-129-122.ec2.internal\" DevicePath \"\""
Sep 10 17:59:10.370400 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:10.370374    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e82663d4-b257-42ee-a0b3-a28b8f3f06e0-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e82663d4-b257-42ee-a0b3-a28b8f3f06e0" (UID: "e82663d4-b257-42ee-a0b3-a28b8f3f06e0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 17:59:10.370804 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:10.370750    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87ba0e8f-9ab6-45be-b6ec-531f2c60b4a8-kube-api-access-bh9bz" (OuterVolumeSpecName: "kube-api-access-bh9bz") pod "87ba0e8f-9ab6-45be-b6ec-531f2c60b4a8" (UID: "87ba0e8f-9ab6-45be-b6ec-531f2c60b4a8"). InnerVolumeSpecName "kube-api-access-bh9bz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:59:10.370880 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:10.370801    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e82663d4-b257-42ee-a0b3-a28b8f3f06e0-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "e82663d4-b257-42ee-a0b3-a28b8f3f06e0" (UID: "e82663d4-b257-42ee-a0b3-a28b8f3f06e0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 17:59:10.370880 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:10.370847    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e82663d4-b257-42ee-a0b3-a28b8f3f06e0-kube-api-access-t6mng" (OuterVolumeSpecName: "kube-api-access-t6mng") pod "e82663d4-b257-42ee-a0b3-a28b8f3f06e0" (UID: "e82663d4-b257-42ee-a0b3-a28b8f3f06e0"). InnerVolumeSpecName "kube-api-access-t6mng". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:59:10.469534 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:10.469451    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e82663d4-b257-42ee-a0b3-a28b8f3f06e0-maas-api-tls\") on node \"ip-10-0-129-122.ec2.internal\" DevicePath \"\""
Sep 10 17:59:10.469534 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:10.469477    2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e82663d4-b257-42ee-a0b3-a28b8f3f06e0-metrics-certs\") on node \"ip-10-0-129-122.ec2.internal\" DevicePath \"\""
Sep 10 17:59:10.469534 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:10.469486    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bh9bz\" (UniqueName: \"kubernetes.io/projected/87ba0e8f-9ab6-45be-b6ec-531f2c60b4a8-kube-api-access-bh9bz\") on node \"ip-10-0-129-122.ec2.internal\" DevicePath \"\""
Sep 10 17:59:10.469534 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:10.469495    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87ba0e8f-9ab6-45be-b6ec-531f2c60b4a8-plugins-config-volume\") on node \"ip-10-0-129-122.ec2.internal\" DevicePath \"\""
Sep 10 17:59:10.469534 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:10.469504    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t6mng\" (UniqueName: \"kubernetes.io/projected/e82663d4-b257-42ee-a0b3-a28b8f3f06e0-kube-api-access-t6mng\") on node \"ip-10-0-129-122.ec2.internal\" DevicePath \"\""
Sep 10 17:59:10.770825 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:10.770727    2579 generic.go:356] "Generic (PLEG): container finished" podID="87ba0e8f-9ab6-45be-b6ec-531f2c60b4a8" containerID="17318ab7581920abacdb128e1c60e3986e2ce58463857c918e85a67244ab74d8" exitCode=0
Sep 10 17:59:10.770825 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:10.770782    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c5c751-75d6b7dfcd-5bhm6"
Sep 10 17:59:10.770825 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:10.770816    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c5c751-75d6b7dfcd-5bhm6" event={"ID":"87ba0e8f-9ab6-45be-b6ec-531f2c60b4a8","Type":"ContainerDied","Data":"17318ab7581920abacdb128e1c60e3986e2ce58463857c918e85a67244ab74d8"}
Sep 10 17:59:10.771126 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:10.770866    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c5c751-75d6b7dfcd-5bhm6" event={"ID":"87ba0e8f-9ab6-45be-b6ec-531f2c60b4a8","Type":"ContainerDied","Data":"da607699c41bc0af59c6d2276b3fc1995a8ffebc570b8ac97b775324be595ffe"}
Sep 10 17:59:10.771126 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:10.770891    2579 scope.go:117] "RemoveContainer" containerID="17318ab7581920abacdb128e1c60e3986e2ce58463857c918e85a67244ab74d8"
Sep 10 17:59:10.772384 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:10.772357    2579 generic.go:356] "Generic (PLEG): container finished" podID="e82663d4-b257-42ee-a0b3-a28b8f3f06e0" containerID="1983c2edd249e06d8a66c9336c862123a3415d80c0b2bc188fd1a94a0e3f9a03" exitCode=0
Sep 10 17:59:10.772520 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:10.772399    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c5c751-7b9d859477-d4jb7"
Sep 10 17:59:10.772520 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:10.772447    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c5c751-7b9d859477-d4jb7" event={"ID":"e82663d4-b257-42ee-a0b3-a28b8f3f06e0","Type":"ContainerDied","Data":"1983c2edd249e06d8a66c9336c862123a3415d80c0b2bc188fd1a94a0e3f9a03"}
Sep 10 17:59:10.772520 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:10.772508    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c5c751-7b9d859477-d4jb7" event={"ID":"e82663d4-b257-42ee-a0b3-a28b8f3f06e0","Type":"ContainerDied","Data":"397c9c67f5f0cb747bc18472bfdebc101fea17275bb604064fa6f7c2dc2b8aac"}
Sep 10 17:59:10.780487 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:10.780454    2579 scope.go:117] "RemoveContainer" containerID="17318ab7581920abacdb128e1c60e3986e2ce58463857c918e85a67244ab74d8"
Sep 10 17:59:10.780760 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:59:10.780742    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17318ab7581920abacdb128e1c60e3986e2ce58463857c918e85a67244ab74d8\": container with ID starting with 17318ab7581920abacdb128e1c60e3986e2ce58463857c918e85a67244ab74d8 not found: ID does not exist" containerID="17318ab7581920abacdb128e1c60e3986e2ce58463857c918e85a67244ab74d8"
Sep 10 17:59:10.780820 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:10.780768    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17318ab7581920abacdb128e1c60e3986e2ce58463857c918e85a67244ab74d8"} err="failed to get container status \"17318ab7581920abacdb128e1c60e3986e2ce58463857c918e85a67244ab74d8\": rpc error: code = NotFound desc = could not find container \"17318ab7581920abacdb128e1c60e3986e2ce58463857c918e85a67244ab74d8\": container with ID starting with 17318ab7581920abacdb128e1c60e3986e2ce58463857c918e85a67244ab74d8 not found: ID does not exist"
Sep 10 17:59:10.780820 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:10.780784    2579 scope.go:117] "RemoveContainer" containerID="1983c2edd249e06d8a66c9336c862123a3415d80c0b2bc188fd1a94a0e3f9a03"
Sep 10 17:59:10.789123 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:10.789103    2579 scope.go:117] "RemoveContainer" containerID="1983c2edd249e06d8a66c9336c862123a3415d80c0b2bc188fd1a94a0e3f9a03"
Sep 10 17:59:10.789421 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:59:10.789397    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1983c2edd249e06d8a66c9336c862123a3415d80c0b2bc188fd1a94a0e3f9a03\": container with ID starting with 1983c2edd249e06d8a66c9336c862123a3415d80c0b2bc188fd1a94a0e3f9a03 not found: ID does not exist" containerID="1983c2edd249e06d8a66c9336c862123a3415d80c0b2bc188fd1a94a0e3f9a03"
Sep 10 17:59:10.789503 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:10.789431    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1983c2edd249e06d8a66c9336c862123a3415d80c0b2bc188fd1a94a0e3f9a03"} err="failed to get container status \"1983c2edd249e06d8a66c9336c862123a3415d80c0b2bc188fd1a94a0e3f9a03\": rpc error: code = NotFound desc = could not find container \"1983c2edd249e06d8a66c9336c862123a3415d80c0b2bc188fd1a94a0e3f9a03\": container with ID starting with 1983c2edd249e06d8a66c9336c862123a3415d80c0b2bc188fd1a94a0e3f9a03 not found: ID does not exist"
Sep 10 17:59:10.813184 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:10.813164    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c5c751-7b9d859477-d4jb7"]
Sep 10 17:59:10.819193 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:10.819174    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c5c751-7b9d859477-d4jb7"]
Sep 10 17:59:10.832237 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:10.832212    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-c5c751-75d6b7dfcd-5bhm6"]
Sep 10 17:59:10.838323 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:10.838305    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-c5c751-75d6b7dfcd-5bhm6"]
Sep 10 17:59:12.477584 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:12.477535    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="87ba0e8f-9ab6-45be-b6ec-531f2c60b4a8" path="/var/lib/kubelet/pods/87ba0e8f-9ab6-45be-b6ec-531f2c60b4a8/volumes"
Sep 10 17:59:12.478123 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:12.478106    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e82663d4-b257-42ee-a0b3-a28b8f3f06e0" path="/var/lib/kubelet/pods/e82663d4-b257-42ee-a0b3-a28b8f3f06e0/volumes"
Sep 10 17:59:17.286130 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:17.286095    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-c5c751-openshift-default-586c478496-llxnx"]
Sep 10 17:59:17.286522 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:17.286316    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-c5c751-openshift-default-586c478496-llxnx" podUID="013e342b-e2dc-4034-b58f-ea9bafb7b379" containerName="istio-proxy" containerID="cri-o://c0f81fd56a99c0f9d2b8a1c9c8e46a92df86d9f269abfae60be5c58961833800" gracePeriod=30
Sep 10 17:59:21.793741 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:21.793703    2579 patch_prober.go:28] interesting pod/e2e-api-a-c5c751-openshift-default-586c478496-llxnx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 10 17:59:21.794173 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:21.793767    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-c5c751-openshift-default-586c478496-llxnx" podUID="013e342b-e2dc-4034-b58f-ea9bafb7b379" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 10 17:59:22.536688 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:22.536663    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-c5c751-openshift-default-586c478496-llxnx"
Sep 10 17:59:22.687185 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:22.687153    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/013e342b-e2dc-4034-b58f-ea9bafb7b379-istio-token\") pod \"013e342b-e2dc-4034-b58f-ea9bafb7b379\" (UID: \"013e342b-e2dc-4034-b58f-ea9bafb7b379\") "
Sep 10 17:59:22.687349 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:22.687197    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/013e342b-e2dc-4034-b58f-ea9bafb7b379-istio-podinfo\") pod \"013e342b-e2dc-4034-b58f-ea9bafb7b379\" (UID: \"013e342b-e2dc-4034-b58f-ea9bafb7b379\") "
Sep 10 17:59:22.687349 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:22.687242    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/013e342b-e2dc-4034-b58f-ea9bafb7b379-istio-envoy\") pod \"013e342b-e2dc-4034-b58f-ea9bafb7b379\" (UID: \"013e342b-e2dc-4034-b58f-ea9bafb7b379\") "
Sep 10 17:59:22.687349 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:22.687268    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s2j5r\" (UniqueName: \"kubernetes.io/projected/013e342b-e2dc-4034-b58f-ea9bafb7b379-kube-api-access-s2j5r\") pod \"013e342b-e2dc-4034-b58f-ea9bafb7b379\" (UID: \"013e342b-e2dc-4034-b58f-ea9bafb7b379\") "
Sep 10 17:59:22.687349 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:22.687298    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/013e342b-e2dc-4034-b58f-ea9bafb7b379-istio-data\") pod \"013e342b-e2dc-4034-b58f-ea9bafb7b379\" (UID: \"013e342b-e2dc-4034-b58f-ea9bafb7b379\") "
Sep 10 17:59:22.687349 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:22.687337    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/013e342b-e2dc-4034-b58f-ea9bafb7b379-credential-socket\") pod \"013e342b-e2dc-4034-b58f-ea9bafb7b379\" (UID: \"013e342b-e2dc-4034-b58f-ea9bafb7b379\") "
Sep 10 17:59:22.687640 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:22.687491    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/013e342b-e2dc-4034-b58f-ea9bafb7b379-workload-certs\") pod \"013e342b-e2dc-4034-b58f-ea9bafb7b379\" (UID: \"013e342b-e2dc-4034-b58f-ea9bafb7b379\") "
Sep 10 17:59:22.687640 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:22.687552    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/013e342b-e2dc-4034-b58f-ea9bafb7b379-istiod-ca-cert\") pod \"013e342b-e2dc-4034-b58f-ea9bafb7b379\" (UID: \"013e342b-e2dc-4034-b58f-ea9bafb7b379\") "
Sep 10 17:59:22.687640 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:22.687620    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/013e342b-e2dc-4034-b58f-ea9bafb7b379-workload-socket\") pod \"013e342b-e2dc-4034-b58f-ea9bafb7b379\" (UID: \"013e342b-e2dc-4034-b58f-ea9bafb7b379\") "
Sep 10 17:59:22.688072 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:22.688041    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/013e342b-e2dc-4034-b58f-ea9bafb7b379-istio-data" (OuterVolumeSpecName: "istio-data") pod "013e342b-e2dc-4034-b58f-ea9bafb7b379" (UID: "013e342b-e2dc-4034-b58f-ea9bafb7b379"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:59:22.688205 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:22.688129    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/013e342b-e2dc-4034-b58f-ea9bafb7b379-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "013e342b-e2dc-4034-b58f-ea9bafb7b379" (UID: "013e342b-e2dc-4034-b58f-ea9bafb7b379"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:59:22.688279 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:22.688220    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/013e342b-e2dc-4034-b58f-ea9bafb7b379-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "013e342b-e2dc-4034-b58f-ea9bafb7b379" (UID: "013e342b-e2dc-4034-b58f-ea9bafb7b379"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:59:22.688433 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:22.688411    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/013e342b-e2dc-4034-b58f-ea9bafb7b379-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "013e342b-e2dc-4034-b58f-ea9bafb7b379" (UID: "013e342b-e2dc-4034-b58f-ea9bafb7b379"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:59:22.688433 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:22.688425    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/013e342b-e2dc-4034-b58f-ea9bafb7b379-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "013e342b-e2dc-4034-b58f-ea9bafb7b379" (UID: "013e342b-e2dc-4034-b58f-ea9bafb7b379"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 17:59:22.689525 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:22.689502    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/013e342b-e2dc-4034-b58f-ea9bafb7b379-istio-token" (OuterVolumeSpecName: "istio-token") pod "013e342b-e2dc-4034-b58f-ea9bafb7b379" (UID: "013e342b-e2dc-4034-b58f-ea9bafb7b379"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:59:22.689871 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:22.689856    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/013e342b-e2dc-4034-b58f-ea9bafb7b379-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "013e342b-e2dc-4034-b58f-ea9bafb7b379" (UID: "013e342b-e2dc-4034-b58f-ea9bafb7b379"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:59:22.690002 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:22.689988    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/013e342b-e2dc-4034-b58f-ea9bafb7b379-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "013e342b-e2dc-4034-b58f-ea9bafb7b379" (UID: "013e342b-e2dc-4034-b58f-ea9bafb7b379"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 10 17:59:22.690286 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:22.690263    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/013e342b-e2dc-4034-b58f-ea9bafb7b379-kube-api-access-s2j5r" (OuterVolumeSpecName: "kube-api-access-s2j5r") pod "013e342b-e2dc-4034-b58f-ea9bafb7b379" (UID: "013e342b-e2dc-4034-b58f-ea9bafb7b379"). InnerVolumeSpecName "kube-api-access-s2j5r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:59:22.788925 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:22.788898    2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/013e342b-e2dc-4034-b58f-ea9bafb7b379-istio-token\") on node \"ip-10-0-129-122.ec2.internal\" DevicePath \"\""
Sep 10 17:59:22.788925 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:22.788925    2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/013e342b-e2dc-4034-b58f-ea9bafb7b379-istio-podinfo\") on node \"ip-10-0-129-122.ec2.internal\" DevicePath \"\""
Sep 10 17:59:22.789091 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:22.788933    2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/013e342b-e2dc-4034-b58f-ea9bafb7b379-istio-envoy\") on node \"ip-10-0-129-122.ec2.internal\" DevicePath \"\""
Sep 10 17:59:22.789091 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:22.788942    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s2j5r\" (UniqueName: \"kubernetes.io/projected/013e342b-e2dc-4034-b58f-ea9bafb7b379-kube-api-access-s2j5r\") on node \"ip-10-0-129-122.ec2.internal\" DevicePath \"\""
Sep 10 17:59:22.789091 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:22.788952    2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/013e342b-e2dc-4034-b58f-ea9bafb7b379-istio-data\") on node \"ip-10-0-129-122.ec2.internal\" DevicePath \"\""
Sep 10 17:59:22.789091 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:22.788960    2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/013e342b-e2dc-4034-b58f-ea9bafb7b379-credential-socket\") on node \"ip-10-0-129-122.ec2.internal\" DevicePath \"\""
Sep 10 17:59:22.789091 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:22.788967    2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/013e342b-e2dc-4034-b58f-ea9bafb7b379-workload-certs\") on node \"ip-10-0-129-122.ec2.internal\" DevicePath \"\""
Sep 10 17:59:22.789091 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:22.788975    2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/013e342b-e2dc-4034-b58f-ea9bafb7b379-istiod-ca-cert\") on node \"ip-10-0-129-122.ec2.internal\" DevicePath \"\""
Sep 10 17:59:22.789091 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:22.788982    2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/013e342b-e2dc-4034-b58f-ea9bafb7b379-workload-socket\") on node \"ip-10-0-129-122.ec2.internal\" DevicePath \"\""
Sep 10 17:59:22.841399 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:22.841368    2579 generic.go:356] "Generic (PLEG): container finished" podID="013e342b-e2dc-4034-b58f-ea9bafb7b379" containerID="c0f81fd56a99c0f9d2b8a1c9c8e46a92df86d9f269abfae60be5c58961833800" exitCode=0
Sep 10 17:59:22.841784 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:22.841417    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-c5c751-openshift-default-586c478496-llxnx"
Sep 10 17:59:22.841784 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:22.841457    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c5c751-openshift-default-586c478496-llxnx" event={"ID":"013e342b-e2dc-4034-b58f-ea9bafb7b379","Type":"ContainerDied","Data":"c0f81fd56a99c0f9d2b8a1c9c8e46a92df86d9f269abfae60be5c58961833800"}
Sep 10 17:59:22.841784 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:22.841508    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c5c751-openshift-default-586c478496-llxnx" event={"ID":"013e342b-e2dc-4034-b58f-ea9bafb7b379","Type":"ContainerDied","Data":"b9cae73f1ba1d5cbfbbc113462998f006e9eb5d4adb933a2ab1c606eac19afc3"}
Sep 10 17:59:22.841784 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:22.841529    2579 scope.go:117] "RemoveContainer" containerID="c0f81fd56a99c0f9d2b8a1c9c8e46a92df86d9f269abfae60be5c58961833800"
Sep 10 17:59:22.854029 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:22.854012    2579 scope.go:117] "RemoveContainer" containerID="e5013314d8a7dec634ead2722945328aedf17c68b85155aeb54ddbe4a0b950c8"
Sep 10 17:59:22.862361 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:22.862341    2579 scope.go:117] "RemoveContainer" containerID="c0f81fd56a99c0f9d2b8a1c9c8e46a92df86d9f269abfae60be5c58961833800"
Sep 10 17:59:22.862651 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:59:22.862633    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0f81fd56a99c0f9d2b8a1c9c8e46a92df86d9f269abfae60be5c58961833800\": container with ID starting with c0f81fd56a99c0f9d2b8a1c9c8e46a92df86d9f269abfae60be5c58961833800 not found: ID does not exist" containerID="c0f81fd56a99c0f9d2b8a1c9c8e46a92df86d9f269abfae60be5c58961833800"
Sep 10 17:59:22.862719 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:22.862659    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0f81fd56a99c0f9d2b8a1c9c8e46a92df86d9f269abfae60be5c58961833800"} err="failed to get container status \"c0f81fd56a99c0f9d2b8a1c9c8e46a92df86d9f269abfae60be5c58961833800\": rpc error: code = NotFound desc = could not find container \"c0f81fd56a99c0f9d2b8a1c9c8e46a92df86d9f269abfae60be5c58961833800\": container with ID starting with c0f81fd56a99c0f9d2b8a1c9c8e46a92df86d9f269abfae60be5c58961833800 not found: ID does not exist"
Sep 10 17:59:22.862719 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:22.862674    2579 scope.go:117] "RemoveContainer" containerID="e5013314d8a7dec634ead2722945328aedf17c68b85155aeb54ddbe4a0b950c8"
Sep 10 17:59:22.862912 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:59:22.862894    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5013314d8a7dec634ead2722945328aedf17c68b85155aeb54ddbe4a0b950c8\": container with ID starting with e5013314d8a7dec634ead2722945328aedf17c68b85155aeb54ddbe4a0b950c8 not found: ID does not exist" containerID="e5013314d8a7dec634ead2722945328aedf17c68b85155aeb54ddbe4a0b950c8"
Sep 10 17:59:22.862953 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:22.862918    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5013314d8a7dec634ead2722945328aedf17c68b85155aeb54ddbe4a0b950c8"} err="failed to get container status \"e5013314d8a7dec634ead2722945328aedf17c68b85155aeb54ddbe4a0b950c8\": rpc error: code = NotFound desc = could not find container \"e5013314d8a7dec634ead2722945328aedf17c68b85155aeb54ddbe4a0b950c8\": container with ID starting with e5013314d8a7dec634ead2722945328aedf17c68b85155aeb54ddbe4a0b950c8 not found: ID does not exist"
Sep 10 17:59:22.868908 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:22.868885    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-c5c751-openshift-default-586c478496-llxnx"]
Sep 10 17:59:22.873400 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:22.873380    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-c5c751-openshift-default-586c478496-llxnx"]
Sep 10 17:59:24.476889 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:24.476855    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="013e342b-e2dc-4034-b58f-ea9bafb7b379" path="/var/lib/kubelet/pods/013e342b-e2dc-4034-b58f-ea9bafb7b379/volumes"
Sep 10 17:59:33.335694 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:33.335659    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-abad1a-6c745c8b5-5tjzp"]
Sep 10 17:59:33.336114 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:33.335894    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-abad1a-6c745c8b5-5tjzp" podUID="2560bb66-7ee8-4285-8d26-913a26f10099" containerName="payload-processing" containerID="cri-o://b236d0de0b9be08ab08db6378ec3201364afb6181d7110a9d1737ce55ed2c696" gracePeriod=30
Sep 10 17:59:33.602090 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:33.602066    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-abad1a-6c745c8b5-5tjzp"
Sep 10 17:59:33.789949 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:33.789909    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jspfm\" (UniqueName: \"kubernetes.io/projected/2560bb66-7ee8-4285-8d26-913a26f10099-kube-api-access-jspfm\") pod \"2560bb66-7ee8-4285-8d26-913a26f10099\" (UID: \"2560bb66-7ee8-4285-8d26-913a26f10099\") "
Sep 10 17:59:33.790085 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:33.790000    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2560bb66-7ee8-4285-8d26-913a26f10099-plugins-config-volume\") pod \"2560bb66-7ee8-4285-8d26-913a26f10099\" (UID: \"2560bb66-7ee8-4285-8d26-913a26f10099\") "
Sep 10 17:59:33.790369 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:33.790337    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2560bb66-7ee8-4285-8d26-913a26f10099-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2560bb66-7ee8-4285-8d26-913a26f10099" (UID: "2560bb66-7ee8-4285-8d26-913a26f10099"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 17:59:33.792167 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:33.792147    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2560bb66-7ee8-4285-8d26-913a26f10099-kube-api-access-jspfm" (OuterVolumeSpecName: "kube-api-access-jspfm") pod "2560bb66-7ee8-4285-8d26-913a26f10099" (UID: "2560bb66-7ee8-4285-8d26-913a26f10099"). InnerVolumeSpecName "kube-api-access-jspfm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:59:33.890773 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:33.890743    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2560bb66-7ee8-4285-8d26-913a26f10099-plugins-config-volume\") on node \"ip-10-0-129-122.ec2.internal\" DevicePath \"\""
Sep 10 17:59:33.890921 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:33.890773    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jspfm\" (UniqueName: \"kubernetes.io/projected/2560bb66-7ee8-4285-8d26-913a26f10099-kube-api-access-jspfm\") on node \"ip-10-0-129-122.ec2.internal\" DevicePath \"\""
Sep 10 17:59:33.899208 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:33.899182    2579 generic.go:356] "Generic (PLEG): container finished" podID="2560bb66-7ee8-4285-8d26-913a26f10099" containerID="b236d0de0b9be08ab08db6378ec3201364afb6181d7110a9d1737ce55ed2c696" exitCode=0
Sep 10 17:59:33.899335 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:33.899219    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-abad1a-6c745c8b5-5tjzp"
Sep 10 17:59:33.899335 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:33.899264    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-abad1a-6c745c8b5-5tjzp" event={"ID":"2560bb66-7ee8-4285-8d26-913a26f10099","Type":"ContainerDied","Data":"b236d0de0b9be08ab08db6378ec3201364afb6181d7110a9d1737ce55ed2c696"}
Sep 10 17:59:33.899335 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:33.899313    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-abad1a-6c745c8b5-5tjzp" event={"ID":"2560bb66-7ee8-4285-8d26-913a26f10099","Type":"ContainerDied","Data":"3293b925bc4d7e456ae3e15a597773d49e15a725b1f43228bd16df72be81466f"}
Sep 10 17:59:33.899335 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:33.899334    2579 scope.go:117] "RemoveContainer" containerID="b236d0de0b9be08ab08db6378ec3201364afb6181d7110a9d1737ce55ed2c696"
Sep 10 17:59:33.913824 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:33.913803    2579 scope.go:117] "RemoveContainer" containerID="b236d0de0b9be08ab08db6378ec3201364afb6181d7110a9d1737ce55ed2c696"
Sep 10 17:59:33.914143 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:59:33.914123    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b236d0de0b9be08ab08db6378ec3201364afb6181d7110a9d1737ce55ed2c696\": container with ID starting with b236d0de0b9be08ab08db6378ec3201364afb6181d7110a9d1737ce55ed2c696 not found: ID does not exist" containerID="b236d0de0b9be08ab08db6378ec3201364afb6181d7110a9d1737ce55ed2c696"
Sep 10 17:59:33.914211 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:33.914176    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b236d0de0b9be08ab08db6378ec3201364afb6181d7110a9d1737ce55ed2c696"} err="failed to get container status \"b236d0de0b9be08ab08db6378ec3201364afb6181d7110a9d1737ce55ed2c696\": rpc error: code = NotFound desc = could not find container \"b236d0de0b9be08ab08db6378ec3201364afb6181d7110a9d1737ce55ed2c696\": container with ID starting with b236d0de0b9be08ab08db6378ec3201364afb6181d7110a9d1737ce55ed2c696 not found: ID does not exist"
Sep 10 17:59:33.927887 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:33.927865    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-abad1a-6c745c8b5-5tjzp"]
Sep 10 17:59:33.932170 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:33.932150    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-abad1a-6c745c8b5-5tjzp"]
Sep 10 17:59:34.475801 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:34.475772    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2560bb66-7ee8-4285-8d26-913a26f10099" path="/var/lib/kubelet/pods/2560bb66-7ee8-4285-8d26-913a26f10099/volumes"
Sep 10 17:59:40.213508 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:40.213476    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-abad1a-openshift-default-7969487574-l69b4"]
Sep 10 17:59:40.214019 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:40.213758    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-abad1a-openshift-default-7969487574-l69b4" podUID="6b252299-8b3e-4fd7-986e-99f19feae0e3" containerName="istio-proxy" containerID="cri-o://72f73be54d9b32da69de161708190152cb9d81380b2726cb4c51522a88a7b3a6" gracePeriod=30
Sep 10 17:59:40.436058 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:40.436022    2579 patch_prober.go:28] interesting pod/e2e-api-b-abad1a-openshift-default-7969487574-l69b4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 10 17:59:40.436203 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:40.436082    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-abad1a-openshift-default-7969487574-l69b4" podUID="6b252299-8b3e-4fd7-986e-99f19feae0e3" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 10 17:59:43.286542 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:43.286510    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-904bbd-openshift-default-69d64c9f6-lttqf"]
Sep 10 17:59:43.287138 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:43.286924    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="013e342b-e2dc-4034-b58f-ea9bafb7b379" containerName="istio-proxy"
Sep 10 17:59:43.287138 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:43.286937    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="013e342b-e2dc-4034-b58f-ea9bafb7b379" containerName="istio-proxy"
Sep 10 17:59:43.287138 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:43.286948    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="87ba0e8f-9ab6-45be-b6ec-531f2c60b4a8" containerName="payload-pre-processing"
Sep 10 17:59:43.287138 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:43.286953    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="87ba0e8f-9ab6-45be-b6ec-531f2c60b4a8" containerName="payload-pre-processing"
Sep 10 17:59:43.287138 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:43.286965    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2560bb66-7ee8-4285-8d26-913a26f10099" containerName="payload-processing"
Sep 10 17:59:43.287138 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:43.286970    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="2560bb66-7ee8-4285-8d26-913a26f10099" containerName="payload-processing"
Sep 10 17:59:43.287138 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:43.286980    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e82663d4-b257-42ee-a0b3-a28b8f3f06e0" containerName="maas-api"
Sep 10 17:59:43.287138 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:43.286985    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="e82663d4-b257-42ee-a0b3-a28b8f3f06e0" containerName="maas-api"
Sep 10 17:59:43.287138 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:43.287054    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="e82663d4-b257-42ee-a0b3-a28b8f3f06e0" containerName="maas-api"
Sep 10 17:59:43.287138 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:43.287062    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="013e342b-e2dc-4034-b58f-ea9bafb7b379" containerName="istio-proxy"
Sep 10 17:59:43.287138 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:43.287070    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="013e342b-e2dc-4034-b58f-ea9bafb7b379" containerName="istio-proxy"
Sep 10 17:59:43.287138 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:43.287080    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="87ba0e8f-9ab6-45be-b6ec-531f2c60b4a8" containerName="payload-pre-processing"
Sep 10 17:59:43.287138 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:43.287089    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="2560bb66-7ee8-4285-8d26-913a26f10099" containerName="payload-processing"
Sep 10 17:59:43.291707 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:43.291686    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-904bbd-openshift-default-69d64c9f6-lttqf"
Sep 10 17:59:43.294527 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:43.294506    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-904bbd-openshift-default-dockercfg-7kz9c\""
Sep 10 17:59:43.302086 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:43.302059    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-904bbd-openshift-default-69d64c9f6-lttqf"]
Sep 10 17:59:43.368122 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:43.368081    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22216a3a-3a36-4fb3-8980-4ea64db8a74b-credential-socket\") pod \"e2e-ipp-a-904bbd-openshift-default-69d64c9f6-lttqf\" (UID: \"22216a3a-3a36-4fb3-8980-4ea64db8a74b\") " pod="openshift-ingress/e2e-ipp-a-904bbd-openshift-default-69d64c9f6-lttqf"
Sep 10 17:59:43.368289 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:43.368148    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22216a3a-3a36-4fb3-8980-4ea64db8a74b-istio-envoy\") pod \"e2e-ipp-a-904bbd-openshift-default-69d64c9f6-lttqf\" (UID: \"22216a3a-3a36-4fb3-8980-4ea64db8a74b\") " pod="openshift-ingress/e2e-ipp-a-904bbd-openshift-default-69d64c9f6-lttqf"
Sep 10 17:59:43.368289 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:43.368183    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hh58d\" (UniqueName: \"kubernetes.io/projected/22216a3a-3a36-4fb3-8980-4ea64db8a74b-kube-api-access-hh58d\") pod \"e2e-ipp-a-904bbd-openshift-default-69d64c9f6-lttqf\" (UID: \"22216a3a-3a36-4fb3-8980-4ea64db8a74b\") " pod="openshift-ingress/e2e-ipp-a-904bbd-openshift-default-69d64c9f6-lttqf"
Sep 10 17:59:43.368289 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:43.368274    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22216a3a-3a36-4fb3-8980-4ea64db8a74b-istiod-ca-cert\") pod \"e2e-ipp-a-904bbd-openshift-default-69d64c9f6-lttqf\" (UID: \"22216a3a-3a36-4fb3-8980-4ea64db8a74b\") " pod="openshift-ingress/e2e-ipp-a-904bbd-openshift-default-69d64c9f6-lttqf"
Sep 10 17:59:43.368463 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:43.368333    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22216a3a-3a36-4fb3-8980-4ea64db8a74b-workload-certs\") pod \"e2e-ipp-a-904bbd-openshift-default-69d64c9f6-lttqf\" (UID: \"22216a3a-3a36-4fb3-8980-4ea64db8a74b\") " pod="openshift-ingress/e2e-ipp-a-904bbd-openshift-default-69d64c9f6-lttqf"
Sep 10 17:59:43.368463 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:43.368362    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22216a3a-3a36-4fb3-8980-4ea64db8a74b-istio-token\") pod \"e2e-ipp-a-904bbd-openshift-default-69d64c9f6-lttqf\" (UID: \"22216a3a-3a36-4fb3-8980-4ea64db8a74b\") " pod="openshift-ingress/e2e-ipp-a-904bbd-openshift-default-69d64c9f6-lttqf"
Sep 10 17:59:43.368463 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:43.368395    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22216a3a-3a36-4fb3-8980-4ea64db8a74b-istio-data\") pod \"e2e-ipp-a-904bbd-openshift-default-69d64c9f6-lttqf\" (UID: \"22216a3a-3a36-4fb3-8980-4ea64db8a74b\") " pod="openshift-ingress/e2e-ipp-a-904bbd-openshift-default-69d64c9f6-lttqf"
Sep 10 17:59:43.368621 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:43.368461    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22216a3a-3a36-4fb3-8980-4ea64db8a74b-istio-podinfo\") pod \"e2e-ipp-a-904bbd-openshift-default-69d64c9f6-lttqf\" (UID: \"22216a3a-3a36-4fb3-8980-4ea64db8a74b\") " pod="openshift-ingress/e2e-ipp-a-904bbd-openshift-default-69d64c9f6-lttqf"
Sep 10 17:59:43.368621 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:43.368489    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22216a3a-3a36-4fb3-8980-4ea64db8a74b-workload-socket\") pod \"e2e-ipp-a-904bbd-openshift-default-69d64c9f6-lttqf\" (UID: \"22216a3a-3a36-4fb3-8980-4ea64db8a74b\") " pod="openshift-ingress/e2e-ipp-a-904bbd-openshift-default-69d64c9f6-lttqf"
Sep 10 17:59:43.469275 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:43.469239    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22216a3a-3a36-4fb3-8980-4ea64db8a74b-istio-envoy\") pod \"e2e-ipp-a-904bbd-openshift-default-69d64c9f6-lttqf\" (UID: \"22216a3a-3a36-4fb3-8980-4ea64db8a74b\") " pod="openshift-ingress/e2e-ipp-a-904bbd-openshift-default-69d64c9f6-lttqf"
Sep 10 17:59:43.469501 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:43.469281    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hh58d\" (UniqueName: \"kubernetes.io/projected/22216a3a-3a36-4fb3-8980-4ea64db8a74b-kube-api-access-hh58d\") pod \"e2e-ipp-a-904bbd-openshift-default-69d64c9f6-lttqf\" (UID: \"22216a3a-3a36-4fb3-8980-4ea64db8a74b\") " pod="openshift-ingress/e2e-ipp-a-904bbd-openshift-default-69d64c9f6-lttqf"
Sep 10 17:59:43.469501 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:43.469316    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22216a3a-3a36-4fb3-8980-4ea64db8a74b-istiod-ca-cert\") pod \"e2e-ipp-a-904bbd-openshift-default-69d64c9f6-lttqf\" (UID: \"22216a3a-3a36-4fb3-8980-4ea64db8a74b\") " pod="openshift-ingress/e2e-ipp-a-904bbd-openshift-default-69d64c9f6-lttqf"
Sep 10 17:59:43.469501 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:43.469341    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22216a3a-3a36-4fb3-8980-4ea64db8a74b-workload-certs\") pod \"e2e-ipp-a-904bbd-openshift-default-69d64c9f6-lttqf\" (UID: \"22216a3a-3a36-4fb3-8980-4ea64db8a74b\") " pod="openshift-ingress/e2e-ipp-a-904bbd-openshift-default-69d64c9f6-lttqf"
Sep 10 17:59:43.469501 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:43.469356    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22216a3a-3a36-4fb3-8980-4ea64db8a74b-istio-token\") pod \"e2e-ipp-a-904bbd-openshift-default-69d64c9f6-lttqf\" (UID: \"22216a3a-3a36-4fb3-8980-4ea64db8a74b\") " pod="openshift-ingress/e2e-ipp-a-904bbd-openshift-default-69d64c9f6-lttqf"
Sep 10 17:59:43.469501 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:43.469374    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22216a3a-3a36-4fb3-8980-4ea64db8a74b-istio-data\") pod \"e2e-ipp-a-904bbd-openshift-default-69d64c9f6-lttqf\" (UID: \"22216a3a-3a36-4fb3-8980-4ea64db8a74b\") " pod="openshift-ingress/e2e-ipp-a-904bbd-openshift-default-69d64c9f6-lttqf"
Sep 10 17:59:43.469501 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:43.469416    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22216a3a-3a36-4fb3-8980-4ea64db8a74b-istio-podinfo\") pod \"e2e-ipp-a-904bbd-openshift-default-69d64c9f6-lttqf\" (UID: \"22216a3a-3a36-4fb3-8980-4ea64db8a74b\") " pod="openshift-ingress/e2e-ipp-a-904bbd-openshift-default-69d64c9f6-lttqf"
Sep 10 17:59:43.469501 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:43.469443    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22216a3a-3a36-4fb3-8980-4ea64db8a74b-workload-socket\") pod \"e2e-ipp-a-904bbd-openshift-default-69d64c9f6-lttqf\" (UID: \"22216a3a-3a36-4fb3-8980-4ea64db8a74b\") " pod="openshift-ingress/e2e-ipp-a-904bbd-openshift-default-69d64c9f6-lttqf"
Sep 10 17:59:43.469501 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:43.469474    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22216a3a-3a36-4fb3-8980-4ea64db8a74b-credential-socket\") pod \"e2e-ipp-a-904bbd-openshift-default-69d64c9f6-lttqf\" (UID: \"22216a3a-3a36-4fb3-8980-4ea64db8a74b\") " pod="openshift-ingress/e2e-ipp-a-904bbd-openshift-default-69d64c9f6-lttqf"
Sep 10 17:59:43.469989 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:43.469792    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22216a3a-3a36-4fb3-8980-4ea64db8a74b-workload-certs\") pod \"e2e-ipp-a-904bbd-openshift-default-69d64c9f6-lttqf\" (UID: \"22216a3a-3a36-4fb3-8980-4ea64db8a74b\") " pod="openshift-ingress/e2e-ipp-a-904bbd-openshift-default-69d64c9f6-lttqf"
Sep 10 17:59:43.469989 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:43.469813    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22216a3a-3a36-4fb3-8980-4ea64db8a74b-credential-socket\") pod \"e2e-ipp-a-904bbd-openshift-default-69d64c9f6-lttqf\" (UID: \"22216a3a-3a36-4fb3-8980-4ea64db8a74b\") " pod="openshift-ingress/e2e-ipp-a-904bbd-openshift-default-69d64c9f6-lttqf"
Sep 10 17:59:43.470102 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:43.470019    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22216a3a-3a36-4fb3-8980-4ea64db8a74b-istio-data\") pod \"e2e-ipp-a-904bbd-openshift-default-69d64c9f6-lttqf\" (UID: \"22216a3a-3a36-4fb3-8980-4ea64db8a74b\") " pod="openshift-ingress/e2e-ipp-a-904bbd-openshift-default-69d64c9f6-lttqf"
Sep 10 17:59:43.470102 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:43.470030    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22216a3a-3a36-4fb3-8980-4ea64db8a74b-workload-socket\") pod \"e2e-ipp-a-904bbd-openshift-default-69d64c9f6-lttqf\" (UID: \"22216a3a-3a36-4fb3-8980-4ea64db8a74b\") " pod="openshift-ingress/e2e-ipp-a-904bbd-openshift-default-69d64c9f6-lttqf"
Sep 10 17:59:43.470442 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:43.470422    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22216a3a-3a36-4fb3-8980-4ea64db8a74b-istiod-ca-cert\") pod \"e2e-ipp-a-904bbd-openshift-default-69d64c9f6-lttqf\" (UID: \"22216a3a-3a36-4fb3-8980-4ea64db8a74b\") " pod="openshift-ingress/e2e-ipp-a-904bbd-openshift-default-69d64c9f6-lttqf"
Sep 10 17:59:43.471942 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:43.471913    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22216a3a-3a36-4fb3-8980-4ea64db8a74b-istio-envoy\") pod \"e2e-ipp-a-904bbd-openshift-default-69d64c9f6-lttqf\" (UID: \"22216a3a-3a36-4fb3-8980-4ea64db8a74b\") " pod="openshift-ingress/e2e-ipp-a-904bbd-openshift-default-69d64c9f6-lttqf"
Sep 10 17:59:43.472298 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:43.472278    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22216a3a-3a36-4fb3-8980-4ea64db8a74b-istio-podinfo\") pod \"e2e-ipp-a-904bbd-openshift-default-69d64c9f6-lttqf\" (UID: \"22216a3a-3a36-4fb3-8980-4ea64db8a74b\") " pod="openshift-ingress/e2e-ipp-a-904bbd-openshift-default-69d64c9f6-lttqf"
Sep 10 17:59:43.478540 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:43.478513    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22216a3a-3a36-4fb3-8980-4ea64db8a74b-istio-token\") pod \"e2e-ipp-a-904bbd-openshift-default-69d64c9f6-lttqf\" (UID: \"22216a3a-3a36-4fb3-8980-4ea64db8a74b\") " pod="openshift-ingress/e2e-ipp-a-904bbd-openshift-default-69d64c9f6-lttqf"
Sep 10 17:59:43.478863 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:43.478845    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hh58d\" (UniqueName: \"kubernetes.io/projected/22216a3a-3a36-4fb3-8980-4ea64db8a74b-kube-api-access-hh58d\") pod \"e2e-ipp-a-904bbd-openshift-default-69d64c9f6-lttqf\" (UID: \"22216a3a-3a36-4fb3-8980-4ea64db8a74b\") " pod="openshift-ingress/e2e-ipp-a-904bbd-openshift-default-69d64c9f6-lttqf"
Sep 10 17:59:43.603496 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:43.603402    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-904bbd-openshift-default-69d64c9f6-lttqf"
Sep 10 17:59:43.884166 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:43.884140    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-904bbd-openshift-default-69d64c9f6-lttqf"]
Sep 10 17:59:43.886627 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:59:43.886600    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22216a3a_3a36_4fb3_8980_4ea64db8a74b.slice/crio-d637a9b011e4e2d6448fbaefafc717d3991437d490034c982c5c93dd1bda0074 WatchSource:0}: Error finding container d637a9b011e4e2d6448fbaefafc717d3991437d490034c982c5c93dd1bda0074: Status 404 returned error can't find the container with id d637a9b011e4e2d6448fbaefafc717d3991437d490034c982c5c93dd1bda0074
Sep 10 17:59:43.888981 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:43.888946    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 10 17:59:43.889092 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:43.889024    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 10 17:59:43.889092 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:43.889051    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 10 17:59:43.950823 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:43.950790    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-904bbd-openshift-default-69d64c9f6-lttqf" event={"ID":"22216a3a-3a36-4fb3-8980-4ea64db8a74b","Type":"ContainerStarted","Data":"d637a9b011e4e2d6448fbaefafc717d3991437d490034c982c5c93dd1bda0074"}
Sep 10 17:59:44.965914 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:44.965867    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-904bbd-openshift-default-69d64c9f6-lttqf" event={"ID":"22216a3a-3a36-4fb3-8980-4ea64db8a74b","Type":"ContainerStarted","Data":"901786dd36540ff4103a97108b80afa7bc58417f57bf9734a803cedd8f8fb70f"}
Sep 10 17:59:44.989787 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:44.989743    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-904bbd-openshift-default-69d64c9f6-lttqf" podStartSLOduration=1.989728991 podStartE2EDuration="1.989728991s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:59:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:59:44.988616925 +0000 UTC m=+1428.988446995" watchObservedRunningTime="2026-09-10 17:59:44.989728991 +0000 UTC m=+1428.989559060"
Sep 10 17:59:45.481907 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:45.481835    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-abad1a-openshift-default-7969487574-l69b4"
Sep 10 17:59:45.587839 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:45.587806    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b252299-8b3e-4fd7-986e-99f19feae0e3-workload-socket\") pod \"6b252299-8b3e-4fd7-986e-99f19feae0e3\" (UID: \"6b252299-8b3e-4fd7-986e-99f19feae0e3\") "
Sep 10 17:59:45.588012 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:45.587872    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b252299-8b3e-4fd7-986e-99f19feae0e3-credential-socket\") pod \"6b252299-8b3e-4fd7-986e-99f19feae0e3\" (UID: \"6b252299-8b3e-4fd7-986e-99f19feae0e3\") "
Sep 10 17:59:45.588012 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:45.587901    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b252299-8b3e-4fd7-986e-99f19feae0e3-istio-data\") pod \"6b252299-8b3e-4fd7-986e-99f19feae0e3\" (UID: \"6b252299-8b3e-4fd7-986e-99f19feae0e3\") "
Sep 10 17:59:45.588012 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:45.587936    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b252299-8b3e-4fd7-986e-99f19feae0e3-istio-token\") pod \"6b252299-8b3e-4fd7-986e-99f19feae0e3\" (UID: \"6b252299-8b3e-4fd7-986e-99f19feae0e3\") "
Sep 10 17:59:45.588012 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:45.587967    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b252299-8b3e-4fd7-986e-99f19feae0e3-istio-envoy\") pod \"6b252299-8b3e-4fd7-986e-99f19feae0e3\" (UID: \"6b252299-8b3e-4fd7-986e-99f19feae0e3\") "
Sep 10 17:59:45.588012 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:45.588000    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b252299-8b3e-4fd7-986e-99f19feae0e3-workload-certs\") pod \"6b252299-8b3e-4fd7-986e-99f19feae0e3\" (UID: \"6b252299-8b3e-4fd7-986e-99f19feae0e3\") "
Sep 10 17:59:45.588286 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:45.588038    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lhbft\" (UniqueName: \"kubernetes.io/projected/6b252299-8b3e-4fd7-986e-99f19feae0e3-kube-api-access-lhbft\") pod \"6b252299-8b3e-4fd7-986e-99f19feae0e3\" (UID: \"6b252299-8b3e-4fd7-986e-99f19feae0e3\") "
Sep 10 17:59:45.588286 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:45.588070    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b252299-8b3e-4fd7-986e-99f19feae0e3-istiod-ca-cert\") pod \"6b252299-8b3e-4fd7-986e-99f19feae0e3\" (UID: \"6b252299-8b3e-4fd7-986e-99f19feae0e3\") "
Sep 10 17:59:45.588286 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:45.588090    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b252299-8b3e-4fd7-986e-99f19feae0e3-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "6b252299-8b3e-4fd7-986e-99f19feae0e3" (UID: "6b252299-8b3e-4fd7-986e-99f19feae0e3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:59:45.588286 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:45.588099    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b252299-8b3e-4fd7-986e-99f19feae0e3-istio-podinfo\") pod \"6b252299-8b3e-4fd7-986e-99f19feae0e3\" (UID: \"6b252299-8b3e-4fd7-986e-99f19feae0e3\") "
Sep 10 17:59:45.588286 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:45.588232    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b252299-8b3e-4fd7-986e-99f19feae0e3-istio-data" (OuterVolumeSpecName: "istio-data") pod "6b252299-8b3e-4fd7-986e-99f19feae0e3" (UID: "6b252299-8b3e-4fd7-986e-99f19feae0e3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:59:45.588286 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:45.588244    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b252299-8b3e-4fd7-986e-99f19feae0e3-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "6b252299-8b3e-4fd7-986e-99f19feae0e3" (UID: "6b252299-8b3e-4fd7-986e-99f19feae0e3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:59:45.588599 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:45.588441    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b252299-8b3e-4fd7-986e-99f19feae0e3-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "6b252299-8b3e-4fd7-986e-99f19feae0e3" (UID: "6b252299-8b3e-4fd7-986e-99f19feae0e3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:59:45.588599 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:45.588535    2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b252299-8b3e-4fd7-986e-99f19feae0e3-istio-data\") on node \"ip-10-0-129-122.ec2.internal\" DevicePath \"\""
Sep 10 17:59:45.588599 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:45.588556    2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b252299-8b3e-4fd7-986e-99f19feae0e3-workload-certs\") on node \"ip-10-0-129-122.ec2.internal\" DevicePath \"\""
Sep 10 17:59:45.588756 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:45.588614    2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b252299-8b3e-4fd7-986e-99f19feae0e3-workload-socket\") on node \"ip-10-0-129-122.ec2.internal\" DevicePath \"\""
Sep 10 17:59:45.588756 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:45.588628    2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b252299-8b3e-4fd7-986e-99f19feae0e3-credential-socket\") on node \"ip-10-0-129-122.ec2.internal\" DevicePath \"\""
Sep 10 17:59:45.588756 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:45.588743    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b252299-8b3e-4fd7-986e-99f19feae0e3-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "6b252299-8b3e-4fd7-986e-99f19feae0e3" (UID: "6b252299-8b3e-4fd7-986e-99f19feae0e3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 17:59:45.590739 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:45.590716    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6b252299-8b3e-4fd7-986e-99f19feae0e3-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "6b252299-8b3e-4fd7-986e-99f19feae0e3" (UID: "6b252299-8b3e-4fd7-986e-99f19feae0e3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 10 17:59:45.590854 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:45.590735    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b252299-8b3e-4fd7-986e-99f19feae0e3-istio-token" (OuterVolumeSpecName: "istio-token") pod "6b252299-8b3e-4fd7-986e-99f19feae0e3" (UID: "6b252299-8b3e-4fd7-986e-99f19feae0e3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:59:45.590854 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:45.590803    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b252299-8b3e-4fd7-986e-99f19feae0e3-kube-api-access-lhbft" (OuterVolumeSpecName: "kube-api-access-lhbft") pod "6b252299-8b3e-4fd7-986e-99f19feae0e3" (UID: "6b252299-8b3e-4fd7-986e-99f19feae0e3"). InnerVolumeSpecName "kube-api-access-lhbft". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:59:45.591005 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:45.590986    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b252299-8b3e-4fd7-986e-99f19feae0e3-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "6b252299-8b3e-4fd7-986e-99f19feae0e3" (UID: "6b252299-8b3e-4fd7-986e-99f19feae0e3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:59:45.604236 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:45.604209    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-904bbd-openshift-default-69d64c9f6-lttqf"
Sep 10 17:59:45.609540 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:45.609522    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-904bbd-openshift-default-69d64c9f6-lttqf"
Sep 10 17:59:45.689721 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:45.689688    2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b252299-8b3e-4fd7-986e-99f19feae0e3-istio-token\") on node \"ip-10-0-129-122.ec2.internal\" DevicePath \"\""
Sep 10 17:59:45.689721 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:45.689717    2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b252299-8b3e-4fd7-986e-99f19feae0e3-istio-envoy\") on node \"ip-10-0-129-122.ec2.internal\" DevicePath \"\""
Sep 10 17:59:45.689721 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:45.689726    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lhbft\" (UniqueName: \"kubernetes.io/projected/6b252299-8b3e-4fd7-986e-99f19feae0e3-kube-api-access-lhbft\") on node \"ip-10-0-129-122.ec2.internal\" DevicePath \"\""
Sep 10 17:59:45.689940 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:45.689735    2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b252299-8b3e-4fd7-986e-99f19feae0e3-istiod-ca-cert\") on node \"ip-10-0-129-122.ec2.internal\" DevicePath \"\""
Sep 10 17:59:45.689940 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:45.689744    2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b252299-8b3e-4fd7-986e-99f19feae0e3-istio-podinfo\") on node \"ip-10-0-129-122.ec2.internal\" DevicePath \"\""
Sep 10 17:59:45.972256 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:45.972221    2579 generic.go:356] "Generic (PLEG): container finished" podID="6b252299-8b3e-4fd7-986e-99f19feae0e3" containerID="72f73be54d9b32da69de161708190152cb9d81380b2726cb4c51522a88a7b3a6" exitCode=0
Sep 10 17:59:45.972759 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:45.972278    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-abad1a-openshift-default-7969487574-l69b4"
Sep 10 17:59:45.972759 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:45.972313    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-abad1a-openshift-default-7969487574-l69b4" event={"ID":"6b252299-8b3e-4fd7-986e-99f19feae0e3","Type":"ContainerDied","Data":"72f73be54d9b32da69de161708190152cb9d81380b2726cb4c51522a88a7b3a6"}
Sep 10 17:59:45.972759 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:45.972372    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-abad1a-openshift-default-7969487574-l69b4" event={"ID":"6b252299-8b3e-4fd7-986e-99f19feae0e3","Type":"ContainerDied","Data":"2c074e93b961cccb55c870539b55712d7d33810652897d468c81249335aa1195"}
Sep 10 17:59:45.972759 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:45.972398    2579 scope.go:117] "RemoveContainer" containerID="72f73be54d9b32da69de161708190152cb9d81380b2726cb4c51522a88a7b3a6"
Sep 10 17:59:45.972984 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:45.972764    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-904bbd-openshift-default-69d64c9f6-lttqf"
Sep 10 17:59:45.973808 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:45.973793    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-904bbd-openshift-default-69d64c9f6-lttqf"
Sep 10 17:59:45.982143 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:45.982123    2579 scope.go:117] "RemoveContainer" containerID="15dd723eed76fa41f52e17247186aaa28000d84d0d7aa4e023f0aff341aec39a"
Sep 10 17:59:45.990468 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:45.990453    2579 scope.go:117] "RemoveContainer" containerID="72f73be54d9b32da69de161708190152cb9d81380b2726cb4c51522a88a7b3a6"
Sep 10 17:59:45.990767 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:59:45.990747    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72f73be54d9b32da69de161708190152cb9d81380b2726cb4c51522a88a7b3a6\": container with ID starting with 72f73be54d9b32da69de161708190152cb9d81380b2726cb4c51522a88a7b3a6 not found: ID does not exist" containerID="72f73be54d9b32da69de161708190152cb9d81380b2726cb4c51522a88a7b3a6"
Sep 10 17:59:45.990850 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:45.990779    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72f73be54d9b32da69de161708190152cb9d81380b2726cb4c51522a88a7b3a6"} err="failed to get container status \"72f73be54d9b32da69de161708190152cb9d81380b2726cb4c51522a88a7b3a6\": rpc error: code = NotFound desc = could not find container \"72f73be54d9b32da69de161708190152cb9d81380b2726cb4c51522a88a7b3a6\": container with ID starting with 72f73be54d9b32da69de161708190152cb9d81380b2726cb4c51522a88a7b3a6 not found: ID does not exist"
Sep 10 17:59:45.990850 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:45.990803    2579 scope.go:117] "RemoveContainer" containerID="15dd723eed76fa41f52e17247186aaa28000d84d0d7aa4e023f0aff341aec39a"
Sep 10 17:59:45.991103 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:59:45.991078    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15dd723eed76fa41f52e17247186aaa28000d84d0d7aa4e023f0aff341aec39a\": container with ID starting with 15dd723eed76fa41f52e17247186aaa28000d84d0d7aa4e023f0aff341aec39a not found: ID does not exist" containerID="15dd723eed76fa41f52e17247186aaa28000d84d0d7aa4e023f0aff341aec39a"
Sep 10 17:59:45.991143 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:45.991115    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15dd723eed76fa41f52e17247186aaa28000d84d0d7aa4e023f0aff341aec39a"} err="failed to get container status \"15dd723eed76fa41f52e17247186aaa28000d84d0d7aa4e023f0aff341aec39a\": rpc error: code = NotFound desc = could not find container \"15dd723eed76fa41f52e17247186aaa28000d84d0d7aa4e023f0aff341aec39a\": container with ID starting with 15dd723eed76fa41f52e17247186aaa28000d84d0d7aa4e023f0aff341aec39a not found: ID does not exist"
Sep 10 17:59:46.029680 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:46.029649    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-abad1a-openshift-default-7969487574-l69b4"]
Sep 10 17:59:46.035587 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:46.035539    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-abad1a-openshift-default-7969487574-l69b4"]
Sep 10 17:59:46.477904 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:46.477859    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6b252299-8b3e-4fd7-986e-99f19feae0e3" path="/var/lib/kubelet/pods/6b252299-8b3e-4fd7-986e-99f19feae0e3/volumes"
Sep 10 17:59:48.136140 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:48.136053    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-904bbd-64f46b78d5-jm2tt"]
Sep 10 17:59:48.136691 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:48.136672    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b252299-8b3e-4fd7-986e-99f19feae0e3" containerName="istio-proxy"
Sep 10 17:59:48.136780 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:48.136694    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b252299-8b3e-4fd7-986e-99f19feae0e3" containerName="istio-proxy"
Sep 10 17:59:48.136780 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:48.136720    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b252299-8b3e-4fd7-986e-99f19feae0e3" containerName="istio-proxy"
Sep 10 17:59:48.136780 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:48.136728    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b252299-8b3e-4fd7-986e-99f19feae0e3" containerName="istio-proxy"
Sep 10 17:59:48.136780 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:48.136747    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="013e342b-e2dc-4034-b58f-ea9bafb7b379" containerName="istio-proxy"
Sep 10 17:59:48.136780 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:48.136755    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="013e342b-e2dc-4034-b58f-ea9bafb7b379" containerName="istio-proxy"
Sep 10 17:59:48.137033 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:48.136855    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="6b252299-8b3e-4fd7-986e-99f19feae0e3" containerName="istio-proxy"
Sep 10 17:59:48.137033 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:48.136871    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="6b252299-8b3e-4fd7-986e-99f19feae0e3" containerName="istio-proxy"
Sep 10 17:59:48.141406 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:48.141379    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-904bbd-64f46b78d5-jm2tt"
Sep 10 17:59:48.144141 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:48.144117    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-904bbd-serving-cert\""
Sep 10 17:59:48.155000 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:48.154976    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-904bbd-64f46b78d5-jm2tt"]
Sep 10 17:59:48.209196 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:48.209163    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e729c2e6-5480-4d68-9b10-20731810e30b-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-904bbd-64f46b78d5-jm2tt\" (UID: \"e729c2e6-5480-4d68-9b10-20731810e30b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-904bbd-64f46b78d5-jm2tt"
Sep 10 17:59:48.209348 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:48.209207    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e729c2e6-5480-4d68-9b10-20731810e30b-metrics-certs\") pod \"maas-api-e2e-ipp-a-904bbd-64f46b78d5-jm2tt\" (UID: \"e729c2e6-5480-4d68-9b10-20731810e30b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-904bbd-64f46b78d5-jm2tt"
Sep 10 17:59:48.209348 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:48.209248    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e729c2e6-5480-4d68-9b10-20731810e30b-maas-api-tls\") pod \"maas-api-e2e-ipp-a-904bbd-64f46b78d5-jm2tt\" (UID: \"e729c2e6-5480-4d68-9b10-20731810e30b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-904bbd-64f46b78d5-jm2tt"
Sep 10 17:59:48.209348 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:48.209325    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5rcc\" (UniqueName: \"kubernetes.io/projected/e729c2e6-5480-4d68-9b10-20731810e30b-kube-api-access-f5rcc\") pod \"maas-api-e2e-ipp-a-904bbd-64f46b78d5-jm2tt\" (UID: \"e729c2e6-5480-4d68-9b10-20731810e30b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-904bbd-64f46b78d5-jm2tt"
Sep 10 17:59:48.310857 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:48.310819    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5rcc\" (UniqueName: \"kubernetes.io/projected/e729c2e6-5480-4d68-9b10-20731810e30b-kube-api-access-f5rcc\") pod \"maas-api-e2e-ipp-a-904bbd-64f46b78d5-jm2tt\" (UID: \"e729c2e6-5480-4d68-9b10-20731810e30b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-904bbd-64f46b78d5-jm2tt"
Sep 10 17:59:48.311056 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:48.310900    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e729c2e6-5480-4d68-9b10-20731810e30b-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-904bbd-64f46b78d5-jm2tt\" (UID: \"e729c2e6-5480-4d68-9b10-20731810e30b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-904bbd-64f46b78d5-jm2tt"
Sep 10 17:59:48.311056 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:48.310939    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e729c2e6-5480-4d68-9b10-20731810e30b-metrics-certs\") pod \"maas-api-e2e-ipp-a-904bbd-64f46b78d5-jm2tt\" (UID: \"e729c2e6-5480-4d68-9b10-20731810e30b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-904bbd-64f46b78d5-jm2tt"
Sep 10 17:59:48.311056 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:48.310996    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e729c2e6-5480-4d68-9b10-20731810e30b-maas-api-tls\") pod \"maas-api-e2e-ipp-a-904bbd-64f46b78d5-jm2tt\" (UID: \"e729c2e6-5480-4d68-9b10-20731810e30b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-904bbd-64f46b78d5-jm2tt"
Sep 10 17:59:48.311224 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:59:48.311107    2579 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-904bbd-serving-cert: secret "maas-api-e2e-ipp-a-904bbd-serving-cert" not found
Sep 10 17:59:48.311224 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:59:48.311187    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e729c2e6-5480-4d68-9b10-20731810e30b-maas-api-tls podName:e729c2e6-5480-4d68-9b10-20731810e30b nodeName:}" failed. No retries permitted until 2026-09-10 17:59:48.811157894 +0000 UTC m=+1432.810987943 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e729c2e6-5480-4d68-9b10-20731810e30b-maas-api-tls") pod "maas-api-e2e-ipp-a-904bbd-64f46b78d5-jm2tt" (UID: "e729c2e6-5480-4d68-9b10-20731810e30b") : secret "maas-api-e2e-ipp-a-904bbd-serving-cert" not found
Sep 10 17:59:48.311508 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:48.311486    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e729c2e6-5480-4d68-9b10-20731810e30b-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-904bbd-64f46b78d5-jm2tt\" (UID: \"e729c2e6-5480-4d68-9b10-20731810e30b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-904bbd-64f46b78d5-jm2tt"
Sep 10 17:59:48.313592 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:48.313554    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e729c2e6-5480-4d68-9b10-20731810e30b-metrics-certs\") pod \"maas-api-e2e-ipp-a-904bbd-64f46b78d5-jm2tt\" (UID: \"e729c2e6-5480-4d68-9b10-20731810e30b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-904bbd-64f46b78d5-jm2tt"
Sep 10 17:59:48.324041 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:48.324013    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5rcc\" (UniqueName: \"kubernetes.io/projected/e729c2e6-5480-4d68-9b10-20731810e30b-kube-api-access-f5rcc\") pod \"maas-api-e2e-ipp-a-904bbd-64f46b78d5-jm2tt\" (UID: \"e729c2e6-5480-4d68-9b10-20731810e30b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-904bbd-64f46b78d5-jm2tt"
Sep 10 17:59:48.815028 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:48.814995    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e729c2e6-5480-4d68-9b10-20731810e30b-maas-api-tls\") pod \"maas-api-e2e-ipp-a-904bbd-64f46b78d5-jm2tt\" (UID: \"e729c2e6-5480-4d68-9b10-20731810e30b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-904bbd-64f46b78d5-jm2tt"
Sep 10 17:59:48.817496 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:48.817468    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e729c2e6-5480-4d68-9b10-20731810e30b-maas-api-tls\") pod \"maas-api-e2e-ipp-a-904bbd-64f46b78d5-jm2tt\" (UID: \"e729c2e6-5480-4d68-9b10-20731810e30b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-904bbd-64f46b78d5-jm2tt"
Sep 10 17:59:48.927161 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:48.927132    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-904bbd-84d8556b86-xl85m"]
Sep 10 17:59:48.930963 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:48.930945    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-904bbd-84d8556b86-xl85m"
Sep 10 17:59:48.933750 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:48.933728    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-904bbd\""
Sep 10 17:59:48.933943 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:48.933908    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-904bbd-dockercfg-b5n5r\""
Sep 10 17:59:48.948276 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:48.946359    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-904bbd-84d8556b86-xl85m"]
Sep 10 17:59:49.016644 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:49.016600    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3bd402f5-7e0b-4f92-a2b1-f007bfc8633b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-904bbd-84d8556b86-xl85m\" (UID: \"3bd402f5-7e0b-4f92-a2b1-f007bfc8633b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-904bbd-84d8556b86-xl85m"
Sep 10 17:59:49.016864 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:49.016651    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6j6ts\" (UniqueName: \"kubernetes.io/projected/3bd402f5-7e0b-4f92-a2b1-f007bfc8633b-kube-api-access-6j6ts\") pod \"payload-pre-processing-e2e-ipp-a-904bbd-84d8556b86-xl85m\" (UID: \"3bd402f5-7e0b-4f92-a2b1-f007bfc8633b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-904bbd-84d8556b86-xl85m"
Sep 10 17:59:49.056069 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:49.056039    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-904bbd-64f46b78d5-jm2tt"
Sep 10 17:59:49.119181 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:49.118367    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3bd402f5-7e0b-4f92-a2b1-f007bfc8633b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-904bbd-84d8556b86-xl85m\" (UID: \"3bd402f5-7e0b-4f92-a2b1-f007bfc8633b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-904bbd-84d8556b86-xl85m"
Sep 10 17:59:49.119181 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:49.118429    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6j6ts\" (UniqueName: \"kubernetes.io/projected/3bd402f5-7e0b-4f92-a2b1-f007bfc8633b-kube-api-access-6j6ts\") pod \"payload-pre-processing-e2e-ipp-a-904bbd-84d8556b86-xl85m\" (UID: \"3bd402f5-7e0b-4f92-a2b1-f007bfc8633b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-904bbd-84d8556b86-xl85m"
Sep 10 17:59:49.119417 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:49.119393    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3bd402f5-7e0b-4f92-a2b1-f007bfc8633b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-904bbd-84d8556b86-xl85m\" (UID: \"3bd402f5-7e0b-4f92-a2b1-f007bfc8633b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-904bbd-84d8556b86-xl85m"
Sep 10 17:59:49.127222 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:49.127176    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6j6ts\" (UniqueName: \"kubernetes.io/projected/3bd402f5-7e0b-4f92-a2b1-f007bfc8633b-kube-api-access-6j6ts\") pod \"payload-pre-processing-e2e-ipp-a-904bbd-84d8556b86-xl85m\" (UID: \"3bd402f5-7e0b-4f92-a2b1-f007bfc8633b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-904bbd-84d8556b86-xl85m"
Sep 10 17:59:49.243720 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:49.243680    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-904bbd-84d8556b86-xl85m"
Sep 10 17:59:49.369795 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:49.369712    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-904bbd-64f46b78d5-jm2tt"]
Sep 10 17:59:49.378290 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:59:49.378264    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode729c2e6_5480_4d68_9b10_20731810e30b.slice/crio-9c579f231d33596c13084f3191066834f3f4f12cb08306024b8d237bf303e177 WatchSource:0}: Error finding container 9c579f231d33596c13084f3191066834f3f4f12cb08306024b8d237bf303e177: Status 404 returned error can't find the container with id 9c579f231d33596c13084f3191066834f3f4f12cb08306024b8d237bf303e177
Sep 10 17:59:49.379700 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:49.379677    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 17:59:49.554334 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:49.554308    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-904bbd-84d8556b86-xl85m"]
Sep 10 17:59:49.555528 ip-10-0-129-122 kubenswrapper[2579]: W0910 17:59:49.555501    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bd402f5_7e0b_4f92_a2b1_f007bfc8633b.slice/crio-44907aaf4ed02e0081f517dd98bb901a68decd0b388c6bb9d98b7c2eeff4c677 WatchSource:0}: Error finding container 44907aaf4ed02e0081f517dd98bb901a68decd0b388c6bb9d98b7c2eeff4c677: Status 404 returned error can't find the container with id 44907aaf4ed02e0081f517dd98bb901a68decd0b388c6bb9d98b7c2eeff4c677
Sep 10 17:59:49.999430 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:49.999392    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-904bbd-64f46b78d5-jm2tt" event={"ID":"e729c2e6-5480-4d68-9b10-20731810e30b","Type":"ContainerStarted","Data":"3ef43a3939c5a40164ea49eb9b0b923b6a30bbf708118b9b95b28378ade7fa9e"}
Sep 10 17:59:49.999642 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:49.999468    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-904bbd-64f46b78d5-jm2tt"
Sep 10 17:59:49.999642 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:49.999491    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-904bbd-64f46b78d5-jm2tt" event={"ID":"e729c2e6-5480-4d68-9b10-20731810e30b","Type":"ContainerStarted","Data":"9c579f231d33596c13084f3191066834f3f4f12cb08306024b8d237bf303e177"}
Sep 10 17:59:50.001208 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:50.001180    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-904bbd-84d8556b86-xl85m" event={"ID":"3bd402f5-7e0b-4f92-a2b1-f007bfc8633b","Type":"ContainerStarted","Data":"974f40dd8519569608b7d8132dc1894db08a7e78643bff6214ac060a64af364d"}
Sep 10 17:59:50.001338 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:50.001222    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-904bbd-84d8556b86-xl85m" event={"ID":"3bd402f5-7e0b-4f92-a2b1-f007bfc8633b","Type":"ContainerStarted","Data":"44907aaf4ed02e0081f517dd98bb901a68decd0b388c6bb9d98b7c2eeff4c677"}
Sep 10 17:59:50.001338 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:50.001243    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-904bbd-84d8556b86-xl85m"
Sep 10 17:59:50.022779 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:50.022730    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-904bbd-64f46b78d5-jm2tt" podStartSLOduration=1.626674182 podStartE2EDuration="2.022715335s" totalImagesPullingTime="396.041153ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:59:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:59:50.022126165 +0000 UTC m=+1434.021956234" watchObservedRunningTime="2026-09-10 17:59:50.022715335 +0000 UTC m=+1434.022545403"
Sep 10 17:59:50.046311 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:50.046269    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-904bbd-84d8556b86-xl85m" podStartSLOduration=2.046254928 podStartE2EDuration="2.046254928s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:59:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:59:50.04451015 +0000 UTC m=+1434.044340229" watchObservedRunningTime="2026-09-10 17:59:50.046254928 +0000 UTC m=+1434.046085028"
Sep 10 17:59:52.164613 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:52.164544    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-9391d2-78f89686b8-n24kz"]
Sep 10 17:59:52.165080 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:52.164855    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-9391d2-78f89686b8-n24kz" podUID="15a7e05e-738a-4789-a718-e8f7c37b6487" containerName="maas-api" containerID="cri-o://b61278981094fe776741fb1ee4a9b6f294fb0e2e39a9cf00533954dfb3977156" gracePeriod=30
Sep 10 17:59:52.420940 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:52.420876    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-9391d2-78f89686b8-n24kz"
Sep 10 17:59:52.448303 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:52.448257    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/15a7e05e-738a-4789-a718-e8f7c37b6487-openshift-service-ca\") pod \"15a7e05e-738a-4789-a718-e8f7c37b6487\" (UID: \"15a7e05e-738a-4789-a718-e8f7c37b6487\") "
Sep 10 17:59:52.448512 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:52.448316    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/15a7e05e-738a-4789-a718-e8f7c37b6487-maas-api-tls\") pod \"15a7e05e-738a-4789-a718-e8f7c37b6487\" (UID: \"15a7e05e-738a-4789-a718-e8f7c37b6487\") "
Sep 10 17:59:52.448512 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:52.448385    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bpf4m\" (UniqueName: \"kubernetes.io/projected/15a7e05e-738a-4789-a718-e8f7c37b6487-kube-api-access-bpf4m\") pod \"15a7e05e-738a-4789-a718-e8f7c37b6487\" (UID: \"15a7e05e-738a-4789-a718-e8f7c37b6487\") "
Sep 10 17:59:52.448512 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:52.448489    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/15a7e05e-738a-4789-a718-e8f7c37b6487-metrics-certs\") pod \"15a7e05e-738a-4789-a718-e8f7c37b6487\" (UID: \"15a7e05e-738a-4789-a718-e8f7c37b6487\") "
Sep 10 17:59:52.448825 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:52.448731    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15a7e05e-738a-4789-a718-e8f7c37b6487-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "15a7e05e-738a-4789-a718-e8f7c37b6487" (UID: "15a7e05e-738a-4789-a718-e8f7c37b6487"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 17:59:52.448955 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:52.448885    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/15a7e05e-738a-4789-a718-e8f7c37b6487-openshift-service-ca\") on node \"ip-10-0-129-122.ec2.internal\" DevicePath \"\""
Sep 10 17:59:52.451641 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:52.451602    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/15a7e05e-738a-4789-a718-e8f7c37b6487-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "15a7e05e-738a-4789-a718-e8f7c37b6487" (UID: "15a7e05e-738a-4789-a718-e8f7c37b6487"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 17:59:52.451778 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:52.451747    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15a7e05e-738a-4789-a718-e8f7c37b6487-kube-api-access-bpf4m" (OuterVolumeSpecName: "kube-api-access-bpf4m") pod "15a7e05e-738a-4789-a718-e8f7c37b6487" (UID: "15a7e05e-738a-4789-a718-e8f7c37b6487"). InnerVolumeSpecName "kube-api-access-bpf4m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:59:52.452419 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:52.452398    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/15a7e05e-738a-4789-a718-e8f7c37b6487-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "15a7e05e-738a-4789-a718-e8f7c37b6487" (UID: "15a7e05e-738a-4789-a718-e8f7c37b6487"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 17:59:52.550281 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:52.550244    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/15a7e05e-738a-4789-a718-e8f7c37b6487-maas-api-tls\") on node \"ip-10-0-129-122.ec2.internal\" DevicePath \"\""
Sep 10 17:59:52.550281 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:52.550280    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bpf4m\" (UniqueName: \"kubernetes.io/projected/15a7e05e-738a-4789-a718-e8f7c37b6487-kube-api-access-bpf4m\") on node \"ip-10-0-129-122.ec2.internal\" DevicePath \"\""
Sep 10 17:59:52.550281 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:52.550291    2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/15a7e05e-738a-4789-a718-e8f7c37b6487-metrics-certs\") on node \"ip-10-0-129-122.ec2.internal\" DevicePath \"\""
Sep 10 17:59:53.018922 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:53.018883    2579 generic.go:356] "Generic (PLEG): container finished" podID="15a7e05e-738a-4789-a718-e8f7c37b6487" containerID="b61278981094fe776741fb1ee4a9b6f294fb0e2e39a9cf00533954dfb3977156" exitCode=0
Sep 10 17:59:53.019109 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:53.018931    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-9391d2-78f89686b8-n24kz"
Sep 10 17:59:53.019109 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:53.018967    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-9391d2-78f89686b8-n24kz" event={"ID":"15a7e05e-738a-4789-a718-e8f7c37b6487","Type":"ContainerDied","Data":"b61278981094fe776741fb1ee4a9b6f294fb0e2e39a9cf00533954dfb3977156"}
Sep 10 17:59:53.019109 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:53.019022    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-9391d2-78f89686b8-n24kz" event={"ID":"15a7e05e-738a-4789-a718-e8f7c37b6487","Type":"ContainerDied","Data":"95a482b6c1b585e6f382618162e39f07adb2dfb0f8eb10cb287272efe453b7e1"}
Sep 10 17:59:53.019109 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:53.019045    2579 scope.go:117] "RemoveContainer" containerID="b61278981094fe776741fb1ee4a9b6f294fb0e2e39a9cf00533954dfb3977156"
Sep 10 17:59:53.028003 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:53.027986    2579 scope.go:117] "RemoveContainer" containerID="b61278981094fe776741fb1ee4a9b6f294fb0e2e39a9cf00533954dfb3977156"
Sep 10 17:59:53.028244 ip-10-0-129-122 kubenswrapper[2579]: E0910 17:59:53.028223    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b61278981094fe776741fb1ee4a9b6f294fb0e2e39a9cf00533954dfb3977156\": container with ID starting with b61278981094fe776741fb1ee4a9b6f294fb0e2e39a9cf00533954dfb3977156 not found: ID does not exist" containerID="b61278981094fe776741fb1ee4a9b6f294fb0e2e39a9cf00533954dfb3977156"
Sep 10 17:59:53.028300 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:53.028254    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b61278981094fe776741fb1ee4a9b6f294fb0e2e39a9cf00533954dfb3977156"} err="failed to get container status \"b61278981094fe776741fb1ee4a9b6f294fb0e2e39a9cf00533954dfb3977156\": rpc error: code = NotFound desc = could not find container \"b61278981094fe776741fb1ee4a9b6f294fb0e2e39a9cf00533954dfb3977156\": container with ID starting with b61278981094fe776741fb1ee4a9b6f294fb0e2e39a9cf00533954dfb3977156 not found: ID does not exist"
Sep 10 17:59:53.042949 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:53.042921    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-9391d2-78f89686b8-n24kz"]
Sep 10 17:59:53.046279 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:53.046261    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-9391d2-78f89686b8-n24kz"]
Sep 10 17:59:54.475317 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:54.475278    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="15a7e05e-738a-4789-a718-e8f7c37b6487" path="/var/lib/kubelet/pods/15a7e05e-738a-4789-a718-e8f7c37b6487/volumes"
Sep 10 17:59:56.012521 ip-10-0-129-122 kubenswrapper[2579]: I0910 17:59:56.012491    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-904bbd-64f46b78d5-jm2tt"
Sep 10 18:00:01.009998 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:00:01.009967    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-904bbd-84d8556b86-xl85m"
Sep 10 18:00:06.089341 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:00:06.089305    2579 generic.go:356] "Generic (PLEG): container finished" podID="22216a3a-3a36-4fb3-8980-4ea64db8a74b" containerID="901786dd36540ff4103a97108b80afa7bc58417f57bf9734a803cedd8f8fb70f" exitCode=0
Sep 10 18:00:06.089798 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:00:06.089361    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-904bbd-openshift-default-69d64c9f6-lttqf" event={"ID":"22216a3a-3a36-4fb3-8980-4ea64db8a74b","Type":"ContainerDied","Data":"901786dd36540ff4103a97108b80afa7bc58417f57bf9734a803cedd8f8fb70f"}
Sep 10 18:00:06.089798 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:00:06.089741    2579 scope.go:117] "RemoveContainer" containerID="901786dd36540ff4103a97108b80afa7bc58417f57bf9734a803cedd8f8fb70f"
Sep 10 18:00:06.090865 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:00:06.090832    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 10 18:00:06.090967 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:00:06.090916    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 10 18:00:06.091030 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:00:06.090966    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 10 18:00:07.099123 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:00:07.099090    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-904bbd-openshift-default-69d64c9f6-lttqf" event={"ID":"22216a3a-3a36-4fb3-8980-4ea64db8a74b","Type":"ContainerStarted","Data":"50379655e1e27d59decaf8249b08c7354f2887ee1680bbcd152f12ab9992e837"}
Sep 10 18:00:07.603840 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:00:07.603796    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-904bbd-openshift-default-69d64c9f6-lttqf"
Sep 10 18:00:07.604091 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:00:07.604066    2579 patch_prober.go:28] interesting pod/e2e-ipp-a-904bbd-openshift-default-69d64c9f6-lttqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body=
Sep 10 18:00:07.604177 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:00:07.604123    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-904bbd-openshift-default-69d64c9f6-lttqf" podUID="22216a3a-3a36-4fb3-8980-4ea64db8a74b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused"
Sep 10 18:00:08.604875 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:00:08.604835    2579 patch_prober.go:28] interesting pod/e2e-ipp-a-904bbd-openshift-default-69d64c9f6-lttqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body=
Sep 10 18:00:08.605266 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:00:08.604893    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-904bbd-openshift-default-69d64c9f6-lttqf" podUID="22216a3a-3a36-4fb3-8980-4ea64db8a74b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused"
Sep 10 18:00:09.604714 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:00:09.604669    2579 patch_prober.go:28] interesting pod/e2e-ipp-a-904bbd-openshift-default-69d64c9f6-lttqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body=
Sep 10 18:00:09.604909 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:00:09.604734    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-904bbd-openshift-default-69d64c9f6-lttqf" podUID="22216a3a-3a36-4fb3-8980-4ea64db8a74b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused"
Sep 10 18:00:09.834414 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:00:09.834355    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-bdf47c-d85f68558-r2bt5"]
Sep 10 18:00:09.834672 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:00:09.834642    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-bdf47c-d85f68558-r2bt5" podUID="6e3058bf-33b7-4c50-81a3-b99147b0ba0f" containerName="payload-pre-processing" containerID="cri-o://24c8d5ddb3058a9e682a9c493b3454753a45637320a1ecac12598210a0f1b131" gracePeriod=30
Sep 10 18:00:10.115475 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:00:10.115389    2579 generic.go:356] "Generic (PLEG): container finished" podID="6e3058bf-33b7-4c50-81a3-b99147b0ba0f" containerID="24c8d5ddb3058a9e682a9c493b3454753a45637320a1ecac12598210a0f1b131" exitCode=0
Sep 10 18:00:10.115475 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:00:10.115456    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-bdf47c-d85f68558-r2bt5" event={"ID":"6e3058bf-33b7-4c50-81a3-b99147b0ba0f","Type":"ContainerDied","Data":"24c8d5ddb3058a9e682a9c493b3454753a45637320a1ecac12598210a0f1b131"}
Sep 10 18:00:10.145537 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:00:10.145506    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-bdf47c-d85f68558-r2bt5"
Sep 10 18:00:10.226276 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:00:10.226240    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e3058bf-33b7-4c50-81a3-b99147b0ba0f-plugins-config-volume\") pod \"6e3058bf-33b7-4c50-81a3-b99147b0ba0f\" (UID: \"6e3058bf-33b7-4c50-81a3-b99147b0ba0f\") "
Sep 10 18:00:10.226477 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:00:10.226346    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bg95w\" (UniqueName: \"kubernetes.io/projected/6e3058bf-33b7-4c50-81a3-b99147b0ba0f-kube-api-access-bg95w\") pod \"6e3058bf-33b7-4c50-81a3-b99147b0ba0f\" (UID: \"6e3058bf-33b7-4c50-81a3-b99147b0ba0f\") "
Sep 10 18:00:10.226699 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:00:10.226674    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e3058bf-33b7-4c50-81a3-b99147b0ba0f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6e3058bf-33b7-4c50-81a3-b99147b0ba0f" (UID: "6e3058bf-33b7-4c50-81a3-b99147b0ba0f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 18:00:10.228635 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:00:10.228613    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e3058bf-33b7-4c50-81a3-b99147b0ba0f-kube-api-access-bg95w" (OuterVolumeSpecName: "kube-api-access-bg95w") pod "6e3058bf-33b7-4c50-81a3-b99147b0ba0f" (UID: "6e3058bf-33b7-4c50-81a3-b99147b0ba0f"). InnerVolumeSpecName "kube-api-access-bg95w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 18:00:10.327554 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:00:10.327509    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bg95w\" (UniqueName: \"kubernetes.io/projected/6e3058bf-33b7-4c50-81a3-b99147b0ba0f-kube-api-access-bg95w\") on node \"ip-10-0-129-122.ec2.internal\" DevicePath \"\""
Sep 10 18:00:10.327554 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:00:10.327546    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e3058bf-33b7-4c50-81a3-b99147b0ba0f-plugins-config-volume\") on node \"ip-10-0-129-122.ec2.internal\" DevicePath \"\""
Sep 10 18:00:10.604883 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:00:10.604844    2579 patch_prober.go:28] interesting pod/e2e-ipp-a-904bbd-openshift-default-69d64c9f6-lttqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body=
Sep 10 18:00:10.605070 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:00:10.604904    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-904bbd-openshift-default-69d64c9f6-lttqf" podUID="22216a3a-3a36-4fb3-8980-4ea64db8a74b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused"
Sep 10 18:00:11.121905 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:00:11.121866    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-bdf47c-d85f68558-r2bt5"
Sep 10 18:00:11.122091 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:00:11.121922    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-bdf47c-d85f68558-r2bt5" event={"ID":"6e3058bf-33b7-4c50-81a3-b99147b0ba0f","Type":"ContainerDied","Data":"da52ae2c2d529747a292f5ce6e09afd40e4be59cca45f9b58efd95a9e42d153b"}
Sep 10 18:00:11.122091 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:00:11.121977    2579 scope.go:117] "RemoveContainer" containerID="24c8d5ddb3058a9e682a9c493b3454753a45637320a1ecac12598210a0f1b131"
Sep 10 18:00:11.144234 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:00:11.144200    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-bdf47c-d85f68558-r2bt5"]
Sep 10 18:00:11.148703 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:00:11.148679    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-bdf47c-d85f68558-r2bt5"]
Sep 10 18:00:11.604030 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:00:11.603996    2579 patch_prober.go:28] interesting pod/e2e-ipp-a-904bbd-openshift-default-69d64c9f6-lttqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body=
Sep 10 18:00:11.604212 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:00:11.604052    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-904bbd-openshift-default-69d64c9f6-lttqf" podUID="22216a3a-3a36-4fb3-8980-4ea64db8a74b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused"
Sep 10 18:00:12.476653 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:00:12.476617    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6e3058bf-33b7-4c50-81a3-b99147b0ba0f" path="/var/lib/kubelet/pods/6e3058bf-33b7-4c50-81a3-b99147b0ba0f/volumes"
Sep 10 18:00:12.608198 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:00:12.608168    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-904bbd-openshift-default-69d64c9f6-lttqf"
Sep 10 18:00:12.608458 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:00:12.608436    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-904bbd-openshift-default-69d64c9f6-lttqf"
Sep 10 18:00:12.609125 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:00:12.609109    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-904bbd-openshift-default-69d64c9f6-lttqf"
Sep 10 18:00:45.724613 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:00:45.724583    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-kpwl9_fcfe8a95-b4b5-4997-8d4b-717a1f3d8149/payload-processing/0.log"
Sep 10 18:00:56.526662 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:00:56.526638    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-rb854_97172b14-b84a-476e-89f1-05d3e70df416/console-operator/1.log"
Sep 10 18:00:56.531012 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:00:56.530986    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-rb854_97172b14-b84a-476e-89f1-05d3e70df416/console-operator/1.log"
Sep 10 18:01:28.857241 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:01:28.857146    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-kpwl9_fcfe8a95-b4b5-4997-8d4b-717a1f3d8149/payload-processing/0.log"
Sep 10 18:01:29.576538 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:01:29.576505    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-05a81f-openshift-default-7584465ddb-kktsm"]
Sep 10 18:01:29.577033 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:01:29.577015    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="15a7e05e-738a-4789-a718-e8f7c37b6487" containerName="maas-api"
Sep 10 18:01:29.577127 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:01:29.577036    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="15a7e05e-738a-4789-a718-e8f7c37b6487" containerName="maas-api"
Sep 10 18:01:29.577127 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:01:29.577056    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e3058bf-33b7-4c50-81a3-b99147b0ba0f" containerName="payload-pre-processing"
Sep 10 18:01:29.577127 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:01:29.577063    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e3058bf-33b7-4c50-81a3-b99147b0ba0f" containerName="payload-pre-processing"
Sep 10 18:01:29.577286 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:01:29.577146    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e3058bf-33b7-4c50-81a3-b99147b0ba0f" containerName="payload-pre-processing"
Sep 10 18:01:29.577286 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:01:29.577164    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="15a7e05e-738a-4789-a718-e8f7c37b6487" containerName="maas-api"
Sep 10 18:01:29.579268 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:01:29.579249    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-05a81f-openshift-default-7584465ddb-kktsm"
Sep 10 18:01:29.582166 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:01:29.582146    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-cleanup-05a81f-openshift-default-dockercfg-x4fqz\""
Sep 10 18:01:29.593202 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:01:29.593180    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-05a81f-openshift-default-7584465ddb-kktsm"]
Sep 10 18:01:29.674094 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:01:29.674060    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e479896-5ea5-4e33-9e1d-aecfd1368324-istio-envoy\") pod \"e2e-ipp-cleanup-05a81f-openshift-default-7584465ddb-kktsm\" (UID: \"3e479896-5ea5-4e33-9e1d-aecfd1368324\") " pod="openshift-ingress/e2e-ipp-cleanup-05a81f-openshift-default-7584465ddb-kktsm"
Sep 10 18:01:29.674094 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:01:29.674101    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e479896-5ea5-4e33-9e1d-aecfd1368324-istio-token\") pod \"e2e-ipp-cleanup-05a81f-openshift-default-7584465ddb-kktsm\" (UID: \"3e479896-5ea5-4e33-9e1d-aecfd1368324\") " pod="openshift-ingress/e2e-ipp-cleanup-05a81f-openshift-default-7584465ddb-kktsm"
Sep 10 18:01:29.674355 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:01:29.674179    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgztc\" (UniqueName: \"kubernetes.io/projected/3e479896-5ea5-4e33-9e1d-aecfd1368324-kube-api-access-dgztc\") pod \"e2e-ipp-cleanup-05a81f-openshift-default-7584465ddb-kktsm\" (UID: \"3e479896-5ea5-4e33-9e1d-aecfd1368324\") " pod="openshift-ingress/e2e-ipp-cleanup-05a81f-openshift-default-7584465ddb-kktsm"
Sep 10 18:01:29.674355 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:01:29.674209    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e479896-5ea5-4e33-9e1d-aecfd1368324-istio-data\") pod \"e2e-ipp-cleanup-05a81f-openshift-default-7584465ddb-kktsm\" (UID: \"3e479896-5ea5-4e33-9e1d-aecfd1368324\") " pod="openshift-ingress/e2e-ipp-cleanup-05a81f-openshift-default-7584465ddb-kktsm"
Sep 10 18:01:29.674355 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:01:29.674231    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e479896-5ea5-4e33-9e1d-aecfd1368324-workload-socket\") pod \"e2e-ipp-cleanup-05a81f-openshift-default-7584465ddb-kktsm\" (UID: \"3e479896-5ea5-4e33-9e1d-aecfd1368324\") " pod="openshift-ingress/e2e-ipp-cleanup-05a81f-openshift-default-7584465ddb-kktsm"
Sep 10 18:01:29.674355 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:01:29.674247    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e479896-5ea5-4e33-9e1d-aecfd1368324-istiod-ca-cert\") pod \"e2e-ipp-cleanup-05a81f-openshift-default-7584465ddb-kktsm\" (UID: \"3e479896-5ea5-4e33-9e1d-aecfd1368324\") " pod="openshift-ingress/e2e-ipp-cleanup-05a81f-openshift-default-7584465ddb-kktsm"
Sep 10 18:01:29.674355 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:01:29.674270    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e479896-5ea5-4e33-9e1d-aecfd1368324-credential-socket\") pod \"e2e-ipp-cleanup-05a81f-openshift-default-7584465ddb-kktsm\" (UID: \"3e479896-5ea5-4e33-9e1d-aecfd1368324\") " pod="openshift-ingress/e2e-ipp-cleanup-05a81f-openshift-default-7584465ddb-kktsm"
Sep 10 18:01:29.674355 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:01:29.674286    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e479896-5ea5-4e33-9e1d-aecfd1368324-istio-podinfo\") pod \"e2e-ipp-cleanup-05a81f-openshift-default-7584465ddb-kktsm\" (UID: \"3e479896-5ea5-4e33-9e1d-aecfd1368324\") " pod="openshift-ingress/e2e-ipp-cleanup-05a81f-openshift-default-7584465ddb-kktsm"
Sep 10 18:01:29.674355 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:01:29.674330    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e479896-5ea5-4e33-9e1d-aecfd1368324-workload-certs\") pod \"e2e-ipp-cleanup-05a81f-openshift-default-7584465ddb-kktsm\" (UID: \"3e479896-5ea5-4e33-9e1d-aecfd1368324\") " pod="openshift-ingress/e2e-ipp-cleanup-05a81f-openshift-default-7584465ddb-kktsm"
Sep 10 18:01:29.775527 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:01:29.775490    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e479896-5ea5-4e33-9e1d-aecfd1368324-workload-certs\") pod \"e2e-ipp-cleanup-05a81f-openshift-default-7584465ddb-kktsm\" (UID: \"3e479896-5ea5-4e33-9e1d-aecfd1368324\") " pod="openshift-ingress/e2e-ipp-cleanup-05a81f-openshift-default-7584465ddb-kktsm"
Sep 10 18:01:29.775738 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:01:29.775546    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e479896-5ea5-4e33-9e1d-aecfd1368324-istio-envoy\") pod \"e2e-ipp-cleanup-05a81f-openshift-default-7584465ddb-kktsm\" (UID: \"3e479896-5ea5-4e33-9e1d-aecfd1368324\") " pod="openshift-ingress/e2e-ipp-cleanup-05a81f-openshift-default-7584465ddb-kktsm"
Sep 10 18:01:29.775738 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:01:29.775591    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e479896-5ea5-4e33-9e1d-aecfd1368324-istio-token\") pod \"e2e-ipp-cleanup-05a81f-openshift-default-7584465ddb-kktsm\" (UID: \"3e479896-5ea5-4e33-9e1d-aecfd1368324\") " pod="openshift-ingress/e2e-ipp-cleanup-05a81f-openshift-default-7584465ddb-kktsm"
Sep 10 18:01:29.775738 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:01:29.775649    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgztc\" (UniqueName: \"kubernetes.io/projected/3e479896-5ea5-4e33-9e1d-aecfd1368324-kube-api-access-dgztc\") pod \"e2e-ipp-cleanup-05a81f-openshift-default-7584465ddb-kktsm\" (UID: \"3e479896-5ea5-4e33-9e1d-aecfd1368324\") " pod="openshift-ingress/e2e-ipp-cleanup-05a81f-openshift-default-7584465ddb-kktsm"
Sep 10 18:01:29.775738 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:01:29.775715    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e479896-5ea5-4e33-9e1d-aecfd1368324-istio-data\") pod \"e2e-ipp-cleanup-05a81f-openshift-default-7584465ddb-kktsm\" (UID: \"3e479896-5ea5-4e33-9e1d-aecfd1368324\") " pod="openshift-ingress/e2e-ipp-cleanup-05a81f-openshift-default-7584465ddb-kktsm"
Sep 10 18:01:29.775956 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:01:29.775746    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e479896-5ea5-4e33-9e1d-aecfd1368324-workload-socket\") pod \"e2e-ipp-cleanup-05a81f-openshift-default-7584465ddb-kktsm\" (UID: \"3e479896-5ea5-4e33-9e1d-aecfd1368324\") " pod="openshift-ingress/e2e-ipp-cleanup-05a81f-openshift-default-7584465ddb-kktsm"
Sep 10 18:01:29.775956 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:01:29.775775    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e479896-5ea5-4e33-9e1d-aecfd1368324-istiod-ca-cert\") pod \"e2e-ipp-cleanup-05a81f-openshift-default-7584465ddb-kktsm\" (UID: \"3e479896-5ea5-4e33-9e1d-aecfd1368324\") " pod="openshift-ingress/e2e-ipp-cleanup-05a81f-openshift-default-7584465ddb-kktsm"
Sep 10 18:01:29.775956 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:01:29.775814    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e479896-5ea5-4e33-9e1d-aecfd1368324-credential-socket\") pod \"e2e-ipp-cleanup-05a81f-openshift-default-7584465ddb-kktsm\" (UID: \"3e479896-5ea5-4e33-9e1d-aecfd1368324\") " pod="openshift-ingress/e2e-ipp-cleanup-05a81f-openshift-default-7584465ddb-kktsm"
Sep 10 18:01:29.775956 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:01:29.775841    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e479896-5ea5-4e33-9e1d-aecfd1368324-istio-podinfo\") pod \"e2e-ipp-cleanup-05a81f-openshift-default-7584465ddb-kktsm\" (UID: \"3e479896-5ea5-4e33-9e1d-aecfd1368324\") " pod="openshift-ingress/e2e-ipp-cleanup-05a81f-openshift-default-7584465ddb-kktsm"
Sep 10 18:01:29.776163 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:01:29.775965    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e479896-5ea5-4e33-9e1d-aecfd1368324-workload-certs\") pod \"e2e-ipp-cleanup-05a81f-openshift-default-7584465ddb-kktsm\" (UID: \"3e479896-5ea5-4e33-9e1d-aecfd1368324\") " pod="openshift-ingress/e2e-ipp-cleanup-05a81f-openshift-default-7584465ddb-kktsm"
Sep 10 18:01:29.776221 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:01:29.776207    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e479896-5ea5-4e33-9e1d-aecfd1368324-istio-data\") pod \"e2e-ipp-cleanup-05a81f-openshift-default-7584465ddb-kktsm\" (UID: \"3e479896-5ea5-4e33-9e1d-aecfd1368324\") " pod="openshift-ingress/e2e-ipp-cleanup-05a81f-openshift-default-7584465ddb-kktsm"
Sep 10 18:01:29.776835 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:01:29.776790    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e479896-5ea5-4e33-9e1d-aecfd1368324-credential-socket\") pod \"e2e-ipp-cleanup-05a81f-openshift-default-7584465ddb-kktsm\" (UID: \"3e479896-5ea5-4e33-9e1d-aecfd1368324\") " pod="openshift-ingress/e2e-ipp-cleanup-05a81f-openshift-default-7584465ddb-kktsm"
Sep 10 18:01:29.776986 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:01:29.776937    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e479896-5ea5-4e33-9e1d-aecfd1368324-istiod-ca-cert\") pod \"e2e-ipp-cleanup-05a81f-openshift-default-7584465ddb-kktsm\" (UID: \"3e479896-5ea5-4e33-9e1d-aecfd1368324\") " pod="openshift-ingress/e2e-ipp-cleanup-05a81f-openshift-default-7584465ddb-kktsm"
Sep 10 18:01:29.777322 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:01:29.777292    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e479896-5ea5-4e33-9e1d-aecfd1368324-workload-socket\") pod \"e2e-ipp-cleanup-05a81f-openshift-default-7584465ddb-kktsm\" (UID: \"3e479896-5ea5-4e33-9e1d-aecfd1368324\") " pod="openshift-ingress/e2e-ipp-cleanup-05a81f-openshift-default-7584465ddb-kktsm"
Sep 10 18:01:29.778579 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:01:29.778547    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e479896-5ea5-4e33-9e1d-aecfd1368324-istio-podinfo\") pod \"e2e-ipp-cleanup-05a81f-openshift-default-7584465ddb-kktsm\" (UID: \"3e479896-5ea5-4e33-9e1d-aecfd1368324\") " pod="openshift-ingress/e2e-ipp-cleanup-05a81f-openshift-default-7584465ddb-kktsm"
Sep 10 18:01:29.778898 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:01:29.778876    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e479896-5ea5-4e33-9e1d-aecfd1368324-istio-envoy\") pod \"e2e-ipp-cleanup-05a81f-openshift-default-7584465ddb-kktsm\" (UID: \"3e479896-5ea5-4e33-9e1d-aecfd1368324\") " pod="openshift-ingress/e2e-ipp-cleanup-05a81f-openshift-default-7584465ddb-kktsm"
Sep 10 18:01:29.785135 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:01:29.785111    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e479896-5ea5-4e33-9e1d-aecfd1368324-istio-token\") pod \"e2e-ipp-cleanup-05a81f-openshift-default-7584465ddb-kktsm\" (UID: \"3e479896-5ea5-4e33-9e1d-aecfd1368324\") " pod="openshift-ingress/e2e-ipp-cleanup-05a81f-openshift-default-7584465ddb-kktsm"
Sep 10 18:01:29.785669 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:01:29.785652    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgztc\" (UniqueName: \"kubernetes.io/projected/3e479896-5ea5-4e33-9e1d-aecfd1368324-kube-api-access-dgztc\") pod \"e2e-ipp-cleanup-05a81f-openshift-default-7584465ddb-kktsm\" (UID: \"3e479896-5ea5-4e33-9e1d-aecfd1368324\") " pod="openshift-ingress/e2e-ipp-cleanup-05a81f-openshift-default-7584465ddb-kktsm"
Sep 10 18:01:29.890082 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:01:29.890048    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-05a81f-openshift-default-7584465ddb-kktsm"
Sep 10 18:01:30.173558 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:01:30.173521    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-05a81f-openshift-default-7584465ddb-kktsm"]
Sep 10 18:01:30.175107 ip-10-0-129-122 kubenswrapper[2579]: W0910 18:01:30.175075    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e479896_5ea5_4e33_9e1d_aecfd1368324.slice/crio-65819b8bd24ee4eed5c4c0b13200d2ae1cbac4859564e9b33fd6f59f9d9f9c96 WatchSource:0}: Error finding container 65819b8bd24ee4eed5c4c0b13200d2ae1cbac4859564e9b33fd6f59f9d9f9c96: Status 404 returned error can't find the container with id 65819b8bd24ee4eed5c4c0b13200d2ae1cbac4859564e9b33fd6f59f9d9f9c96
Sep 10 18:01:30.177608 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:01:30.177552    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 10 18:01:30.177705 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:01:30.177651    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 10 18:01:30.177750 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:01:30.177716    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 10 18:01:30.532702 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:01:30.532607    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-05a81f-openshift-default-7584465ddb-kktsm" event={"ID":"3e479896-5ea5-4e33-9e1d-aecfd1368324","Type":"ContainerStarted","Data":"f8370dc3b6859c8ad3aad0a200e220406585fc693b3f486bf66f50d63ef160de"}
Sep 10 18:01:30.532702 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:01:30.532664    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-05a81f-openshift-default-7584465ddb-kktsm" event={"ID":"3e479896-5ea5-4e33-9e1d-aecfd1368324","Type":"ContainerStarted","Data":"65819b8bd24ee4eed5c4c0b13200d2ae1cbac4859564e9b33fd6f59f9d9f9c96"}
Sep 10 18:01:30.554603 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:01:30.554535    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-cleanup-05a81f-openshift-default-7584465ddb-kktsm" podStartSLOduration=1.554521319 podStartE2EDuration="1.554521319s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 18:01:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 18:01:30.552904072 +0000 UTC m=+1534.552734137" watchObservedRunningTime="2026-09-10 18:01:30.554521319 +0000 UTC m=+1534.554351386"
Sep 10 18:01:30.890166 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:01:30.890130    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-05a81f-openshift-default-7584465ddb-kktsm"
Sep 10 18:01:31.896395 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:01:31.896364    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-05a81f-openshift-default-7584465ddb-kktsm"
Sep 10 18:01:32.542842 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:01:32.542812    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-05a81f-openshift-default-7584465ddb-kktsm"
Sep 10 18:01:32.543776 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:01:32.543759    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-05a81f-openshift-default-7584465ddb-kktsm"
Sep 10 18:01:34.286630 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:01:34.286593    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-05a81f-6f99d6f749-mc8wh"]
Sep 10 18:01:34.289426 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:01:34.289403    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-05a81f-6f99d6f749-mc8wh"
Sep 10 18:01:34.292042 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:01:34.292015    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-05a81f-dockercfg-hmhh9\""
Sep 10 18:01:34.292222 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:01:34.292072    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-05a81f\""
Sep 10 18:01:34.300853 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:01:34.300831    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-05a81f-6f99d6f749-mc8wh"]
Sep 10 18:01:34.421234 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:01:34.421200    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cca34d8a-faa3-4b7e-9359-6729829fa8d2-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-05a81f-6f99d6f749-mc8wh\" (UID: \"cca34d8a-faa3-4b7e-9359-6729829fa8d2\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-05a81f-6f99d6f749-mc8wh"
Sep 10 18:01:34.421446 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:01:34.421276    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2lkdv\" (UniqueName: \"kubernetes.io/projected/cca34d8a-faa3-4b7e-9359-6729829fa8d2-kube-api-access-2lkdv\") pod \"payload-processing-e2e-ipp-cleanup-05a81f-6f99d6f749-mc8wh\" (UID: \"cca34d8a-faa3-4b7e-9359-6729829fa8d2\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-05a81f-6f99d6f749-mc8wh"
Sep 10 18:01:34.522752 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:01:34.522722    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lkdv\" (UniqueName: \"kubernetes.io/projected/cca34d8a-faa3-4b7e-9359-6729829fa8d2-kube-api-access-2lkdv\") pod \"payload-processing-e2e-ipp-cleanup-05a81f-6f99d6f749-mc8wh\" (UID: \"cca34d8a-faa3-4b7e-9359-6729829fa8d2\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-05a81f-6f99d6f749-mc8wh"
Sep 10 18:01:34.522907 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:01:34.522819    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cca34d8a-faa3-4b7e-9359-6729829fa8d2-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-05a81f-6f99d6f749-mc8wh\" (UID: \"cca34d8a-faa3-4b7e-9359-6729829fa8d2\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-05a81f-6f99d6f749-mc8wh"
Sep 10 18:01:34.523465 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:01:34.523445    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cca34d8a-faa3-4b7e-9359-6729829fa8d2-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-05a81f-6f99d6f749-mc8wh\" (UID: \"cca34d8a-faa3-4b7e-9359-6729829fa8d2\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-05a81f-6f99d6f749-mc8wh"
Sep 10 18:01:34.531076 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:01:34.531052    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2lkdv\" (UniqueName: \"kubernetes.io/projected/cca34d8a-faa3-4b7e-9359-6729829fa8d2-kube-api-access-2lkdv\") pod \"payload-processing-e2e-ipp-cleanup-05a81f-6f99d6f749-mc8wh\" (UID: \"cca34d8a-faa3-4b7e-9359-6729829fa8d2\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-05a81f-6f99d6f749-mc8wh"
Sep 10 18:01:34.600353 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:01:34.600269    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-05a81f-6f99d6f749-mc8wh"
Sep 10 18:01:34.873158 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:01:34.873131    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-05a81f-6f99d6f749-mc8wh"]
Sep 10 18:01:34.875254 ip-10-0-129-122 kubenswrapper[2579]: W0910 18:01:34.875220    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcca34d8a_faa3_4b7e_9359_6729829fa8d2.slice/crio-309b08c83ebb73279c99d9098e6b8971ebe071b6d979b61e89d0699860f91c87 WatchSource:0}: Error finding container 309b08c83ebb73279c99d9098e6b8971ebe071b6d979b61e89d0699860f91c87: Status 404 returned error can't find the container with id 309b08c83ebb73279c99d9098e6b8971ebe071b6d979b61e89d0699860f91c87
Sep 10 18:01:35.558460 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:01:35.558427    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-05a81f-6f99d6f749-mc8wh" event={"ID":"cca34d8a-faa3-4b7e-9359-6729829fa8d2","Type":"ContainerStarted","Data":"6617e355638c04e8a6e848cf0063d745bae77cc37a27ef42d5c520fbee820acd"}
Sep 10 18:01:35.558925 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:01:35.558484    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-05a81f-6f99d6f749-mc8wh"
Sep 10 18:01:35.558925 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:01:35.558501    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-05a81f-6f99d6f749-mc8wh" event={"ID":"cca34d8a-faa3-4b7e-9359-6729829fa8d2","Type":"ContainerStarted","Data":"309b08c83ebb73279c99d9098e6b8971ebe071b6d979b61e89d0699860f91c87"}
Sep 10 18:01:35.574415 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:01:35.574356    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-05a81f-6f99d6f749-mc8wh" podStartSLOduration=1.574336076 podStartE2EDuration="1.574336076s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 18:01:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 18:01:35.573550334 +0000 UTC m=+1539.573380401" watchObservedRunningTime="2026-09-10 18:01:35.574336076 +0000 UTC m=+1539.574166145"
Sep 10 18:01:46.566053 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:01:46.566021    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-05a81f-6f99d6f749-mc8wh"
Sep 10 18:01:47.777911 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:01:47.777606    2579 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-05a81f-openshift-default-7584465ddb-kktsm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": read tcp 10.132.0.2:53576->10.132.0.83:15021: read: connection reset by peer" start-of-body=
Sep 10 18:01:47.777911 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:01:47.777689    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-cleanup-05a81f-openshift-default-7584465ddb-kktsm" podUID="3e479896-5ea5-4e33-9e1d-aecfd1368324" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": read tcp 10.132.0.2:53576->10.132.0.83:15021: read: connection reset by peer"
Sep 10 18:01:48.635847 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:01:48.635817    2579 generic.go:356] "Generic (PLEG): container finished" podID="3e479896-5ea5-4e33-9e1d-aecfd1368324" containerID="f8370dc3b6859c8ad3aad0a200e220406585fc693b3f486bf66f50d63ef160de" exitCode=0
Sep 10 18:01:48.636020 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:01:48.635889    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-05a81f-openshift-default-7584465ddb-kktsm" event={"ID":"3e479896-5ea5-4e33-9e1d-aecfd1368324","Type":"ContainerDied","Data":"f8370dc3b6859c8ad3aad0a200e220406585fc693b3f486bf66f50d63ef160de"}
Sep 10 18:01:48.636339 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:01:48.636322    2579 scope.go:117] "RemoveContainer" containerID="f8370dc3b6859c8ad3aad0a200e220406585fc693b3f486bf66f50d63ef160de"
Sep 10 18:01:48.637523 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:01:48.637493    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 10 18:01:48.637633 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:01:48.637577    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 10 18:01:48.637633 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:01:48.637612    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 10 18:01:49.643135 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:01:49.643093    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-05a81f-openshift-default-7584465ddb-kktsm" event={"ID":"3e479896-5ea5-4e33-9e1d-aecfd1368324","Type":"ContainerStarted","Data":"88ab550fd400fd294f04d08dfaa978a285e3250eb2ee693907b66d0450ef173d"}
Sep 10 18:01:49.891232 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:01:49.890762    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-05a81f-openshift-default-7584465ddb-kktsm"
Sep 10 18:01:49.891232 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:01:49.890893    2579 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-05a81f-openshift-default-7584465ddb-kktsm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body=
Sep 10 18:01:49.891232 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:01:49.890954    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-05a81f-openshift-default-7584465ddb-kktsm" podUID="3e479896-5ea5-4e33-9e1d-aecfd1368324" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused"
Sep 10 18:01:50.890589 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:01:50.890524    2579 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-05a81f-openshift-default-7584465ddb-kktsm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body=
Sep 10 18:01:50.891090 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:01:50.890625    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-05a81f-openshift-default-7584465ddb-kktsm" podUID="3e479896-5ea5-4e33-9e1d-aecfd1368324" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused"
Sep 10 18:01:51.891383 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:01:51.891344    2579 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-05a81f-openshift-default-7584465ddb-kktsm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body=
Sep 10 18:01:51.892039 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:01:51.892005    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-05a81f-openshift-default-7584465ddb-kktsm" podUID="3e479896-5ea5-4e33-9e1d-aecfd1368324" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused"
Sep 10 18:01:52.891222 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:01:52.891178    2579 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-05a81f-openshift-default-7584465ddb-kktsm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body=
Sep 10 18:01:52.891589 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:01:52.891525    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-05a81f-openshift-default-7584465ddb-kktsm" podUID="3e479896-5ea5-4e33-9e1d-aecfd1368324" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused"
Sep 10 18:01:53.891672 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:01:53.890716    2579 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-05a81f-openshift-default-7584465ddb-kktsm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body=
Sep 10 18:01:53.891672 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:01:53.890780    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-05a81f-openshift-default-7584465ddb-kktsm" podUID="3e479896-5ea5-4e33-9e1d-aecfd1368324" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused"
Sep 10 18:01:54.891318 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:01:54.891272    2579 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-05a81f-openshift-default-7584465ddb-kktsm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body=
Sep 10 18:01:54.891687 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:01:54.891650    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-05a81f-openshift-default-7584465ddb-kktsm" podUID="3e479896-5ea5-4e33-9e1d-aecfd1368324" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused"
Sep 10 18:01:56.055687 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:01:56.055649    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-05a81f-openshift-default-7584465ddb-kktsm"
Sep 10 18:01:56.056106 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:01:56.056014    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-05a81f-openshift-default-7584465ddb-kktsm"
Sep 10 18:01:56.056823 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:01:56.056806    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-05a81f-openshift-default-7584465ddb-kktsm"
Sep 10 18:02:04.141459 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:02:04.141423    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-05a81f-6f99d6f749-mc8wh"]
Sep 10 18:02:04.142122 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:02:04.141723    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-05a81f-6f99d6f749-mc8wh" podUID="cca34d8a-faa3-4b7e-9359-6729829fa8d2" containerName="payload-processing" containerID="cri-o://6617e355638c04e8a6e848cf0063d745bae77cc37a27ef42d5c520fbee820acd" gracePeriod=30
Sep 10 18:02:04.398192 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:02:04.398132    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-05a81f-6f99d6f749-mc8wh"
Sep 10 18:02:04.424481 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:02:04.424445    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cca34d8a-faa3-4b7e-9359-6729829fa8d2-plugins-config-volume\") pod \"cca34d8a-faa3-4b7e-9359-6729829fa8d2\" (UID: \"cca34d8a-faa3-4b7e-9359-6729829fa8d2\") "
Sep 10 18:02:04.424682 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:02:04.424498    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2lkdv\" (UniqueName: \"kubernetes.io/projected/cca34d8a-faa3-4b7e-9359-6729829fa8d2-kube-api-access-2lkdv\") pod \"cca34d8a-faa3-4b7e-9359-6729829fa8d2\" (UID: \"cca34d8a-faa3-4b7e-9359-6729829fa8d2\") "
Sep 10 18:02:04.424837 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:02:04.424815    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cca34d8a-faa3-4b7e-9359-6729829fa8d2-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "cca34d8a-faa3-4b7e-9359-6729829fa8d2" (UID: "cca34d8a-faa3-4b7e-9359-6729829fa8d2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 18:02:04.426739 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:02:04.426711    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cca34d8a-faa3-4b7e-9359-6729829fa8d2-kube-api-access-2lkdv" (OuterVolumeSpecName: "kube-api-access-2lkdv") pod "cca34d8a-faa3-4b7e-9359-6729829fa8d2" (UID: "cca34d8a-faa3-4b7e-9359-6729829fa8d2"). InnerVolumeSpecName "kube-api-access-2lkdv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 18:02:04.526227 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:02:04.526190    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cca34d8a-faa3-4b7e-9359-6729829fa8d2-plugins-config-volume\") on node \"ip-10-0-129-122.ec2.internal\" DevicePath \"\""
Sep 10 18:02:04.526227 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:02:04.526219    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2lkdv\" (UniqueName: \"kubernetes.io/projected/cca34d8a-faa3-4b7e-9359-6729829fa8d2-kube-api-access-2lkdv\") on node \"ip-10-0-129-122.ec2.internal\" DevicePath \"\""
Sep 10 18:02:04.729002 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:02:04.728914    2579 generic.go:356] "Generic (PLEG): container finished" podID="cca34d8a-faa3-4b7e-9359-6729829fa8d2" containerID="6617e355638c04e8a6e848cf0063d745bae77cc37a27ef42d5c520fbee820acd" exitCode=0
Sep 10 18:02:04.729002 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:02:04.728959    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-05a81f-6f99d6f749-mc8wh"
Sep 10 18:02:04.729002 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:02:04.728977    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-05a81f-6f99d6f749-mc8wh" event={"ID":"cca34d8a-faa3-4b7e-9359-6729829fa8d2","Type":"ContainerDied","Data":"6617e355638c04e8a6e848cf0063d745bae77cc37a27ef42d5c520fbee820acd"}
Sep 10 18:02:04.729232 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:02:04.729030    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-05a81f-6f99d6f749-mc8wh" event={"ID":"cca34d8a-faa3-4b7e-9359-6729829fa8d2","Type":"ContainerDied","Data":"309b08c83ebb73279c99d9098e6b8971ebe071b6d979b61e89d0699860f91c87"}
Sep 10 18:02:04.729232 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:02:04.729061    2579 scope.go:117] "RemoveContainer" containerID="6617e355638c04e8a6e848cf0063d745bae77cc37a27ef42d5c520fbee820acd"
Sep 10 18:02:04.738039 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:02:04.738019    2579 scope.go:117] "RemoveContainer" containerID="6617e355638c04e8a6e848cf0063d745bae77cc37a27ef42d5c520fbee820acd"
Sep 10 18:02:04.738318 ip-10-0-129-122 kubenswrapper[2579]: E0910 18:02:04.738294    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6617e355638c04e8a6e848cf0063d745bae77cc37a27ef42d5c520fbee820acd\": container with ID starting with 6617e355638c04e8a6e848cf0063d745bae77cc37a27ef42d5c520fbee820acd not found: ID does not exist" containerID="6617e355638c04e8a6e848cf0063d745bae77cc37a27ef42d5c520fbee820acd"
Sep 10 18:02:04.738423 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:02:04.738323    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6617e355638c04e8a6e848cf0063d745bae77cc37a27ef42d5c520fbee820acd"} err="failed to get container status \"6617e355638c04e8a6e848cf0063d745bae77cc37a27ef42d5c520fbee820acd\": rpc error: code = NotFound desc = could not find container \"6617e355638c04e8a6e848cf0063d745bae77cc37a27ef42d5c520fbee820acd\": container with ID starting with 6617e355638c04e8a6e848cf0063d745bae77cc37a27ef42d5c520fbee820acd not found: ID does not exist"
Sep 10 18:02:04.746610 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:02:04.746584    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-05a81f-6f99d6f749-mc8wh"]
Sep 10 18:02:04.750095 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:02:04.750073    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-05a81f-6f99d6f749-mc8wh"]
Sep 10 18:02:06.477805 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:02:06.477775    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cca34d8a-faa3-4b7e-9359-6729829fa8d2" path="/var/lib/kubelet/pods/cca34d8a-faa3-4b7e-9359-6729829fa8d2/volumes"
Sep 10 18:02:11.190287 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:02:11.190253    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-05a81f-openshift-default-7584465ddb-kktsm"]
Sep 10 18:02:11.191112 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:02:11.191074    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-cleanup-05a81f-openshift-default-7584465ddb-kktsm" podUID="3e479896-5ea5-4e33-9e1d-aecfd1368324" containerName="istio-proxy" containerID="cri-o://88ab550fd400fd294f04d08dfaa978a285e3250eb2ee693907b66d0450ef173d" gracePeriod=30
Sep 10 18:02:16.456122 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:02:16.456057    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-05a81f-openshift-default-7584465ddb-kktsm"
Sep 10 18:02:16.535785 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:02:16.535756    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e479896-5ea5-4e33-9e1d-aecfd1368324-workload-socket\") pod \"3e479896-5ea5-4e33-9e1d-aecfd1368324\" (UID: \"3e479896-5ea5-4e33-9e1d-aecfd1368324\") "
Sep 10 18:02:16.535966 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:02:16.535799    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e479896-5ea5-4e33-9e1d-aecfd1368324-istio-envoy\") pod \"3e479896-5ea5-4e33-9e1d-aecfd1368324\" (UID: \"3e479896-5ea5-4e33-9e1d-aecfd1368324\") "
Sep 10 18:02:16.535966 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:02:16.535821    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e479896-5ea5-4e33-9e1d-aecfd1368324-istio-podinfo\") pod \"3e479896-5ea5-4e33-9e1d-aecfd1368324\" (UID: \"3e479896-5ea5-4e33-9e1d-aecfd1368324\") "
Sep 10 18:02:16.535966 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:02:16.535865    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dgztc\" (UniqueName: \"kubernetes.io/projected/3e479896-5ea5-4e33-9e1d-aecfd1368324-kube-api-access-dgztc\") pod \"3e479896-5ea5-4e33-9e1d-aecfd1368324\" (UID: \"3e479896-5ea5-4e33-9e1d-aecfd1368324\") "
Sep 10 18:02:16.535966 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:02:16.535886    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e479896-5ea5-4e33-9e1d-aecfd1368324-istio-data\") pod \"3e479896-5ea5-4e33-9e1d-aecfd1368324\" (UID: \"3e479896-5ea5-4e33-9e1d-aecfd1368324\") "
Sep 10 18:02:16.535966 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:02:16.535901    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e479896-5ea5-4e33-9e1d-aecfd1368324-credential-socket\") pod \"3e479896-5ea5-4e33-9e1d-aecfd1368324\" (UID: \"3e479896-5ea5-4e33-9e1d-aecfd1368324\") "
Sep 10 18:02:16.535966 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:02:16.535917    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e479896-5ea5-4e33-9e1d-aecfd1368324-istio-token\") pod \"3e479896-5ea5-4e33-9e1d-aecfd1368324\" (UID: \"3e479896-5ea5-4e33-9e1d-aecfd1368324\") "
Sep 10 18:02:16.536277 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:02:16.536132    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e479896-5ea5-4e33-9e1d-aecfd1368324-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "3e479896-5ea5-4e33-9e1d-aecfd1368324" (UID: "3e479896-5ea5-4e33-9e1d-aecfd1368324"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 18:02:16.536277 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:02:16.536240    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e479896-5ea5-4e33-9e1d-aecfd1368324-workload-certs\") pod \"3e479896-5ea5-4e33-9e1d-aecfd1368324\" (UID: \"3e479896-5ea5-4e33-9e1d-aecfd1368324\") "
Sep 10 18:02:16.536277 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:02:16.536237    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e479896-5ea5-4e33-9e1d-aecfd1368324-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "3e479896-5ea5-4e33-9e1d-aecfd1368324" (UID: "3e479896-5ea5-4e33-9e1d-aecfd1368324"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 18:02:16.536277 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:02:16.536251    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e479896-5ea5-4e33-9e1d-aecfd1368324-istio-data" (OuterVolumeSpecName: "istio-data") pod "3e479896-5ea5-4e33-9e1d-aecfd1368324" (UID: "3e479896-5ea5-4e33-9e1d-aecfd1368324"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 18:02:16.536277 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:02:16.536272    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e479896-5ea5-4e33-9e1d-aecfd1368324-istiod-ca-cert\") pod \"3e479896-5ea5-4e33-9e1d-aecfd1368324\" (UID: \"3e479896-5ea5-4e33-9e1d-aecfd1368324\") "
Sep 10 18:02:16.536596 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:02:16.536436    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e479896-5ea5-4e33-9e1d-aecfd1368324-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "3e479896-5ea5-4e33-9e1d-aecfd1368324" (UID: "3e479896-5ea5-4e33-9e1d-aecfd1368324"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 18:02:16.536596 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:02:16.536558    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e479896-5ea5-4e33-9e1d-aecfd1368324-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "3e479896-5ea5-4e33-9e1d-aecfd1368324" (UID: "3e479896-5ea5-4e33-9e1d-aecfd1368324"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 18:02:16.537044 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:02:16.537024    2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e479896-5ea5-4e33-9e1d-aecfd1368324-workload-socket\") on node \"ip-10-0-129-122.ec2.internal\" DevicePath \"\""
Sep 10 18:02:16.537102 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:02:16.537052    2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e479896-5ea5-4e33-9e1d-aecfd1368324-istio-data\") on node \"ip-10-0-129-122.ec2.internal\" DevicePath \"\""
Sep 10 18:02:16.537102 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:02:16.537067    2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e479896-5ea5-4e33-9e1d-aecfd1368324-credential-socket\") on node \"ip-10-0-129-122.ec2.internal\" DevicePath \"\""
Sep 10 18:02:16.537102 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:02:16.537080    2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e479896-5ea5-4e33-9e1d-aecfd1368324-workload-certs\") on node \"ip-10-0-129-122.ec2.internal\" DevicePath \"\""
Sep 10 18:02:16.537102 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:02:16.537095    2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e479896-5ea5-4e33-9e1d-aecfd1368324-istiod-ca-cert\") on node \"ip-10-0-129-122.ec2.internal\" DevicePath \"\""
Sep 10 18:02:16.538263 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:02:16.538233    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e479896-5ea5-4e33-9e1d-aecfd1368324-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "3e479896-5ea5-4e33-9e1d-aecfd1368324" (UID: "3e479896-5ea5-4e33-9e1d-aecfd1368324"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 18:02:16.538472 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:02:16.538454    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e479896-5ea5-4e33-9e1d-aecfd1368324-istio-token" (OuterVolumeSpecName: "istio-token") pod "3e479896-5ea5-4e33-9e1d-aecfd1368324" (UID: "3e479896-5ea5-4e33-9e1d-aecfd1368324"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 18:02:16.538783 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:02:16.538763    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3e479896-5ea5-4e33-9e1d-aecfd1368324-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "3e479896-5ea5-4e33-9e1d-aecfd1368324" (UID: "3e479896-5ea5-4e33-9e1d-aecfd1368324"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 10 18:02:16.538862 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:02:16.538814    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e479896-5ea5-4e33-9e1d-aecfd1368324-kube-api-access-dgztc" (OuterVolumeSpecName: "kube-api-access-dgztc") pod "3e479896-5ea5-4e33-9e1d-aecfd1368324" (UID: "3e479896-5ea5-4e33-9e1d-aecfd1368324"). InnerVolumeSpecName "kube-api-access-dgztc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 18:02:16.638463 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:02:16.638430    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dgztc\" (UniqueName: \"kubernetes.io/projected/3e479896-5ea5-4e33-9e1d-aecfd1368324-kube-api-access-dgztc\") on node \"ip-10-0-129-122.ec2.internal\" DevicePath \"\""
Sep 10 18:02:16.638463 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:02:16.638457    2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e479896-5ea5-4e33-9e1d-aecfd1368324-istio-token\") on node \"ip-10-0-129-122.ec2.internal\" DevicePath \"\""
Sep 10 18:02:16.638463 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:02:16.638466    2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e479896-5ea5-4e33-9e1d-aecfd1368324-istio-envoy\") on node \"ip-10-0-129-122.ec2.internal\" DevicePath \"\""
Sep 10 18:02:16.638716 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:02:16.638476    2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e479896-5ea5-4e33-9e1d-aecfd1368324-istio-podinfo\") on node \"ip-10-0-129-122.ec2.internal\" DevicePath \"\""
Sep 10 18:02:16.792915 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:02:16.792827    2579 generic.go:356] "Generic (PLEG): container finished" podID="3e479896-5ea5-4e33-9e1d-aecfd1368324" containerID="88ab550fd400fd294f04d08dfaa978a285e3250eb2ee693907b66d0450ef173d" exitCode=0
Sep 10 18:02:16.792915 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:02:16.792880    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-05a81f-openshift-default-7584465ddb-kktsm"
Sep 10 18:02:16.793107 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:02:16.792913    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-05a81f-openshift-default-7584465ddb-kktsm" event={"ID":"3e479896-5ea5-4e33-9e1d-aecfd1368324","Type":"ContainerDied","Data":"88ab550fd400fd294f04d08dfaa978a285e3250eb2ee693907b66d0450ef173d"}
Sep 10 18:02:16.793107 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:02:16.792967    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-05a81f-openshift-default-7584465ddb-kktsm" event={"ID":"3e479896-5ea5-4e33-9e1d-aecfd1368324","Type":"ContainerDied","Data":"65819b8bd24ee4eed5c4c0b13200d2ae1cbac4859564e9b33fd6f59f9d9f9c96"}
Sep 10 18:02:16.793107 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:02:16.792988    2579 scope.go:117] "RemoveContainer" containerID="88ab550fd400fd294f04d08dfaa978a285e3250eb2ee693907b66d0450ef173d"
Sep 10 18:02:16.802270 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:02:16.802254    2579 scope.go:117] "RemoveContainer" containerID="f8370dc3b6859c8ad3aad0a200e220406585fc693b3f486bf66f50d63ef160de"
Sep 10 18:02:16.815926 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:02:16.815900    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-05a81f-openshift-default-7584465ddb-kktsm"]
Sep 10 18:02:16.816213 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:02:16.816188    2579 scope.go:117] "RemoveContainer" containerID="88ab550fd400fd294f04d08dfaa978a285e3250eb2ee693907b66d0450ef173d"
Sep 10 18:02:16.816490 ip-10-0-129-122 kubenswrapper[2579]: E0910 18:02:16.816469    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88ab550fd400fd294f04d08dfaa978a285e3250eb2ee693907b66d0450ef173d\": container with ID starting with 88ab550fd400fd294f04d08dfaa978a285e3250eb2ee693907b66d0450ef173d not found: ID does not exist" containerID="88ab550fd400fd294f04d08dfaa978a285e3250eb2ee693907b66d0450ef173d"
Sep 10 18:02:16.816556 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:02:16.816499    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88ab550fd400fd294f04d08dfaa978a285e3250eb2ee693907b66d0450ef173d"} err="failed to get container status \"88ab550fd400fd294f04d08dfaa978a285e3250eb2ee693907b66d0450ef173d\": rpc error: code = NotFound desc = could not find container \"88ab550fd400fd294f04d08dfaa978a285e3250eb2ee693907b66d0450ef173d\": container with ID starting with 88ab550fd400fd294f04d08dfaa978a285e3250eb2ee693907b66d0450ef173d not found: ID does not exist"
Sep 10 18:02:16.816556 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:02:16.816515    2579 scope.go:117] "RemoveContainer" containerID="f8370dc3b6859c8ad3aad0a200e220406585fc693b3f486bf66f50d63ef160de"
Sep 10 18:02:16.817001 ip-10-0-129-122 kubenswrapper[2579]: E0910 18:02:16.816976    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8370dc3b6859c8ad3aad0a200e220406585fc693b3f486bf66f50d63ef160de\": container with ID starting with f8370dc3b6859c8ad3aad0a200e220406585fc693b3f486bf66f50d63ef160de not found: ID does not exist" containerID="f8370dc3b6859c8ad3aad0a200e220406585fc693b3f486bf66f50d63ef160de"
Sep 10 18:02:16.817157 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:02:16.817007    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8370dc3b6859c8ad3aad0a200e220406585fc693b3f486bf66f50d63ef160de"} err="failed to get container status \"f8370dc3b6859c8ad3aad0a200e220406585fc693b3f486bf66f50d63ef160de\": rpc error: code = NotFound desc = could not find container \"f8370dc3b6859c8ad3aad0a200e220406585fc693b3f486bf66f50d63ef160de\": container with ID starting with f8370dc3b6859c8ad3aad0a200e220406585fc693b3f486bf66f50d63ef160de not found: ID does not exist"
Sep 10 18:02:16.819376 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:02:16.819356    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-05a81f-openshift-default-7584465ddb-kktsm"]
Sep 10 18:02:18.475691 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:02:18.475658    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3e479896-5ea5-4e33-9e1d-aecfd1368324" path="/var/lib/kubelet/pods/3e479896-5ea5-4e33-9e1d-aecfd1368324/volumes"
Sep 10 18:02:30.433377 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:02:30.433321    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-904bbd-64f46b78d5-jm2tt"]
Sep 10 18:02:30.433877 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:02:30.433602    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-904bbd-64f46b78d5-jm2tt" podUID="e729c2e6-5480-4d68-9b10-20731810e30b" containerName="maas-api" containerID="cri-o://3ef43a3939c5a40164ea49eb9b0b923b6a30bbf708118b9b95b28378ade7fa9e" gracePeriod=30
Sep 10 18:02:30.497729 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:02:30.497693    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-904bbd-84d8556b86-xl85m"]
Sep 10 18:02:30.497963 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:02:30.497911    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-904bbd-84d8556b86-xl85m" podUID="3bd402f5-7e0b-4f92-a2b1-f007bfc8633b" containerName="payload-pre-processing" containerID="cri-o://974f40dd8519569608b7d8132dc1894db08a7e78643bff6214ac060a64af364d" gracePeriod=30
Sep 10 18:02:30.764408 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:02:30.764378    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-904bbd-64f46b78d5-jm2tt"
Sep 10 18:02:30.799816 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:02:30.799788    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-904bbd-84d8556b86-xl85m"
Sep 10 18:02:30.864739 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:02:30.864709    2579 generic.go:356] "Generic (PLEG): container finished" podID="e729c2e6-5480-4d68-9b10-20731810e30b" containerID="3ef43a3939c5a40164ea49eb9b0b923b6a30bbf708118b9b95b28378ade7fa9e" exitCode=0
Sep 10 18:02:30.864926 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:02:30.864757    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-904bbd-64f46b78d5-jm2tt"
Sep 10 18:02:30.864926 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:02:30.864795    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-904bbd-64f46b78d5-jm2tt" event={"ID":"e729c2e6-5480-4d68-9b10-20731810e30b","Type":"ContainerDied","Data":"3ef43a3939c5a40164ea49eb9b0b923b6a30bbf708118b9b95b28378ade7fa9e"}
Sep 10 18:02:30.864926 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:02:30.864858    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-904bbd-64f46b78d5-jm2tt" event={"ID":"e729c2e6-5480-4d68-9b10-20731810e30b","Type":"ContainerDied","Data":"9c579f231d33596c13084f3191066834f3f4f12cb08306024b8d237bf303e177"}
Sep 10 18:02:30.864926 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:02:30.864891    2579 scope.go:117] "RemoveContainer" containerID="3ef43a3939c5a40164ea49eb9b0b923b6a30bbf708118b9b95b28378ade7fa9e"
Sep 10 18:02:30.866152 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:02:30.866131    2579 generic.go:356] "Generic (PLEG): container finished" podID="3bd402f5-7e0b-4f92-a2b1-f007bfc8633b" containerID="974f40dd8519569608b7d8132dc1894db08a7e78643bff6214ac060a64af364d" exitCode=0
Sep 10 18:02:30.866249 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:02:30.866157    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-904bbd-84d8556b86-xl85m" event={"ID":"3bd402f5-7e0b-4f92-a2b1-f007bfc8633b","Type":"ContainerDied","Data":"974f40dd8519569608b7d8132dc1894db08a7e78643bff6214ac060a64af364d"}
Sep 10 18:02:30.866249 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:02:30.866203    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-904bbd-84d8556b86-xl85m" event={"ID":"3bd402f5-7e0b-4f92-a2b1-f007bfc8633b","Type":"ContainerDied","Data":"44907aaf4ed02e0081f517dd98bb901a68decd0b388c6bb9d98b7c2eeff4c677"}
Sep 10 18:02:30.866249 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:02:30.866170    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-904bbd-84d8556b86-xl85m"
Sep 10 18:02:30.868251 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:02:30.868228    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e729c2e6-5480-4d68-9b10-20731810e30b-metrics-certs\") pod \"e729c2e6-5480-4d68-9b10-20731810e30b\" (UID: \"e729c2e6-5480-4d68-9b10-20731810e30b\") "
Sep 10 18:02:30.868425 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:02:30.868304    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e729c2e6-5480-4d68-9b10-20731810e30b-maas-api-tls\") pod \"e729c2e6-5480-4d68-9b10-20731810e30b\" (UID: \"e729c2e6-5480-4d68-9b10-20731810e30b\") "
Sep 10 18:02:30.868425 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:02:30.868351    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f5rcc\" (UniqueName: \"kubernetes.io/projected/e729c2e6-5480-4d68-9b10-20731810e30b-kube-api-access-f5rcc\") pod \"e729c2e6-5480-4d68-9b10-20731810e30b\" (UID: \"e729c2e6-5480-4d68-9b10-20731810e30b\") "
Sep 10 18:02:30.868425 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:02:30.868404    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e729c2e6-5480-4d68-9b10-20731810e30b-openshift-service-ca\") pod \"e729c2e6-5480-4d68-9b10-20731810e30b\" (UID: \"e729c2e6-5480-4d68-9b10-20731810e30b\") "
Sep 10 18:02:30.868868 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:02:30.868833    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e729c2e6-5480-4d68-9b10-20731810e30b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e729c2e6-5480-4d68-9b10-20731810e30b" (UID: "e729c2e6-5480-4d68-9b10-20731810e30b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 18:02:30.870752 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:02:30.870729    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e729c2e6-5480-4d68-9b10-20731810e30b-kube-api-access-f5rcc" (OuterVolumeSpecName: "kube-api-access-f5rcc") pod "e729c2e6-5480-4d68-9b10-20731810e30b" (UID: "e729c2e6-5480-4d68-9b10-20731810e30b"). InnerVolumeSpecName "kube-api-access-f5rcc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 18:02:30.870857 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:02:30.870809    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e729c2e6-5480-4d68-9b10-20731810e30b-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "e729c2e6-5480-4d68-9b10-20731810e30b" (UID: "e729c2e6-5480-4d68-9b10-20731810e30b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 18:02:30.871042 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:02:30.871028    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e729c2e6-5480-4d68-9b10-20731810e30b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e729c2e6-5480-4d68-9b10-20731810e30b" (UID: "e729c2e6-5480-4d68-9b10-20731810e30b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 18:02:30.874485 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:02:30.874466    2579 scope.go:117] "RemoveContainer" containerID="3ef43a3939c5a40164ea49eb9b0b923b6a30bbf708118b9b95b28378ade7fa9e"
Sep 10 18:02:30.874840 ip-10-0-129-122 kubenswrapper[2579]: E0910 18:02:30.874815    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ef43a3939c5a40164ea49eb9b0b923b6a30bbf708118b9b95b28378ade7fa9e\": container with ID starting with 3ef43a3939c5a40164ea49eb9b0b923b6a30bbf708118b9b95b28378ade7fa9e not found: ID does not exist" containerID="3ef43a3939c5a40164ea49eb9b0b923b6a30bbf708118b9b95b28378ade7fa9e"
Sep 10 18:02:30.874954 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:02:30.874851    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ef43a3939c5a40164ea49eb9b0b923b6a30bbf708118b9b95b28378ade7fa9e"} err="failed to get container status \"3ef43a3939c5a40164ea49eb9b0b923b6a30bbf708118b9b95b28378ade7fa9e\": rpc error: code = NotFound desc = could not find container \"3ef43a3939c5a40164ea49eb9b0b923b6a30bbf708118b9b95b28378ade7fa9e\": container with ID starting with 3ef43a3939c5a40164ea49eb9b0b923b6a30bbf708118b9b95b28378ade7fa9e not found: ID does not exist"
Sep 10 18:02:30.874954 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:02:30.874868    2579 scope.go:117] "RemoveContainer" containerID="974f40dd8519569608b7d8132dc1894db08a7e78643bff6214ac060a64af364d"
Sep 10 18:02:30.883158 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:02:30.883137    2579 scope.go:117] "RemoveContainer" containerID="974f40dd8519569608b7d8132dc1894db08a7e78643bff6214ac060a64af364d"
Sep 10 18:02:30.883447 ip-10-0-129-122 kubenswrapper[2579]: E0910 18:02:30.883429    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"974f40dd8519569608b7d8132dc1894db08a7e78643bff6214ac060a64af364d\": container with ID starting with 974f40dd8519569608b7d8132dc1894db08a7e78643bff6214ac060a64af364d not found: ID does not exist" containerID="974f40dd8519569608b7d8132dc1894db08a7e78643bff6214ac060a64af364d"
Sep 10 18:02:30.883501 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:02:30.883460    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"974f40dd8519569608b7d8132dc1894db08a7e78643bff6214ac060a64af364d"} err="failed to get container status \"974f40dd8519569608b7d8132dc1894db08a7e78643bff6214ac060a64af364d\": rpc error: code = NotFound desc = could not find container \"974f40dd8519569608b7d8132dc1894db08a7e78643bff6214ac060a64af364d\": container with ID starting with 974f40dd8519569608b7d8132dc1894db08a7e78643bff6214ac060a64af364d not found: ID does not exist"
Sep 10 18:02:30.969691 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:02:30.969590    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6j6ts\" (UniqueName: \"kubernetes.io/projected/3bd402f5-7e0b-4f92-a2b1-f007bfc8633b-kube-api-access-6j6ts\") pod \"3bd402f5-7e0b-4f92-a2b1-f007bfc8633b\" (UID: \"3bd402f5-7e0b-4f92-a2b1-f007bfc8633b\") "
Sep 10 18:02:30.969878 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:02:30.969706    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3bd402f5-7e0b-4f92-a2b1-f007bfc8633b-plugins-config-volume\") pod \"3bd402f5-7e0b-4f92-a2b1-f007bfc8633b\" (UID: \"3bd402f5-7e0b-4f92-a2b1-f007bfc8633b\") "
Sep 10 18:02:30.969878 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:02:30.969855    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f5rcc\" (UniqueName: \"kubernetes.io/projected/e729c2e6-5480-4d68-9b10-20731810e30b-kube-api-access-f5rcc\") on node \"ip-10-0-129-122.ec2.internal\" DevicePath \"\""
Sep 10 18:02:30.969878 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:02:30.969868    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e729c2e6-5480-4d68-9b10-20731810e30b-openshift-service-ca\") on node \"ip-10-0-129-122.ec2.internal\" DevicePath \"\""
Sep 10 18:02:30.969878 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:02:30.969879    2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e729c2e6-5480-4d68-9b10-20731810e30b-metrics-certs\") on node \"ip-10-0-129-122.ec2.internal\" DevicePath \"\""
Sep 10 18:02:30.970099 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:02:30.969888    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e729c2e6-5480-4d68-9b10-20731810e30b-maas-api-tls\") on node \"ip-10-0-129-122.ec2.internal\" DevicePath \"\""
Sep 10 18:02:30.970135 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:02:30.970092    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3bd402f5-7e0b-4f92-a2b1-f007bfc8633b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3bd402f5-7e0b-4f92-a2b1-f007bfc8633b" (UID: "3bd402f5-7e0b-4f92-a2b1-f007bfc8633b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 18:02:30.971960 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:02:30.971938    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bd402f5-7e0b-4f92-a2b1-f007bfc8633b-kube-api-access-6j6ts" (OuterVolumeSpecName: "kube-api-access-6j6ts") pod "3bd402f5-7e0b-4f92-a2b1-f007bfc8633b" (UID: "3bd402f5-7e0b-4f92-a2b1-f007bfc8633b"). InnerVolumeSpecName "kube-api-access-6j6ts". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 18:02:31.070350 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:02:31.070311    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3bd402f5-7e0b-4f92-a2b1-f007bfc8633b-plugins-config-volume\") on node \"ip-10-0-129-122.ec2.internal\" DevicePath \"\""
Sep 10 18:02:31.070350 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:02:31.070343    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6j6ts\" (UniqueName: \"kubernetes.io/projected/3bd402f5-7e0b-4f92-a2b1-f007bfc8633b-kube-api-access-6j6ts\") on node \"ip-10-0-129-122.ec2.internal\" DevicePath \"\""
Sep 10 18:02:31.192936 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:02:31.192907    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-904bbd-64f46b78d5-jm2tt"]
Sep 10 18:02:31.199308 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:02:31.199284    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-904bbd-64f46b78d5-jm2tt"]
Sep 10 18:02:31.211580 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:02:31.211317    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-904bbd-84d8556b86-xl85m"]
Sep 10 18:02:31.213369 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:02:31.213349    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-904bbd-84d8556b86-xl85m"]
Sep 10 18:02:31.217028 ip-10-0-129-122 kubenswrapper[2579]: E0910 18:02:31.217002    2579 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode729c2e6_5480_4d68_9b10_20731810e30b.slice/crio-9c579f231d33596c13084f3191066834f3f4f12cb08306024b8d237bf303e177\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode729c2e6_5480_4d68_9b10_20731810e30b.slice\": RecentStats: unable to find data in memory cache]"
Sep 10 18:02:32.475995 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:02:32.475964    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3bd402f5-7e0b-4f92-a2b1-f007bfc8633b" path="/var/lib/kubelet/pods/3bd402f5-7e0b-4f92-a2b1-f007bfc8633b/volumes"
Sep 10 18:02:32.476381 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:02:32.476360    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e729c2e6-5480-4d68-9b10-20731810e30b" path="/var/lib/kubelet/pods/e729c2e6-5480-4d68-9b10-20731810e30b/volumes"
Sep 10 18:03:34.071590 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:03:34.071478    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-904bbd-openshift-default-69d64c9f6-lttqf"]
Sep 10 18:03:34.072044 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:03:34.071754    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-904bbd-openshift-default-69d64c9f6-lttqf" podUID="22216a3a-3a36-4fb3-8980-4ea64db8a74b" containerName="istio-proxy" containerID="cri-o://50379655e1e27d59decaf8249b08c7354f2887ee1680bbcd152f12ab9992e837" gracePeriod=30
Sep 10 18:03:39.204677 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:03:39.204640    2579 generic.go:356] "Generic (PLEG): container finished" podID="22216a3a-3a36-4fb3-8980-4ea64db8a74b" containerID="50379655e1e27d59decaf8249b08c7354f2887ee1680bbcd152f12ab9992e837" exitCode=0
Sep 10 18:03:39.205069 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:03:39.204681    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-904bbd-openshift-default-69d64c9f6-lttqf" event={"ID":"22216a3a-3a36-4fb3-8980-4ea64db8a74b","Type":"ContainerDied","Data":"50379655e1e27d59decaf8249b08c7354f2887ee1680bbcd152f12ab9992e837"}
Sep 10 18:03:39.205069 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:03:39.204766    2579 scope.go:117] "RemoveContainer" containerID="901786dd36540ff4103a97108b80afa7bc58417f57bf9734a803cedd8f8fb70f"
Sep 10 18:03:39.329859 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:03:39.329833    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-904bbd-openshift-default-69d64c9f6-lttqf"
Sep 10 18:03:39.394115 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:03:39.394086    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22216a3a-3a36-4fb3-8980-4ea64db8a74b-istio-envoy\") pod \"22216a3a-3a36-4fb3-8980-4ea64db8a74b\" (UID: \"22216a3a-3a36-4fb3-8980-4ea64db8a74b\") "
Sep 10 18:03:39.394238 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:03:39.394124    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22216a3a-3a36-4fb3-8980-4ea64db8a74b-workload-certs\") pod \"22216a3a-3a36-4fb3-8980-4ea64db8a74b\" (UID: \"22216a3a-3a36-4fb3-8980-4ea64db8a74b\") "
Sep 10 18:03:39.394238 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:03:39.394157    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22216a3a-3a36-4fb3-8980-4ea64db8a74b-istio-token\") pod \"22216a3a-3a36-4fb3-8980-4ea64db8a74b\" (UID: \"22216a3a-3a36-4fb3-8980-4ea64db8a74b\") "
Sep 10 18:03:39.394323 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:03:39.394240    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22216a3a-3a36-4fb3-8980-4ea64db8a74b-istiod-ca-cert\") pod \"22216a3a-3a36-4fb3-8980-4ea64db8a74b\" (UID: \"22216a3a-3a36-4fb3-8980-4ea64db8a74b\") "
Sep 10 18:03:39.394323 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:03:39.394299    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22216a3a-3a36-4fb3-8980-4ea64db8a74b-credential-socket\") pod \"22216a3a-3a36-4fb3-8980-4ea64db8a74b\" (UID: \"22216a3a-3a36-4fb3-8980-4ea64db8a74b\") "
Sep 10 18:03:39.394419 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:03:39.394326    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22216a3a-3a36-4fb3-8980-4ea64db8a74b-istio-podinfo\") pod \"22216a3a-3a36-4fb3-8980-4ea64db8a74b\" (UID: \"22216a3a-3a36-4fb3-8980-4ea64db8a74b\") "
Sep 10 18:03:39.394419 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:03:39.394350    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hh58d\" (UniqueName: \"kubernetes.io/projected/22216a3a-3a36-4fb3-8980-4ea64db8a74b-kube-api-access-hh58d\") pod \"22216a3a-3a36-4fb3-8980-4ea64db8a74b\" (UID: \"22216a3a-3a36-4fb3-8980-4ea64db8a74b\") "
Sep 10 18:03:39.394419 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:03:39.394380    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22216a3a-3a36-4fb3-8980-4ea64db8a74b-workload-socket\") pod \"22216a3a-3a36-4fb3-8980-4ea64db8a74b\" (UID: \"22216a3a-3a36-4fb3-8980-4ea64db8a74b\") "
Sep 10 18:03:39.394419 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:03:39.394398    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22216a3a-3a36-4fb3-8980-4ea64db8a74b-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "22216a3a-3a36-4fb3-8980-4ea64db8a74b" (UID: "22216a3a-3a36-4fb3-8980-4ea64db8a74b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 18:03:39.394664 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:03:39.394446    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22216a3a-3a36-4fb3-8980-4ea64db8a74b-istio-data\") pod \"22216a3a-3a36-4fb3-8980-4ea64db8a74b\" (UID: \"22216a3a-3a36-4fb3-8980-4ea64db8a74b\") "
Sep 10 18:03:39.394721 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:03:39.394692    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22216a3a-3a36-4fb3-8980-4ea64db8a74b-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "22216a3a-3a36-4fb3-8980-4ea64db8a74b" (UID: "22216a3a-3a36-4fb3-8980-4ea64db8a74b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 18:03:39.394786 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:03:39.394740    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22216a3a-3a36-4fb3-8980-4ea64db8a74b-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "22216a3a-3a36-4fb3-8980-4ea64db8a74b" (UID: "22216a3a-3a36-4fb3-8980-4ea64db8a74b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 18:03:39.394844 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:03:39.394803    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22216a3a-3a36-4fb3-8980-4ea64db8a74b-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "22216a3a-3a36-4fb3-8980-4ea64db8a74b" (UID: "22216a3a-3a36-4fb3-8980-4ea64db8a74b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 18:03:39.394844 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:03:39.394817    2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22216a3a-3a36-4fb3-8980-4ea64db8a74b-workload-certs\") on node \"ip-10-0-129-122.ec2.internal\" DevicePath \"\""
Sep 10 18:03:39.394947 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:03:39.394893    2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22216a3a-3a36-4fb3-8980-4ea64db8a74b-istiod-ca-cert\") on node \"ip-10-0-129-122.ec2.internal\" DevicePath \"\""
Sep 10 18:03:39.394947 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:03:39.394907    2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22216a3a-3a36-4fb3-8980-4ea64db8a74b-workload-socket\") on node \"ip-10-0-129-122.ec2.internal\" DevicePath \"\""
Sep 10 18:03:39.394947 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:03:39.394909    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22216a3a-3a36-4fb3-8980-4ea64db8a74b-istio-data" (OuterVolumeSpecName: "istio-data") pod "22216a3a-3a36-4fb3-8980-4ea64db8a74b" (UID: "22216a3a-3a36-4fb3-8980-4ea64db8a74b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 18:03:39.396733 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:03:39.396702    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/22216a3a-3a36-4fb3-8980-4ea64db8a74b-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "22216a3a-3a36-4fb3-8980-4ea64db8a74b" (UID: "22216a3a-3a36-4fb3-8980-4ea64db8a74b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 10 18:03:39.396864 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:03:39.396751    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22216a3a-3a36-4fb3-8980-4ea64db8a74b-kube-api-access-hh58d" (OuterVolumeSpecName: "kube-api-access-hh58d") pod "22216a3a-3a36-4fb3-8980-4ea64db8a74b" (UID: "22216a3a-3a36-4fb3-8980-4ea64db8a74b"). InnerVolumeSpecName "kube-api-access-hh58d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 18:03:39.396989 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:03:39.396972    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22216a3a-3a36-4fb3-8980-4ea64db8a74b-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "22216a3a-3a36-4fb3-8980-4ea64db8a74b" (UID: "22216a3a-3a36-4fb3-8980-4ea64db8a74b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 18:03:39.397089 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:03:39.397002    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22216a3a-3a36-4fb3-8980-4ea64db8a74b-istio-token" (OuterVolumeSpecName: "istio-token") pod "22216a3a-3a36-4fb3-8980-4ea64db8a74b" (UID: "22216a3a-3a36-4fb3-8980-4ea64db8a74b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 18:03:39.496130 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:03:39.496104    2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22216a3a-3a36-4fb3-8980-4ea64db8a74b-istio-envoy\") on node \"ip-10-0-129-122.ec2.internal\" DevicePath \"\""
Sep 10 18:03:39.496130 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:03:39.496127    2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22216a3a-3a36-4fb3-8980-4ea64db8a74b-istio-token\") on node \"ip-10-0-129-122.ec2.internal\" DevicePath \"\""
Sep 10 18:03:39.496292 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:03:39.496136    2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22216a3a-3a36-4fb3-8980-4ea64db8a74b-credential-socket\") on node \"ip-10-0-129-122.ec2.internal\" DevicePath \"\""
Sep 10 18:03:39.496292 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:03:39.496146    2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22216a3a-3a36-4fb3-8980-4ea64db8a74b-istio-podinfo\") on node \"ip-10-0-129-122.ec2.internal\" DevicePath \"\""
Sep 10 18:03:39.496292 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:03:39.496154    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hh58d\" (UniqueName: \"kubernetes.io/projected/22216a3a-3a36-4fb3-8980-4ea64db8a74b-kube-api-access-hh58d\") on node \"ip-10-0-129-122.ec2.internal\" DevicePath \"\""
Sep 10 18:03:39.496292 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:03:39.496162    2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22216a3a-3a36-4fb3-8980-4ea64db8a74b-istio-data\") on node \"ip-10-0-129-122.ec2.internal\" DevicePath \"\""
Sep 10 18:03:40.211017 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:03:40.210985    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-904bbd-openshift-default-69d64c9f6-lttqf" event={"ID":"22216a3a-3a36-4fb3-8980-4ea64db8a74b","Type":"ContainerDied","Data":"d637a9b011e4e2d6448fbaefafc717d3991437d490034c982c5c93dd1bda0074"}
Sep 10 18:03:40.211017 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:03:40.211010    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-904bbd-openshift-default-69d64c9f6-lttqf"
Sep 10 18:03:40.211540 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:03:40.211039    2579 scope.go:117] "RemoveContainer" containerID="50379655e1e27d59decaf8249b08c7354f2887ee1680bbcd152f12ab9992e837"
Sep 10 18:03:40.231832 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:03:40.231799    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-904bbd-openshift-default-69d64c9f6-lttqf"]
Sep 10 18:03:40.236919 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:03:40.236899    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-904bbd-openshift-default-69d64c9f6-lttqf"]
Sep 10 18:03:40.474439 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:03:40.474368    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="22216a3a-3a36-4fb3-8980-4ea64db8a74b" path="/var/lib/kubelet/pods/22216a3a-3a36-4fb3-8980-4ea64db8a74b/volumes"
Sep 10 18:04:02.762697 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:04:02.762658    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-18ecb2-78c8777656-2lncg"]
Sep 10 18:04:02.763703 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:04:02.763648    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-18ecb2-78c8777656-2lncg" podUID="0ce9649a-4505-48dc-84c0-e3ee22ca3e7e" containerName="maas-api" containerID="cri-o://e221ffb0e453abe05f8ef7c3d43ea008350ff30742b7a99c66305415145ff2b2" gracePeriod=30
Sep 10 18:04:03.022524 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:04:03.022455    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-18ecb2-78c8777656-2lncg"
Sep 10 18:04:03.206642 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:04:03.206607    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ce9649a-4505-48dc-84c0-e3ee22ca3e7e-openshift-service-ca\") pod \"0ce9649a-4505-48dc-84c0-e3ee22ca3e7e\" (UID: \"0ce9649a-4505-48dc-84c0-e3ee22ca3e7e\") "
Sep 10 18:04:03.206815 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:04:03.206665    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ce9649a-4505-48dc-84c0-e3ee22ca3e7e-maas-api-tls\") pod \"0ce9649a-4505-48dc-84c0-e3ee22ca3e7e\" (UID: \"0ce9649a-4505-48dc-84c0-e3ee22ca3e7e\") "
Sep 10 18:04:03.206815 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:04:03.206727    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ffrsk\" (UniqueName: \"kubernetes.io/projected/0ce9649a-4505-48dc-84c0-e3ee22ca3e7e-kube-api-access-ffrsk\") pod \"0ce9649a-4505-48dc-84c0-e3ee22ca3e7e\" (UID: \"0ce9649a-4505-48dc-84c0-e3ee22ca3e7e\") "
Sep 10 18:04:03.206815 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:04:03.206788    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ce9649a-4505-48dc-84c0-e3ee22ca3e7e-metrics-certs\") pod \"0ce9649a-4505-48dc-84c0-e3ee22ca3e7e\" (UID: \"0ce9649a-4505-48dc-84c0-e3ee22ca3e7e\") "
Sep 10 18:04:03.207063 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:04:03.207036    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ce9649a-4505-48dc-84c0-e3ee22ca3e7e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "0ce9649a-4505-48dc-84c0-e3ee22ca3e7e" (UID: "0ce9649a-4505-48dc-84c0-e3ee22ca3e7e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 18:04:03.209048 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:04:03.209018    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ce9649a-4505-48dc-84c0-e3ee22ca3e7e-kube-api-access-ffrsk" (OuterVolumeSpecName: "kube-api-access-ffrsk") pod "0ce9649a-4505-48dc-84c0-e3ee22ca3e7e" (UID: "0ce9649a-4505-48dc-84c0-e3ee22ca3e7e"). InnerVolumeSpecName "kube-api-access-ffrsk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 18:04:03.209204 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:04:03.209105    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ce9649a-4505-48dc-84c0-e3ee22ca3e7e-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "0ce9649a-4505-48dc-84c0-e3ee22ca3e7e" (UID: "0ce9649a-4505-48dc-84c0-e3ee22ca3e7e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 18:04:03.209204 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:04:03.209117    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ce9649a-4505-48dc-84c0-e3ee22ca3e7e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "0ce9649a-4505-48dc-84c0-e3ee22ca3e7e" (UID: "0ce9649a-4505-48dc-84c0-e3ee22ca3e7e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 18:04:03.307972 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:04:03.307882    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ffrsk\" (UniqueName: \"kubernetes.io/projected/0ce9649a-4505-48dc-84c0-e3ee22ca3e7e-kube-api-access-ffrsk\") on node \"ip-10-0-129-122.ec2.internal\" DevicePath \"\""
Sep 10 18:04:03.307972 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:04:03.307916    2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ce9649a-4505-48dc-84c0-e3ee22ca3e7e-metrics-certs\") on node \"ip-10-0-129-122.ec2.internal\" DevicePath \"\""
Sep 10 18:04:03.307972 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:04:03.307924    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ce9649a-4505-48dc-84c0-e3ee22ca3e7e-openshift-service-ca\") on node \"ip-10-0-129-122.ec2.internal\" DevicePath \"\""
Sep 10 18:04:03.307972 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:04:03.307932    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ce9649a-4505-48dc-84c0-e3ee22ca3e7e-maas-api-tls\") on node \"ip-10-0-129-122.ec2.internal\" DevicePath \"\""
Sep 10 18:04:03.329292 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:04:03.329252    2579 generic.go:356] "Generic (PLEG): container finished" podID="0ce9649a-4505-48dc-84c0-e3ee22ca3e7e" containerID="e221ffb0e453abe05f8ef7c3d43ea008350ff30742b7a99c66305415145ff2b2" exitCode=0
Sep 10 18:04:03.329497 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:04:03.329303    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-18ecb2-78c8777656-2lncg"
Sep 10 18:04:03.329497 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:04:03.329341    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-18ecb2-78c8777656-2lncg" event={"ID":"0ce9649a-4505-48dc-84c0-e3ee22ca3e7e","Type":"ContainerDied","Data":"e221ffb0e453abe05f8ef7c3d43ea008350ff30742b7a99c66305415145ff2b2"}
Sep 10 18:04:03.329497 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:04:03.329395    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-18ecb2-78c8777656-2lncg" event={"ID":"0ce9649a-4505-48dc-84c0-e3ee22ca3e7e","Type":"ContainerDied","Data":"8fc209ecbb8c6ca007dc2ff5a38ef55c8d0faf1f109968001b8886fb114a2620"}
Sep 10 18:04:03.329497 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:04:03.329417    2579 scope.go:117] "RemoveContainer" containerID="e221ffb0e453abe05f8ef7c3d43ea008350ff30742b7a99c66305415145ff2b2"
Sep 10 18:04:03.338910 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:04:03.338890    2579 scope.go:117] "RemoveContainer" containerID="e221ffb0e453abe05f8ef7c3d43ea008350ff30742b7a99c66305415145ff2b2"
Sep 10 18:04:03.339193 ip-10-0-129-122 kubenswrapper[2579]: E0910 18:04:03.339172    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e221ffb0e453abe05f8ef7c3d43ea008350ff30742b7a99c66305415145ff2b2\": container with ID starting with e221ffb0e453abe05f8ef7c3d43ea008350ff30742b7a99c66305415145ff2b2 not found: ID does not exist" containerID="e221ffb0e453abe05f8ef7c3d43ea008350ff30742b7a99c66305415145ff2b2"
Sep 10 18:04:03.339268 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:04:03.339206    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e221ffb0e453abe05f8ef7c3d43ea008350ff30742b7a99c66305415145ff2b2"} err="failed to get container status \"e221ffb0e453abe05f8ef7c3d43ea008350ff30742b7a99c66305415145ff2b2\": rpc error: code = NotFound desc = could not find container \"e221ffb0e453abe05f8ef7c3d43ea008350ff30742b7a99c66305415145ff2b2\": container with ID starting with e221ffb0e453abe05f8ef7c3d43ea008350ff30742b7a99c66305415145ff2b2 not found: ID does not exist"
Sep 10 18:04:03.350361 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:04:03.350317    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-18ecb2-78c8777656-2lncg"]
Sep 10 18:04:03.354071 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:04:03.354048    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-18ecb2-78c8777656-2lncg"]
Sep 10 18:04:04.475152 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:04:04.475114    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0ce9649a-4505-48dc-84c0-e3ee22ca3e7e" path="/var/lib/kubelet/pods/0ce9649a-4505-48dc-84c0-e3ee22ca3e7e/volumes"
Sep 10 18:04:09.718713 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:04:09.718678    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-18ecb2-openshift-default-86c6bff59d-9zh8x"]
Sep 10 18:04:09.719114 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:04:09.718879    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-18ecb2-openshift-default-86c6bff59d-9zh8x" podUID="d5268e09-33ea-4f48-97cc-24afd8b14c63" containerName="istio-proxy" containerID="cri-o://ebf45602388eca065559eca96fb0f0b02802def6553a3dfb38b4c419f735c86d" gracePeriod=30
Sep 10 18:04:12.378922 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:04:12.378884    2579 patch_prober.go:28] interesting pod/e2e-shared-a-w3-18ecb2-openshift-default-86c6bff59d-9zh8x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 10 18:04:12.379296 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:04:12.378939    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-18ecb2-openshift-default-86c6bff59d-9zh8x" podUID="d5268e09-33ea-4f48-97cc-24afd8b14c63" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 10 18:04:14.978928 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:04:14.978899    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-18ecb2-openshift-default-86c6bff59d-9zh8x"
Sep 10 18:04:15.001629 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:04:15.001529    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5268e09-33ea-4f48-97cc-24afd8b14c63-istiod-ca-cert\") pod \"d5268e09-33ea-4f48-97cc-24afd8b14c63\" (UID: \"d5268e09-33ea-4f48-97cc-24afd8b14c63\") "
Sep 10 18:04:15.001629 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:04:15.001599    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5268e09-33ea-4f48-97cc-24afd8b14c63-workload-certs\") pod \"d5268e09-33ea-4f48-97cc-24afd8b14c63\" (UID: \"d5268e09-33ea-4f48-97cc-24afd8b14c63\") "
Sep 10 18:04:15.001629 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:04:15.001627    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5268e09-33ea-4f48-97cc-24afd8b14c63-istio-data\") pod \"d5268e09-33ea-4f48-97cc-24afd8b14c63\" (UID: \"d5268e09-33ea-4f48-97cc-24afd8b14c63\") "
Sep 10 18:04:15.001912 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:04:15.001653    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5268e09-33ea-4f48-97cc-24afd8b14c63-credential-socket\") pod \"d5268e09-33ea-4f48-97cc-24afd8b14c63\" (UID: \"d5268e09-33ea-4f48-97cc-24afd8b14c63\") "
Sep 10 18:04:15.001912 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:04:15.001717    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5268e09-33ea-4f48-97cc-24afd8b14c63-istio-envoy\") pod \"d5268e09-33ea-4f48-97cc-24afd8b14c63\" (UID: \"d5268e09-33ea-4f48-97cc-24afd8b14c63\") "
Sep 10 18:04:15.001912 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:04:15.001745    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5268e09-33ea-4f48-97cc-24afd8b14c63-istio-podinfo\") pod \"d5268e09-33ea-4f48-97cc-24afd8b14c63\" (UID: \"d5268e09-33ea-4f48-97cc-24afd8b14c63\") "
Sep 10 18:04:15.001912 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:04:15.001790    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5268e09-33ea-4f48-97cc-24afd8b14c63-istio-token\") pod \"d5268e09-33ea-4f48-97cc-24afd8b14c63\" (UID: \"d5268e09-33ea-4f48-97cc-24afd8b14c63\") "
Sep 10 18:04:15.001912 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:04:15.001826    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-68rm9\" (UniqueName: \"kubernetes.io/projected/d5268e09-33ea-4f48-97cc-24afd8b14c63-kube-api-access-68rm9\") pod \"d5268e09-33ea-4f48-97cc-24afd8b14c63\" (UID: \"d5268e09-33ea-4f48-97cc-24afd8b14c63\") "
Sep 10 18:04:15.001912 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:04:15.001854    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5268e09-33ea-4f48-97cc-24afd8b14c63-workload-socket\") pod \"d5268e09-33ea-4f48-97cc-24afd8b14c63\" (UID: \"d5268e09-33ea-4f48-97cc-24afd8b14c63\") "
Sep 10 18:04:15.001912 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:04:15.001863    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5268e09-33ea-4f48-97cc-24afd8b14c63-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "d5268e09-33ea-4f48-97cc-24afd8b14c63" (UID: "d5268e09-33ea-4f48-97cc-24afd8b14c63"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 18:04:15.002271 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:04:15.001955    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5268e09-33ea-4f48-97cc-24afd8b14c63-istio-data" (OuterVolumeSpecName: "istio-data") pod "d5268e09-33ea-4f48-97cc-24afd8b14c63" (UID: "d5268e09-33ea-4f48-97cc-24afd8b14c63"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 18:04:15.002271 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:04:15.002014    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5268e09-33ea-4f48-97cc-24afd8b14c63-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "d5268e09-33ea-4f48-97cc-24afd8b14c63" (UID: "d5268e09-33ea-4f48-97cc-24afd8b14c63"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 18:04:15.002271 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:04:15.002111    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5268e09-33ea-4f48-97cc-24afd8b14c63-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "d5268e09-33ea-4f48-97cc-24afd8b14c63" (UID: "d5268e09-33ea-4f48-97cc-24afd8b14c63"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 18:04:15.002271 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:04:15.002225    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5268e09-33ea-4f48-97cc-24afd8b14c63-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "d5268e09-33ea-4f48-97cc-24afd8b14c63" (UID: "d5268e09-33ea-4f48-97cc-24afd8b14c63"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 18:04:15.002492 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:04:15.002303    2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5268e09-33ea-4f48-97cc-24afd8b14c63-workload-socket\") on node \"ip-10-0-129-122.ec2.internal\" DevicePath \"\""
Sep 10 18:04:15.002492 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:04:15.002321    2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5268e09-33ea-4f48-97cc-24afd8b14c63-istiod-ca-cert\") on node \"ip-10-0-129-122.ec2.internal\" DevicePath \"\""
Sep 10 18:04:15.002492 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:04:15.002336    2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5268e09-33ea-4f48-97cc-24afd8b14c63-workload-certs\") on node \"ip-10-0-129-122.ec2.internal\" DevicePath \"\""
Sep 10 18:04:15.002492 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:04:15.002349    2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5268e09-33ea-4f48-97cc-24afd8b14c63-istio-data\") on node \"ip-10-0-129-122.ec2.internal\" DevicePath \"\""
Sep 10 18:04:15.002492 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:04:15.002362    2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5268e09-33ea-4f48-97cc-24afd8b14c63-credential-socket\") on node \"ip-10-0-129-122.ec2.internal\" DevicePath \"\""
Sep 10 18:04:15.004671 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:04:15.004636    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d5268e09-33ea-4f48-97cc-24afd8b14c63-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "d5268e09-33ea-4f48-97cc-24afd8b14c63" (UID: "d5268e09-33ea-4f48-97cc-24afd8b14c63"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 10 18:04:15.004954 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:04:15.004926    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5268e09-33ea-4f48-97cc-24afd8b14c63-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "d5268e09-33ea-4f48-97cc-24afd8b14c63" (UID: "d5268e09-33ea-4f48-97cc-24afd8b14c63"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 18:04:15.005176 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:04:15.005146    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5268e09-33ea-4f48-97cc-24afd8b14c63-kube-api-access-68rm9" (OuterVolumeSpecName: "kube-api-access-68rm9") pod "d5268e09-33ea-4f48-97cc-24afd8b14c63" (UID: "d5268e09-33ea-4f48-97cc-24afd8b14c63"). InnerVolumeSpecName "kube-api-access-68rm9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 18:04:15.006459 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:04:15.006436    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5268e09-33ea-4f48-97cc-24afd8b14c63-istio-token" (OuterVolumeSpecName: "istio-token") pod "d5268e09-33ea-4f48-97cc-24afd8b14c63" (UID: "d5268e09-33ea-4f48-97cc-24afd8b14c63"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 18:04:15.103552 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:04:15.103516    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-68rm9\" (UniqueName: \"kubernetes.io/projected/d5268e09-33ea-4f48-97cc-24afd8b14c63-kube-api-access-68rm9\") on node \"ip-10-0-129-122.ec2.internal\" DevicePath \"\""
Sep 10 18:04:15.103552 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:04:15.103548    2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5268e09-33ea-4f48-97cc-24afd8b14c63-istio-envoy\") on node \"ip-10-0-129-122.ec2.internal\" DevicePath \"\""
Sep 10 18:04:15.103763 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:04:15.103558    2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5268e09-33ea-4f48-97cc-24afd8b14c63-istio-podinfo\") on node \"ip-10-0-129-122.ec2.internal\" DevicePath \"\""
Sep 10 18:04:15.103763 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:04:15.103594    2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5268e09-33ea-4f48-97cc-24afd8b14c63-istio-token\") on node \"ip-10-0-129-122.ec2.internal\" DevicePath \"\""
Sep 10 18:04:15.393760 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:04:15.393724    2579 generic.go:356] "Generic (PLEG): container finished" podID="d5268e09-33ea-4f48-97cc-24afd8b14c63" containerID="ebf45602388eca065559eca96fb0f0b02802def6553a3dfb38b4c419f735c86d" exitCode=0
Sep 10 18:04:15.393931 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:04:15.393775    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-18ecb2-openshift-default-86c6bff59d-9zh8x"
Sep 10 18:04:15.393931 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:04:15.393771    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-18ecb2-openshift-default-86c6bff59d-9zh8x" event={"ID":"d5268e09-33ea-4f48-97cc-24afd8b14c63","Type":"ContainerDied","Data":"ebf45602388eca065559eca96fb0f0b02802def6553a3dfb38b4c419f735c86d"}
Sep 10 18:04:15.393931 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:04:15.393882    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-18ecb2-openshift-default-86c6bff59d-9zh8x" event={"ID":"d5268e09-33ea-4f48-97cc-24afd8b14c63","Type":"ContainerDied","Data":"f83d262084740b89096fa9e668b0992023b26c3fcdac9d5ff06c8655c24a4007"}
Sep 10 18:04:15.393931 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:04:15.393914    2579 scope.go:117] "RemoveContainer" containerID="ebf45602388eca065559eca96fb0f0b02802def6553a3dfb38b4c419f735c86d"
Sep 10 18:04:15.403280 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:04:15.403257    2579 scope.go:117] "RemoveContainer" containerID="cd4206ead03e557afa1f81fea8e64f913743ca377d6317d6902815a52d76f86b"
Sep 10 18:04:15.414093 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:04:15.414044    2579 scope.go:117] "RemoveContainer" containerID="ebf45602388eca065559eca96fb0f0b02802def6553a3dfb38b4c419f735c86d"
Sep 10 18:04:15.414438 ip-10-0-129-122 kubenswrapper[2579]: E0910 18:04:15.414404    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ebf45602388eca065559eca96fb0f0b02802def6553a3dfb38b4c419f735c86d\": container with ID starting with ebf45602388eca065559eca96fb0f0b02802def6553a3dfb38b4c419f735c86d not found: ID does not exist" containerID="ebf45602388eca065559eca96fb0f0b02802def6553a3dfb38b4c419f735c86d"
Sep 10 18:04:15.414438 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:04:15.414444    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ebf45602388eca065559eca96fb0f0b02802def6553a3dfb38b4c419f735c86d"} err="failed to get container status \"ebf45602388eca065559eca96fb0f0b02802def6553a3dfb38b4c419f735c86d\": rpc error: code = NotFound desc = could not find container \"ebf45602388eca065559eca96fb0f0b02802def6553a3dfb38b4c419f735c86d\": container with ID starting with ebf45602388eca065559eca96fb0f0b02802def6553a3dfb38b4c419f735c86d not found: ID does not exist"
Sep 10 18:04:15.414725 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:04:15.414470    2579 scope.go:117] "RemoveContainer" containerID="cd4206ead03e557afa1f81fea8e64f913743ca377d6317d6902815a52d76f86b"
Sep 10 18:04:15.415293 ip-10-0-129-122 kubenswrapper[2579]: E0910 18:04:15.415266    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd4206ead03e557afa1f81fea8e64f913743ca377d6317d6902815a52d76f86b\": container with ID starting with cd4206ead03e557afa1f81fea8e64f913743ca377d6317d6902815a52d76f86b not found: ID does not exist" containerID="cd4206ead03e557afa1f81fea8e64f913743ca377d6317d6902815a52d76f86b"
Sep 10 18:04:15.415376 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:04:15.415300    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd4206ead03e557afa1f81fea8e64f913743ca377d6317d6902815a52d76f86b"} err="failed to get container status \"cd4206ead03e557afa1f81fea8e64f913743ca377d6317d6902815a52d76f86b\": rpc error: code = NotFound desc = could not find container \"cd4206ead03e557afa1f81fea8e64f913743ca377d6317d6902815a52d76f86b\": container with ID starting with cd4206ead03e557afa1f81fea8e64f913743ca377d6317d6902815a52d76f86b not found: ID does not exist"
Sep 10 18:04:15.416880 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:04:15.416859    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-18ecb2-openshift-default-86c6bff59d-9zh8x"]
Sep 10 18:04:15.420656 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:04:15.420635    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-18ecb2-openshift-default-86c6bff59d-9zh8x"]
Sep 10 18:04:16.474740 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:04:16.474706    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d5268e09-33ea-4f48-97cc-24afd8b14c63" path="/var/lib/kubelet/pods/d5268e09-33ea-4f48-97cc-24afd8b14c63/volumes"
Sep 10 18:04:20.807861 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:04:20.807769    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-57474d-644fb5dd75-mrfnj"]
Sep 10 18:04:20.808420 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:04:20.807977    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-57474d-644fb5dd75-mrfnj" podUID="aa4f0384-9c16-43c4-b52f-0e4f67a24d25" containerName="payload-processing" containerID="cri-o://1bbce74b004946cfa3140aeecaefa32f517251c86ac2e68421ad76d38161c78b" gracePeriod=30
Sep 10 18:04:20.813207 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:04:20.813177    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-57474d-d7f965c5d-zgnx6"]
Sep 10 18:04:20.813589 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:04:20.813472    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-57474d-d7f965c5d-zgnx6" podUID="f879e029-a958-4467-b001-d5eebb380ceb" containerName="payload-pre-processing" containerID="cri-o://7ab2ff8d1fb104321b7d21ef948c275a27403ec84423e4dfbb032f3b5888c697" gracePeriod=30
Sep 10 18:04:21.098064 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:04:21.098042    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-57474d-644fb5dd75-mrfnj"
Sep 10 18:04:21.100925 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:04:21.100893    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-57474d-d7f965c5d-zgnx6"
Sep 10 18:04:21.158944 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:04:21.158915    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mb782\" (UniqueName: \"kubernetes.io/projected/f879e029-a958-4467-b001-d5eebb380ceb-kube-api-access-mb782\") pod \"f879e029-a958-4467-b001-d5eebb380ceb\" (UID: \"f879e029-a958-4467-b001-d5eebb380ceb\") "
Sep 10 18:04:21.159111 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:04:21.158953    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa4f0384-9c16-43c4-b52f-0e4f67a24d25-plugins-config-volume\") pod \"aa4f0384-9c16-43c4-b52f-0e4f67a24d25\" (UID: \"aa4f0384-9c16-43c4-b52f-0e4f67a24d25\") "
Sep 10 18:04:21.159111 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:04:21.158978    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f879e029-a958-4467-b001-d5eebb380ceb-plugins-config-volume\") pod \"f879e029-a958-4467-b001-d5eebb380ceb\" (UID: \"f879e029-a958-4467-b001-d5eebb380ceb\") "
Sep 10 18:04:21.159111 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:04:21.159049    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2qlr8\" (UniqueName: \"kubernetes.io/projected/aa4f0384-9c16-43c4-b52f-0e4f67a24d25-kube-api-access-2qlr8\") pod \"aa4f0384-9c16-43c4-b52f-0e4f67a24d25\" (UID: \"aa4f0384-9c16-43c4-b52f-0e4f67a24d25\") "
Sep 10 18:04:21.159300 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:04:21.159276    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa4f0384-9c16-43c4-b52f-0e4f67a24d25-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "aa4f0384-9c16-43c4-b52f-0e4f67a24d25" (UID: "aa4f0384-9c16-43c4-b52f-0e4f67a24d25"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 18:04:21.159348 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:04:21.159318    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f879e029-a958-4467-b001-d5eebb380ceb-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f879e029-a958-4467-b001-d5eebb380ceb" (UID: "f879e029-a958-4467-b001-d5eebb380ceb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 18:04:21.159395 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:04:21.159351    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa4f0384-9c16-43c4-b52f-0e4f67a24d25-plugins-config-volume\") on node \"ip-10-0-129-122.ec2.internal\" DevicePath \"\""
Sep 10 18:04:21.161182 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:04:21.161159    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa4f0384-9c16-43c4-b52f-0e4f67a24d25-kube-api-access-2qlr8" (OuterVolumeSpecName: "kube-api-access-2qlr8") pod "aa4f0384-9c16-43c4-b52f-0e4f67a24d25" (UID: "aa4f0384-9c16-43c4-b52f-0e4f67a24d25"). InnerVolumeSpecName "kube-api-access-2qlr8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 18:04:21.161248 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:04:21.161178    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f879e029-a958-4467-b001-d5eebb380ceb-kube-api-access-mb782" (OuterVolumeSpecName: "kube-api-access-mb782") pod "f879e029-a958-4467-b001-d5eebb380ceb" (UID: "f879e029-a958-4467-b001-d5eebb380ceb"). InnerVolumeSpecName "kube-api-access-mb782". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 18:04:21.260311 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:04:21.260281    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mb782\" (UniqueName: \"kubernetes.io/projected/f879e029-a958-4467-b001-d5eebb380ceb-kube-api-access-mb782\") on node \"ip-10-0-129-122.ec2.internal\" DevicePath \"\""
Sep 10 18:04:21.260311 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:04:21.260308    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f879e029-a958-4467-b001-d5eebb380ceb-plugins-config-volume\") on node \"ip-10-0-129-122.ec2.internal\" DevicePath \"\""
Sep 10 18:04:21.260311 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:04:21.260318    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2qlr8\" (UniqueName: \"kubernetes.io/projected/aa4f0384-9c16-43c4-b52f-0e4f67a24d25-kube-api-access-2qlr8\") on node \"ip-10-0-129-122.ec2.internal\" DevicePath \"\""
Sep 10 18:04:21.423457 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:04:21.423428    2579 generic.go:356] "Generic (PLEG): container finished" podID="aa4f0384-9c16-43c4-b52f-0e4f67a24d25" containerID="1bbce74b004946cfa3140aeecaefa32f517251c86ac2e68421ad76d38161c78b" exitCode=0
Sep 10 18:04:21.423673 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:04:21.423476    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-57474d-644fb5dd75-mrfnj"
Sep 10 18:04:21.423673 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:04:21.423503    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-57474d-644fb5dd75-mrfnj" event={"ID":"aa4f0384-9c16-43c4-b52f-0e4f67a24d25","Type":"ContainerDied","Data":"1bbce74b004946cfa3140aeecaefa32f517251c86ac2e68421ad76d38161c78b"}
Sep 10 18:04:21.423673 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:04:21.423622    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-57474d-644fb5dd75-mrfnj" event={"ID":"aa4f0384-9c16-43c4-b52f-0e4f67a24d25","Type":"ContainerDied","Data":"27c614edc8ce8c4950f7073828246f21451dbc7b3c2e8b7602f70826846c82e5"}
Sep 10 18:04:21.423673 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:04:21.423656    2579 scope.go:117] "RemoveContainer" containerID="1bbce74b004946cfa3140aeecaefa32f517251c86ac2e68421ad76d38161c78b"
Sep 10 18:04:21.424925 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:04:21.424907    2579 generic.go:356] "Generic (PLEG): container finished" podID="f879e029-a958-4467-b001-d5eebb380ceb" containerID="7ab2ff8d1fb104321b7d21ef948c275a27403ec84423e4dfbb032f3b5888c697" exitCode=0
Sep 10 18:04:21.425008 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:04:21.424949    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-57474d-d7f965c5d-zgnx6"
Sep 10 18:04:21.425008 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:04:21.424981    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-57474d-d7f965c5d-zgnx6" event={"ID":"f879e029-a958-4467-b001-d5eebb380ceb","Type":"ContainerDied","Data":"7ab2ff8d1fb104321b7d21ef948c275a27403ec84423e4dfbb032f3b5888c697"}
Sep 10 18:04:21.425127 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:04:21.425026    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-57474d-d7f965c5d-zgnx6" event={"ID":"f879e029-a958-4467-b001-d5eebb380ceb","Type":"ContainerDied","Data":"1e13ba4e27e7a5cf570ca9a74b8d59d718016d7a128b35bbbbd83b34617c46aa"}
Sep 10 18:04:21.432703 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:04:21.432651    2579 scope.go:117] "RemoveContainer" containerID="1bbce74b004946cfa3140aeecaefa32f517251c86ac2e68421ad76d38161c78b"
Sep 10 18:04:21.432994 ip-10-0-129-122 kubenswrapper[2579]: E0910 18:04:21.432971    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1bbce74b004946cfa3140aeecaefa32f517251c86ac2e68421ad76d38161c78b\": container with ID starting with 1bbce74b004946cfa3140aeecaefa32f517251c86ac2e68421ad76d38161c78b not found: ID does not exist" containerID="1bbce74b004946cfa3140aeecaefa32f517251c86ac2e68421ad76d38161c78b"
Sep 10 18:04:21.433047 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:04:21.433005    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1bbce74b004946cfa3140aeecaefa32f517251c86ac2e68421ad76d38161c78b"} err="failed to get container status \"1bbce74b004946cfa3140aeecaefa32f517251c86ac2e68421ad76d38161c78b\": rpc error: code = NotFound desc = could not find container \"1bbce74b004946cfa3140aeecaefa32f517251c86ac2e68421ad76d38161c78b\": container with ID starting with 1bbce74b004946cfa3140aeecaefa32f517251c86ac2e68421ad76d38161c78b not found: ID does not exist"
Sep 10 18:04:21.433047 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:04:21.433029    2579 scope.go:117] "RemoveContainer" containerID="7ab2ff8d1fb104321b7d21ef948c275a27403ec84423e4dfbb032f3b5888c697"
Sep 10 18:04:21.440892 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:04:21.440876    2579 scope.go:117] "RemoveContainer" containerID="7ab2ff8d1fb104321b7d21ef948c275a27403ec84423e4dfbb032f3b5888c697"
Sep 10 18:04:21.441152 ip-10-0-129-122 kubenswrapper[2579]: E0910 18:04:21.441132    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ab2ff8d1fb104321b7d21ef948c275a27403ec84423e4dfbb032f3b5888c697\": container with ID starting with 7ab2ff8d1fb104321b7d21ef948c275a27403ec84423e4dfbb032f3b5888c697 not found: ID does not exist" containerID="7ab2ff8d1fb104321b7d21ef948c275a27403ec84423e4dfbb032f3b5888c697"
Sep 10 18:04:21.441190 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:04:21.441159    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ab2ff8d1fb104321b7d21ef948c275a27403ec84423e4dfbb032f3b5888c697"} err="failed to get container status \"7ab2ff8d1fb104321b7d21ef948c275a27403ec84423e4dfbb032f3b5888c697\": rpc error: code = NotFound desc = could not find container \"7ab2ff8d1fb104321b7d21ef948c275a27403ec84423e4dfbb032f3b5888c697\": container with ID starting with 7ab2ff8d1fb104321b7d21ef948c275a27403ec84423e4dfbb032f3b5888c697 not found: ID does not exist"
Sep 10 18:04:21.458780 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:04:21.458753    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-57474d-644fb5dd75-mrfnj"]
Sep 10 18:04:21.468112 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:04:21.468090    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-57474d-644fb5dd75-mrfnj"]
Sep 10 18:04:21.481773 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:04:21.481754    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-57474d-d7f965c5d-zgnx6"]
Sep 10 18:04:21.486317 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:04:21.486297    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-57474d-d7f965c5d-zgnx6"]
Sep 10 18:04:22.475195 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:04:22.475160    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aa4f0384-9c16-43c4-b52f-0e4f67a24d25" path="/var/lib/kubelet/pods/aa4f0384-9c16-43c4-b52f-0e4f67a24d25/volumes"
Sep 10 18:04:22.475581 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:04:22.475501    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f879e029-a958-4467-b001-d5eebb380ceb" path="/var/lib/kubelet/pods/f879e029-a958-4467-b001-d5eebb380ceb/volumes"
Sep 10 18:05:32.391555 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:05:32.391519    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-75895f5d8-ngkvs"]
Sep 10 18:05:32.392158 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:05:32.392031    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22216a3a-3a36-4fb3-8980-4ea64db8a74b" containerName="istio-proxy"
Sep 10 18:05:32.392158 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:05:32.392051    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="22216a3a-3a36-4fb3-8980-4ea64db8a74b" containerName="istio-proxy"
Sep 10 18:05:32.392158 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:05:32.392062    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3bd402f5-7e0b-4f92-a2b1-f007bfc8633b" containerName="payload-pre-processing"
Sep 10 18:05:32.392158 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:05:32.392071    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="3bd402f5-7e0b-4f92-a2b1-f007bfc8633b" containerName="payload-pre-processing"
Sep 10 18:05:32.392158 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:05:32.392085    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e729c2e6-5480-4d68-9b10-20731810e30b" containerName="maas-api"
Sep 10 18:05:32.392158 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:05:32.392094    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="e729c2e6-5480-4d68-9b10-20731810e30b" containerName="maas-api"
Sep 10 18:05:32.392158 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:05:32.392107    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22216a3a-3a36-4fb3-8980-4ea64db8a74b" containerName="istio-proxy"
Sep 10 18:05:32.392158 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:05:32.392114    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="22216a3a-3a36-4fb3-8980-4ea64db8a74b" containerName="istio-proxy"
Sep 10 18:05:32.392158 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:05:32.392133    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cca34d8a-faa3-4b7e-9359-6729829fa8d2" containerName="payload-processing"
Sep 10 18:05:32.392158 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:05:32.392142    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="cca34d8a-faa3-4b7e-9359-6729829fa8d2" containerName="payload-processing"
Sep 10 18:05:32.392158 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:05:32.392155    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa4f0384-9c16-43c4-b52f-0e4f67a24d25" containerName="payload-processing"
Sep 10 18:05:32.392158 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:05:32.392163    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa4f0384-9c16-43c4-b52f-0e4f67a24d25" containerName="payload-processing"
Sep 10 18:05:32.392798 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:05:32.392177    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5268e09-33ea-4f48-97cc-24afd8b14c63" containerName="istio-proxy"
Sep 10 18:05:32.392798 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:05:32.392185    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5268e09-33ea-4f48-97cc-24afd8b14c63" containerName="istio-proxy"
Sep 10 18:05:32.392798 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:05:32.392193    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3e479896-5ea5-4e33-9e1d-aecfd1368324" containerName="istio-proxy"
Sep 10 18:05:32.392798 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:05:32.392202    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="3e479896-5ea5-4e33-9e1d-aecfd1368324" containerName="istio-proxy"
Sep 10 18:05:32.392798 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:05:32.392214    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5268e09-33ea-4f48-97cc-24afd8b14c63" containerName="istio-proxy"
Sep 10 18:05:32.392798 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:05:32.392222    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5268e09-33ea-4f48-97cc-24afd8b14c63" containerName="istio-proxy"
Sep 10 18:05:32.392798 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:05:32.392232    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0ce9649a-4505-48dc-84c0-e3ee22ca3e7e" containerName="maas-api"
Sep 10 18:05:32.392798 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:05:32.392240    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="0ce9649a-4505-48dc-84c0-e3ee22ca3e7e" containerName="maas-api"
Sep 10 18:05:32.392798 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:05:32.392255    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f879e029-a958-4467-b001-d5eebb380ceb" containerName="payload-pre-processing"
Sep 10 18:05:32.392798 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:05:32.392264    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="f879e029-a958-4467-b001-d5eebb380ceb" containerName="payload-pre-processing"
Sep 10 18:05:32.392798 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:05:32.392273    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3e479896-5ea5-4e33-9e1d-aecfd1368324" containerName="istio-proxy"
Sep 10 18:05:32.392798 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:05:32.392281    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="3e479896-5ea5-4e33-9e1d-aecfd1368324" containerName="istio-proxy"
Sep 10 18:05:32.392798 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:05:32.392360    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="22216a3a-3a36-4fb3-8980-4ea64db8a74b" containerName="istio-proxy"
Sep 10 18:05:32.392798 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:05:32.392371    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="cca34d8a-faa3-4b7e-9359-6729829fa8d2" containerName="payload-processing"
Sep 10 18:05:32.392798 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:05:32.392380    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="3e479896-5ea5-4e33-9e1d-aecfd1368324" containerName="istio-proxy"
Sep 10 18:05:32.392798 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:05:32.392390    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="aa4f0384-9c16-43c4-b52f-0e4f67a24d25" containerName="payload-processing"
Sep 10 18:05:32.392798 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:05:32.392399    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="e729c2e6-5480-4d68-9b10-20731810e30b" containerName="maas-api"
Sep 10 18:05:32.392798 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:05:32.392412    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="d5268e09-33ea-4f48-97cc-24afd8b14c63" containerName="istio-proxy"
Sep 10 18:05:32.392798 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:05:32.392420    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="3bd402f5-7e0b-4f92-a2b1-f007bfc8633b" containerName="payload-pre-processing"
Sep 10 18:05:32.392798 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:05:32.392430    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="0ce9649a-4505-48dc-84c0-e3ee22ca3e7e" containerName="maas-api"
Sep 10 18:05:32.392798 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:05:32.392441    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="f879e029-a958-4467-b001-d5eebb380ceb" containerName="payload-pre-processing"
Sep 10 18:05:32.395802 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:05:32.395784    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-75895f5d8-ngkvs"
Sep 10 18:05:32.399371 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:05:32.399340    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 10 18:05:32.399371 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:05:32.399359    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 10 18:05:32.399555 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:05:32.399385    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 10 18:05:32.399555 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:05:32.399465    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-bbp7v\""
Sep 10 18:05:32.407747 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:05:32.407726    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-75895f5d8-ngkvs"]
Sep 10 18:05:32.466494 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:05:32.466464    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gthl6\" (UniqueName: \"kubernetes.io/projected/9aa7132f-d1a1-49e5-a1ec-c3bb62984205-kube-api-access-gthl6\") pod \"maas-controller-75895f5d8-ngkvs\" (UID: \"9aa7132f-d1a1-49e5-a1ec-c3bb62984205\") " pod="opendatahub/maas-controller-75895f5d8-ngkvs"
Sep 10 18:05:32.466666 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:05:32.466504    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9aa7132f-d1a1-49e5-a1ec-c3bb62984205-webhook-cert\") pod \"maas-controller-75895f5d8-ngkvs\" (UID: \"9aa7132f-d1a1-49e5-a1ec-c3bb62984205\") " pod="opendatahub/maas-controller-75895f5d8-ngkvs"
Sep 10 18:05:32.466666 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:05:32.466530    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9aa7132f-d1a1-49e5-a1ec-c3bb62984205-metrics-certs\") pod \"maas-controller-75895f5d8-ngkvs\" (UID: \"9aa7132f-d1a1-49e5-a1ec-c3bb62984205\") " pod="opendatahub/maas-controller-75895f5d8-ngkvs"
Sep 10 18:05:32.567237 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:05:32.567202    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gthl6\" (UniqueName: \"kubernetes.io/projected/9aa7132f-d1a1-49e5-a1ec-c3bb62984205-kube-api-access-gthl6\") pod \"maas-controller-75895f5d8-ngkvs\" (UID: \"9aa7132f-d1a1-49e5-a1ec-c3bb62984205\") " pod="opendatahub/maas-controller-75895f5d8-ngkvs"
Sep 10 18:05:32.567440 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:05:32.567258    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9aa7132f-d1a1-49e5-a1ec-c3bb62984205-webhook-cert\") pod \"maas-controller-75895f5d8-ngkvs\" (UID: \"9aa7132f-d1a1-49e5-a1ec-c3bb62984205\") " pod="opendatahub/maas-controller-75895f5d8-ngkvs"
Sep 10 18:05:32.567440 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:05:32.567274    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9aa7132f-d1a1-49e5-a1ec-c3bb62984205-metrics-certs\") pod \"maas-controller-75895f5d8-ngkvs\" (UID: \"9aa7132f-d1a1-49e5-a1ec-c3bb62984205\") " pod="opendatahub/maas-controller-75895f5d8-ngkvs"
Sep 10 18:05:32.569878 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:05:32.569853    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9aa7132f-d1a1-49e5-a1ec-c3bb62984205-metrics-certs\") pod \"maas-controller-75895f5d8-ngkvs\" (UID: \"9aa7132f-d1a1-49e5-a1ec-c3bb62984205\") " pod="opendatahub/maas-controller-75895f5d8-ngkvs"
Sep 10 18:05:32.570013 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:05:32.569973    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9aa7132f-d1a1-49e5-a1ec-c3bb62984205-webhook-cert\") pod \"maas-controller-75895f5d8-ngkvs\" (UID: \"9aa7132f-d1a1-49e5-a1ec-c3bb62984205\") " pod="opendatahub/maas-controller-75895f5d8-ngkvs"
Sep 10 18:05:32.575422 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:05:32.575400    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gthl6\" (UniqueName: \"kubernetes.io/projected/9aa7132f-d1a1-49e5-a1ec-c3bb62984205-kube-api-access-gthl6\") pod \"maas-controller-75895f5d8-ngkvs\" (UID: \"9aa7132f-d1a1-49e5-a1ec-c3bb62984205\") " pod="opendatahub/maas-controller-75895f5d8-ngkvs"
Sep 10 18:05:32.706999 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:05:32.706912    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-75895f5d8-ngkvs"
Sep 10 18:05:32.978046 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:05:32.978021    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-75895f5d8-ngkvs"]
Sep 10 18:05:32.980535 ip-10-0-129-122 kubenswrapper[2579]: W0910 18:05:32.980502    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9aa7132f_d1a1_49e5_a1ec_c3bb62984205.slice/crio-dac986a71a6a1c04d9027c2521d1a7cd16195591dc9a5f1ddde73794eecb7e20 WatchSource:0}: Error finding container dac986a71a6a1c04d9027c2521d1a7cd16195591dc9a5f1ddde73794eecb7e20: Status 404 returned error can't find the container with id dac986a71a6a1c04d9027c2521d1a7cd16195591dc9a5f1ddde73794eecb7e20
Sep 10 18:05:32.981975 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:05:32.981953    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 18:05:33.765160 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:05:33.765066    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-75895f5d8-ngkvs" event={"ID":"9aa7132f-d1a1-49e5-a1ec-c3bb62984205","Type":"ContainerStarted","Data":"b0f5569f6c85306ada748f4ea78c6cccb7bc779e920f96fb98ddcf55e36706d5"}
Sep 10 18:05:33.765160 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:05:33.765118    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-75895f5d8-ngkvs" event={"ID":"9aa7132f-d1a1-49e5-a1ec-c3bb62984205","Type":"ContainerStarted","Data":"dac986a71a6a1c04d9027c2521d1a7cd16195591dc9a5f1ddde73794eecb7e20"}
Sep 10 18:05:33.765160 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:05:33.765138    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-75895f5d8-ngkvs"
Sep 10 18:05:33.782781 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:05:33.782730    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-75895f5d8-ngkvs" podStartSLOduration=1.3259028769999999 podStartE2EDuration="1.782716325s" totalImagesPullingTime="456.813448ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 18:05:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 18:05:33.781467662 +0000 UTC m=+1777.781297729" watchObservedRunningTime="2026-09-10 18:05:33.782716325 +0000 UTC m=+1777.782546408"
Sep 10 18:05:44.771396 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:05:44.771363    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-75895f5d8-ngkvs"
Sep 10 18:05:54.099638 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:05:54.099539    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-75895f5d8-ngkvs"]
Sep 10 18:05:54.100094 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:05:54.099852    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-75895f5d8-ngkvs" podUID="9aa7132f-d1a1-49e5-a1ec-c3bb62984205" containerName="manager" containerID="cri-o://b0f5569f6c85306ada748f4ea78c6cccb7bc779e920f96fb98ddcf55e36706d5" gracePeriod=30
Sep 10 18:05:54.344758 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:05:54.344735    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-75895f5d8-ngkvs"
Sep 10 18:05:54.457195 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:05:54.457158    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gthl6\" (UniqueName: \"kubernetes.io/projected/9aa7132f-d1a1-49e5-a1ec-c3bb62984205-kube-api-access-gthl6\") pod \"9aa7132f-d1a1-49e5-a1ec-c3bb62984205\" (UID: \"9aa7132f-d1a1-49e5-a1ec-c3bb62984205\") "
Sep 10 18:05:54.457384 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:05:54.457211    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9aa7132f-d1a1-49e5-a1ec-c3bb62984205-webhook-cert\") pod \"9aa7132f-d1a1-49e5-a1ec-c3bb62984205\" (UID: \"9aa7132f-d1a1-49e5-a1ec-c3bb62984205\") "
Sep 10 18:05:54.457384 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:05:54.457264    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9aa7132f-d1a1-49e5-a1ec-c3bb62984205-metrics-certs\") pod \"9aa7132f-d1a1-49e5-a1ec-c3bb62984205\" (UID: \"9aa7132f-d1a1-49e5-a1ec-c3bb62984205\") "
Sep 10 18:05:54.459378 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:05:54.459346    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9aa7132f-d1a1-49e5-a1ec-c3bb62984205-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "9aa7132f-d1a1-49e5-a1ec-c3bb62984205" (UID: "9aa7132f-d1a1-49e5-a1ec-c3bb62984205"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 18:05:54.459378 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:05:54.459347    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9aa7132f-d1a1-49e5-a1ec-c3bb62984205-kube-api-access-gthl6" (OuterVolumeSpecName: "kube-api-access-gthl6") pod "9aa7132f-d1a1-49e5-a1ec-c3bb62984205" (UID: "9aa7132f-d1a1-49e5-a1ec-c3bb62984205"). InnerVolumeSpecName "kube-api-access-gthl6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 18:05:54.459518 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:05:54.459377    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9aa7132f-d1a1-49e5-a1ec-c3bb62984205-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "9aa7132f-d1a1-49e5-a1ec-c3bb62984205" (UID: "9aa7132f-d1a1-49e5-a1ec-c3bb62984205"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 18:05:54.558487 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:05:54.558462    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gthl6\" (UniqueName: \"kubernetes.io/projected/9aa7132f-d1a1-49e5-a1ec-c3bb62984205-kube-api-access-gthl6\") on node \"ip-10-0-129-122.ec2.internal\" DevicePath \"\""
Sep 10 18:05:54.558487 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:05:54.558484    2579 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9aa7132f-d1a1-49e5-a1ec-c3bb62984205-webhook-cert\") on node \"ip-10-0-129-122.ec2.internal\" DevicePath \"\""
Sep 10 18:05:54.558675 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:05:54.558494    2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9aa7132f-d1a1-49e5-a1ec-c3bb62984205-metrics-certs\") on node \"ip-10-0-129-122.ec2.internal\" DevicePath \"\""
Sep 10 18:05:54.866341 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:05:54.866256    2579 generic.go:356] "Generic (PLEG): container finished" podID="9aa7132f-d1a1-49e5-a1ec-c3bb62984205" containerID="b0f5569f6c85306ada748f4ea78c6cccb7bc779e920f96fb98ddcf55e36706d5" exitCode=0
Sep 10 18:05:54.866341 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:05:54.866305    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-75895f5d8-ngkvs"
Sep 10 18:05:54.866522 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:05:54.866341    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-75895f5d8-ngkvs" event={"ID":"9aa7132f-d1a1-49e5-a1ec-c3bb62984205","Type":"ContainerDied","Data":"b0f5569f6c85306ada748f4ea78c6cccb7bc779e920f96fb98ddcf55e36706d5"}
Sep 10 18:05:54.866522 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:05:54.866396    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-75895f5d8-ngkvs" event={"ID":"9aa7132f-d1a1-49e5-a1ec-c3bb62984205","Type":"ContainerDied","Data":"dac986a71a6a1c04d9027c2521d1a7cd16195591dc9a5f1ddde73794eecb7e20"}
Sep 10 18:05:54.866522 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:05:54.866418    2579 scope.go:117] "RemoveContainer" containerID="b0f5569f6c85306ada748f4ea78c6cccb7bc779e920f96fb98ddcf55e36706d5"
Sep 10 18:05:54.874909 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:05:54.874888    2579 scope.go:117] "RemoveContainer" containerID="b0f5569f6c85306ada748f4ea78c6cccb7bc779e920f96fb98ddcf55e36706d5"
Sep 10 18:05:54.875172 ip-10-0-129-122 kubenswrapper[2579]: E0910 18:05:54.875153    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0f5569f6c85306ada748f4ea78c6cccb7bc779e920f96fb98ddcf55e36706d5\": container with ID starting with b0f5569f6c85306ada748f4ea78c6cccb7bc779e920f96fb98ddcf55e36706d5 not found: ID does not exist" containerID="b0f5569f6c85306ada748f4ea78c6cccb7bc779e920f96fb98ddcf55e36706d5"
Sep 10 18:05:54.875241 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:05:54.875181    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0f5569f6c85306ada748f4ea78c6cccb7bc779e920f96fb98ddcf55e36706d5"} err="failed to get container status \"b0f5569f6c85306ada748f4ea78c6cccb7bc779e920f96fb98ddcf55e36706d5\": rpc error: code = NotFound desc = could not find container \"b0f5569f6c85306ada748f4ea78c6cccb7bc779e920f96fb98ddcf55e36706d5\": container with ID starting with b0f5569f6c85306ada748f4ea78c6cccb7bc779e920f96fb98ddcf55e36706d5 not found: ID does not exist"
Sep 10 18:05:54.892934 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:05:54.892909    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-75895f5d8-ngkvs"]
Sep 10 18:05:54.905255 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:05:54.905235    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-75895f5d8-ngkvs"]
Sep 10 18:05:56.475490 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:05:56.475456    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9aa7132f-d1a1-49e5-a1ec-c3bb62984205" path="/var/lib/kubelet/pods/9aa7132f-d1a1-49e5-a1ec-c3bb62984205/volumes"
Sep 10 18:05:56.554339 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:05:56.554309    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-rb854_97172b14-b84a-476e-89f1-05d3e70df416/console-operator/1.log"
Sep 10 18:05:56.562600 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:05:56.562556    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-rb854_97172b14-b84a-476e-89f1-05d3e70df416/console-operator/1.log"
Sep 10 18:07:06.153577 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:07:06.153533    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-2m796"]
Sep 10 18:07:06.155995 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:07:06.153808    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-2m796" podUID="14e9bbf5-88d2-4de8-a033-cebd387ee6b7" containerName="manager" containerID="cri-o://f70116414612aad2d72e73d9596438dd8a47ff803eeac0f85d038636dbe147be" gracePeriod=10
Sep 10 18:07:06.402482 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:07:06.402459    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-2m796"
Sep 10 18:07:06.460209 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:07:06.460115    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fnt9n\" (UniqueName: \"kubernetes.io/projected/14e9bbf5-88d2-4de8-a033-cebd387ee6b7-kube-api-access-fnt9n\") pod \"14e9bbf5-88d2-4de8-a033-cebd387ee6b7\" (UID: \"14e9bbf5-88d2-4de8-a033-cebd387ee6b7\") "
Sep 10 18:07:06.460374 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:07:06.460219    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/14e9bbf5-88d2-4de8-a033-cebd387ee6b7-extensions-socket-volume\") pod \"14e9bbf5-88d2-4de8-a033-cebd387ee6b7\" (UID: \"14e9bbf5-88d2-4de8-a033-cebd387ee6b7\") "
Sep 10 18:07:06.460616 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:07:06.460589    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14e9bbf5-88d2-4de8-a033-cebd387ee6b7-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "14e9bbf5-88d2-4de8-a033-cebd387ee6b7" (UID: "14e9bbf5-88d2-4de8-a033-cebd387ee6b7"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 18:07:06.462351 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:07:06.462322    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14e9bbf5-88d2-4de8-a033-cebd387ee6b7-kube-api-access-fnt9n" (OuterVolumeSpecName: "kube-api-access-fnt9n") pod "14e9bbf5-88d2-4de8-a033-cebd387ee6b7" (UID: "14e9bbf5-88d2-4de8-a033-cebd387ee6b7"). InnerVolumeSpecName "kube-api-access-fnt9n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 18:07:06.561298 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:07:06.561261    2579 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/14e9bbf5-88d2-4de8-a033-cebd387ee6b7-extensions-socket-volume\") on node \"ip-10-0-129-122.ec2.internal\" DevicePath \"\""
Sep 10 18:07:06.561298 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:07:06.561291    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fnt9n\" (UniqueName: \"kubernetes.io/projected/14e9bbf5-88d2-4de8-a033-cebd387ee6b7-kube-api-access-fnt9n\") on node \"ip-10-0-129-122.ec2.internal\" DevicePath \"\""
Sep 10 18:07:07.207059 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:07:07.207025    2579 generic.go:356] "Generic (PLEG): container finished" podID="14e9bbf5-88d2-4de8-a033-cebd387ee6b7" containerID="f70116414612aad2d72e73d9596438dd8a47ff803eeac0f85d038636dbe147be" exitCode=0
Sep 10 18:07:07.207589 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:07:07.207072    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-2m796"
Sep 10 18:07:07.207589 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:07:07.207113    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-2m796" event={"ID":"14e9bbf5-88d2-4de8-a033-cebd387ee6b7","Type":"ContainerDied","Data":"f70116414612aad2d72e73d9596438dd8a47ff803eeac0f85d038636dbe147be"}
Sep 10 18:07:07.207589 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:07:07.207168    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-2m796" event={"ID":"14e9bbf5-88d2-4de8-a033-cebd387ee6b7","Type":"ContainerDied","Data":"657e978ee644b4d10cf47bc81fe3175a95fa91c4a19ea24c13a029ddfe0f6d87"}
Sep 10 18:07:07.207589 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:07:07.207191    2579 scope.go:117] "RemoveContainer" containerID="f70116414612aad2d72e73d9596438dd8a47ff803eeac0f85d038636dbe147be"
Sep 10 18:07:07.217032 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:07:07.217005    2579 scope.go:117] "RemoveContainer" containerID="f70116414612aad2d72e73d9596438dd8a47ff803eeac0f85d038636dbe147be"
Sep 10 18:07:07.217325 ip-10-0-129-122 kubenswrapper[2579]: E0910 18:07:07.217304    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f70116414612aad2d72e73d9596438dd8a47ff803eeac0f85d038636dbe147be\": container with ID starting with f70116414612aad2d72e73d9596438dd8a47ff803eeac0f85d038636dbe147be not found: ID does not exist" containerID="f70116414612aad2d72e73d9596438dd8a47ff803eeac0f85d038636dbe147be"
Sep 10 18:07:07.217391 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:07:07.217333    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f70116414612aad2d72e73d9596438dd8a47ff803eeac0f85d038636dbe147be"} err="failed to get container status \"f70116414612aad2d72e73d9596438dd8a47ff803eeac0f85d038636dbe147be\": rpc error: code = NotFound desc = could not find container \"f70116414612aad2d72e73d9596438dd8a47ff803eeac0f85d038636dbe147be\": container with ID starting with f70116414612aad2d72e73d9596438dd8a47ff803eeac0f85d038636dbe147be not found: ID does not exist"
Sep 10 18:07:07.240221 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:07:07.240196    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-2m796"]
Sep 10 18:07:07.241709 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:07:07.241692    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-2m796"]
Sep 10 18:07:08.474277 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:07:08.474239    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="14e9bbf5-88d2-4de8-a033-cebd387ee6b7" path="/var/lib/kubelet/pods/14e9bbf5-88d2-4de8-a033-cebd387ee6b7/volumes"
Sep 10 18:08:06.838950 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:08:06.838907    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-dljf5"]
Sep 10 18:08:06.839464 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:08:06.839247    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="14e9bbf5-88d2-4de8-a033-cebd387ee6b7" containerName="manager"
Sep 10 18:08:06.839464 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:08:06.839261    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="14e9bbf5-88d2-4de8-a033-cebd387ee6b7" containerName="manager"
Sep 10 18:08:06.839464 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:08:06.839271    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9aa7132f-d1a1-49e5-a1ec-c3bb62984205" containerName="manager"
Sep 10 18:08:06.839464 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:08:06.839276    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="9aa7132f-d1a1-49e5-a1ec-c3bb62984205" containerName="manager"
Sep 10 18:08:06.839464 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:08:06.839330    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="22216a3a-3a36-4fb3-8980-4ea64db8a74b" containerName="istio-proxy"
Sep 10 18:08:06.839464 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:08:06.839338    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="3e479896-5ea5-4e33-9e1d-aecfd1368324" containerName="istio-proxy"
Sep 10 18:08:06.839464 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:08:06.839345    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="14e9bbf5-88d2-4de8-a033-cebd387ee6b7" containerName="manager"
Sep 10 18:08:06.839464 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:08:06.839352    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="d5268e09-33ea-4f48-97cc-24afd8b14c63" containerName="istio-proxy"
Sep 10 18:08:06.839464 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:08:06.839358    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="9aa7132f-d1a1-49e5-a1ec-c3bb62984205" containerName="manager"
Sep 10 18:08:06.842219 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:08:06.842192    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-dljf5"
Sep 10 18:08:06.845013 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:08:06.844988    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 10 18:08:06.845166 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:08:06.845146    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-85c54\""
Sep 10 18:08:06.845248 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:08:06.845191    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 10 18:08:06.856500 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:08:06.856476    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-dljf5"]
Sep 10 18:08:06.863665 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:08:06.863639    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b8f43ec3-634d-458d-9b05-c8c6d160d121-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-dljf5\" (UID: \"b8f43ec3-634d-458d-9b05-c8c6d160d121\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-dljf5"
Sep 10 18:08:06.863764 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:08:06.863719    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25jp8\" (UniqueName: \"kubernetes.io/projected/b8f43ec3-634d-458d-9b05-c8c6d160d121-kube-api-access-25jp8\") pod \"kuadrant-operator-controller-manager-5769f99d9f-dljf5\" (UID: \"b8f43ec3-634d-458d-9b05-c8c6d160d121\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-dljf5"
Sep 10 18:08:06.964376 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:08:06.964342    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b8f43ec3-634d-458d-9b05-c8c6d160d121-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-dljf5\" (UID: \"b8f43ec3-634d-458d-9b05-c8c6d160d121\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-dljf5"
Sep 10 18:08:06.964556 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:08:06.964442    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25jp8\" (UniqueName: \"kubernetes.io/projected/b8f43ec3-634d-458d-9b05-c8c6d160d121-kube-api-access-25jp8\") pod \"kuadrant-operator-controller-manager-5769f99d9f-dljf5\" (UID: \"b8f43ec3-634d-458d-9b05-c8c6d160d121\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-dljf5"
Sep 10 18:08:06.964726 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:08:06.964706    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b8f43ec3-634d-458d-9b05-c8c6d160d121-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-dljf5\" (UID: \"b8f43ec3-634d-458d-9b05-c8c6d160d121\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-dljf5"
Sep 10 18:08:06.973316 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:08:06.973294    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-25jp8\" (UniqueName: \"kubernetes.io/projected/b8f43ec3-634d-458d-9b05-c8c6d160d121-kube-api-access-25jp8\") pod \"kuadrant-operator-controller-manager-5769f99d9f-dljf5\" (UID: \"b8f43ec3-634d-458d-9b05-c8c6d160d121\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-dljf5"
Sep 10 18:08:07.152762 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:08:07.152727    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-dljf5"
Sep 10 18:08:07.427861 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:08:07.427838    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-dljf5"]
Sep 10 18:08:07.430263 ip-10-0-129-122 kubenswrapper[2579]: W0910 18:08:07.430230    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8f43ec3_634d_458d_9b05_c8c6d160d121.slice/crio-b428dd4bfc885cd896dfae73fbdea89daf148ad893473dcfe39e730c34cb3674 WatchSource:0}: Error finding container b428dd4bfc885cd896dfae73fbdea89daf148ad893473dcfe39e730c34cb3674: Status 404 returned error can't find the container with id b428dd4bfc885cd896dfae73fbdea89daf148ad893473dcfe39e730c34cb3674
Sep 10 18:08:07.492987 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:08:07.492957    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-dljf5" event={"ID":"b8f43ec3-634d-458d-9b05-c8c6d160d121","Type":"ContainerStarted","Data":"b428dd4bfc885cd896dfae73fbdea89daf148ad893473dcfe39e730c34cb3674"}
Sep 10 18:08:08.499020 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:08:08.498978    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-dljf5" event={"ID":"b8f43ec3-634d-458d-9b05-c8c6d160d121","Type":"ContainerStarted","Data":"631a902f8b25a0d3fa50c68aa65a420c6f40935ac325019c3cd523cfb2ff7c48"}
Sep 10 18:08:08.499411 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:08:08.499099    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-dljf5"
Sep 10 18:08:08.523279 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:08:08.523235    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-dljf5" podStartSLOduration=2.523221649 podStartE2EDuration="2.523221649s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 18:08:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 18:08:08.520683968 +0000 UTC m=+1932.520514036" watchObservedRunningTime="2026-09-10 18:08:08.523221649 +0000 UTC m=+1932.523051716"
Sep 10 18:08:19.505416 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:08:19.505381    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-dljf5"
Sep 10 18:08:41.818597 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:08:41.818545    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-d37e65ae-kserve-7b4ff7d575-xpp5t"]
Sep 10 18:08:41.822087 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:08:41.822061    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-d37e65ae-kserve-7b4ff7d575-xpp5t"
Sep 10 18:08:41.824895 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:08:41.824861    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-d37e65ae-kserve-self-signed-certs\""
Sep 10 18:08:41.832150 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:08:41.832124    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-d37e65ae-kserve-7b4ff7d575-xpp5t"]
Sep 10 18:08:41.972135 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:08:41.972100    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/50e8e7ff-d502-41f5-bf88-1fcd64f0a3cd-model-cache\") pod \"e2e-same-id-a-d37e65ae-kserve-7b4ff7d575-xpp5t\" (UID: \"50e8e7ff-d502-41f5-bf88-1fcd64f0a3cd\") " pod="llm/e2e-same-id-a-d37e65ae-kserve-7b4ff7d575-xpp5t"
Sep 10 18:08:41.972135 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:08:41.972140    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50e8e7ff-d502-41f5-bf88-1fcd64f0a3cd-tls-certs\") pod \"e2e-same-id-a-d37e65ae-kserve-7b4ff7d575-xpp5t\" (UID: \"50e8e7ff-d502-41f5-bf88-1fcd64f0a3cd\") " pod="llm/e2e-same-id-a-d37e65ae-kserve-7b4ff7d575-xpp5t"
Sep 10 18:08:41.972340 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:08:41.972159    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/50e8e7ff-d502-41f5-bf88-1fcd64f0a3cd-home\") pod \"e2e-same-id-a-d37e65ae-kserve-7b4ff7d575-xpp5t\" (UID: \"50e8e7ff-d502-41f5-bf88-1fcd64f0a3cd\") " pod="llm/e2e-same-id-a-d37e65ae-kserve-7b4ff7d575-xpp5t"
Sep 10 18:08:41.972340 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:08:41.972267    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/50e8e7ff-d502-41f5-bf88-1fcd64f0a3cd-dshm\") pod \"e2e-same-id-a-d37e65ae-kserve-7b4ff7d575-xpp5t\" (UID: \"50e8e7ff-d502-41f5-bf88-1fcd64f0a3cd\") " pod="llm/e2e-same-id-a-d37e65ae-kserve-7b4ff7d575-xpp5t"
Sep 10 18:08:41.972340 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:08:41.972321    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zc9kz\" (UniqueName: \"kubernetes.io/projected/50e8e7ff-d502-41f5-bf88-1fcd64f0a3cd-kube-api-access-zc9kz\") pod \"e2e-same-id-a-d37e65ae-kserve-7b4ff7d575-xpp5t\" (UID: \"50e8e7ff-d502-41f5-bf88-1fcd64f0a3cd\") " pod="llm/e2e-same-id-a-d37e65ae-kserve-7b4ff7d575-xpp5t"
Sep 10 18:08:41.972443 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:08:41.972376    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50e8e7ff-d502-41f5-bf88-1fcd64f0a3cd-tmp-dir\") pod \"e2e-same-id-a-d37e65ae-kserve-7b4ff7d575-xpp5t\" (UID: \"50e8e7ff-d502-41f5-bf88-1fcd64f0a3cd\") " pod="llm/e2e-same-id-a-d37e65ae-kserve-7b4ff7d575-xpp5t"
Sep 10 18:08:42.073369 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:08:42.073266    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50e8e7ff-d502-41f5-bf88-1fcd64f0a3cd-tmp-dir\") pod \"e2e-same-id-a-d37e65ae-kserve-7b4ff7d575-xpp5t\" (UID: \"50e8e7ff-d502-41f5-bf88-1fcd64f0a3cd\") " pod="llm/e2e-same-id-a-d37e65ae-kserve-7b4ff7d575-xpp5t"
Sep 10 18:08:42.073369 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:08:42.073337    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/50e8e7ff-d502-41f5-bf88-1fcd64f0a3cd-model-cache\") pod \"e2e-same-id-a-d37e65ae-kserve-7b4ff7d575-xpp5t\" (UID: \"50e8e7ff-d502-41f5-bf88-1fcd64f0a3cd\") " pod="llm/e2e-same-id-a-d37e65ae-kserve-7b4ff7d575-xpp5t"
Sep 10 18:08:42.073369 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:08:42.073366    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50e8e7ff-d502-41f5-bf88-1fcd64f0a3cd-tls-certs\") pod \"e2e-same-id-a-d37e65ae-kserve-7b4ff7d575-xpp5t\" (UID: \"50e8e7ff-d502-41f5-bf88-1fcd64f0a3cd\") " pod="llm/e2e-same-id-a-d37e65ae-kserve-7b4ff7d575-xpp5t"
Sep 10 18:08:42.073712 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:08:42.073383    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/50e8e7ff-d502-41f5-bf88-1fcd64f0a3cd-home\") pod \"e2e-same-id-a-d37e65ae-kserve-7b4ff7d575-xpp5t\" (UID: \"50e8e7ff-d502-41f5-bf88-1fcd64f0a3cd\") " pod="llm/e2e-same-id-a-d37e65ae-kserve-7b4ff7d575-xpp5t"
Sep 10 18:08:42.073712 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:08:42.073413    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/50e8e7ff-d502-41f5-bf88-1fcd64f0a3cd-dshm\") pod \"e2e-same-id-a-d37e65ae-kserve-7b4ff7d575-xpp5t\" (UID: \"50e8e7ff-d502-41f5-bf88-1fcd64f0a3cd\") " pod="llm/e2e-same-id-a-d37e65ae-kserve-7b4ff7d575-xpp5t"
Sep 10 18:08:42.073712 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:08:42.073452    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zc9kz\" (UniqueName: \"kubernetes.io/projected/50e8e7ff-d502-41f5-bf88-1fcd64f0a3cd-kube-api-access-zc9kz\") pod \"e2e-same-id-a-d37e65ae-kserve-7b4ff7d575-xpp5t\" (UID: \"50e8e7ff-d502-41f5-bf88-1fcd64f0a3cd\") " pod="llm/e2e-same-id-a-d37e65ae-kserve-7b4ff7d575-xpp5t"
Sep 10 18:08:42.073876 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:08:42.073760    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50e8e7ff-d502-41f5-bf88-1fcd64f0a3cd-tmp-dir\") pod \"e2e-same-id-a-d37e65ae-kserve-7b4ff7d575-xpp5t\" (UID: \"50e8e7ff-d502-41f5-bf88-1fcd64f0a3cd\") " pod="llm/e2e-same-id-a-d37e65ae-kserve-7b4ff7d575-xpp5t"
Sep 10 18:08:42.073876 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:08:42.073833    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/50e8e7ff-d502-41f5-bf88-1fcd64f0a3cd-model-cache\") pod \"e2e-same-id-a-d37e65ae-kserve-7b4ff7d575-xpp5t\" (UID: \"50e8e7ff-d502-41f5-bf88-1fcd64f0a3cd\") " pod="llm/e2e-same-id-a-d37e65ae-kserve-7b4ff7d575-xpp5t"
Sep 10 18:08:42.073876 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:08:42.073835    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/50e8e7ff-d502-41f5-bf88-1fcd64f0a3cd-home\") pod \"e2e-same-id-a-d37e65ae-kserve-7b4ff7d575-xpp5t\" (UID: \"50e8e7ff-d502-41f5-bf88-1fcd64f0a3cd\") " pod="llm/e2e-same-id-a-d37e65ae-kserve-7b4ff7d575-xpp5t"
Sep 10 18:08:42.076028 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:08:42.076004    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/50e8e7ff-d502-41f5-bf88-1fcd64f0a3cd-dshm\") pod \"e2e-same-id-a-d37e65ae-kserve-7b4ff7d575-xpp5t\" (UID: \"50e8e7ff-d502-41f5-bf88-1fcd64f0a3cd\") " pod="llm/e2e-same-id-a-d37e65ae-kserve-7b4ff7d575-xpp5t"
Sep 10 18:08:42.076318 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:08:42.076296    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50e8e7ff-d502-41f5-bf88-1fcd64f0a3cd-tls-certs\") pod \"e2e-same-id-a-d37e65ae-kserve-7b4ff7d575-xpp5t\" (UID: \"50e8e7ff-d502-41f5-bf88-1fcd64f0a3cd\") " pod="llm/e2e-same-id-a-d37e65ae-kserve-7b4ff7d575-xpp5t"
Sep 10 18:08:42.081812 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:08:42.081789    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zc9kz\" (UniqueName: \"kubernetes.io/projected/50e8e7ff-d502-41f5-bf88-1fcd64f0a3cd-kube-api-access-zc9kz\") pod \"e2e-same-id-a-d37e65ae-kserve-7b4ff7d575-xpp5t\" (UID: \"50e8e7ff-d502-41f5-bf88-1fcd64f0a3cd\") " pod="llm/e2e-same-id-a-d37e65ae-kserve-7b4ff7d575-xpp5t"
Sep 10 18:08:42.133491 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:08:42.133450    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-d37e65ae-kserve-7b4ff7d575-xpp5t"
Sep 10 18:08:42.405739 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:08:42.405710    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-d37e65ae-kserve-7b4ff7d575-xpp5t"]
Sep 10 18:08:42.407920 ip-10-0-129-122 kubenswrapper[2579]: W0910 18:08:42.407890    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod50e8e7ff_d502_41f5_bf88_1fcd64f0a3cd.slice/crio-770b508dd8e9a440c1a60f48a0a6d4db0677cb486abfb5fcc6f7543a19ebf2ea WatchSource:0}: Error finding container 770b508dd8e9a440c1a60f48a0a6d4db0677cb486abfb5fcc6f7543a19ebf2ea: Status 404 returned error can't find the container with id 770b508dd8e9a440c1a60f48a0a6d4db0677cb486abfb5fcc6f7543a19ebf2ea
Sep 10 18:08:42.648995 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:08:42.648954    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-d37e65ae-kserve-7b4ff7d575-xpp5t" event={"ID":"50e8e7ff-d502-41f5-bf88-1fcd64f0a3cd","Type":"ContainerStarted","Data":"9ff9735d855c4bfa8a4cea786d11cfecbb255b5b5265ae099df877b0eb27ff3d"}
Sep 10 18:08:42.649147 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:08:42.649002    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-d37e65ae-kserve-7b4ff7d575-xpp5t" event={"ID":"50e8e7ff-d502-41f5-bf88-1fcd64f0a3cd","Type":"ContainerStarted","Data":"770b508dd8e9a440c1a60f48a0a6d4db0677cb486abfb5fcc6f7543a19ebf2ea"}
Sep 10 18:08:42.665702 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:08:42.665611    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-d37e65ae-kserve-7b4ff7d575-xpp5t" podStartSLOduration=1.665598066 podStartE2EDuration="1.665598066s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 18:08:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 18:08:42.664030658 +0000 UTC m=+1966.663860716" watchObservedRunningTime="2026-09-10 18:08:42.665598066 +0000 UTC m=+1966.665428134"
Sep 10 18:08:52.134006 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:08:52.133915    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-d37e65ae-kserve-7b4ff7d575-xpp5t"
Sep 10 18:08:52.134006 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:08:52.133968    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-d37e65ae-kserve-7b4ff7d575-xpp5t"
Sep 10 18:08:52.146637 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:08:52.146615    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-d37e65ae-kserve-7b4ff7d575-xpp5t"
Sep 10 18:08:52.707616 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:08:52.707584    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-d37e65ae-kserve-7b4ff7d575-xpp5t"
Sep 10 18:09:11.085160 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:09:11.085124    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-d37e65ae-kserve-7b4ff7d575-xpp5t"]
Sep 10 18:09:26.106186 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:09:26.106122    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-d37e65ae-kserve-7b4ff7d575-xpp5t" podUID="50e8e7ff-d502-41f5-bf88-1fcd64f0a3cd" containerName="main" containerID="cri-o://9ff9735d855c4bfa8a4cea786d11cfecbb255b5b5265ae099df877b0eb27ff3d" gracePeriod=45
Sep 10 18:09:26.364541 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:09:26.364478    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-d37e65ae-kserve-7b4ff7d575-xpp5t"
Sep 10 18:09:26.438519 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:09:26.438475    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/50e8e7ff-d502-41f5-bf88-1fcd64f0a3cd-home\") pod \"50e8e7ff-d502-41f5-bf88-1fcd64f0a3cd\" (UID: \"50e8e7ff-d502-41f5-bf88-1fcd64f0a3cd\") "
Sep 10 18:09:26.438519 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:09:26.438525    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zc9kz\" (UniqueName: \"kubernetes.io/projected/50e8e7ff-d502-41f5-bf88-1fcd64f0a3cd-kube-api-access-zc9kz\") pod \"50e8e7ff-d502-41f5-bf88-1fcd64f0a3cd\" (UID: \"50e8e7ff-d502-41f5-bf88-1fcd64f0a3cd\") "
Sep 10 18:09:26.438806 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:09:26.438546    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/50e8e7ff-d502-41f5-bf88-1fcd64f0a3cd-model-cache\") pod \"50e8e7ff-d502-41f5-bf88-1fcd64f0a3cd\" (UID: \"50e8e7ff-d502-41f5-bf88-1fcd64f0a3cd\") "
Sep 10 18:09:26.438806 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:09:26.438600    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50e8e7ff-d502-41f5-bf88-1fcd64f0a3cd-tmp-dir\") pod \"50e8e7ff-d502-41f5-bf88-1fcd64f0a3cd\" (UID: \"50e8e7ff-d502-41f5-bf88-1fcd64f0a3cd\") "
Sep 10 18:09:26.438806 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:09:26.438618    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/50e8e7ff-d502-41f5-bf88-1fcd64f0a3cd-dshm\") pod \"50e8e7ff-d502-41f5-bf88-1fcd64f0a3cd\" (UID: \"50e8e7ff-d502-41f5-bf88-1fcd64f0a3cd\") "
Sep 10 18:09:26.438806 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:09:26.438657    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50e8e7ff-d502-41f5-bf88-1fcd64f0a3cd-tls-certs\") pod \"50e8e7ff-d502-41f5-bf88-1fcd64f0a3cd\" (UID: \"50e8e7ff-d502-41f5-bf88-1fcd64f0a3cd\") "
Sep 10 18:09:26.439019 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:09:26.438814    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50e8e7ff-d502-41f5-bf88-1fcd64f0a3cd-home" (OuterVolumeSpecName: "home") pod "50e8e7ff-d502-41f5-bf88-1fcd64f0a3cd" (UID: "50e8e7ff-d502-41f5-bf88-1fcd64f0a3cd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 18:09:26.439019 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:09:26.438878    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50e8e7ff-d502-41f5-bf88-1fcd64f0a3cd-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "50e8e7ff-d502-41f5-bf88-1fcd64f0a3cd" (UID: "50e8e7ff-d502-41f5-bf88-1fcd64f0a3cd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 18:09:26.439019 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:09:26.438930    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50e8e7ff-d502-41f5-bf88-1fcd64f0a3cd-model-cache" (OuterVolumeSpecName: "model-cache") pod "50e8e7ff-d502-41f5-bf88-1fcd64f0a3cd" (UID: "50e8e7ff-d502-41f5-bf88-1fcd64f0a3cd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 18:09:26.441006 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:09:26.440979    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50e8e7ff-d502-41f5-bf88-1fcd64f0a3cd-dshm" (OuterVolumeSpecName: "dshm") pod "50e8e7ff-d502-41f5-bf88-1fcd64f0a3cd" (UID: "50e8e7ff-d502-41f5-bf88-1fcd64f0a3cd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 18:09:26.441102 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:09:26.441069    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50e8e7ff-d502-41f5-bf88-1fcd64f0a3cd-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "50e8e7ff-d502-41f5-bf88-1fcd64f0a3cd" (UID: "50e8e7ff-d502-41f5-bf88-1fcd64f0a3cd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 18:09:26.441162 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:09:26.441143    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50e8e7ff-d502-41f5-bf88-1fcd64f0a3cd-kube-api-access-zc9kz" (OuterVolumeSpecName: "kube-api-access-zc9kz") pod "50e8e7ff-d502-41f5-bf88-1fcd64f0a3cd" (UID: "50e8e7ff-d502-41f5-bf88-1fcd64f0a3cd"). InnerVolumeSpecName "kube-api-access-zc9kz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 18:09:26.540233 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:09:26.540190    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zc9kz\" (UniqueName: \"kubernetes.io/projected/50e8e7ff-d502-41f5-bf88-1fcd64f0a3cd-kube-api-access-zc9kz\") on node \"ip-10-0-129-122.ec2.internal\" DevicePath \"\""
Sep 10 18:09:26.540233 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:09:26.540225    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/50e8e7ff-d502-41f5-bf88-1fcd64f0a3cd-model-cache\") on node \"ip-10-0-129-122.ec2.internal\" DevicePath \"\""
Sep 10 18:09:26.540233 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:09:26.540237    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50e8e7ff-d502-41f5-bf88-1fcd64f0a3cd-tmp-dir\") on node \"ip-10-0-129-122.ec2.internal\" DevicePath \"\""
Sep 10 18:09:26.540489 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:09:26.540248    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/50e8e7ff-d502-41f5-bf88-1fcd64f0a3cd-dshm\") on node \"ip-10-0-129-122.ec2.internal\" DevicePath \"\""
Sep 10 18:09:26.540489 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:09:26.540259    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50e8e7ff-d502-41f5-bf88-1fcd64f0a3cd-tls-certs\") on node \"ip-10-0-129-122.ec2.internal\" DevicePath \"\""
Sep 10 18:09:26.540489 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:09:26.540269    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/50e8e7ff-d502-41f5-bf88-1fcd64f0a3cd-home\") on node \"ip-10-0-129-122.ec2.internal\" DevicePath \"\""
Sep 10 18:09:26.857384 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:09:26.857353    2579 generic.go:356] "Generic (PLEG): container finished" podID="50e8e7ff-d502-41f5-bf88-1fcd64f0a3cd" containerID="9ff9735d855c4bfa8a4cea786d11cfecbb255b5b5265ae099df877b0eb27ff3d" exitCode=0
Sep 10 18:09:26.857582 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:09:26.857400    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-d37e65ae-kserve-7b4ff7d575-xpp5t"
Sep 10 18:09:26.857582 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:09:26.857446    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-d37e65ae-kserve-7b4ff7d575-xpp5t" event={"ID":"50e8e7ff-d502-41f5-bf88-1fcd64f0a3cd","Type":"ContainerDied","Data":"9ff9735d855c4bfa8a4cea786d11cfecbb255b5b5265ae099df877b0eb27ff3d"}
Sep 10 18:09:26.857582 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:09:26.857495    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-d37e65ae-kserve-7b4ff7d575-xpp5t" event={"ID":"50e8e7ff-d502-41f5-bf88-1fcd64f0a3cd","Type":"ContainerDied","Data":"770b508dd8e9a440c1a60f48a0a6d4db0677cb486abfb5fcc6f7543a19ebf2ea"}
Sep 10 18:09:26.857582 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:09:26.857515    2579 scope.go:117] "RemoveContainer" containerID="9ff9735d855c4bfa8a4cea786d11cfecbb255b5b5265ae099df877b0eb27ff3d"
Sep 10 18:09:26.866177 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:09:26.866157    2579 scope.go:117] "RemoveContainer" containerID="9ff9735d855c4bfa8a4cea786d11cfecbb255b5b5265ae099df877b0eb27ff3d"
Sep 10 18:09:26.866492 ip-10-0-129-122 kubenswrapper[2579]: E0910 18:09:26.866469    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ff9735d855c4bfa8a4cea786d11cfecbb255b5b5265ae099df877b0eb27ff3d\": container with ID starting with 9ff9735d855c4bfa8a4cea786d11cfecbb255b5b5265ae099df877b0eb27ff3d not found: ID does not exist" containerID="9ff9735d855c4bfa8a4cea786d11cfecbb255b5b5265ae099df877b0eb27ff3d"
Sep 10 18:09:26.866595 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:09:26.866502    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ff9735d855c4bfa8a4cea786d11cfecbb255b5b5265ae099df877b0eb27ff3d"} err="failed to get container status \"9ff9735d855c4bfa8a4cea786d11cfecbb255b5b5265ae099df877b0eb27ff3d\": rpc error: code = NotFound desc = could not find container \"9ff9735d855c4bfa8a4cea786d11cfecbb255b5b5265ae099df877b0eb27ff3d\": container with ID starting with 9ff9735d855c4bfa8a4cea786d11cfecbb255b5b5265ae099df877b0eb27ff3d not found: ID does not exist"
Sep 10 18:09:26.874684 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:09:26.874654    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-d37e65ae-kserve-7b4ff7d575-xpp5t"]
Sep 10 18:09:26.879924 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:09:26.879896    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-d37e65ae-kserve-7b4ff7d575-xpp5t"]
Sep 10 18:09:28.474693 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:09:28.474659    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="50e8e7ff-d502-41f5-bf88-1fcd64f0a3cd" path="/var/lib/kubelet/pods/50e8e7ff-d502-41f5-bf88-1fcd64f0a3cd/volumes"
Sep 10 18:09:30.071702 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:09:30.071664    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-84b747cc96-k689b"]
Sep 10 18:09:30.072189 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:09:30.071999    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50e8e7ff-d502-41f5-bf88-1fcd64f0a3cd" containerName="main"
Sep 10 18:09:30.072189 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:09:30.072009    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="50e8e7ff-d502-41f5-bf88-1fcd64f0a3cd" containerName="main"
Sep 10 18:09:30.072189 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:09:30.072075    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="50e8e7ff-d502-41f5-bf88-1fcd64f0a3cd" containerName="main"
Sep 10 18:09:30.075020 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:09:30.075000    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-84b747cc96-k689b"
Sep 10 18:09:30.077431 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:09:30.077410    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-tdclw\""
Sep 10 18:09:30.086491 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:09:30.086468    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-84b747cc96-k689b"]
Sep 10 18:09:30.175506 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:09:30.175473    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/57a51c10-f155-42a6-9b85-5ca71fec3668-openshift-service-ca\") pod \"maas-api-84b747cc96-k689b\" (UID: \"57a51c10-f155-42a6-9b85-5ca71fec3668\") " pod="odh-ai-gateway-infra/maas-api-84b747cc96-k689b"
Sep 10 18:09:30.175687 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:09:30.175509    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cv7h2\" (UniqueName: \"kubernetes.io/projected/57a51c10-f155-42a6-9b85-5ca71fec3668-kube-api-access-cv7h2\") pod \"maas-api-84b747cc96-k689b\" (UID: \"57a51c10-f155-42a6-9b85-5ca71fec3668\") " pod="odh-ai-gateway-infra/maas-api-84b747cc96-k689b"
Sep 10 18:09:30.175687 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:09:30.175535    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57a51c10-f155-42a6-9b85-5ca71fec3668-metrics-certs\") pod \"maas-api-84b747cc96-k689b\" (UID: \"57a51c10-f155-42a6-9b85-5ca71fec3668\") " pod="odh-ai-gateway-infra/maas-api-84b747cc96-k689b"
Sep 10 18:09:30.175687 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:09:30.175618    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/57a51c10-f155-42a6-9b85-5ca71fec3668-maas-api-tls\") pod \"maas-api-84b747cc96-k689b\" (UID: \"57a51c10-f155-42a6-9b85-5ca71fec3668\") " pod="odh-ai-gateway-infra/maas-api-84b747cc96-k689b"
Sep 10 18:09:30.276486 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:09:30.276446    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/57a51c10-f155-42a6-9b85-5ca71fec3668-openshift-service-ca\") pod \"maas-api-84b747cc96-k689b\" (UID: \"57a51c10-f155-42a6-9b85-5ca71fec3668\") " pod="odh-ai-gateway-infra/maas-api-84b747cc96-k689b"
Sep 10 18:09:30.276486 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:09:30.276485    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cv7h2\" (UniqueName: \"kubernetes.io/projected/57a51c10-f155-42a6-9b85-5ca71fec3668-kube-api-access-cv7h2\") pod \"maas-api-84b747cc96-k689b\" (UID: \"57a51c10-f155-42a6-9b85-5ca71fec3668\") " pod="odh-ai-gateway-infra/maas-api-84b747cc96-k689b"
Sep 10 18:09:30.276742 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:09:30.276504    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57a51c10-f155-42a6-9b85-5ca71fec3668-metrics-certs\") pod \"maas-api-84b747cc96-k689b\" (UID: \"57a51c10-f155-42a6-9b85-5ca71fec3668\") " pod="odh-ai-gateway-infra/maas-api-84b747cc96-k689b"
Sep 10 18:09:30.276742 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:09:30.276534    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/57a51c10-f155-42a6-9b85-5ca71fec3668-maas-api-tls\") pod \"maas-api-84b747cc96-k689b\" (UID: \"57a51c10-f155-42a6-9b85-5ca71fec3668\") " pod="odh-ai-gateway-infra/maas-api-84b747cc96-k689b"
Sep 10 18:09:30.277131 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:09:30.277109    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/57a51c10-f155-42a6-9b85-5ca71fec3668-openshift-service-ca\") pod \"maas-api-84b747cc96-k689b\" (UID: \"57a51c10-f155-42a6-9b85-5ca71fec3668\") " pod="odh-ai-gateway-infra/maas-api-84b747cc96-k689b"
Sep 10 18:09:30.279233 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:09:30.279210    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57a51c10-f155-42a6-9b85-5ca71fec3668-metrics-certs\") pod \"maas-api-84b747cc96-k689b\" (UID: \"57a51c10-f155-42a6-9b85-5ca71fec3668\") " pod="odh-ai-gateway-infra/maas-api-84b747cc96-k689b"
Sep 10 18:09:30.279341 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:09:30.279232    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/57a51c10-f155-42a6-9b85-5ca71fec3668-maas-api-tls\") pod \"maas-api-84b747cc96-k689b\" (UID: \"57a51c10-f155-42a6-9b85-5ca71fec3668\") " pod="odh-ai-gateway-infra/maas-api-84b747cc96-k689b"
Sep 10 18:09:30.284011 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:09:30.283990    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cv7h2\" (UniqueName: \"kubernetes.io/projected/57a51c10-f155-42a6-9b85-5ca71fec3668-kube-api-access-cv7h2\") pod \"maas-api-84b747cc96-k689b\" (UID: \"57a51c10-f155-42a6-9b85-5ca71fec3668\") " pod="odh-ai-gateway-infra/maas-api-84b747cc96-k689b"
Sep 10 18:09:30.386326 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:09:30.386290    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-84b747cc96-k689b"
Sep 10 18:09:30.657612 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:09:30.656405    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-84b747cc96-k689b"]
Sep 10 18:09:30.878820 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:09:30.878783    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-84b747cc96-k689b" event={"ID":"57a51c10-f155-42a6-9b85-5ca71fec3668","Type":"ContainerStarted","Data":"6e350e0a3631521ae247327e7968ea6297eb64a205b5881af3738426da837d16"}
Sep 10 18:09:31.885327 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:09:31.885293    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-84b747cc96-k689b" event={"ID":"57a51c10-f155-42a6-9b85-5ca71fec3668","Type":"ContainerStarted","Data":"595eb52c8733c437c89e2c326cbc17becc2cf718686ad5a797473ae4b56b0f28"}
Sep 10 18:09:31.885744 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:09:31.885413    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-84b747cc96-k689b"
Sep 10 18:09:31.901685 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:09:31.901634    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-84b747cc96-k689b" podStartSLOduration=1.536431366 podStartE2EDuration="1.90161751s" totalImagesPullingTime="365.186144ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 18:09:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 18:09:31.901406444 +0000 UTC m=+2015.901236511" watchObservedRunningTime="2026-09-10 18:09:31.90161751 +0000 UTC m=+2015.901447577"
Sep 10 18:09:37.895762 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:09:37.895733    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-84b747cc96-k689b"
Sep 10 18:09:37.942944 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:09:37.942909    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6ff67d7f48-b46rw"]
Sep 10 18:09:37.943203 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:09:37.943160    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-6ff67d7f48-b46rw" podUID="4ba160dc-d064-4f2d-8bd1-dcebd0855f10" containerName="maas-api" containerID="cri-o://25c5e471389bfaccb91cb9cc72f2eaffd805f7798757f2eabefe0978b7b2fa31" gracePeriod=30
Sep 10 18:09:38.088937 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:09:38.088895    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-6ff67d7f48-b46rw" podUID="4ba160dc-d064-4f2d-8bd1-dcebd0855f10" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.38:8443/health\": dial tcp 10.132.0.38:8443: connect: connection refused"
Sep 10 18:09:39.205585 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:09:39.205540    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6ff67d7f48-b46rw"
Sep 10 18:09:39.353487 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:09:39.353396    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ba160dc-d064-4f2d-8bd1-dcebd0855f10-openshift-service-ca\") pod \"4ba160dc-d064-4f2d-8bd1-dcebd0855f10\" (UID: \"4ba160dc-d064-4f2d-8bd1-dcebd0855f10\") "
Sep 10 18:09:39.353487 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:09:39.353434    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4ba160dc-d064-4f2d-8bd1-dcebd0855f10-maas-api-tls\") pod \"4ba160dc-d064-4f2d-8bd1-dcebd0855f10\" (UID: \"4ba160dc-d064-4f2d-8bd1-dcebd0855f10\") "
Sep 10 18:09:39.353714 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:09:39.353498    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6bhb6\" (UniqueName: \"kubernetes.io/projected/4ba160dc-d064-4f2d-8bd1-dcebd0855f10-kube-api-access-6bhb6\") pod \"4ba160dc-d064-4f2d-8bd1-dcebd0855f10\" (UID: \"4ba160dc-d064-4f2d-8bd1-dcebd0855f10\") "
Sep 10 18:09:39.353714 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:09:39.353551    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4ba160dc-d064-4f2d-8bd1-dcebd0855f10-metrics-certs\") pod \"4ba160dc-d064-4f2d-8bd1-dcebd0855f10\" (UID: \"4ba160dc-d064-4f2d-8bd1-dcebd0855f10\") "
Sep 10 18:09:39.353880 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:09:39.353852    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ba160dc-d064-4f2d-8bd1-dcebd0855f10-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "4ba160dc-d064-4f2d-8bd1-dcebd0855f10" (UID: "4ba160dc-d064-4f2d-8bd1-dcebd0855f10"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 18:09:39.355938 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:09:39.355916    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ba160dc-d064-4f2d-8bd1-dcebd0855f10-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "4ba160dc-d064-4f2d-8bd1-dcebd0855f10" (UID: "4ba160dc-d064-4f2d-8bd1-dcebd0855f10"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 18:09:39.356051 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:09:39.355939    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ba160dc-d064-4f2d-8bd1-dcebd0855f10-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "4ba160dc-d064-4f2d-8bd1-dcebd0855f10" (UID: "4ba160dc-d064-4f2d-8bd1-dcebd0855f10"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 18:09:39.356051 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:09:39.355954    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ba160dc-d064-4f2d-8bd1-dcebd0855f10-kube-api-access-6bhb6" (OuterVolumeSpecName: "kube-api-access-6bhb6") pod "4ba160dc-d064-4f2d-8bd1-dcebd0855f10" (UID: "4ba160dc-d064-4f2d-8bd1-dcebd0855f10"). InnerVolumeSpecName "kube-api-access-6bhb6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 18:09:39.454328 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:09:39.454292    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6bhb6\" (UniqueName: \"kubernetes.io/projected/4ba160dc-d064-4f2d-8bd1-dcebd0855f10-kube-api-access-6bhb6\") on node \"ip-10-0-129-122.ec2.internal\" DevicePath \"\""
Sep 10 18:09:39.454328 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:09:39.454322    2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4ba160dc-d064-4f2d-8bd1-dcebd0855f10-metrics-certs\") on node \"ip-10-0-129-122.ec2.internal\" DevicePath \"\""
Sep 10 18:09:39.454328 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:09:39.454335    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ba160dc-d064-4f2d-8bd1-dcebd0855f10-openshift-service-ca\") on node \"ip-10-0-129-122.ec2.internal\" DevicePath \"\""
Sep 10 18:09:39.454556 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:09:39.454347    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4ba160dc-d064-4f2d-8bd1-dcebd0855f10-maas-api-tls\") on node \"ip-10-0-129-122.ec2.internal\" DevicePath \"\""
Sep 10 18:09:39.925646 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:09:39.925610    2579 generic.go:356] "Generic (PLEG): container finished" podID="4ba160dc-d064-4f2d-8bd1-dcebd0855f10" containerID="25c5e471389bfaccb91cb9cc72f2eaffd805f7798757f2eabefe0978b7b2fa31" exitCode=0
Sep 10 18:09:39.925853 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:09:39.925663    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6ff67d7f48-b46rw"
Sep 10 18:09:39.925853 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:09:39.925694    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6ff67d7f48-b46rw" event={"ID":"4ba160dc-d064-4f2d-8bd1-dcebd0855f10","Type":"ContainerDied","Data":"25c5e471389bfaccb91cb9cc72f2eaffd805f7798757f2eabefe0978b7b2fa31"}
Sep 10 18:09:39.925853 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:09:39.925748    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6ff67d7f48-b46rw" event={"ID":"4ba160dc-d064-4f2d-8bd1-dcebd0855f10","Type":"ContainerDied","Data":"eb0329a74c16ab13600d2a950f7c4f99e36974b4e691601944f1bab5b3996d98"}
Sep 10 18:09:39.925853 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:09:39.925771    2579 scope.go:117] "RemoveContainer" containerID="25c5e471389bfaccb91cb9cc72f2eaffd805f7798757f2eabefe0978b7b2fa31"
Sep 10 18:09:39.934731 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:09:39.934715    2579 scope.go:117] "RemoveContainer" containerID="25c5e471389bfaccb91cb9cc72f2eaffd805f7798757f2eabefe0978b7b2fa31"
Sep 10 18:09:39.934981 ip-10-0-129-122 kubenswrapper[2579]: E0910 18:09:39.934963    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25c5e471389bfaccb91cb9cc72f2eaffd805f7798757f2eabefe0978b7b2fa31\": container with ID starting with 25c5e471389bfaccb91cb9cc72f2eaffd805f7798757f2eabefe0978b7b2fa31 not found: ID does not exist" containerID="25c5e471389bfaccb91cb9cc72f2eaffd805f7798757f2eabefe0978b7b2fa31"
Sep 10 18:09:39.935027 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:09:39.934990    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25c5e471389bfaccb91cb9cc72f2eaffd805f7798757f2eabefe0978b7b2fa31"} err="failed to get container status \"25c5e471389bfaccb91cb9cc72f2eaffd805f7798757f2eabefe0978b7b2fa31\": rpc error: code = NotFound desc = could not find container \"25c5e471389bfaccb91cb9cc72f2eaffd805f7798757f2eabefe0978b7b2fa31\": container with ID starting with 25c5e471389bfaccb91cb9cc72f2eaffd805f7798757f2eabefe0978b7b2fa31 not found: ID does not exist"
Sep 10 18:09:39.946659 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:09:39.946635    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6ff67d7f48-b46rw"]
Sep 10 18:09:39.950374 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:09:39.950352    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-6ff67d7f48-b46rw"]
Sep 10 18:09:40.475615 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:09:40.475585    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4ba160dc-d064-4f2d-8bd1-dcebd0855f10" path="/var/lib/kubelet/pods/4ba160dc-d064-4f2d-8bd1-dcebd0855f10/volumes"
Sep 10 18:10:05.947239 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:10:05.947183    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-dbf885bd4-txcxf_19472691-efec-4186-9c11-10dc71432e71/copy-manifests/0.log"
Sep 10 18:10:05.959720 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:10:05.959694    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-dbf885bd4-txcxf_19472691-efec-4186-9c11-10dc71432e71/manager/0.log"
Sep 10 18:10:06.474543 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:10:06.474511    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-84b747cc96-k689b_57a51c10-f155-42a6-9b85-5ca71fec3668/maas-api/0.log"
Sep 10 18:10:06.792742 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:10:06.792665    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-f765s_bf74a5e6-2847-46e9-8131-f44fad693339/postgres/0.log"
Sep 10 18:10:08.007778 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:10:08.007747    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-dljf5_b8f43ec3-634d-458d-9b05-c8c6d160d121/manager/0.log"
Sep 10 18:10:09.340053 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:10:09.340006    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-kpwl9_fcfe8a95-b4b5-4997-8d4b-717a1f3d8149/payload-processing/0.log"
Sep 10 18:10:09.455295 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:10:09.455256    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-d7f986b9d-xvnsc_678c4830-c56e-4af8-85a2-6f26a4e03aa8/router/0.log"
Sep 10 18:10:10.116652 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:10:10.116623    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-4jfr8_254aaaed-8976-4976-88e6-d0aa5cb2a84e/main/0.log"
Sep 10 18:10:10.229431 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:10:10.229403    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f76js2_98b3e5d2-a3c9-41fe-9a22-e36ba7760daa/main/0.log"
Sep 10 18:10:16.203650 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:10:16.203614    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-pqhnx/must-gather-4k5lk"]
Sep 10 18:10:16.204038 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:10:16.203973    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ba160dc-d064-4f2d-8bd1-dcebd0855f10" containerName="maas-api"
Sep 10 18:10:16.204038 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:10:16.203985    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ba160dc-d064-4f2d-8bd1-dcebd0855f10" containerName="maas-api"
Sep 10 18:10:16.204038 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:10:16.204037    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="4ba160dc-d064-4f2d-8bd1-dcebd0855f10" containerName="maas-api"
Sep 10 18:10:16.207242 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:10:16.207221    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-pqhnx/must-gather-4k5lk"
Sep 10 18:10:16.209978 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:10:16.209952    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-pqhnx\"/\"default-dockercfg-6n2nd\""
Sep 10 18:10:16.211020 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:10:16.211004    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-pqhnx\"/\"kube-root-ca.crt\""
Sep 10 18:10:16.211091 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:10:16.211034    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-pqhnx\"/\"openshift-service-ca.crt\""
Sep 10 18:10:16.215352 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:10:16.215325    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-pqhnx/must-gather-4k5lk"]
Sep 10 18:10:16.277543 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:10:16.277506    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/ec576099-adae-46df-b135-832272404594-must-gather-output\") pod \"must-gather-4k5lk\" (UID: \"ec576099-adae-46df-b135-832272404594\") " pod="openshift-must-gather-pqhnx/must-gather-4k5lk"
Sep 10 18:10:16.277733 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:10:16.277557    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mmbd\" (UniqueName: \"kubernetes.io/projected/ec576099-adae-46df-b135-832272404594-kube-api-access-2mmbd\") pod \"must-gather-4k5lk\" (UID: \"ec576099-adae-46df-b135-832272404594\") " pod="openshift-must-gather-pqhnx/must-gather-4k5lk"
Sep 10 18:10:16.378375 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:10:16.378337    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/ec576099-adae-46df-b135-832272404594-must-gather-output\") pod \"must-gather-4k5lk\" (UID: \"ec576099-adae-46df-b135-832272404594\") " pod="openshift-must-gather-pqhnx/must-gather-4k5lk"
Sep 10 18:10:16.378513 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:10:16.378382    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mmbd\" (UniqueName: \"kubernetes.io/projected/ec576099-adae-46df-b135-832272404594-kube-api-access-2mmbd\") pod \"must-gather-4k5lk\" (UID: \"ec576099-adae-46df-b135-832272404594\") " pod="openshift-must-gather-pqhnx/must-gather-4k5lk"
Sep 10 18:10:16.378705 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:10:16.378684    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/ec576099-adae-46df-b135-832272404594-must-gather-output\") pod \"must-gather-4k5lk\" (UID: \"ec576099-adae-46df-b135-832272404594\") " pod="openshift-must-gather-pqhnx/must-gather-4k5lk"
Sep 10 18:10:16.408497 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:10:16.408460    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mmbd\" (UniqueName: \"kubernetes.io/projected/ec576099-adae-46df-b135-832272404594-kube-api-access-2mmbd\") pod \"must-gather-4k5lk\" (UID: \"ec576099-adae-46df-b135-832272404594\") " pod="openshift-must-gather-pqhnx/must-gather-4k5lk"
Sep 10 18:10:16.516724 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:10:16.516625    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-pqhnx/must-gather-4k5lk"
Sep 10 18:10:16.788402 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:10:16.788346    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-pqhnx/must-gather-4k5lk"]
Sep 10 18:10:16.790792 ip-10-0-129-122 kubenswrapper[2579]: W0910 18:10:16.790763    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podec576099_adae_46df_b135_832272404594.slice/crio-8715bfef70330ed845b15f89531360d71653e437937386d8b27dd0583d0a3d45 WatchSource:0}: Error finding container 8715bfef70330ed845b15f89531360d71653e437937386d8b27dd0583d0a3d45: Status 404 returned error can't find the container with id 8715bfef70330ed845b15f89531360d71653e437937386d8b27dd0583d0a3d45
Sep 10 18:10:17.106690 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:10:17.106598    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-pqhnx/must-gather-4k5lk" event={"ID":"ec576099-adae-46df-b135-832272404594","Type":"ContainerStarted","Data":"8715bfef70330ed845b15f89531360d71653e437937386d8b27dd0583d0a3d45"}
Sep 10 18:10:18.115622 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:10:18.115587    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-pqhnx/must-gather-4k5lk" event={"ID":"ec576099-adae-46df-b135-832272404594","Type":"ContainerStarted","Data":"71f02d8bd7d878aca5e9c40246189a7f11174f165c54fcd03dfcb93b52f0efc9"}
Sep 10 18:10:18.116003 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:10:18.115631    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-pqhnx/must-gather-4k5lk" event={"ID":"ec576099-adae-46df-b135-832272404594","Type":"ContainerStarted","Data":"a2e4cb76c0794fefa53495321c124bbe1bc783a10bd80276179b1319f75d16ee"}
Sep 10 18:10:18.132829 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:10:18.132776    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-pqhnx/must-gather-4k5lk" podStartSLOduration=1.38670679 podStartE2EDuration="2.132760451s" totalImagesPullingTime="746.053661ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 18:10:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 18:10:18.130552661 +0000 UTC m=+2062.130382741" watchObservedRunningTime="2026-09-10 18:10:18.132760451 +0000 UTC m=+2062.132590517"
Sep 10 18:10:19.173371 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:10:19.173340    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-m4npt_e0fe9300-104f-489d-9962-908000359645/global-pull-secret-syncer/0.log"
Sep 10 18:10:19.297398 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:10:19.297365    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-lpm4k_1cd8f80e-f5c5-4e80-9abb-b43af15ed1d1/konnectivity-agent/0.log"
Sep 10 18:10:19.317443 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:10:19.317416    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-129-122.ec2.internal_bd750ec1329c03eb8935ed3e9791d542/haproxy/0.log"
Sep 10 18:10:24.335354 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:10:24.335265    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-dljf5_b8f43ec3-634d-458d-9b05-c8c6d160d121/manager/0.log"
Sep 10 18:10:25.977363 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:10:25.977320    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6cdcc96965-5cbrn_6976caf5-0ff1-4604-9e30-aab464b54edf/cluster-monitoring-operator/0.log"
Sep 10 18:10:26.219887 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:10:26.219845    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-pjp7g_b1681ff4-3ace-4de2-b0de-56cf8551c284/node-exporter/0.log"
Sep 10 18:10:26.260123 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:10:26.260037    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-pjp7g_b1681ff4-3ace-4de2-b0de-56cf8551c284/kube-rbac-proxy/0.log"
Sep 10 18:10:26.280195 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:10:26.280170    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-pjp7g_b1681ff4-3ace-4de2-b0de-56cf8551c284/init-textfile/0.log"
Sep 10 18:10:26.771612 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:10:26.771551    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-78f599db79-6kgsw_c1fc82a8-9158-4f94-be7e-e5dbea38a73d/prometheus-operator-admission-webhook/0.log"
Sep 10 18:10:28.490289 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:10:28.490250    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-79977f9844-5klhd_510bea53-93c2-4158-851a-0b6ec3e6bee6/networking-console-plugin/0.log"
Sep 10 18:10:29.103021 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:10:29.102985    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-rb854_97172b14-b84a-476e-89f1-05d3e70df416/console-operator/1.log"
Sep 10 18:10:29.109178 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:10:29.109147    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-rb854_97172b14-b84a-476e-89f1-05d3e70df416/console-operator/2.log"
Sep 10 18:10:29.295913 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:10:29.295858    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-pqhnx/perf-node-gather-daemonset-gp52j"]
Sep 10 18:10:29.303994 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:10:29.303959    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-pqhnx/perf-node-gather-daemonset-gp52j"
Sep 10 18:10:29.306061 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:10:29.306030    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-pqhnx/perf-node-gather-daemonset-gp52j"]
Sep 10 18:10:29.411851 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:10:29.411800    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/9f356d2b-20bb-4b15-ad44-a4788dce956a-podres\") pod \"perf-node-gather-daemonset-gp52j\" (UID: \"9f356d2b-20bb-4b15-ad44-a4788dce956a\") " pod="openshift-must-gather-pqhnx/perf-node-gather-daemonset-gp52j"
Sep 10 18:10:29.412027 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:10:29.411884    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tw4sz\" (UniqueName: \"kubernetes.io/projected/9f356d2b-20bb-4b15-ad44-a4788dce956a-kube-api-access-tw4sz\") pod \"perf-node-gather-daemonset-gp52j\" (UID: \"9f356d2b-20bb-4b15-ad44-a4788dce956a\") " pod="openshift-must-gather-pqhnx/perf-node-gather-daemonset-gp52j"
Sep 10 18:10:29.412027 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:10:29.411921    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9f356d2b-20bb-4b15-ad44-a4788dce956a-lib-modules\") pod \"perf-node-gather-daemonset-gp52j\" (UID: \"9f356d2b-20bb-4b15-ad44-a4788dce956a\") " pod="openshift-must-gather-pqhnx/perf-node-gather-daemonset-gp52j"
Sep 10 18:10:29.412027 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:10:29.411996    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/9f356d2b-20bb-4b15-ad44-a4788dce956a-proc\") pod \"perf-node-gather-daemonset-gp52j\" (UID: \"9f356d2b-20bb-4b15-ad44-a4788dce956a\") " pod="openshift-must-gather-pqhnx/perf-node-gather-daemonset-gp52j"
Sep 10 18:10:29.412027 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:10:29.412021    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9f356d2b-20bb-4b15-ad44-a4788dce956a-sys\") pod \"perf-node-gather-daemonset-gp52j\" (UID: \"9f356d2b-20bb-4b15-ad44-a4788dce956a\") " pod="openshift-must-gather-pqhnx/perf-node-gather-daemonset-gp52j"
Sep 10 18:10:29.513699 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:10:29.513424    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/9f356d2b-20bb-4b15-ad44-a4788dce956a-proc\") pod \"perf-node-gather-daemonset-gp52j\" (UID: \"9f356d2b-20bb-4b15-ad44-a4788dce956a\") " pod="openshift-must-gather-pqhnx/perf-node-gather-daemonset-gp52j"
Sep 10 18:10:29.514206 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:10:29.513740    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9f356d2b-20bb-4b15-ad44-a4788dce956a-sys\") pod \"perf-node-gather-daemonset-gp52j\" (UID: \"9f356d2b-20bb-4b15-ad44-a4788dce956a\") " pod="openshift-must-gather-pqhnx/perf-node-gather-daemonset-gp52j"
Sep 10 18:10:29.514206 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:10:29.514083    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/9f356d2b-20bb-4b15-ad44-a4788dce956a-podres\") pod \"perf-node-gather-daemonset-gp52j\" (UID: \"9f356d2b-20bb-4b15-ad44-a4788dce956a\") " pod="openshift-must-gather-pqhnx/perf-node-gather-daemonset-gp52j"
Sep 10 18:10:29.514469 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:10:29.514199    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tw4sz\" (UniqueName: \"kubernetes.io/projected/9f356d2b-20bb-4b15-ad44-a4788dce956a-kube-api-access-tw4sz\") pod \"perf-node-gather-daemonset-gp52j\" (UID: \"9f356d2b-20bb-4b15-ad44-a4788dce956a\") " pod="openshift-must-gather-pqhnx/perf-node-gather-daemonset-gp52j"
Sep 10 18:10:29.514690 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:10:29.514662    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9f356d2b-20bb-4b15-ad44-a4788dce956a-sys\") pod \"perf-node-gather-daemonset-gp52j\" (UID: \"9f356d2b-20bb-4b15-ad44-a4788dce956a\") " pod="openshift-must-gather-pqhnx/perf-node-gather-daemonset-gp52j"
Sep 10 18:10:29.514781 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:10:29.514758    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/9f356d2b-20bb-4b15-ad44-a4788dce956a-proc\") pod \"perf-node-gather-daemonset-gp52j\" (UID: \"9f356d2b-20bb-4b15-ad44-a4788dce956a\") " pod="openshift-must-gather-pqhnx/perf-node-gather-daemonset-gp52j"
Sep 10 18:10:29.514862 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:10:29.514684    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9f356d2b-20bb-4b15-ad44-a4788dce956a-lib-modules\") pod \"perf-node-gather-daemonset-gp52j\" (UID: \"9f356d2b-20bb-4b15-ad44-a4788dce956a\") " pod="openshift-must-gather-pqhnx/perf-node-gather-daemonset-gp52j"
Sep 10 18:10:29.515008 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:10:29.514986    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9f356d2b-20bb-4b15-ad44-a4788dce956a-lib-modules\") pod \"perf-node-gather-daemonset-gp52j\" (UID: \"9f356d2b-20bb-4b15-ad44-a4788dce956a\") " pod="openshift-must-gather-pqhnx/perf-node-gather-daemonset-gp52j"
Sep 10 18:10:29.515071 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:10:29.514911    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/9f356d2b-20bb-4b15-ad44-a4788dce956a-podres\") pod \"perf-node-gather-daemonset-gp52j\" (UID: \"9f356d2b-20bb-4b15-ad44-a4788dce956a\") " pod="openshift-must-gather-pqhnx/perf-node-gather-daemonset-gp52j"
Sep 10 18:10:29.525545 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:10:29.525513    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tw4sz\" (UniqueName: \"kubernetes.io/projected/9f356d2b-20bb-4b15-ad44-a4788dce956a-kube-api-access-tw4sz\") pod \"perf-node-gather-daemonset-gp52j\" (UID: \"9f356d2b-20bb-4b15-ad44-a4788dce956a\") " pod="openshift-must-gather-pqhnx/perf-node-gather-daemonset-gp52j"
Sep 10 18:10:29.618383 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:10:29.618341    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-pqhnx/perf-node-gather-daemonset-gp52j"
Sep 10 18:10:29.717604 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:10:29.716731    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-85d98c998f-fvbsc_023584ce-2746-4d1c-9163-704c37e4cff1/download-server/0.log"
Sep 10 18:10:29.904198 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:10:29.904151    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-pqhnx/perf-node-gather-daemonset-gp52j"]
Sep 10 18:10:29.906031 ip-10-0-129-122 kubenswrapper[2579]: W0910 18:10:29.905998    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod9f356d2b_20bb_4b15_ad44_a4788dce956a.slice/crio-dffa6d11f106407163d2cfc035548504b3c42e8cbbcd32bf4a2ea182d7ef3bac WatchSource:0}: Error finding container dffa6d11f106407163d2cfc035548504b3c42e8cbbcd32bf4a2ea182d7ef3bac: Status 404 returned error can't find the container with id dffa6d11f106407163d2cfc035548504b3c42e8cbbcd32bf4a2ea182d7ef3bac
Sep 10 18:10:30.203273 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:10:30.202219    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-pqhnx/perf-node-gather-daemonset-gp52j" event={"ID":"9f356d2b-20bb-4b15-ad44-a4788dce956a","Type":"ContainerStarted","Data":"f71029b3a8f1a886cddbac145e643cd946a419f4c6c546ba9a3723269546e72c"}
Sep 10 18:10:30.203273 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:10:30.202286    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-pqhnx/perf-node-gather-daemonset-gp52j" event={"ID":"9f356d2b-20bb-4b15-ad44-a4788dce956a","Type":"ContainerStarted","Data":"dffa6d11f106407163d2cfc035548504b3c42e8cbbcd32bf4a2ea182d7ef3bac"}
Sep 10 18:10:30.203273 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:10:30.203157    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-pqhnx/perf-node-gather-daemonset-gp52j"
Sep 10 18:10:30.219729 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:10:30.219667    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-pqhnx/perf-node-gather-daemonset-gp52j" podStartSLOduration=1.219645139 podStartE2EDuration="1.219645139s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 18:10:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 18:10:30.217779049 +0000 UTC m=+2074.217609120" watchObservedRunningTime="2026-09-10 18:10:30.219645139 +0000 UTC m=+2074.219475207"
Sep 10 18:10:30.382069 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:10:30.382039    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-695c895bd6-8dmcm_7f13ef98-e00e-4fe2-8f80-41eb99b0c7f7/volume-data-source-validator/0.log"
Sep 10 18:10:31.541027 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:10:31.540999    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-dxqxz_06ecbee3-6c6f-467b-831a-d9545f4588bd/dns/0.log"
Sep 10 18:10:31.560301 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:10:31.560268    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-dxqxz_06ecbee3-6c6f-467b-831a-d9545f4588bd/kube-rbac-proxy/0.log"
Sep 10 18:10:31.673500 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:10:31.673470    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-t4l6s_3dc43868-1f6c-48fa-85fc-59607a50c103/dns-node-resolver/0.log"
Sep 10 18:10:32.273744 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:10:32.273714    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-rkzwg_eca4abd0-e29b-4c77-9a3b-c9c32e5b5101/node-ca/0.log"
Sep 10 18:10:33.799358 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:10:33.799317    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-kpwl9_fcfe8a95-b4b5-4997-8d4b-717a1f3d8149/payload-processing/0.log"
Sep 10 18:10:33.826744 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:10:33.826716    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-d7f986b9d-xvnsc_678c4830-c56e-4af8-85a2-6f26a4e03aa8/router/0.log"
Sep 10 18:10:34.717630 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:10:34.717599    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-xctgk_bfa6c6a2-4e67-4c3c-b445-e07834780f92/serve-healthcheck-canary/0.log"
Sep 10 18:10:35.180151 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:10:35.180116    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-76bcd6ccf5-rh6vv_fd21a8ca-3f90-4aa5-b1de-78ddebecf928/insights-operator/0.log"
Sep 10 18:10:35.180752 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:10:35.180683    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-76bcd6ccf5-rh6vv_fd21a8ca-3f90-4aa5-b1de-78ddebecf928/insights-operator/1.log"
Sep 10 18:10:35.263280 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:10:35.263243    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-sxbzl_98c74a0b-1bd4-46f2-b78c-97a6469856ba/kube-rbac-proxy/0.log"
Sep 10 18:10:35.282951 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:10:35.282924    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-sxbzl_98c74a0b-1bd4-46f2-b78c-97a6469856ba/exporter/0.log"
Sep 10 18:10:35.303450 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:10:35.303417    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-sxbzl_98c74a0b-1bd4-46f2-b78c-97a6469856ba/extractor/0.log"
Sep 10 18:10:37.227230 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:10:37.227195    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-pqhnx/perf-node-gather-daemonset-gp52j"
Sep 10 18:10:38.255003 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:10:38.254956    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-dbf885bd4-txcxf_19472691-efec-4186-9c11-10dc71432e71/manager/0.log"
Sep 10 18:10:38.273909 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:10:38.273872    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-dbf885bd4-txcxf_19472691-efec-4186-9c11-10dc71432e71/copy-manifests/0.log"
Sep 10 18:10:39.690170 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:10:39.690071    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-57bb875478-k5dn6_719743db-5de0-4ea1-a365-75fca5745d64/manager/0.log"
Sep 10 18:10:40.382269 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:10:40.382207    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-kpwl9_fcfe8a95-b4b5-4997-8d4b-717a1f3d8149/payload-processing/0.log"
Sep 10 18:10:40.397081 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:10:40.397051    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-d7f986b9d-xvnsc_678c4830-c56e-4af8-85a2-6f26a4e03aa8/router/0.log"
Sep 10 18:10:42.439724 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:10:42.439674    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-kpwl9_fcfe8a95-b4b5-4997-8d4b-717a1f3d8149/payload-processing/0.log"
Sep 10 18:10:42.456213 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:10:42.456185    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-d7f986b9d-xvnsc_678c4830-c56e-4af8-85a2-6f26a4e03aa8/router/0.log"
Sep 10 18:10:44.606193 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:10:44.606163    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-fdbb8fdff-kzxj2_7ff85e45-8b8f-432b-b51a-f1ebf3019da3/migrator/0.log"
Sep 10 18:10:44.625086 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:10:44.625056    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-fdbb8fdff-kzxj2_7ff85e45-8b8f-432b-b51a-f1ebf3019da3/graceful-termination/0.log"
Sep 10 18:10:44.969869 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:10:44.969835    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-8d7dd4675-wt6xs_c8f186e0-4c0d-4422-be62-96685cca0258/kube-storage-version-migrator-operator/1.log"
Sep 10 18:10:44.970684 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:10:44.970663    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-8d7dd4675-wt6xs_c8f186e0-4c0d-4422-be62-96685cca0258/kube-storage-version-migrator-operator/0.log"
Sep 10 18:10:46.104356 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:10:46.104323    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-drhxj_48d6ee6d-6025-4ab7-a095-6a31180ac9f7/kube-multus-additional-cni-plugins/0.log"
Sep 10 18:10:46.126299 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:10:46.126252    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-drhxj_48d6ee6d-6025-4ab7-a095-6a31180ac9f7/egress-router-binary-copy/0.log"
Sep 10 18:10:46.145538 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:10:46.145492    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-drhxj_48d6ee6d-6025-4ab7-a095-6a31180ac9f7/cni-plugins/0.log"
Sep 10 18:10:46.164140 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:10:46.164113    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-drhxj_48d6ee6d-6025-4ab7-a095-6a31180ac9f7/bond-cni-plugin/0.log"
Sep 10 18:10:46.183054 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:10:46.183027    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-drhxj_48d6ee6d-6025-4ab7-a095-6a31180ac9f7/routeoverride-cni/0.log"
Sep 10 18:10:46.201984 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:10:46.201941    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-drhxj_48d6ee6d-6025-4ab7-a095-6a31180ac9f7/whereabouts-cni-bincopy/0.log"
Sep 10 18:10:46.221956 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:10:46.221926    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-drhxj_48d6ee6d-6025-4ab7-a095-6a31180ac9f7/whereabouts-cni/0.log"
Sep 10 18:10:46.459710 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:10:46.459684    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-x2sqz_85fae606-5227-404b-95a8-d7de60dc1041/kube-multus/0.log"
Sep 10 18:10:46.630592 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:10:46.630543    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-ncmn7_3fcacac9-c23e-4ba5-8b1d-dfeede6b57e3/network-metrics-daemon/0.log"
Sep 10 18:10:46.646172 ip-10-0-129-122 kubenswrapper[2579]: I0910 18:10:46.646133    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-ncmn7_3fcacac9-c23e-4ba5-8b1d-dfeede6b57e3/kube-rbac-proxy/0.log"