Sep 11 00:57:45.239281 ip-10-0-129-222 systemd[1]: Starting Kubernetes Kubelet...
Sep 11 00:57:45.671858 ip-10-0-129-222 kubenswrapper[2569]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 11 00:57:45.671858 ip-10-0-129-222 kubenswrapper[2569]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 11 00:57:45.671858 ip-10-0-129-222 kubenswrapper[2569]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 11 00:57:45.671858 ip-10-0-129-222 kubenswrapper[2569]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 11 00:57:45.671858 ip-10-0-129-222 kubenswrapper[2569]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 11 00:57:45.672627 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.671931    2569 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 11 00:57:45.675811 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.675648    2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 00:57:45.675811 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.675807    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 00:57:45.675811 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.675813    2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 00:57:45.675811 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.675817    2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 00:57:45.676077 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.675822    2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 00:57:45.676077 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.675827    2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 00:57:45.676077 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.675831    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 00:57:45.676077 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.675835    2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 00:57:45.676077 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.675840    2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 00:57:45.676077 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.675844    2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 00:57:45.676077 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.675847    2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 00:57:45.676077 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.675850    2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 00:57:45.676077 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.675862    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 00:57:45.676077 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.675866    2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 00:57:45.676077 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.675885    2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 00:57:45.676077 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.675890    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 00:57:45.676077 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.675893    2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 00:57:45.676077 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.675897    2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 00:57:45.676077 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.675901    2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 00:57:45.676077 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.675905    2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 00:57:45.676077 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.675909    2569 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 00:57:45.676077 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.675912    2569 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 00:57:45.676077 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.675917    2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 00:57:45.676077 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.675921    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 00:57:45.676929 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.675924    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 00:57:45.676929 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.675928    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 00:57:45.676929 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.675932    2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 00:57:45.676929 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.675937    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 00:57:45.676929 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.675941    2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 00:57:45.676929 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.675947    2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 00:57:45.676929 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.675952    2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 00:57:45.676929 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.675956    2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 00:57:45.676929 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.675960    2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 00:57:45.676929 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.675964    2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 00:57:45.676929 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.675969    2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 00:57:45.676929 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.675973    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 00:57:45.676929 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.675977    2569 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 00:57:45.676929 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.675983    2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 00:57:45.676929 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.675987    2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 00:57:45.676929 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.675993    2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 00:57:45.676929 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.675998    2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 00:57:45.676929 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.676002    2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 00:57:45.676929 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.676010    2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 00:57:45.677516 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.676016    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 00:57:45.677516 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.676023    2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 00:57:45.677516 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.676028    2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 00:57:45.677516 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.676033    2569 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 00:57:45.677516 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.676037    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 00:57:45.677516 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.676042    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 00:57:45.677516 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.676046    2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 00:57:45.677516 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.676050    2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 00:57:45.677516 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.676054    2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 00:57:45.677516 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.676058    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 00:57:45.677516 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.676063    2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 00:57:45.677516 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.676067    2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 00:57:45.677516 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.676071    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 00:57:45.677516 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.676075    2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 00:57:45.677516 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.676079    2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 00:57:45.677516 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.676083    2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 00:57:45.677516 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.676091    2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 00:57:45.677516 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.676096    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 00:57:45.677516 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.676101    2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 00:57:45.678031 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.676106    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 00:57:45.678031 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.676111    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 00:57:45.678031 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.676116    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 00:57:45.678031 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.676121    2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 00:57:45.678031 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.676125    2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 00:57:45.678031 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.676130    2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 00:57:45.678031 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.676135    2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 00:57:45.678031 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.676140    2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 00:57:45.678031 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.676145    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 00:57:45.678031 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.676150    2569 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 00:57:45.678031 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.676155    2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 00:57:45.678031 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.676160    2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 00:57:45.678031 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.676164    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 00:57:45.678031 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.676168    2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 00:57:45.678031 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.676173    2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 00:57:45.678031 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.676178    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 00:57:45.678031 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.676183    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 00:57:45.678031 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.676187    2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 00:57:45.678031 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.676191    2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 00:57:45.678031 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.676196    2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 00:57:45.678902 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.676200    2569 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 00:57:45.678902 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.676204    2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 00:57:45.678902 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.676209    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 00:57:45.678902 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.676213    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 00:57:45.678902 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.676217    2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 00:57:45.678902 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.676221    2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 00:57:45.678902 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.676225    2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 00:57:45.678902 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.676229    2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 00:57:45.678902 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.676233    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 00:57:45.678902 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.676237    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 00:57:45.678902 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.676242    2569 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 00:57:45.678902 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.676246    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 00:57:45.678902 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.676250    2569 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 00:57:45.678902 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.676254    2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 00:57:45.678902 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.676258    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 00:57:45.678902 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.676263    2569 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 00:57:45.678902 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.676267    2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 00:57:45.678902 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.678241    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 00:57:45.678902 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.678256    2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 00:57:45.678902 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.678261    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 00:57:45.679754 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.678266    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 00:57:45.679754 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.678270    2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 00:57:45.679754 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.678274    2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 00:57:45.679754 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.678279    2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 00:57:45.679754 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.678283    2569 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 00:57:45.679754 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.678287    2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 00:57:45.679754 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.678291    2569 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 00:57:45.679754 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.678295    2569 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 00:57:45.679754 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.678300    2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 00:57:45.679754 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.678305    2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 00:57:45.679754 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.678309    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 00:57:45.679754 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.678313    2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 00:57:45.679754 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.678318    2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 00:57:45.679754 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.678323    2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 00:57:45.679754 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.678328    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 00:57:45.679754 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.678332    2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 00:57:45.679754 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.678336    2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 00:57:45.679754 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.678340    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 00:57:45.679754 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.678345    2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 00:57:45.679754 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.678349    2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 00:57:45.680650 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.678353    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 00:57:45.680650 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.678357    2569 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 00:57:45.680650 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.678361    2569 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 00:57:45.680650 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.678365    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 00:57:45.680650 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.678369    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 00:57:45.680650 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.678373    2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 00:57:45.680650 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.678378    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 00:57:45.680650 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.678382    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 00:57:45.680650 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.678387    2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 00:57:45.680650 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.678391    2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 00:57:45.680650 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.678395    2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 00:57:45.680650 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.678399    2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 00:57:45.680650 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.678403    2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 00:57:45.680650 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.678407    2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 00:57:45.680650 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.678414    2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 00:57:45.680650 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.678419    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 00:57:45.680650 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.678424    2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 00:57:45.680650 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.678428    2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 00:57:45.680650 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.678433    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 00:57:45.681485 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.678437    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 00:57:45.681485 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.678441    2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 00:57:45.681485 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.678451    2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 00:57:45.681485 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.678456    2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 00:57:45.681485 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.678460    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 00:57:45.681485 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.678464    2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 00:57:45.681485 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.678471    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 00:57:45.681485 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.678475    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 00:57:45.681485 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.678479    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 00:57:45.681485 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.678483    2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 00:57:45.681485 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.678488    2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 00:57:45.681485 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.678492    2569 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 00:57:45.681485 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.678497    2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 00:57:45.681485 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.678502    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 00:57:45.681485 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.678506    2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 00:57:45.681485 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.678511    2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 00:57:45.681485 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.678515    2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 00:57:45.681485 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.678519    2569 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 00:57:45.681485 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.678523    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 00:57:45.681485 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.678527    2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 00:57:45.682001 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.678534    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 00:57:45.682001 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.678538    2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 00:57:45.682001 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.678543    2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 00:57:45.682001 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.678549    2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 00:57:45.682001 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.678556    2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 00:57:45.682001 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.678561    2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 00:57:45.682001 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.678566    2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 00:57:45.682001 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.678570    2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 00:57:45.682001 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.678575    2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 00:57:45.682001 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.678579    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 00:57:45.682001 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.678583    2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 00:57:45.682001 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.678587    2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 00:57:45.682001 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.678591    2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 00:57:45.682001 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.678595    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 00:57:45.682001 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.678599    2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 00:57:45.682001 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.678606    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 00:57:45.682001 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.678611    2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 00:57:45.682001 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.678615    2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 00:57:45.682001 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.678620    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 00:57:45.682563 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.678626    2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 00:57:45.682563 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.678632    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 00:57:45.682563 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.678637    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 00:57:45.682563 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.678641    2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 00:57:45.682563 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.678646    2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 00:57:45.682563 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.678650    2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 00:57:45.682563 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.678654    2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 00:57:45.682563 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.678658    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 00:57:45.682563 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.678662    2569 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 00:57:45.682563 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.678666    2569 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 00:57:45.682563 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.678670    2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 00:57:45.682563 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.678674    2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 00:57:45.682563 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.678678    2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 00:57:45.682563 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.678683    2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 00:57:45.682563 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.678687    2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 00:57:45.682563 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.678692    2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 00:57:45.682563 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.678697    2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 00:57:45.682563 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.678702    2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 00:57:45.682563 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.679335    2569 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 00:57:45.682563 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.679344    2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 00:57:45.683200 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.679348    2569 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 00:57:45.683200 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.679353    2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 00:57:45.683200 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.679357    2569 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 00:57:45.683200 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.679361    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 00:57:45.683200 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.679365    2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 00:57:45.683200 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.679370    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 00:57:45.683200 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.679374    2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 00:57:45.683200 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.679378    2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 00:57:45.683200 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.679384    2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 00:57:45.683200 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.679388    2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 00:57:45.683200 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.679393    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 00:57:45.683200 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.679397    2569 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 00:57:45.683200 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.679401    2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 00:57:45.683200 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.679406    2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 00:57:45.683200 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.679412    2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 00:57:45.683200 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.679418    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 00:57:45.683200 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.679423    2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 00:57:45.683200 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.679427    2569 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 00:57:45.683200 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.679431    2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 00:57:45.683200 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.679435    2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 00:57:45.683934 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.679440    2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 00:57:45.683934 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.679444    2569 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 00:57:45.683934 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.679448    2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 00:57:45.683934 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.679452    2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 00:57:45.683934 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.679456    2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 00:57:45.683934 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.679460    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 00:57:45.683934 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.679464    2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 00:57:45.683934 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.679469    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 00:57:45.683934 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.679473    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 00:57:45.683934 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.679478    2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 00:57:45.683934 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.679485    2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 00:57:45.683934 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.679490    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 00:57:45.683934 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.679495    2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 00:57:45.683934 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.679499    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 00:57:45.683934 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.679504    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 00:57:45.683934 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.679508    2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 00:57:45.683934 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.679512    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 00:57:45.683934 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.679517    2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 00:57:45.683934 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.679521    2569 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 00:57:45.684431 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.679526    2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 00:57:45.684431 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.679530    2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 00:57:45.684431 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.679535    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 00:57:45.684431 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.679539    2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 00:57:45.684431 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.679544    2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 00:57:45.684431 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.679548    2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 00:57:45.684431 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.679554    2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 00:57:45.684431 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.679559    2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 00:57:45.684431 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.679563    2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 00:57:45.684431 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.679567    2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 00:57:45.684431 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.679571    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 00:57:45.684431 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.679575    2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 00:57:45.684431 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.679580    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 00:57:45.684431 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.679584    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 00:57:45.684431 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.679588    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 00:57:45.684431 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.679594    2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 00:57:45.684431 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.679600    2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 00:57:45.684431 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.679605    2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 00:57:45.684431 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.679609    2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 00:57:45.684924 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.679614    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 00:57:45.684924 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.679620    2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 00:57:45.684924 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.679624    2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 00:57:45.684924 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.679629    2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 00:57:45.684924 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.679633    2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 00:57:45.684924 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.679638    2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 00:57:45.684924 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.679642    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 00:57:45.684924 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.679646    2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 00:57:45.684924 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.679652    2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 00:57:45.684924 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.679656    2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 00:57:45.684924 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.679660    2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 00:57:45.684924 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.679666    2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 00:57:45.684924 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.679670    2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 00:57:45.684924 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.679675    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 00:57:45.684924 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.679679    2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 00:57:45.684924 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.679684    2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 00:57:45.684924 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.679689    2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 00:57:45.684924 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.679693    2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 00:57:45.684924 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.679698    2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 00:57:45.684924 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.679702    2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 00:57:45.685552 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.679706    2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 00:57:45.685552 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.679709    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 00:57:45.685552 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.679713    2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 00:57:45.685552 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.679718    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 00:57:45.685552 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.679721    2569 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 00:57:45.685552 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.679726    2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 00:57:45.685552 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.679730    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 00:57:45.685552 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.679734    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 00:57:45.685552 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.679738    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 00:57:45.685552 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.679742    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 00:57:45.685552 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.679746    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 00:57:45.685552 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.679750    2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 00:57:45.685552 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.679757    2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 00:57:45.685552 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.679761    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 00:57:45.685552 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.679766    2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 00:57:45.685552 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.679769    2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 00:57:45.685552 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.679773    2569 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 00:57:45.685552 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.679777    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 00:57:45.685552 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.679781    2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 00:57:45.685552 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.679907    2569 flags.go:64] FLAG: --address="0.0.0.0"
Sep 11 00:57:45.685552 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.679960    2569 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 11 00:57:45.686095 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.679972    2569 flags.go:64] FLAG: --anonymous-auth="true"
Sep 11 00:57:45.686095 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.679978    2569 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 11 00:57:45.686095 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.679984    2569 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 11 00:57:45.686095 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.679990    2569 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 11 00:57:45.686095 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.679996    2569 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 11 00:57:45.686095 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.680003    2569 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 11 00:57:45.686095 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.680008    2569 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 11 00:57:45.686095 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.680013    2569 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 11 00:57:45.686095 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.680020    2569 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 11 00:57:45.686095 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.680025    2569 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 11 00:57:45.686095 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.680030    2569 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 11 00:57:45.686095 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.680035    2569 flags.go:64] FLAG: --cgroup-root=""
Sep 11 00:57:45.686095 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.680039    2569 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 11 00:57:45.686095 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.680044    2569 flags.go:64] FLAG: --client-ca-file=""
Sep 11 00:57:45.686095 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.680049    2569 flags.go:64] FLAG: --cloud-provider="external"
Sep 11 00:57:45.686095 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.680054    2569 flags.go:64] FLAG: --cluster-dns="[]"
Sep 11 00:57:45.686095 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.680060    2569 flags.go:64] FLAG: --cluster-domain=""
Sep 11 00:57:45.686095 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.680064    2569 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 11 00:57:45.686095 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.680069    2569 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 11 00:57:45.686095 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.680074    2569 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 11 00:57:45.686095 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.680079    2569 flags.go:64] FLAG: --container-log-max-files="5"
Sep 11 00:57:45.686095 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.680085    2569 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 11 00:57:45.686095 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.680090    2569 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 11 00:57:45.686655 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.680097    2569 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 11 00:57:45.686655 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.680102    2569 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 11 00:57:45.686655 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.680107    2569 flags.go:64] FLAG: --contention-profiling="false"
Sep 11 00:57:45.686655 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.680112    2569 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 11 00:57:45.686655 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.680116    2569 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 11 00:57:45.686655 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.680121    2569 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 11 00:57:45.686655 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.680125    2569 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 11 00:57:45.686655 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.680132    2569 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 11 00:57:45.686655 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.680137    2569 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 11 00:57:45.686655 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.680141    2569 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 11 00:57:45.686655 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.680146    2569 flags.go:64] FLAG: --enable-load-reader="false"
Sep 11 00:57:45.686655 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.680150    2569 flags.go:64] FLAG: --enable-server="true"
Sep 11 00:57:45.686655 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.680155    2569 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 11 00:57:45.686655 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.680163    2569 flags.go:64] FLAG: --event-burst="100"
Sep 11 00:57:45.686655 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.680167    2569 flags.go:64] FLAG: --event-qps="50"
Sep 11 00:57:45.686655 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.680172    2569 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 11 00:57:45.686655 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.680177    2569 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 11 00:57:45.686655 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.680182    2569 flags.go:64] FLAG: --eviction-hard=""
Sep 11 00:57:45.686655 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.680189    2569 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 11 00:57:45.686655 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.680194    2569 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 11 00:57:45.686655 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.680200    2569 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 11 00:57:45.686655 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.680205    2569 flags.go:64] FLAG: --eviction-soft=""
Sep 11 00:57:45.686655 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.680210    2569 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 11 00:57:45.686655 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.680215    2569 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 11 00:57:45.686655 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.680219    2569 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 11 00:57:45.687293 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.680224    2569 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 11 00:57:45.687293 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.680229    2569 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 11 00:57:45.687293 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.680234    2569 flags.go:64] FLAG: --fail-swap-on="true"
Sep 11 00:57:45.687293 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.680238    2569 flags.go:64] FLAG: --feature-gates=""
Sep 11 00:57:45.687293 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.680244    2569 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 11 00:57:45.687293 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.680249    2569 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 11 00:57:45.687293 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.680254    2569 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 11 00:57:45.687293 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.680261    2569 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 11 00:57:45.687293 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.680266    2569 flags.go:64] FLAG: --healthz-port="10248"
Sep 11 00:57:45.687293 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.680271    2569 flags.go:64] FLAG: --help="false"
Sep 11 00:57:45.687293 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.680275    2569 flags.go:64] FLAG: --hostname-override="ip-10-0-129-222.ec2.internal"
Sep 11 00:57:45.687293 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.680280    2569 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 11 00:57:45.687293 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.680285    2569 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 11 00:57:45.687293 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.680290    2569 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 11 00:57:45.687293 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.680296    2569 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 11 00:57:45.687293 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.680301    2569 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 11 00:57:45.687293 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.680306    2569 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 11 00:57:45.687293 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.680310    2569 flags.go:64] FLAG: --image-service-endpoint=""
Sep 11 00:57:45.687293 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.680315    2569 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 11 00:57:45.687293 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.680320    2569 flags.go:64] FLAG: --kube-api-burst="100"
Sep 11 00:57:45.687293 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.680325    2569 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 11 00:57:45.687293 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.680330    2569 flags.go:64] FLAG: --kube-api-qps="50"
Sep 11 00:57:45.687293 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.680335    2569 flags.go:64] FLAG: --kube-reserved=""
Sep 11 00:57:45.687293 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.680340    2569 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 11 00:57:45.687908 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.680344    2569 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 11 00:57:45.687908 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.680349    2569 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 11 00:57:45.687908 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.680355    2569 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 11 00:57:45.687908 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.680360    2569 flags.go:64] FLAG: --lock-file=""
Sep 11 00:57:45.687908 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.680364    2569 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 11 00:57:45.687908 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.680369    2569 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 11 00:57:45.687908 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.680374    2569 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 11 00:57:45.687908 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.680382    2569 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 11 00:57:45.687908 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.680387    2569 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 11 00:57:45.687908 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.680391    2569 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 11 00:57:45.687908 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.680396    2569 flags.go:64] FLAG: --logging-format="text"
Sep 11 00:57:45.687908 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.680400    2569 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 11 00:57:45.687908 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.680406    2569 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 11 00:57:45.687908 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.680410    2569 flags.go:64] FLAG: --manifest-url=""
Sep 11 00:57:45.687908 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.680415    2569 flags.go:64] FLAG: --manifest-url-header=""
Sep 11 00:57:45.687908 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.680423    2569 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 11 00:57:45.687908 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.680429    2569 flags.go:64] FLAG: --max-open-files="1000000"
Sep 11 00:57:45.687908 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.680435    2569 flags.go:64] FLAG: --max-pods="110"
Sep 11 00:57:45.687908 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.680440    2569 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 11 00:57:45.687908 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.680444    2569 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 11 00:57:45.687908 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.680449    2569 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 11 00:57:45.687908 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.680454    2569 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 11 00:57:45.687908 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.680459    2569 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 11 00:57:45.687908 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.680464    2569 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 11 00:57:45.687908 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.680468    2569 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 11 00:57:45.688591 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.680479    2569 flags.go:64] FLAG: --node-status-max-images="50"
Sep 11 00:57:45.688591 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.680484    2569 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 11 00:57:45.688591 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.680489    2569 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 11 00:57:45.688591 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.680494    2569 flags.go:64] FLAG: --pod-cidr=""
Sep 11 00:57:45.688591 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.680499    2569 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10"
Sep 11 00:57:45.688591 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.680507    2569 flags.go:64] FLAG: --pod-manifest-path=""
Sep 11 00:57:45.688591 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.680512    2569 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 11 00:57:45.688591 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.680516    2569 flags.go:64] FLAG: --pods-per-core="0"
Sep 11 00:57:45.688591 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.680521    2569 flags.go:64] FLAG: --port="10250"
Sep 11 00:57:45.688591 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.680526    2569 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 11 00:57:45.688591 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.680532    2569 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-01bb95a412d5dec7d"
Sep 11 00:57:45.688591 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.680537    2569 flags.go:64] FLAG: --qos-reserved=""
Sep 11 00:57:45.688591 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.680542    2569 flags.go:64] FLAG: --read-only-port="10255"
Sep 11 00:57:45.688591 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.680547    2569 flags.go:64] FLAG: --register-node="true"
Sep 11 00:57:45.688591 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.680552    2569 flags.go:64] FLAG: --register-with-taints=""
Sep 11 00:57:45.688591 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.680561    2569 flags.go:64] FLAG: --registry-burst="10"
Sep 11 00:57:45.688591 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.680566    2569 flags.go:64] FLAG: --registry-qps="5"
Sep 11 00:57:45.688591 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.680570    2569 flags.go:64] FLAG: --reserved-cpus=""
Sep 11 00:57:45.688591 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.680575    2569 flags.go:64] FLAG: --reserved-memory=""
Sep 11 00:57:45.688591 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.680581    2569 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 11 00:57:45.688591 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.680586    2569 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 11 00:57:45.688591 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.680591    2569 flags.go:64] FLAG: --rotate-certificates="false"
Sep 11 00:57:45.688591 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.680597    2569 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 11 00:57:45.688591 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.680602    2569 flags.go:64] FLAG: --runonce="false"
Sep 11 00:57:45.688591 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.680606    2569 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 11 00:57:45.689334 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.680611    2569 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 11 00:57:45.689334 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.680616    2569 flags.go:64] FLAG: --seccomp-default="false"
Sep 11 00:57:45.689334 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.680621    2569 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 11 00:57:45.689334 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.680626    2569 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 11 00:57:45.689334 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.680631    2569 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 11 00:57:45.689334 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.680635    2569 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 11 00:57:45.689334 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.680640    2569 flags.go:64] FLAG: --storage-driver-password="root"
Sep 11 00:57:45.689334 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.680645    2569 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 11 00:57:45.689334 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.680649    2569 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 11 00:57:45.689334 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.680653    2569 flags.go:64] FLAG: --storage-driver-user="root"
Sep 11 00:57:45.689334 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.680658    2569 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 11 00:57:45.689334 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.680662    2569 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 11 00:57:45.689334 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.680667    2569 flags.go:64] FLAG: --system-cgroups=""
Sep 11 00:57:45.689334 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.680672    2569 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 11 00:57:45.689334 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.680680    2569 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 11 00:57:45.689334 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.680687    2569 flags.go:64] FLAG: --tls-cert-file=""
Sep 11 00:57:45.689334 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.680692    2569 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 11 00:57:45.689334 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.680698    2569 flags.go:64] FLAG: --tls-min-version=""
Sep 11 00:57:45.689334 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.680704    2569 flags.go:64] FLAG: --tls-private-key-file=""
Sep 11 00:57:45.689334 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.680708    2569 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 11 00:57:45.689334 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.680713    2569 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 11 00:57:45.689334 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.680718    2569 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 11 00:57:45.689334 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.680722    2569 flags.go:64] FLAG: --v="2"
Sep 11 00:57:45.689334 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.680728    2569 flags.go:64] FLAG: --version="false"
Sep 11 00:57:45.689334 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.680735    2569 flags.go:64] FLAG: --vmodule=""
Sep 11 00:57:45.690339 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.680741    2569 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 11 00:57:45.690339 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.680746    2569 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 11 00:57:45.690339 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.680938    2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 00:57:45.690339 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.680945    2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 00:57:45.690339 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.680953    2569 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 00:57:45.690339 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.680957    2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 00:57:45.690339 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.680962    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 00:57:45.690339 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.680966    2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 00:57:45.690339 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.680971    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 00:57:45.690339 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.680976    2569 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 00:57:45.690339 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.680982    2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 00:57:45.690339 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.680986    2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 00:57:45.690339 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.680991    2569 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 00:57:45.690339 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.680995    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 00:57:45.690339 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.680999    2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 00:57:45.690339 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.681003    2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 00:57:45.690339 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.681008    2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 00:57:45.690339 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.681012    2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 00:57:45.690339 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.681017    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 00:57:45.690339 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.681021    2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 00:57:45.690910 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.681026    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 00:57:45.690910 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.681030    2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 00:57:45.690910 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.681035    2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 00:57:45.690910 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.681040    2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 00:57:45.690910 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.681044    2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 00:57:45.690910 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.681048    2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 00:57:45.690910 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.681053    2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 00:57:45.690910 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.681057    2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 00:57:45.690910 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.681062    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 00:57:45.690910 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.681066    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 00:57:45.690910 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.681070    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 00:57:45.690910 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.681074    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 00:57:45.690910 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.681078    2569 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 00:57:45.690910 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.681083    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 00:57:45.690910 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.681087    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 00:57:45.690910 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.681091    2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 00:57:45.690910 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.681097    2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 00:57:45.690910 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.681101    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 00:57:45.690910 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.681106    2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 00:57:45.691382 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.681110    2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 00:57:45.691382 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.681114    2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 00:57:45.691382 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.681118    2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 00:57:45.691382 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.681122    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 00:57:45.691382 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.681126    2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 00:57:45.691382 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.681130    2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 00:57:45.691382 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.681134    2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 00:57:45.691382 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.681138    2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 00:57:45.691382 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.681142    2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 00:57:45.691382 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.681147    2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 00:57:45.691382 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.681151    2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 00:57:45.691382 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.681155    2569 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 00:57:45.691382 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.681159    2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 00:57:45.691382 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.681163    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 00:57:45.691382 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.681167    2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 00:57:45.691382 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.681173    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 00:57:45.691382 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.681177    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 00:57:45.691382 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.681184    2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 00:57:45.691382 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.681189    2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 00:57:45.691849 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.681194    2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 00:57:45.691849 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.681199    2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 00:57:45.691849 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.681204    2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 00:57:45.691849 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.681208    2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 00:57:45.691849 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.681212    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 00:57:45.691849 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.681216    2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 00:57:45.691849 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.681220    2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 00:57:45.691849 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.681224    2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 00:57:45.691849 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.681229    2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 00:57:45.691849 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.681233    2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 00:57:45.691849 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.681238    2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 00:57:45.691849 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.681243    2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 00:57:45.691849 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.681247    2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 00:57:45.691849 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.681253    2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 00:57:45.691849 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.681265    2569 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 00:57:45.691849 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.681270    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 00:57:45.691849 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.681274    2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 00:57:45.691849 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.681280    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 00:57:45.691849 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.681284    2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 00:57:45.692343 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.681289    2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 00:57:45.692343 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.681293    2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 00:57:45.692343 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.681297    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 00:57:45.692343 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.681302    2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 00:57:45.692343 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.681306    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 00:57:45.692343 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.681310    2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 00:57:45.692343 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.681314    2569 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 00:57:45.692343 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.681318    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 00:57:45.692343 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.681323    2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 00:57:45.692343 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.681328    2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 00:57:45.692343 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.681333    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 00:57:45.692343 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.681339    2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 00:57:45.692343 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.681344    2569 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 00:57:45.692343 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.681349    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 00:57:45.692343 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.681354    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 00:57:45.692343 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.681360    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 00:57:45.692343 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.681365    2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 00:57:45.692343 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.681369    2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 00:57:45.692343 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.681374    2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 00:57:45.692811 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.681378    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 00:57:45.692811 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.681382    2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 00:57:45.692811 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.681386    2569 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 00:57:45.692811 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.681391    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 00:57:45.692811 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.681397    2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 00:57:45.692811 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.682127    2569 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 11 00:57:45.692811 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.691399    2569 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 11 00:57:45.692811 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.691414    2569 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 11 00:57:45.692811 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.692264    2569 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 11 00:57:45.692811 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.692272    2569 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 11 00:57:45.692811 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.692328    2569 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 00:57:45.692811 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.692335    2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 00:57:45.692811 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.692339    2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 00:57:45.692811 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.692343    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 00:57:45.693176 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.692346    2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 00:57:45.693176 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.692350    2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 00:57:45.693176 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.692353    2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 00:57:45.693176 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.692356    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 00:57:45.693176 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.692359    2569 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 00:57:45.693176 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.692362    2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 00:57:45.693176 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.692365    2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 00:57:45.693176 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.692368    2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 00:57:45.693176 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.692371    2569 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 00:57:45.693176 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.692374    2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 00:57:45.693176 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.692377    2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 00:57:45.693176 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.692380    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 00:57:45.693176 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.692382    2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 00:57:45.693176 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.692385    2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 00:57:45.693176 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.692387    2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 00:57:45.693176 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.692390    2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 00:57:45.693176 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.692393    2569 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 00:57:45.693176 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.692396    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 00:57:45.693176 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.692398    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 00:57:45.693176 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.692401    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 00:57:45.693680 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.692404    2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 00:57:45.693680 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.692407    2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 00:57:45.693680 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.692409    2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 00:57:45.693680 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.692412    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 00:57:45.693680 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.692414    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 00:57:45.693680 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.692418    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 00:57:45.693680 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.692421    2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 00:57:45.693680 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.692423    2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 00:57:45.693680 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.692426    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 00:57:45.693680 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.692428    2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 00:57:45.693680 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.692431    2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 00:57:45.693680 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.692433    2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 00:57:45.693680 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.692436    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 00:57:45.693680 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.692439    2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 00:57:45.693680 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.692441    2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 00:57:45.693680 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.692444    2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 00:57:45.693680 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.692446    2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 00:57:45.693680 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.692449    2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 00:57:45.693680 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.692452    2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 00:57:45.694162 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.692456    2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 00:57:45.694162 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.692459    2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 00:57:45.694162 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.692462    2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 00:57:45.694162 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.692464    2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 00:57:45.694162 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.692467    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 00:57:45.694162 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.692470    2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 00:57:45.694162 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.692473    2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 00:57:45.694162 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.692475    2569 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 00:57:45.694162 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.692478    2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 00:57:45.694162 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.692480    2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 00:57:45.694162 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.692483    2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 00:57:45.694162 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.692485    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 00:57:45.694162 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.692488    2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 00:57:45.694162 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.692490    2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 00:57:45.694162 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.692493    2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 00:57:45.694162 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.692495    2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 00:57:45.694162 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.692498    2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 00:57:45.694162 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.692501    2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 00:57:45.694162 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.692504    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 00:57:45.694637 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.692507    2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 00:57:45.694637 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.692510    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 00:57:45.694637 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.692513    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 00:57:45.694637 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.692516    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 00:57:45.694637 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.692518    2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 00:57:45.694637 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.692521    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 00:57:45.694637 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.692523    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 00:57:45.694637 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.692526    2569 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 00:57:45.694637 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.692528    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 00:57:45.694637 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.692531    2569 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 00:57:45.694637 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.692534    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 00:57:45.694637 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.692536    2569 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 00:57:45.694637 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.692539    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 00:57:45.694637 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.692541    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 00:57:45.694637 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.692544    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 00:57:45.694637 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.692547    2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 00:57:45.694637 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.692549    2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 00:57:45.694637 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.692554    2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 00:57:45.694637 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.692557    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 00:57:45.694637 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.692560    2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 00:57:45.695155 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.692563    2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 00:57:45.695155 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.692566    2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 00:57:45.695155 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.692569    2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 00:57:45.695155 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.692572    2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 00:57:45.695155 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.692575    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 00:57:45.695155 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.692577    2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 00:57:45.695155 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.692580    2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 00:57:45.695155 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.692583    2569 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 00:57:45.695155 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.692585    2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 00:57:45.695155 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.692588    2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 00:57:45.695155 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.692590    2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 00:57:45.695155 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.692593    2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 00:57:45.695155 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.692596    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 00:57:45.695155 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.692599    2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 00:57:45.695155 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.692602    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 00:57:45.695155 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.692605    2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 00:57:45.695155 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.692607    2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 00:57:45.695575 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.692613    2569 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 11 00:57:45.695575 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.692732    2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 00:57:45.695575 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.692738    2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 00:57:45.695575 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.692741    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 00:57:45.695575 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.692744    2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 00:57:45.695575 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.692747    2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 00:57:45.695575 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.692750    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 00:57:45.695575 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.692752    2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 00:57:45.695575 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.692755    2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 00:57:45.695575 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.692758    2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 00:57:45.695575 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.692760    2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 00:57:45.695575 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.692763    2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 00:57:45.695575 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.692765    2569 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 00:57:45.695575 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.692769    2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 00:57:45.695575 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.692772    2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 00:57:45.696095 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.692774    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 00:57:45.696095 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.692777    2569 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 00:57:45.696095 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.692780    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 00:57:45.696095 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.692782    2569 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 00:57:45.696095 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.692785    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 00:57:45.696095 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.692787    2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 00:57:45.696095 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.692790    2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 00:57:45.696095 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.692792    2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 00:57:45.696095 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.692795    2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 00:57:45.696095 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.692797    2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 00:57:45.696095 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.692800    2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 00:57:45.696095 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.692804    2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 00:57:45.696095 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.692809    2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 00:57:45.696095 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.692812    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 00:57:45.696095 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.692815    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 00:57:45.696095 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.692819    2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 00:57:45.696095 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.692822    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 00:57:45.696095 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.692824    2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 00:57:45.696095 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.692827    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 00:57:45.696605 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.692830    2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 00:57:45.696605 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.692832    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 00:57:45.696605 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.692835    2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 00:57:45.696605 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.692838    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 00:57:45.696605 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.692841    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 00:57:45.696605 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.692843    2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 00:57:45.696605 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.692846    2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 00:57:45.696605 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.692848    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 00:57:45.696605 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.692851    2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 00:57:45.696605 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.692854    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 00:57:45.696605 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.692856    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 00:57:45.696605 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.692859    2569 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 00:57:45.696605 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.692863    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 00:57:45.696605 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.692865    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 00:57:45.696605 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.692879    2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 00:57:45.696605 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.692882    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 00:57:45.696605 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.692885    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 00:57:45.696605 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.692887    2569 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 00:57:45.696605 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.692891    2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 00:57:45.696605 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.692895    2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 00:57:45.697162 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.692897    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 00:57:45.697162 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.692901    2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 00:57:45.697162 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.692905    2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 00:57:45.697162 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.692907    2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 00:57:45.697162 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.692911    2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 00:57:45.697162 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.692913    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 00:57:45.697162 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.692916    2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 00:57:45.697162 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.692919    2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 00:57:45.697162 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.692922    2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 00:57:45.697162 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.692925    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 00:57:45.697162 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.692927    2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 00:57:45.697162 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.692930    2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 00:57:45.697162 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.692933    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 00:57:45.697162 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.692935    2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 00:57:45.697162 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.692938    2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 00:57:45.697162 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.692940    2569 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 00:57:45.697162 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.692943    2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 00:57:45.697162 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.692946    2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 00:57:45.697162 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.692948    2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 00:57:45.697633 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.692951    2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 00:57:45.697633 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.692953    2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 00:57:45.697633 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.692956    2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 00:57:45.697633 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.692959    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 00:57:45.697633 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.692961    2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 00:57:45.697633 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.692964    2569 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 00:57:45.697633 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.692967    2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 00:57:45.697633 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.692970    2569 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 00:57:45.697633 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.692972    2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 00:57:45.697633 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.692975    2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 00:57:45.697633 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.692977    2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 00:57:45.697633 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.692980    2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 00:57:45.697633 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.692982    2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 00:57:45.697633 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.692985    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 00:57:45.697633 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.692988    2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 00:57:45.697633 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.692990    2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 00:57:45.697633 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.692993    2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 00:57:45.697633 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.692995    2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 00:57:45.697633 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.692998    2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 00:57:45.697633 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.693001    2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 00:57:45.698238 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.693004    2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 00:57:45.698238 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.693006    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 00:57:45.698238 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.693009    2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 00:57:45.698238 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.693012    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 00:57:45.698238 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.693014    2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 00:57:45.698238 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.693017    2569 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 00:57:45.698238 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:45.693020    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 00:57:45.698238 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.693025    2569 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 11 00:57:45.698238 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.693132    2569 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 11 00:57:45.698238 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.695426    2569 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 11 00:57:45.698238 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.696431    2569 server.go:1019] "Starting client certificate rotation"
Sep 11 00:57:45.698238 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.696555    2569 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 11 00:57:45.698238 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.696597    2569 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 11 00:57:45.699289 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.699273    2569 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 11 00:57:45.700398 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.700379    2569 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 11 00:57:45.708989 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.708970    2569 log.go:25] "Validated CRI v1 runtime API"
Sep 11 00:57:45.715215 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.715200    2569 log.go:25] "Validated CRI v1 image API"
Sep 11 00:57:45.716090 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.716073    2569 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 11 00:57:45.718703 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.718681    2569 fs.go:135] Filesystem UUIDs: map[071d40c2-0fae-42f5-ab23-134d4d49b981:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 c196ecd5-3ae0-4fb1-9ca9-138248e87d74:/dev/nvme0n1p4]
Sep 11 00:57:45.718782 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.718702    2569 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 11 00:57:45.724824 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.724434    2569 manager.go:216] Machine: {Timestamp:2026-09-11 00:57:45.723032606 +0000 UTC m=+0.374406847 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3110117 MemoryCapacity:33164476416 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec21e00350c9a279dc6daff5443f8945 SystemUUID:ec21e003-50c9-a279-dc6d-aff5443f8945 BootID:36994ea9-e2ac-4fc6-b0d3-433ce04ce191 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582238208 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:35:9f:9e:5a:6d Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:35:9f:9e:5a:6d Speed:0 Mtu:9001} {Name:ovs-system MacAddress:1a:96:97:00:fd:3a Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164476416 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 11 00:57:45.724824 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.724815    2569 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 11 00:57:45.725039 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.724989    2569 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 11 00:57:45.726209 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.726173    2569 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 11 00:57:45.727451 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.727265    2569 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 11 00:57:45.727588 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.727454    2569 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-129-222.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 11 00:57:45.728401 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.728388    2569 topology_manager.go:138] "Creating topology manager with none policy"
Sep 11 00:57:45.728401 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.728404    2569 container_manager_linux.go:309] "Creating device plugin manager"
Sep 11 00:57:45.728483 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.728412    2569 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 11 00:57:45.729174 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.729164    2569 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 11 00:57:45.730382 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.730371    2569 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 11 00:57:45.735784 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.735772    2569 state_mem.go:36] "Initialized new in-memory state store"
Sep 11 00:57:45.735913 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.735904    2569 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 11 00:57:45.735961 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.735936    2569 kubelet.go:486] "Attempting to sync node with API server"
Sep 11 00:57:45.735961 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.735945    2569 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 11 00:57:45.735961 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.735958    2569 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 11 00:57:45.736067 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.735967    2569 kubelet.go:398] "Adding apiserver pod source"
Sep 11 00:57:45.736067 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.735975    2569 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 11 00:57:45.736354 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.736343    2569 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 11 00:57:45.736392 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.736364    2569 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 11 00:57:45.741636 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.741618    2569 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 11 00:57:45.744595 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.744579    2569 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 11 00:57:45.744673 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.744608    2569 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 11 00:57:45.747368 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.747354    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 11 00:57:45.747425 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.747374    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 11 00:57:45.747425 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.747380    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 11 00:57:45.747425 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.747386    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 11 00:57:45.747425 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.747394    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 11 00:57:45.747425 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.747399    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 11 00:57:45.747425 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.747413    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 11 00:57:45.747425 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.747418    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 11 00:57:45.747425 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.747425    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 11 00:57:45.747621 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.747431    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 11 00:57:45.747621 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.747439    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 11 00:57:45.747621 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.747444    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 11 00:57:45.748233 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.748223    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 11 00:57:45.748233 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.748232    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 11 00:57:45.752014 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.751973    2569 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-129-222.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 11 00:57:45.752085 ip-10-0-129-222 kubenswrapper[2569]: E0911 00:57:45.752040    2569 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-129-222.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 11 00:57:45.752128 ip-10-0-129-222 kubenswrapper[2569]: E0911 00:57:45.752078    2569 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 11 00:57:45.752188 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.752177    2569 server.go:1268] "Started kubelet"
Sep 11 00:57:45.752235 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.752212    2569 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 11 00:57:45.752307 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.752254    2569 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 11 00:57:45.752359 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.752333    2569 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 11 00:57:45.752942 ip-10-0-129-222 systemd[1]: Started Kubernetes Kubelet.
Sep 11 00:57:45.753835 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.753810    2569 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 11 00:57:45.754231 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.754217    2569 server.go:310] "Adding debug handlers to kubelet server"
Sep 11 00:57:45.757071 ip-10-0-129-222 kubenswrapper[2569]: E0911 00:57:45.757056    2569 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 11 00:57:45.758886 ip-10-0-129-222 kubenswrapper[2569]: E0911 00:57:45.757755    2569 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-222.ec2.internal.18d41e4c77fa0bf5  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-222.ec2.internal,UID:ip-10-0-129-222.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-129-222.ec2.internal,},FirstTimestamp:2026-09-11 00:57:45.752153077 +0000 UTC m=+0.403527317,LastTimestamp:2026-09-11 00:57:45.752153077 +0000 UTC m=+0.403527317,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-222.ec2.internal,}"
Sep 11 00:57:45.762500 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.762475    2569 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 11 00:57:45.763166 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.763146    2569 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 11 00:57:45.763822 ip-10-0-129-222 kubenswrapper[2569]: E0911 00:57:45.763380    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-222.ec2.internal\" not found"
Sep 11 00:57:45.764036 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.764015    2569 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 11 00:57:45.764103 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.764039    2569 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 11 00:57:45.764103 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.764015    2569 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 11 00:57:45.764202 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.764153    2569 reconstruct.go:97] "Volume reconstruction finished"
Sep 11 00:57:45.764202 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.764165    2569 reconciler.go:29] "Reconciler: start to sync state"
Sep 11 00:57:45.764388 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.764369    2569 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 11 00:57:45.764423 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.764390    2569 factory.go:55] Registering systemd factory
Sep 11 00:57:45.764423 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.764400    2569 factory.go:223] Registration of the systemd container factory successfully
Sep 11 00:57:45.764615 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.764605    2569 factory.go:153] Registering CRI-O factory
Sep 11 00:57:45.764615 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.764616    2569 factory.go:223] Registration of the crio container factory successfully
Sep 11 00:57:45.764682 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.764630    2569 factory.go:103] Registering Raw factory
Sep 11 00:57:45.764682 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.764642    2569 manager.go:1196] Started watching for new ooms in manager
Sep 11 00:57:45.764933 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.764924    2569 manager.go:349] Starting recovery of all containers
Sep 11 00:57:45.766128 ip-10-0-129-222 kubenswrapper[2569]: E0911 00:57:45.766104    2569 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-129-222.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 11 00:57:45.766236 ip-10-0-129-222 kubenswrapper[2569]: E0911 00:57:45.766215    2569 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 11 00:57:45.772861 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.772836    2569 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-zj8kn"
Sep 11 00:57:45.776044 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.775670    2569 manager.go:354] Recovery completed
Sep 11 00:57:45.779992 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.779976    2569 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-zj8kn"
Sep 11 00:57:45.781526 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.781515    2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 00:57:45.783409 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.783388    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-222.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 00:57:45.783494 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.783421    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-222.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 00:57:45.783494 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.783434    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-222.ec2.internal" event="NodeHasSufficientPID"
Sep 11 00:57:45.784033 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.784016    2569 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 11 00:57:45.784033 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.784031    2569 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 11 00:57:45.784188 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.784034    2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 00:57:45.784188 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.784049    2569 state_mem.go:36] "Initialized new in-memory state store"
Sep 11 00:57:45.784790 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.784771    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-222.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 00:57:45.784866 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.784802    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-222.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 00:57:45.784866 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.784818    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-222.ec2.internal" event="NodeHasSufficientPID"
Sep 11 00:57:45.785445 ip-10-0-129-222 kubenswrapper[2569]: E0911 00:57:45.785365    2569 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-222.ec2.internal.18d41e4c79d6f1b7  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-222.ec2.internal,UID:ip-10-0-129-222.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-129-222.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-129-222.ec2.internal,},FirstTimestamp:2026-09-11 00:57:45.783407031 +0000 UTC m=+0.434781270,LastTimestamp:2026-09-11 00:57:45.783407031 +0000 UTC m=+0.434781270,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-222.ec2.internal,}"
Sep 11 00:57:45.786160 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.786146    2569 policy_none.go:49] "None policy: Start"
Sep 11 00:57:45.786230 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.786165    2569 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 11 00:57:45.786230 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.786178    2569 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 11 00:57:45.788361 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.788350    2569 policy_none.go:47] "Start"
Sep 11 00:57:45.813428 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.813415    2569 manager.go:337] "Starting Device Plugin manager"
Sep 11 00:57:45.813536 ip-10-0-129-222 kubenswrapper[2569]: E0911 00:57:45.813448    2569 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 11 00:57:45.813536 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.813460    2569 server.go:87] "Starting device plugin registration server"
Sep 11 00:57:45.813710 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.813703    2569 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 11 00:57:45.813753 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.813716    2569 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 11 00:57:45.813900 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.813883    2569 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 11 00:57:45.814184 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.814171    2569 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 11 00:57:45.814277 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.814263    2569 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 11 00:57:45.814466 ip-10-0-129-222 kubenswrapper[2569]: E0911 00:57:45.814446    2569 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 11 00:57:45.814517 ip-10-0-129-222 kubenswrapper[2569]: E0911 00:57:45.814491    2569 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-129-222.ec2.internal\" not found"
Sep 11 00:57:45.862676 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.862651    2569 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 11 00:57:45.864775 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.863812    2569 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 11 00:57:45.864775 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.863856    2569 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 11 00:57:45.864775 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.863894    2569 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 11 00:57:45.864775 ip-10-0-129-222 kubenswrapper[2569]: E0911 00:57:45.863938    2569 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 11 00:57:45.866481 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.866465    2569 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 00:57:45.914520 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.914489    2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 00:57:45.915824 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.915808    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-222.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 00:57:45.915919 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.915841    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-222.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 00:57:45.915919 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.915855    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-222.ec2.internal" event="NodeHasSufficientPID"
Sep 11 00:57:45.915919 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.915895    2569 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-129-222.ec2.internal"
Sep 11 00:57:45.924606 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.924543    2569 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-129-222.ec2.internal"
Sep 11 00:57:45.924606 ip-10-0-129-222 kubenswrapper[2569]: E0911 00:57:45.924564    2569 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-129-222.ec2.internal\": node \"ip-10-0-129-222.ec2.internal\" not found"
Sep 11 00:57:45.948408 ip-10-0-129-222 kubenswrapper[2569]: E0911 00:57:45.948378    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-222.ec2.internal\" not found"
Sep 11 00:57:45.964476 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.964457    2569 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-222.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-129-222.ec2.internal"]
Sep 11 00:57:45.964534 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.964517    2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 00:57:45.965363 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.965347    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-222.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 00:57:45.965441 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.965378    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-222.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 00:57:45.965441 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.965392    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-222.ec2.internal" event="NodeHasSufficientPID"
Sep 11 00:57:45.967736 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.967722    2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 00:57:45.967852 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.967835    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-222.ec2.internal"
Sep 11 00:57:45.967917 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.967885    2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 00:57:45.968433 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.968413    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-222.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 00:57:45.968492 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.968438    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-222.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 00:57:45.968492 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.968448    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-222.ec2.internal" event="NodeHasSufficientPID"
Sep 11 00:57:45.968551 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.968414    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-222.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 00:57:45.968551 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.968521    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-222.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 00:57:45.968551 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.968532    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-222.ec2.internal" event="NodeHasSufficientPID"
Sep 11 00:57:45.970740 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.970721    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-222.ec2.internal"
Sep 11 00:57:45.970831 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.970748    2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 00:57:45.971483 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.971467    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-222.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 00:57:45.971587 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.971488    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-222.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 00:57:45.971587 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:45.971502    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-222.ec2.internal" event="NodeHasSufficientPID"
Sep 11 00:57:46.000550 ip-10-0-129-222 kubenswrapper[2569]: E0911 00:57:46.000535    2569 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-222.ec2.internal\" not found" node="ip-10-0-129-222.ec2.internal"
Sep 11 00:57:46.004817 ip-10-0-129-222 kubenswrapper[2569]: E0911 00:57:46.004799    2569 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-222.ec2.internal\" not found" node="ip-10-0-129-222.ec2.internal"
Sep 11 00:57:46.048605 ip-10-0-129-222 kubenswrapper[2569]: E0911 00:57:46.048584    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-222.ec2.internal\" not found"
Sep 11 00:57:46.149190 ip-10-0-129-222 kubenswrapper[2569]: E0911 00:57:46.149168    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-222.ec2.internal\" not found"
Sep 11 00:57:46.165468 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:46.165446    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bef6176fe69d54950289897ea3150444-config\") pod \"kube-apiserver-proxy-ip-10-0-129-222.ec2.internal\" (UID: \"bef6176fe69d54950289897ea3150444\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-222.ec2.internal"
Sep 11 00:57:46.165535 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:46.165472    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c00e6e1a91842597c59d08786ac60421-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-222.ec2.internal\" (UID: \"c00e6e1a91842597c59d08786ac60421\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-222.ec2.internal"
Sep 11 00:57:46.165535 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:46.165498    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c00e6e1a91842597c59d08786ac60421-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-222.ec2.internal\" (UID: \"c00e6e1a91842597c59d08786ac60421\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-222.ec2.internal"
Sep 11 00:57:46.249867 ip-10-0-129-222 kubenswrapper[2569]: E0911 00:57:46.249818    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-222.ec2.internal\" not found"
Sep 11 00:57:46.266166 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:46.266139    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bef6176fe69d54950289897ea3150444-config\") pod \"kube-apiserver-proxy-ip-10-0-129-222.ec2.internal\" (UID: \"bef6176fe69d54950289897ea3150444\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-222.ec2.internal"
Sep 11 00:57:46.266234 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:46.266171    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c00e6e1a91842597c59d08786ac60421-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-222.ec2.internal\" (UID: \"c00e6e1a91842597c59d08786ac60421\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-222.ec2.internal"
Sep 11 00:57:46.266234 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:46.266188    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c00e6e1a91842597c59d08786ac60421-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-222.ec2.internal\" (UID: \"c00e6e1a91842597c59d08786ac60421\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-222.ec2.internal"
Sep 11 00:57:46.266234 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:46.266221    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c00e6e1a91842597c59d08786ac60421-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-222.ec2.internal\" (UID: \"c00e6e1a91842597c59d08786ac60421\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-222.ec2.internal"
Sep 11 00:57:46.266345 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:46.266251    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c00e6e1a91842597c59d08786ac60421-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-222.ec2.internal\" (UID: \"c00e6e1a91842597c59d08786ac60421\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-222.ec2.internal"
Sep 11 00:57:46.266345 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:46.266250    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bef6176fe69d54950289897ea3150444-config\") pod \"kube-apiserver-proxy-ip-10-0-129-222.ec2.internal\" (UID: \"bef6176fe69d54950289897ea3150444\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-222.ec2.internal"
Sep 11 00:57:46.303270 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:46.303255    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-222.ec2.internal"
Sep 11 00:57:46.308170 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:46.307905    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-222.ec2.internal"
Sep 11 00:57:46.350575 ip-10-0-129-222 kubenswrapper[2569]: E0911 00:57:46.350549    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-222.ec2.internal\" not found"
Sep 11 00:57:46.452970 ip-10-0-129-222 kubenswrapper[2569]: E0911 00:57:46.452943    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-222.ec2.internal\" not found"
Sep 11 00:57:46.554802 ip-10-0-129-222 kubenswrapper[2569]: E0911 00:57:46.554728    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-222.ec2.internal\" not found"
Sep 11 00:57:46.661319 ip-10-0-129-222 kubenswrapper[2569]: E0911 00:57:46.661290    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-222.ec2.internal\" not found"
Sep 11 00:57:46.696098 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:46.696065    2569 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 11 00:57:46.696563 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:46.696239    2569 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 11 00:57:46.762319 ip-10-0-129-222 kubenswrapper[2569]: E0911 00:57:46.762290    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-222.ec2.internal\" not found"
Sep 11 00:57:46.785676 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:46.785633    2569 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-10 00:52:45 +0000 UTC" deadline="2028-06-23 19:36:09.434852878 +0000 UTC"
Sep 11 00:57:46.785676 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:46.785673    2569 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15642h38m22.64918352s"
Sep 11 00:57:46.832254 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:46.832062    2569 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 00:57:46.864103 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:46.864078    2569 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-222.ec2.internal"
Sep 11 00:57:46.871968 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:46.871936    2569 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 11 00:57:46.873837 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:46.873823    2569 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-222.ec2.internal"
Sep 11 00:57:46.885877 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:46.885861    2569 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 11 00:57:46.926126 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:46.926100    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbef6176fe69d54950289897ea3150444.slice/crio-b4ecbf9fab6c003578751e487cc0658919c934e0d851dda3b8b73ee6060dc357 WatchSource:0}: Error finding container b4ecbf9fab6c003578751e487cc0658919c934e0d851dda3b8b73ee6060dc357: Status 404 returned error can't find the container with id b4ecbf9fab6c003578751e487cc0658919c934e0d851dda3b8b73ee6060dc357
Sep 11 00:57:46.927527 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:46.927482    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc00e6e1a91842597c59d08786ac60421.slice/crio-0ec8ee7972fda6f7abf714aa33dc331210f0e55e10647e61c7a58d645b4d237d WatchSource:0}: Error finding container 0ec8ee7972fda6f7abf714aa33dc331210f0e55e10647e61c7a58d645b4d237d: Status 404 returned error can't find the container with id 0ec8ee7972fda6f7abf714aa33dc331210f0e55e10647e61c7a58d645b4d237d
Sep 11 00:57:46.930696 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:46.930681    2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 00:57:47.276463 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.276385    2569 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 00:57:47.300485 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.300464    2569 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 00:57:47.736738 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.736657    2569 apiserver.go:52] "Watching apiserver"
Sep 11 00:57:47.743547 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.743517    2569 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 11 00:57:47.745245 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.745211    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-qgmsz","openshift-dns/node-resolver-2dvjn","openshift-image-registry/node-ca-jrszh","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-222.ec2.internal","openshift-multus/multus-8vxxt","openshift-multus/multus-additional-cni-plugins-8bg6q","openshift-multus/network-metrics-daemon-r8n96","openshift-network-diagnostics/network-check-target-bbbq6","kube-system/konnectivity-agent-st6p9","kube-system/kube-apiserver-proxy-ip-10-0-129-222.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t42lc","openshift-cluster-node-tuning-operator/tuned-m8lrq","openshift-network-operator/iptables-alerter-j862p","openshift-ovn-kubernetes/ovnkube-node-qsqkd"]
Sep 11 00:57:47.748085 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.748049    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qgmsz"
Sep 11 00:57:47.748200 ip-10-0-129-222 kubenswrapper[2569]: E0911 00:57:47.748152    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qgmsz" podUID="d5d763bb-6142-409f-8155-7143a6d91dc6"
Sep 11 00:57:47.750360 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.750338    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-2dvjn"
Sep 11 00:57:47.752859 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.752808    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-l9lcz\""
Sep 11 00:57:47.753072 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.753023    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 11 00:57:47.753072 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.753025    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 11 00:57:47.755018 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.754999    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-jrszh"
Sep 11 00:57:47.757366 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.757342    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-8vxxt"
Sep 11 00:57:47.758014 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.757992    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-h5msj\""
Sep 11 00:57:47.758107 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.758023    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 11 00:57:47.758231 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.758196    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 11 00:57:47.758542 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.758524    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 11 00:57:47.759823 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.759800    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 11 00:57:47.760289 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.760267    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 11 00:57:47.760500 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.760478    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-xzpqj\""
Sep 11 00:57:47.760711 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.760684    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 11 00:57:47.760828 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.760696    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 11 00:57:47.761018 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.760958    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-8bg6q"
Sep 11 00:57:47.763668 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.763222    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r8n96"
Sep 11 00:57:47.763668 ip-10-0-129-222 kubenswrapper[2569]: E0911 00:57:47.763288    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r8n96" podUID="85c82c9f-52d0-49cd-afd9-b791628e1a50"
Sep 11 00:57:47.763898 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.763877    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 11 00:57:47.764202 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.764186    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-4kzqs\""
Sep 11 00:57:47.764408 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.764393    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 11 00:57:47.765848 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.765803    2569 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 11 00:57:47.765963 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.765849    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bbbq6"
Sep 11 00:57:47.765963 ip-10-0-129-222 kubenswrapper[2569]: E0911 00:57:47.765908    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bbbq6" podUID="239b25bd-4932-4b5b-b286-45d3e2ff9f1c"
Sep 11 00:57:47.768271 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.768250    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-st6p9"
Sep 11 00:57:47.770458 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.770434    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-hccqm\""
Sep 11 00:57:47.770607 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.770588    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 11 00:57:47.770745 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.770726    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 11 00:57:47.770909 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.770890    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t42lc"
Sep 11 00:57:47.773571 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.773552    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-m8lrq"
Sep 11 00:57:47.774693 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.773940    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bda35fc9-c767-4b21-9509-994a67ffdeaf-multus-cni-dir\") pod \"multus-8vxxt\" (UID: \"bda35fc9-c767-4b21-9509-994a67ffdeaf\") " pod="openshift-multus/multus-8vxxt"
Sep 11 00:57:47.774693 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.773975    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/bda35fc9-c767-4b21-9509-994a67ffdeaf-host-run-k8s-cni-cncf-io\") pod \"multus-8vxxt\" (UID: \"bda35fc9-c767-4b21-9509-994a67ffdeaf\") " pod="openshift-multus/multus-8vxxt"
Sep 11 00:57:47.774693 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.774003    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bda35fc9-c767-4b21-9509-994a67ffdeaf-multus-conf-dir\") pod \"multus-8vxxt\" (UID: \"bda35fc9-c767-4b21-9509-994a67ffdeaf\") " pod="openshift-multus/multus-8vxxt"
Sep 11 00:57:47.774693 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.774027    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d5d763bb-6142-409f-8155-7143a6d91dc6-kubelet-config\") pod \"global-pull-secret-syncer-qgmsz\" (UID: \"d5d763bb-6142-409f-8155-7143a6d91dc6\") " pod="kube-system/global-pull-secret-syncer-qgmsz"
Sep 11 00:57:47.774693 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.774051    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/799942b1-a6d8-4f5a-8d17-e7b2761b68b7-cnibin\") pod \"multus-additional-cni-plugins-8bg6q\" (UID: \"799942b1-a6d8-4f5a-8d17-e7b2761b68b7\") " pod="openshift-multus/multus-additional-cni-plugins-8bg6q"
Sep 11 00:57:47.774693 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.774078    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/799942b1-a6d8-4f5a-8d17-e7b2761b68b7-cni-binary-copy\") pod \"multus-additional-cni-plugins-8bg6q\" (UID: \"799942b1-a6d8-4f5a-8d17-e7b2761b68b7\") " pod="openshift-multus/multus-additional-cni-plugins-8bg6q"
Sep 11 00:57:47.774693 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.774102    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/304c1b0f-f5f9-4c4c-8041-ac384ad21516-serviceca\") pod \"node-ca-jrszh\" (UID: \"304c1b0f-f5f9-4c4c-8041-ac384ad21516\") " pod="openshift-image-registry/node-ca-jrszh"
Sep 11 00:57:47.774693 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.774124    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bda35fc9-c767-4b21-9509-994a67ffdeaf-cnibin\") pod \"multus-8vxxt\" (UID: \"bda35fc9-c767-4b21-9509-994a67ffdeaf\") " pod="openshift-multus/multus-8vxxt"
Sep 11 00:57:47.774693 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.774148    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/bda35fc9-c767-4b21-9509-994a67ffdeaf-hostroot\") pod \"multus-8vxxt\" (UID: \"bda35fc9-c767-4b21-9509-994a67ffdeaf\") " pod="openshift-multus/multus-8vxxt"
Sep 11 00:57:47.774693 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.774173    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t29m9\" (UniqueName: \"kubernetes.io/projected/bda35fc9-c767-4b21-9509-994a67ffdeaf-kube-api-access-t29m9\") pod \"multus-8vxxt\" (UID: \"bda35fc9-c767-4b21-9509-994a67ffdeaf\") " pod="openshift-multus/multus-8vxxt"
Sep 11 00:57:47.774693 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.774196    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bda35fc9-c767-4b21-9509-994a67ffdeaf-host-var-lib-cni-bin\") pod \"multus-8vxxt\" (UID: \"bda35fc9-c767-4b21-9509-994a67ffdeaf\") " pod="openshift-multus/multus-8vxxt"
Sep 11 00:57:47.774693 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.774220    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bda35fc9-c767-4b21-9509-994a67ffdeaf-etc-kubernetes\") pod \"multus-8vxxt\" (UID: \"bda35fc9-c767-4b21-9509-994a67ffdeaf\") " pod="openshift-multus/multus-8vxxt"
Sep 11 00:57:47.774693 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.774262    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/799942b1-a6d8-4f5a-8d17-e7b2761b68b7-system-cni-dir\") pod \"multus-additional-cni-plugins-8bg6q\" (UID: \"799942b1-a6d8-4f5a-8d17-e7b2761b68b7\") " pod="openshift-multus/multus-additional-cni-plugins-8bg6q"
Sep 11 00:57:47.774693 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.774300    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85c82c9f-52d0-49cd-afd9-b791628e1a50-metrics-certs\") pod \"network-metrics-daemon-r8n96\" (UID: \"85c82c9f-52d0-49cd-afd9-b791628e1a50\") " pod="openshift-multus/network-metrics-daemon-r8n96"
Sep 11 00:57:47.774693 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.774383    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 11 00:57:47.774693 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.774411    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 11 00:57:47.774693 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.774417    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-d56n4\""
Sep 11 00:57:47.774693 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.774444    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/304c1b0f-f5f9-4c4c-8041-ac384ad21516-host\") pod \"node-ca-jrszh\" (UID: \"304c1b0f-f5f9-4c4c-8041-ac384ad21516\") " pod="openshift-image-registry/node-ca-jrszh"
Sep 11 00:57:47.775718 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.774473    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/bda35fc9-c767-4b21-9509-994a67ffdeaf-host-run-multus-certs\") pod \"multus-8vxxt\" (UID: \"bda35fc9-c767-4b21-9509-994a67ffdeaf\") " pod="openshift-multus/multus-8vxxt"
Sep 11 00:57:47.775718 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.774524    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bda35fc9-c767-4b21-9509-994a67ffdeaf-host-run-netns\") pod \"multus-8vxxt\" (UID: \"bda35fc9-c767-4b21-9509-994a67ffdeaf\") " pod="openshift-multus/multus-8vxxt"
Sep 11 00:57:47.775718 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.774566    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/bda35fc9-c767-4b21-9509-994a67ffdeaf-multus-daemon-config\") pod \"multus-8vxxt\" (UID: \"bda35fc9-c767-4b21-9509-994a67ffdeaf\") " pod="openshift-multus/multus-8vxxt"
Sep 11 00:57:47.775718 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.774597    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7486f1f-9947-4405-9186-66dfa5621253-tmp-dir\") pod \"node-resolver-2dvjn\" (UID: \"c7486f1f-9947-4405-9186-66dfa5621253\") " pod="openshift-dns/node-resolver-2dvjn"
Sep 11 00:57:47.775718 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.774626    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64jxj\" (UniqueName: \"kubernetes.io/projected/c7486f1f-9947-4405-9186-66dfa5621253-kube-api-access-64jxj\") pod \"node-resolver-2dvjn\" (UID: \"c7486f1f-9947-4405-9186-66dfa5621253\") " pod="openshift-dns/node-resolver-2dvjn"
Sep 11 00:57:47.775718 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.774632    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 11 00:57:47.775718 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.774652    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gc78g\" (UniqueName: \"kubernetes.io/projected/799942b1-a6d8-4f5a-8d17-e7b2761b68b7-kube-api-access-gc78g\") pod \"multus-additional-cni-plugins-8bg6q\" (UID: \"799942b1-a6d8-4f5a-8d17-e7b2761b68b7\") " pod="openshift-multus/multus-additional-cni-plugins-8bg6q"
Sep 11 00:57:47.775718 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.774679    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/bda35fc9-c767-4b21-9509-994a67ffdeaf-multus-socket-dir-parent\") pod \"multus-8vxxt\" (UID: \"bda35fc9-c767-4b21-9509-994a67ffdeaf\") " pod="openshift-multus/multus-8vxxt"
Sep 11 00:57:47.775718 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.774703    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bda35fc9-c767-4b21-9509-994a67ffdeaf-host-var-lib-kubelet\") pod \"multus-8vxxt\" (UID: \"bda35fc9-c767-4b21-9509-994a67ffdeaf\") " pod="openshift-multus/multus-8vxxt"
Sep 11 00:57:47.775718 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.774727    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d5d763bb-6142-409f-8155-7143a6d91dc6-dbus\") pod \"global-pull-secret-syncer-qgmsz\" (UID: \"d5d763bb-6142-409f-8155-7143a6d91dc6\") " pod="kube-system/global-pull-secret-syncer-qgmsz"
Sep 11 00:57:47.775718 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.774752    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d5d763bb-6142-409f-8155-7143a6d91dc6-original-pull-secret\") pod \"global-pull-secret-syncer-qgmsz\" (UID: \"d5d763bb-6142-409f-8155-7143a6d91dc6\") " pod="kube-system/global-pull-secret-syncer-qgmsz"
Sep 11 00:57:47.775718 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.774775    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c7486f1f-9947-4405-9186-66dfa5621253-hosts-file\") pod \"node-resolver-2dvjn\" (UID: \"c7486f1f-9947-4405-9186-66dfa5621253\") " pod="openshift-dns/node-resolver-2dvjn"
Sep 11 00:57:47.775718 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.774820    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/799942b1-a6d8-4f5a-8d17-e7b2761b68b7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-8bg6q\" (UID: \"799942b1-a6d8-4f5a-8d17-e7b2761b68b7\") " pod="openshift-multus/multus-additional-cni-plugins-8bg6q"
Sep 11 00:57:47.775718 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.774892    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9z79r\" (UniqueName: \"kubernetes.io/projected/85c82c9f-52d0-49cd-afd9-b791628e1a50-kube-api-access-9z79r\") pod \"network-metrics-daemon-r8n96\" (UID: \"85c82c9f-52d0-49cd-afd9-b791628e1a50\") " pod="openshift-multus/network-metrics-daemon-r8n96"
Sep 11 00:57:47.775718 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.774917    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bda35fc9-c767-4b21-9509-994a67ffdeaf-system-cni-dir\") pod \"multus-8vxxt\" (UID: \"bda35fc9-c767-4b21-9509-994a67ffdeaf\") " pod="openshift-multus/multus-8vxxt"
Sep 11 00:57:47.775718 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.774955    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bda35fc9-c767-4b21-9509-994a67ffdeaf-cni-binary-copy\") pod \"multus-8vxxt\" (UID: \"bda35fc9-c767-4b21-9509-994a67ffdeaf\") " pod="openshift-multus/multus-8vxxt"
Sep 11 00:57:47.775718 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.774977    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/bda35fc9-c767-4b21-9509-994a67ffdeaf-host-var-lib-cni-multus\") pod \"multus-8vxxt\" (UID: \"bda35fc9-c767-4b21-9509-994a67ffdeaf\") " pod="openshift-multus/multus-8vxxt"
Sep 11 00:57:47.776594 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.775003    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwg5s\" (UniqueName: \"kubernetes.io/projected/239b25bd-4932-4b5b-b286-45d3e2ff9f1c-kube-api-access-qwg5s\") pod \"network-check-target-bbbq6\" (UID: \"239b25bd-4932-4b5b-b286-45d3e2ff9f1c\") " pod="openshift-network-diagnostics/network-check-target-bbbq6"
Sep 11 00:57:47.776594 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.775023    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/799942b1-a6d8-4f5a-8d17-e7b2761b68b7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-8bg6q\" (UID: \"799942b1-a6d8-4f5a-8d17-e7b2761b68b7\") " pod="openshift-multus/multus-additional-cni-plugins-8bg6q"
Sep 11 00:57:47.776594 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.775040    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mlxmt\" (UniqueName: \"kubernetes.io/projected/304c1b0f-f5f9-4c4c-8041-ac384ad21516-kube-api-access-mlxmt\") pod \"node-ca-jrszh\" (UID: \"304c1b0f-f5f9-4c4c-8041-ac384ad21516\") " pod="openshift-image-registry/node-ca-jrszh"
Sep 11 00:57:47.776594 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.775057    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/799942b1-a6d8-4f5a-8d17-e7b2761b68b7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-8bg6q\" (UID: \"799942b1-a6d8-4f5a-8d17-e7b2761b68b7\") " pod="openshift-multus/multus-additional-cni-plugins-8bg6q"
Sep 11 00:57:47.776594 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.775491    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-7ltgl\""
Sep 11 00:57:47.776594 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.775976    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 11 00:57:47.778339 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.778317    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 11 00:57:47.780535 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.780508    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-qsqkd"
Sep 11 00:57:47.780633 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.780551    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-j862p"
Sep 11 00:57:47.782806 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.782780    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 11 00:57:47.783849 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.783376    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 11 00:57:47.783849 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.783646    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 11 00:57:47.785972 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.785037    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 11 00:57:47.785972 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.785264    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-z7kn5\""
Sep 11 00:57:47.785972 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.785456    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 11 00:57:47.785972 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.785630    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-cmg5c\""
Sep 11 00:57:47.785972 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.785915    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 11 00:57:47.787561 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.786449    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 11 00:57:47.787561 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.786657    2569 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 11 00:57:47.787561 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.787001    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 11 00:57:47.787561 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.787216    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 11 00:57:47.808925 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.808903    2569 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-t26l5"
Sep 11 00:57:47.816836 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.816812    2569 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-t26l5"
Sep 11 00:57:47.866662 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.866096    2569 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 11 00:57:47.870233 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.870179    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-222.ec2.internal" event={"ID":"bef6176fe69d54950289897ea3150444","Type":"ContainerStarted","Data":"b4ecbf9fab6c003578751e487cc0658919c934e0d851dda3b8b73ee6060dc357"}
Sep 11 00:57:47.871696 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.871668    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-222.ec2.internal" event={"ID":"c00e6e1a91842597c59d08786ac60421","Type":"ContainerStarted","Data":"0ec8ee7972fda6f7abf714aa33dc331210f0e55e10647e61c7a58d645b4d237d"}
Sep 11 00:57:47.876036 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.876014    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/920866ad-d7f7-46e4-8624-7ae8011a2b76-registration-dir\") pod \"aws-ebs-csi-driver-node-t42lc\" (UID: \"920866ad-d7f7-46e4-8624-7ae8011a2b76\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t42lc"
Sep 11 00:57:47.876164 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.876057    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/bda35fc9-c767-4b21-9509-994a67ffdeaf-multus-socket-dir-parent\") pod \"multus-8vxxt\" (UID: \"bda35fc9-c767-4b21-9509-994a67ffdeaf\") " pod="openshift-multus/multus-8vxxt"
Sep 11 00:57:47.876164 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.876087    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bda35fc9-c767-4b21-9509-994a67ffdeaf-host-var-lib-kubelet\") pod \"multus-8vxxt\" (UID: \"bda35fc9-c767-4b21-9509-994a67ffdeaf\") " pod="openshift-multus/multus-8vxxt"
Sep 11 00:57:47.876164 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.876112    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d5d763bb-6142-409f-8155-7143a6d91dc6-dbus\") pod \"global-pull-secret-syncer-qgmsz\" (UID: \"d5d763bb-6142-409f-8155-7143a6d91dc6\") " pod="kube-system/global-pull-secret-syncer-qgmsz"
Sep 11 00:57:47.876164 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.876139    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/799942b1-a6d8-4f5a-8d17-e7b2761b68b7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-8bg6q\" (UID: \"799942b1-a6d8-4f5a-8d17-e7b2761b68b7\") " pod="openshift-multus/multus-additional-cni-plugins-8bg6q"
Sep 11 00:57:47.876370 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.876165    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/37ee44c2-7e75-4918-bc66-636974fa9b94-node-log\") pod \"ovnkube-node-qsqkd\" (UID: \"37ee44c2-7e75-4918-bc66-636974fa9b94\") " pod="openshift-ovn-kubernetes/ovnkube-node-qsqkd"
Sep 11 00:57:47.876370 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.876191    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/920866ad-d7f7-46e4-8624-7ae8011a2b76-device-dir\") pod \"aws-ebs-csi-driver-node-t42lc\" (UID: \"920866ad-d7f7-46e4-8624-7ae8011a2b76\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t42lc"
Sep 11 00:57:47.876370 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.876213    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/920866ad-d7f7-46e4-8624-7ae8011a2b76-sys-fs\") pod \"aws-ebs-csi-driver-node-t42lc\" (UID: \"920866ad-d7f7-46e4-8624-7ae8011a2b76\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t42lc"
Sep 11 00:57:47.876370 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.876242    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9z79r\" (UniqueName: \"kubernetes.io/projected/85c82c9f-52d0-49cd-afd9-b791628e1a50-kube-api-access-9z79r\") pod \"network-metrics-daemon-r8n96\" (UID: \"85c82c9f-52d0-49cd-afd9-b791628e1a50\") " pod="openshift-multus/network-metrics-daemon-r8n96"
Sep 11 00:57:47.876370 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.876269    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bda35fc9-c767-4b21-9509-994a67ffdeaf-cni-binary-copy\") pod \"multus-8vxxt\" (UID: \"bda35fc9-c767-4b21-9509-994a67ffdeaf\") " pod="openshift-multus/multus-8vxxt"
Sep 11 00:57:47.876370 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.876295    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/bda35fc9-c767-4b21-9509-994a67ffdeaf-host-var-lib-cni-multus\") pod \"multus-8vxxt\" (UID: \"bda35fc9-c767-4b21-9509-994a67ffdeaf\") " pod="openshift-multus/multus-8vxxt"
Sep 11 00:57:47.876370 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.876320    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwg5s\" (UniqueName: \"kubernetes.io/projected/239b25bd-4932-4b5b-b286-45d3e2ff9f1c-kube-api-access-qwg5s\") pod \"network-check-target-bbbq6\" (UID: \"239b25bd-4932-4b5b-b286-45d3e2ff9f1c\") " pod="openshift-network-diagnostics/network-check-target-bbbq6"
Sep 11 00:57:47.876370 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.876355    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f27a49c0-52c5-42cf-9640-77fb72ef0998-etc-systemd\") pod \"tuned-m8lrq\" (UID: \"f27a49c0-52c5-42cf-9640-77fb72ef0998\") " pod="openshift-cluster-node-tuning-operator/tuned-m8lrq"
Sep 11 00:57:47.876801 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.876409    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/37ee44c2-7e75-4918-bc66-636974fa9b94-ovnkube-script-lib\") pod \"ovnkube-node-qsqkd\" (UID: \"37ee44c2-7e75-4918-bc66-636974fa9b94\") " pod="openshift-ovn-kubernetes/ovnkube-node-qsqkd"
Sep 11 00:57:47.876801 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.876440    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/799942b1-a6d8-4f5a-8d17-e7b2761b68b7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-8bg6q\" (UID: \"799942b1-a6d8-4f5a-8d17-e7b2761b68b7\") " pod="openshift-multus/multus-additional-cni-plugins-8bg6q"
Sep 11 00:57:47.876801 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.876465    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f27a49c0-52c5-42cf-9640-77fb72ef0998-sys\") pod \"tuned-m8lrq\" (UID: \"f27a49c0-52c5-42cf-9640-77fb72ef0998\") " pod="openshift-cluster-node-tuning-operator/tuned-m8lrq"
Sep 11 00:57:47.876801 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.876483    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/bda35fc9-c767-4b21-9509-994a67ffdeaf-multus-socket-dir-parent\") pod \"multus-8vxxt\" (UID: \"bda35fc9-c767-4b21-9509-994a67ffdeaf\") " pod="openshift-multus/multus-8vxxt"
Sep 11 00:57:47.876801 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.876493    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/37ee44c2-7e75-4918-bc66-636974fa9b94-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qsqkd\" (UID: \"37ee44c2-7e75-4918-bc66-636974fa9b94\") " pod="openshift-ovn-kubernetes/ovnkube-node-qsqkd"
Sep 11 00:57:47.876801 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.876526    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d5d763bb-6142-409f-8155-7143a6d91dc6-kubelet-config\") pod \"global-pull-secret-syncer-qgmsz\" (UID: \"d5d763bb-6142-409f-8155-7143a6d91dc6\") " pod="kube-system/global-pull-secret-syncer-qgmsz"
Sep 11 00:57:47.876801 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.876555    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/799942b1-a6d8-4f5a-8d17-e7b2761b68b7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-8bg6q\" (UID: \"799942b1-a6d8-4f5a-8d17-e7b2761b68b7\") " pod="openshift-multus/multus-additional-cni-plugins-8bg6q"
Sep 11 00:57:47.876801 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.876569    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/37ee44c2-7e75-4918-bc66-636974fa9b94-ovn-node-metrics-cert\") pod \"ovnkube-node-qsqkd\" (UID: \"37ee44c2-7e75-4918-bc66-636974fa9b94\") " pod="openshift-ovn-kubernetes/ovnkube-node-qsqkd"
Sep 11 00:57:47.876801 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.876573    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bda35fc9-c767-4b21-9509-994a67ffdeaf-host-var-lib-kubelet\") pod \"multus-8vxxt\" (UID: \"bda35fc9-c767-4b21-9509-994a67ffdeaf\") " pod="openshift-multus/multus-8vxxt"
Sep 11 00:57:47.876801 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.876596    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bda35fc9-c767-4b21-9509-994a67ffdeaf-cnibin\") pod \"multus-8vxxt\" (UID: \"bda35fc9-c767-4b21-9509-994a67ffdeaf\") " pod="openshift-multus/multus-8vxxt"
Sep 11 00:57:47.876801 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.876624    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d5d763bb-6142-409f-8155-7143a6d91dc6-dbus\") pod \"global-pull-secret-syncer-qgmsz\" (UID: \"d5d763bb-6142-409f-8155-7143a6d91dc6\") " pod="kube-system/global-pull-secret-syncer-qgmsz"
Sep 11 00:57:47.876801 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.876691    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/bda35fc9-c767-4b21-9509-994a67ffdeaf-hostroot\") pod \"multus-8vxxt\" (UID: \"bda35fc9-c767-4b21-9509-994a67ffdeaf\") " pod="openshift-multus/multus-8vxxt"
Sep 11 00:57:47.876801 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.876728    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t29m9\" (UniqueName: \"kubernetes.io/projected/bda35fc9-c767-4b21-9509-994a67ffdeaf-kube-api-access-t29m9\") pod \"multus-8vxxt\" (UID: \"bda35fc9-c767-4b21-9509-994a67ffdeaf\") " pod="openshift-multus/multus-8vxxt"
Sep 11 00:57:47.876801 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.876757    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/37ee44c2-7e75-4918-bc66-636974fa9b94-host-kubelet\") pod \"ovnkube-node-qsqkd\" (UID: \"37ee44c2-7e75-4918-bc66-636974fa9b94\") " pod="openshift-ovn-kubernetes/ovnkube-node-qsqkd"
Sep 11 00:57:47.876801 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.876784    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/37ee44c2-7e75-4918-bc66-636974fa9b94-host-cni-bin\") pod \"ovnkube-node-qsqkd\" (UID: \"37ee44c2-7e75-4918-bc66-636974fa9b94\") " pod="openshift-ovn-kubernetes/ovnkube-node-qsqkd"
Sep 11 00:57:47.877588 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.876810    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/920866ad-d7f7-46e4-8624-7ae8011a2b76-socket-dir\") pod \"aws-ebs-csi-driver-node-t42lc\" (UID: \"920866ad-d7f7-46e4-8624-7ae8011a2b76\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t42lc"
Sep 11 00:57:47.877588 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.876837    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bda35fc9-c767-4b21-9509-994a67ffdeaf-etc-kubernetes\") pod \"multus-8vxxt\" (UID: \"bda35fc9-c767-4b21-9509-994a67ffdeaf\") " pod="openshift-multus/multus-8vxxt"
Sep 11 00:57:47.877588 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.876865    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/799942b1-a6d8-4f5a-8d17-e7b2761b68b7-system-cni-dir\") pod \"multus-additional-cni-plugins-8bg6q\" (UID: \"799942b1-a6d8-4f5a-8d17-e7b2761b68b7\") " pod="openshift-multus/multus-additional-cni-plugins-8bg6q"
Sep 11 00:57:47.877588 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.876893    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-856bx\" (UniqueName: \"kubernetes.io/projected/f2ed1cf4-6159-47f1-aff0-8915d23dc932-kube-api-access-856bx\") pod \"iptables-alerter-j862p\" (UID: \"f2ed1cf4-6159-47f1-aff0-8915d23dc932\") " pod="openshift-network-operator/iptables-alerter-j862p"
Sep 11 00:57:47.877588 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.876943    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/bda35fc9-c767-4b21-9509-994a67ffdeaf-host-run-multus-certs\") pod \"multus-8vxxt\" (UID: \"bda35fc9-c767-4b21-9509-994a67ffdeaf\") " pod="openshift-multus/multus-8vxxt"
Sep 11 00:57:47.877588 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.876971    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f27a49c0-52c5-42cf-9640-77fb72ef0998-var-lib-kubelet\") pod \"tuned-m8lrq\" (UID: \"f27a49c0-52c5-42cf-9640-77fb72ef0998\") " pod="openshift-cluster-node-tuning-operator/tuned-m8lrq"
Sep 11 00:57:47.877588 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.876998    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bda35fc9-c767-4b21-9509-994a67ffdeaf-host-run-netns\") pod \"multus-8vxxt\" (UID: \"bda35fc9-c767-4b21-9509-994a67ffdeaf\") " pod="openshift-multus/multus-8vxxt"
Sep 11 00:57:47.877588 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.877098    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64jxj\" (UniqueName: \"kubernetes.io/projected/c7486f1f-9947-4405-9186-66dfa5621253-kube-api-access-64jxj\") pod \"node-resolver-2dvjn\" (UID: \"c7486f1f-9947-4405-9186-66dfa5621253\") " pod="openshift-dns/node-resolver-2dvjn"
Sep 11 00:57:47.877588 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.877128    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gc78g\" (UniqueName: \"kubernetes.io/projected/799942b1-a6d8-4f5a-8d17-e7b2761b68b7-kube-api-access-gc78g\") pod \"multus-additional-cni-plugins-8bg6q\" (UID: \"799942b1-a6d8-4f5a-8d17-e7b2761b68b7\") " pod="openshift-multus/multus-additional-cni-plugins-8bg6q"
Sep 11 00:57:47.877588 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.877160    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bda35fc9-c767-4b21-9509-994a67ffdeaf-cnibin\") pod \"multus-8vxxt\" (UID: \"bda35fc9-c767-4b21-9509-994a67ffdeaf\") " pod="openshift-multus/multus-8vxxt"
Sep 11 00:57:47.877588 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.877170    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/799942b1-a6d8-4f5a-8d17-e7b2761b68b7-system-cni-dir\") pod \"multus-additional-cni-plugins-8bg6q\" (UID: \"799942b1-a6d8-4f5a-8d17-e7b2761b68b7\") " pod="openshift-multus/multus-additional-cni-plugins-8bg6q"
Sep 11 00:57:47.877588 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.877183    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/bda35fc9-c767-4b21-9509-994a67ffdeaf-host-run-multus-certs\") pod \"multus-8vxxt\" (UID: \"bda35fc9-c767-4b21-9509-994a67ffdeaf\") " pod="openshift-multus/multus-8vxxt"
Sep 11 00:57:47.877588 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.877217    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bda35fc9-c767-4b21-9509-994a67ffdeaf-host-run-netns\") pod \"multus-8vxxt\" (UID: \"bda35fc9-c767-4b21-9509-994a67ffdeaf\") " pod="openshift-multus/multus-8vxxt"
Sep 11 00:57:47.877588 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.877243    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bda35fc9-c767-4b21-9509-994a67ffdeaf-etc-kubernetes\") pod \"multus-8vxxt\" (UID: \"bda35fc9-c767-4b21-9509-994a67ffdeaf\") " pod="openshift-multus/multus-8vxxt"
Sep 11 00:57:47.877588 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.877255    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/bda35fc9-c767-4b21-9509-994a67ffdeaf-host-var-lib-cni-multus\") pod \"multus-8vxxt\" (UID: \"bda35fc9-c767-4b21-9509-994a67ffdeaf\") " pod="openshift-multus/multus-8vxxt"
Sep 11 00:57:47.877588 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.877279    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f27a49c0-52c5-42cf-9640-77fb72ef0998-etc-modprobe-d\") pod \"tuned-m8lrq\" (UID: \"f27a49c0-52c5-42cf-9640-77fb72ef0998\") " pod="openshift-cluster-node-tuning-operator/tuned-m8lrq"
Sep 11 00:57:47.877588 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.877317    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d5d763bb-6142-409f-8155-7143a6d91dc6-kubelet-config\") pod \"global-pull-secret-syncer-qgmsz\" (UID: \"d5d763bb-6142-409f-8155-7143a6d91dc6\") " pod="kube-system/global-pull-secret-syncer-qgmsz"
Sep 11 00:57:47.878445 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.877323    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f27a49c0-52c5-42cf-9640-77fb72ef0998-etc-sysconfig\") pod \"tuned-m8lrq\" (UID: \"f27a49c0-52c5-42cf-9640-77fb72ef0998\") " pod="openshift-cluster-node-tuning-operator/tuned-m8lrq"
Sep 11 00:57:47.878445 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.877442    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f27a49c0-52c5-42cf-9640-77fb72ef0998-etc-kubernetes\") pod \"tuned-m8lrq\" (UID: \"f27a49c0-52c5-42cf-9640-77fb72ef0998\") " pod="openshift-cluster-node-tuning-operator/tuned-m8lrq"
Sep 11 00:57:47.878445 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.877473    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/37ee44c2-7e75-4918-bc66-636974fa9b94-systemd-units\") pod \"ovnkube-node-qsqkd\" (UID: \"37ee44c2-7e75-4918-bc66-636974fa9b94\") " pod="openshift-ovn-kubernetes/ovnkube-node-qsqkd"
Sep 11 00:57:47.878445 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.877483    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/bda35fc9-c767-4b21-9509-994a67ffdeaf-hostroot\") pod \"multus-8vxxt\" (UID: \"bda35fc9-c767-4b21-9509-994a67ffdeaf\") " pod="openshift-multus/multus-8vxxt"
Sep 11 00:57:47.878445 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.877512    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/37ee44c2-7e75-4918-bc66-636974fa9b94-host-run-netns\") pod \"ovnkube-node-qsqkd\" (UID: \"37ee44c2-7e75-4918-bc66-636974fa9b94\") " pod="openshift-ovn-kubernetes/ovnkube-node-qsqkd"
Sep 11 00:57:47.878445 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.877548    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d5d763bb-6142-409f-8155-7143a6d91dc6-original-pull-secret\") pod \"global-pull-secret-syncer-qgmsz\" (UID: \"d5d763bb-6142-409f-8155-7143a6d91dc6\") " pod="kube-system/global-pull-secret-syncer-qgmsz"
Sep 11 00:57:47.878445 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.877606    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c7486f1f-9947-4405-9186-66dfa5621253-hosts-file\") pod \"node-resolver-2dvjn\" (UID: \"c7486f1f-9947-4405-9186-66dfa5621253\") " pod="openshift-dns/node-resolver-2dvjn"
Sep 11 00:57:47.878445 ip-10-0-129-222 kubenswrapper[2569]: E0911 00:57:47.877633    2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 00:57:47.878445 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.877641    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f27a49c0-52c5-42cf-9640-77fb72ef0998-etc-sysctl-d\") pod \"tuned-m8lrq\" (UID: \"f27a49c0-52c5-42cf-9640-77fb72ef0998\") " pod="openshift-cluster-node-tuning-operator/tuned-m8lrq"
Sep 11 00:57:47.878445 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.877671    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/37ee44c2-7e75-4918-bc66-636974fa9b94-run-openvswitch\") pod \"ovnkube-node-qsqkd\" (UID: \"37ee44c2-7e75-4918-bc66-636974fa9b94\") " pod="openshift-ovn-kubernetes/ovnkube-node-qsqkd"
Sep 11 00:57:47.878445 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.877689    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c7486f1f-9947-4405-9186-66dfa5621253-hosts-file\") pod \"node-resolver-2dvjn\" (UID: \"c7486f1f-9947-4405-9186-66dfa5621253\") " pod="openshift-dns/node-resolver-2dvjn"
Sep 11 00:57:47.878445 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.877713    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/37ee44c2-7e75-4918-bc66-636974fa9b94-host-run-ovn-kubernetes\") pod \"ovnkube-node-qsqkd\" (UID: \"37ee44c2-7e75-4918-bc66-636974fa9b94\") " pod="openshift-ovn-kubernetes/ovnkube-node-qsqkd"
Sep 11 00:57:47.878445 ip-10-0-129-222 kubenswrapper[2569]: E0911 00:57:47.877760    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d5d763bb-6142-409f-8155-7143a6d91dc6-original-pull-secret podName:d5d763bb-6142-409f-8155-7143a6d91dc6 nodeName:}" failed. No retries permitted until 2026-09-11 00:57:48.377738677 +0000 UTC m=+3.029112905 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d5d763bb-6142-409f-8155-7143a6d91dc6-original-pull-secret") pod "global-pull-secret-syncer-qgmsz" (UID: "d5d763bb-6142-409f-8155-7143a6d91dc6") : object "kube-system"/"original-pull-secret" not registered
Sep 11 00:57:47.878445 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.877785    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vr7z\" (UniqueName: \"kubernetes.io/projected/920866ad-d7f7-46e4-8624-7ae8011a2b76-kube-api-access-5vr7z\") pod \"aws-ebs-csi-driver-node-t42lc\" (UID: \"920866ad-d7f7-46e4-8624-7ae8011a2b76\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t42lc"
Sep 11 00:57:47.878445 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.877784    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bda35fc9-c767-4b21-9509-994a67ffdeaf-cni-binary-copy\") pod \"multus-8vxxt\" (UID: \"bda35fc9-c767-4b21-9509-994a67ffdeaf\") " pod="openshift-multus/multus-8vxxt"
Sep 11 00:57:47.878445 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.877838    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bda35fc9-c767-4b21-9509-994a67ffdeaf-system-cni-dir\") pod \"multus-8vxxt\" (UID: \"bda35fc9-c767-4b21-9509-994a67ffdeaf\") " pod="openshift-multus/multus-8vxxt"
Sep 11 00:57:47.878445 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.877877    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/799942b1-a6d8-4f5a-8d17-e7b2761b68b7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-8bg6q\" (UID: \"799942b1-a6d8-4f5a-8d17-e7b2761b68b7\") " pod="openshift-multus/multus-additional-cni-plugins-8bg6q"
Sep 11 00:57:47.879324 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.878017    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bda35fc9-c767-4b21-9509-994a67ffdeaf-system-cni-dir\") pod \"multus-8vxxt\" (UID: \"bda35fc9-c767-4b21-9509-994a67ffdeaf\") " pod="openshift-multus/multus-8vxxt"
Sep 11 00:57:47.879324 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.878080    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/37ee44c2-7e75-4918-bc66-636974fa9b94-host-slash\") pod \"ovnkube-node-qsqkd\" (UID: \"37ee44c2-7e75-4918-bc66-636974fa9b94\") " pod="openshift-ovn-kubernetes/ovnkube-node-qsqkd"
Sep 11 00:57:47.879324 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.878192    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/37ee44c2-7e75-4918-bc66-636974fa9b94-host-cni-netd\") pod \"ovnkube-node-qsqkd\" (UID: \"37ee44c2-7e75-4918-bc66-636974fa9b94\") " pod="openshift-ovn-kubernetes/ovnkube-node-qsqkd"
Sep 11 00:57:47.879324 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.878230    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/37ee44c2-7e75-4918-bc66-636974fa9b94-env-overrides\") pod \"ovnkube-node-qsqkd\" (UID: \"37ee44c2-7e75-4918-bc66-636974fa9b94\") " pod="openshift-ovn-kubernetes/ovnkube-node-qsqkd"
Sep 11 00:57:47.879324 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.878258    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/920866ad-d7f7-46e4-8624-7ae8011a2b76-etc-selinux\") pod \"aws-ebs-csi-driver-node-t42lc\" (UID: \"920866ad-d7f7-46e4-8624-7ae8011a2b76\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t42lc"
Sep 11 00:57:47.879324 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.878290    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlxmt\" (UniqueName: \"kubernetes.io/projected/304c1b0f-f5f9-4c4c-8041-ac384ad21516-kube-api-access-mlxmt\") pod \"node-ca-jrszh\" (UID: \"304c1b0f-f5f9-4c4c-8041-ac384ad21516\") " pod="openshift-image-registry/node-ca-jrszh"
Sep 11 00:57:47.879324 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.878316    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f27a49c0-52c5-42cf-9640-77fb72ef0998-tmp\") pod \"tuned-m8lrq\" (UID: \"f27a49c0-52c5-42cf-9640-77fb72ef0998\") " pod="openshift-cluster-node-tuning-operator/tuned-m8lrq"
Sep 11 00:57:47.879324 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.878399    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/799942b1-a6d8-4f5a-8d17-e7b2761b68b7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-8bg6q\" (UID: \"799942b1-a6d8-4f5a-8d17-e7b2761b68b7\") " pod="openshift-multus/multus-additional-cni-plugins-8bg6q"
Sep 11 00:57:47.879324 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.878344    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/37ee44c2-7e75-4918-bc66-636974fa9b94-ovnkube-config\") pod \"ovnkube-node-qsqkd\" (UID: \"37ee44c2-7e75-4918-bc66-636974fa9b94\") " pod="openshift-ovn-kubernetes/ovnkube-node-qsqkd"
Sep 11 00:57:47.879324 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.878815    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bda35fc9-c767-4b21-9509-994a67ffdeaf-multus-cni-dir\") pod \"multus-8vxxt\" (UID: \"bda35fc9-c767-4b21-9509-994a67ffdeaf\") " pod="openshift-multus/multus-8vxxt"
Sep 11 00:57:47.879324 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.878829    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/799942b1-a6d8-4f5a-8d17-e7b2761b68b7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-8bg6q\" (UID: \"799942b1-a6d8-4f5a-8d17-e7b2761b68b7\") " pod="openshift-multus/multus-additional-cni-plugins-8bg6q"
Sep 11 00:57:47.879324 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.878843    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/bda35fc9-c767-4b21-9509-994a67ffdeaf-host-run-k8s-cni-cncf-io\") pod \"multus-8vxxt\" (UID: \"bda35fc9-c767-4b21-9509-994a67ffdeaf\") " pod="openshift-multus/multus-8vxxt"
Sep 11 00:57:47.879324 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.878886    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bda35fc9-c767-4b21-9509-994a67ffdeaf-multus-conf-dir\") pod \"multus-8vxxt\" (UID: \"bda35fc9-c767-4b21-9509-994a67ffdeaf\") " pod="openshift-multus/multus-8vxxt"
Sep 11 00:57:47.879324 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.878911    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/799942b1-a6d8-4f5a-8d17-e7b2761b68b7-cnibin\") pod \"multus-additional-cni-plugins-8bg6q\" (UID: \"799942b1-a6d8-4f5a-8d17-e7b2761b68b7\") " pod="openshift-multus/multus-additional-cni-plugins-8bg6q"
Sep 11 00:57:47.879324 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.878937    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/799942b1-a6d8-4f5a-8d17-e7b2761b68b7-cni-binary-copy\") pod \"multus-additional-cni-plugins-8bg6q\" (UID: \"799942b1-a6d8-4f5a-8d17-e7b2761b68b7\") " pod="openshift-multus/multus-additional-cni-plugins-8bg6q"
Sep 11 00:57:47.879324 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.878975    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f27a49c0-52c5-42cf-9640-77fb72ef0998-etc-sysctl-conf\") pod \"tuned-m8lrq\" (UID: \"f27a49c0-52c5-42cf-9640-77fb72ef0998\") " pod="openshift-cluster-node-tuning-operator/tuned-m8lrq"
Sep 11 00:57:47.879324 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.879000    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/bbc61763-126c-46a6-8607-9f1071004770-konnectivity-ca\") pod \"konnectivity-agent-st6p9\" (UID: \"bbc61763-126c-46a6-8607-9f1071004770\") " pod="kube-system/konnectivity-agent-st6p9"
Sep 11 00:57:47.880253 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.879014    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bda35fc9-c767-4b21-9509-994a67ffdeaf-multus-cni-dir\") pod \"multus-8vxxt\" (UID: \"bda35fc9-c767-4b21-9509-994a67ffdeaf\") " pod="openshift-multus/multus-8vxxt"
Sep 11 00:57:47.880253 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.879025    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/304c1b0f-f5f9-4c4c-8041-ac384ad21516-serviceca\") pod \"node-ca-jrszh\" (UID: \"304c1b0f-f5f9-4c4c-8041-ac384ad21516\") " pod="openshift-image-registry/node-ca-jrszh"
Sep 11 00:57:47.880253 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.879048    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f27a49c0-52c5-42cf-9640-77fb72ef0998-host\") pod \"tuned-m8lrq\" (UID: \"f27a49c0-52c5-42cf-9640-77fb72ef0998\") " pod="openshift-cluster-node-tuning-operator/tuned-m8lrq"
Sep 11 00:57:47.880253 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.879069    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/799942b1-a6d8-4f5a-8d17-e7b2761b68b7-cnibin\") pod \"multus-additional-cni-plugins-8bg6q\" (UID: \"799942b1-a6d8-4f5a-8d17-e7b2761b68b7\") " pod="openshift-multus/multus-additional-cni-plugins-8bg6q"
Sep 11 00:57:47.880253 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.879077    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/37ee44c2-7e75-4918-bc66-636974fa9b94-etc-openvswitch\") pod \"ovnkube-node-qsqkd\" (UID: \"37ee44c2-7e75-4918-bc66-636974fa9b94\") " pod="openshift-ovn-kubernetes/ovnkube-node-qsqkd"
Sep 11 00:57:47.880253 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.879093    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/37ee44c2-7e75-4918-bc66-636974fa9b94-log-socket\") pod \"ovnkube-node-qsqkd\" (UID: \"37ee44c2-7e75-4918-bc66-636974fa9b94\") " pod="openshift-ovn-kubernetes/ovnkube-node-qsqkd"
Sep 11 00:57:47.880253 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.879108    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bda35fc9-c767-4b21-9509-994a67ffdeaf-host-var-lib-cni-bin\") pod \"multus-8vxxt\" (UID: \"bda35fc9-c767-4b21-9509-994a67ffdeaf\") " pod="openshift-multus/multus-8vxxt"
Sep 11 00:57:47.880253 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.879116    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/bda35fc9-c767-4b21-9509-994a67ffdeaf-host-run-k8s-cni-cncf-io\") pod \"multus-8vxxt\" (UID: \"bda35fc9-c767-4b21-9509-994a67ffdeaf\") " pod="openshift-multus/multus-8vxxt"
Sep 11 00:57:47.880253 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.879125    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f27a49c0-52c5-42cf-9640-77fb72ef0998-run\") pod \"tuned-m8lrq\" (UID: \"f27a49c0-52c5-42cf-9640-77fb72ef0998\") " pod="openshift-cluster-node-tuning-operator/tuned-m8lrq"
Sep 11 00:57:47.880253 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.879166    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85c82c9f-52d0-49cd-afd9-b791628e1a50-metrics-certs\") pod \"network-metrics-daemon-r8n96\" (UID: \"85c82c9f-52d0-49cd-afd9-b791628e1a50\") " pod="openshift-multus/network-metrics-daemon-r8n96"
Sep 11 00:57:47.880253 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.879167    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bda35fc9-c767-4b21-9509-994a67ffdeaf-multus-conf-dir\") pod \"multus-8vxxt\" (UID: \"bda35fc9-c767-4b21-9509-994a67ffdeaf\") " pod="openshift-multus/multus-8vxxt"
Sep 11 00:57:47.880253 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.879198    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/304c1b0f-f5f9-4c4c-8041-ac384ad21516-host\") pod \"node-ca-jrszh\" (UID: \"304c1b0f-f5f9-4c4c-8041-ac384ad21516\") " pod="openshift-image-registry/node-ca-jrszh"
Sep 11 00:57:47.880253 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.879229    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9l8xf\" (UniqueName: \"kubernetes.io/projected/f27a49c0-52c5-42cf-9640-77fb72ef0998-kube-api-access-9l8xf\") pod \"tuned-m8lrq\" (UID: \"f27a49c0-52c5-42cf-9640-77fb72ef0998\") " pod="openshift-cluster-node-tuning-operator/tuned-m8lrq"
Sep 11 00:57:47.880253 ip-10-0-129-222 kubenswrapper[2569]: E0911 00:57:47.879274    2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 00:57:47.880253 ip-10-0-129-222 kubenswrapper[2569]: E0911 00:57:47.879310    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85c82c9f-52d0-49cd-afd9-b791628e1a50-metrics-certs podName:85c82c9f-52d0-49cd-afd9-b791628e1a50 nodeName:}" failed. No retries permitted until 2026-09-11 00:57:48.379299048 +0000 UTC m=+3.030673277 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/85c82c9f-52d0-49cd-afd9-b791628e1a50-metrics-certs") pod "network-metrics-daemon-r8n96" (UID: "85c82c9f-52d0-49cd-afd9-b791628e1a50") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 00:57:47.880253 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.879309    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f2ed1cf4-6159-47f1-aff0-8915d23dc932-iptables-alerter-script\") pod \"iptables-alerter-j862p\" (UID: \"f2ed1cf4-6159-47f1-aff0-8915d23dc932\") " pod="openshift-network-operator/iptables-alerter-j862p"
Sep 11 00:57:47.880253 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.879337    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/37ee44c2-7e75-4918-bc66-636974fa9b94-run-systemd\") pod \"ovnkube-node-qsqkd\" (UID: \"37ee44c2-7e75-4918-bc66-636974fa9b94\") " pod="openshift-ovn-kubernetes/ovnkube-node-qsqkd"
Sep 11 00:57:47.881140 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.879362    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/37ee44c2-7e75-4918-bc66-636974fa9b94-var-lib-openvswitch\") pod \"ovnkube-node-qsqkd\" (UID: \"37ee44c2-7e75-4918-bc66-636974fa9b94\") " pod="openshift-ovn-kubernetes/ovnkube-node-qsqkd"
Sep 11 00:57:47.881140 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.879385    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/bbc61763-126c-46a6-8607-9f1071004770-agent-certs\") pod \"konnectivity-agent-st6p9\" (UID: \"bbc61763-126c-46a6-8607-9f1071004770\") " pod="kube-system/konnectivity-agent-st6p9"
Sep 11 00:57:47.881140 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.879409    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/920866ad-d7f7-46e4-8624-7ae8011a2b76-kubelet-dir\") pod \"aws-ebs-csi-driver-node-t42lc\" (UID: \"920866ad-d7f7-46e4-8624-7ae8011a2b76\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t42lc"
Sep 11 00:57:47.881140 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.879434    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f27a49c0-52c5-42cf-9640-77fb72ef0998-lib-modules\") pod \"tuned-m8lrq\" (UID: \"f27a49c0-52c5-42cf-9640-77fb72ef0998\") " pod="openshift-cluster-node-tuning-operator/tuned-m8lrq"
Sep 11 00:57:47.881140 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.879459    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f27a49c0-52c5-42cf-9640-77fb72ef0998-etc-tuned\") pod \"tuned-m8lrq\" (UID: \"f27a49c0-52c5-42cf-9640-77fb72ef0998\") " pod="openshift-cluster-node-tuning-operator/tuned-m8lrq"
Sep 11 00:57:47.881140 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.879483    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f2ed1cf4-6159-47f1-aff0-8915d23dc932-host-slash\") pod \"iptables-alerter-j862p\" (UID: \"f2ed1cf4-6159-47f1-aff0-8915d23dc932\") " pod="openshift-network-operator/iptables-alerter-j862p"
Sep 11 00:57:47.881140 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.879511    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/bda35fc9-c767-4b21-9509-994a67ffdeaf-multus-daemon-config\") pod \"multus-8vxxt\" (UID: \"bda35fc9-c767-4b21-9509-994a67ffdeaf\") " pod="openshift-multus/multus-8vxxt"
Sep 11 00:57:47.881140 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.879537    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7486f1f-9947-4405-9186-66dfa5621253-tmp-dir\") pod \"node-resolver-2dvjn\" (UID: \"c7486f1f-9947-4405-9186-66dfa5621253\") " pod="openshift-dns/node-resolver-2dvjn"
Sep 11 00:57:47.881140 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.879562    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/37ee44c2-7e75-4918-bc66-636974fa9b94-run-ovn\") pod \"ovnkube-node-qsqkd\" (UID: \"37ee44c2-7e75-4918-bc66-636974fa9b94\") " pod="openshift-ovn-kubernetes/ovnkube-node-qsqkd"
Sep 11 00:57:47.881140 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.879589    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jpqdb\" (UniqueName: \"kubernetes.io/projected/37ee44c2-7e75-4918-bc66-636974fa9b94-kube-api-access-jpqdb\") pod \"ovnkube-node-qsqkd\" (UID: \"37ee44c2-7e75-4918-bc66-636974fa9b94\") " pod="openshift-ovn-kubernetes/ovnkube-node-qsqkd"
Sep 11 00:57:47.881140 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.879601    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/304c1b0f-f5f9-4c4c-8041-ac384ad21516-serviceca\") pod \"node-ca-jrszh\" (UID: \"304c1b0f-f5f9-4c4c-8041-ac384ad21516\") " pod="openshift-image-registry/node-ca-jrszh"
Sep 11 00:57:47.881140 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.879673    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/304c1b0f-f5f9-4c4c-8041-ac384ad21516-host\") pod \"node-ca-jrszh\" (UID: \"304c1b0f-f5f9-4c4c-8041-ac384ad21516\") " pod="openshift-image-registry/node-ca-jrszh"
Sep 11 00:57:47.881140 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.879718    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bda35fc9-c767-4b21-9509-994a67ffdeaf-host-var-lib-cni-bin\") pod \"multus-8vxxt\" (UID: \"bda35fc9-c767-4b21-9509-994a67ffdeaf\") " pod="openshift-multus/multus-8vxxt"
Sep 11 00:57:47.881140 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.879984    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/799942b1-a6d8-4f5a-8d17-e7b2761b68b7-cni-binary-copy\") pod \"multus-additional-cni-plugins-8bg6q\" (UID: \"799942b1-a6d8-4f5a-8d17-e7b2761b68b7\") " pod="openshift-multus/multus-additional-cni-plugins-8bg6q"
Sep 11 00:57:47.881140 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.880217    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7486f1f-9947-4405-9186-66dfa5621253-tmp-dir\") pod \"node-resolver-2dvjn\" (UID: \"c7486f1f-9947-4405-9186-66dfa5621253\") " pod="openshift-dns/node-resolver-2dvjn"
Sep 11 00:57:47.881140 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.880255    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/bda35fc9-c767-4b21-9509-994a67ffdeaf-multus-daemon-config\") pod \"multus-8vxxt\" (UID: \"bda35fc9-c767-4b21-9509-994a67ffdeaf\") " pod="openshift-multus/multus-8vxxt"
Sep 11 00:57:47.883597 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.883570    2569 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 11 00:57:47.886334 ip-10-0-129-222 kubenswrapper[2569]: E0911 00:57:47.886313    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 00:57:47.886334 ip-10-0-129-222 kubenswrapper[2569]: E0911 00:57:47.886342    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 00:57:47.886566 ip-10-0-129-222 kubenswrapper[2569]: E0911 00:57:47.886355    2569 projected.go:196] Error preparing data for projected volume kube-api-access-qwg5s for pod openshift-network-diagnostics/network-check-target-bbbq6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 00:57:47.886566 ip-10-0-129-222 kubenswrapper[2569]: E0911 00:57:47.886415    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/239b25bd-4932-4b5b-b286-45d3e2ff9f1c-kube-api-access-qwg5s podName:239b25bd-4932-4b5b-b286-45d3e2ff9f1c nodeName:}" failed. No retries permitted until 2026-09-11 00:57:48.386399753 +0000 UTC m=+3.037774026 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-qwg5s" (UniqueName: "kubernetes.io/projected/239b25bd-4932-4b5b-b286-45d3e2ff9f1c-kube-api-access-qwg5s") pod "network-check-target-bbbq6" (UID: "239b25bd-4932-4b5b-b286-45d3e2ff9f1c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 00:57:47.889728 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.889703    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9z79r\" (UniqueName: \"kubernetes.io/projected/85c82c9f-52d0-49cd-afd9-b791628e1a50-kube-api-access-9z79r\") pod \"network-metrics-daemon-r8n96\" (UID: \"85c82c9f-52d0-49cd-afd9-b791628e1a50\") " pod="openshift-multus/network-metrics-daemon-r8n96"
Sep 11 00:57:47.889862 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.889713    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-64jxj\" (UniqueName: \"kubernetes.io/projected/c7486f1f-9947-4405-9186-66dfa5621253-kube-api-access-64jxj\") pod \"node-resolver-2dvjn\" (UID: \"c7486f1f-9947-4405-9186-66dfa5621253\") " pod="openshift-dns/node-resolver-2dvjn"
Sep 11 00:57:47.890193 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.890170    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gc78g\" (UniqueName: \"kubernetes.io/projected/799942b1-a6d8-4f5a-8d17-e7b2761b68b7-kube-api-access-gc78g\") pod \"multus-additional-cni-plugins-8bg6q\" (UID: \"799942b1-a6d8-4f5a-8d17-e7b2761b68b7\") " pod="openshift-multus/multus-additional-cni-plugins-8bg6q"
Sep 11 00:57:47.891530 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.891170    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t29m9\" (UniqueName: \"kubernetes.io/projected/bda35fc9-c767-4b21-9509-994a67ffdeaf-kube-api-access-t29m9\") pod \"multus-8vxxt\" (UID: \"bda35fc9-c767-4b21-9509-994a67ffdeaf\") " pod="openshift-multus/multus-8vxxt"
Sep 11 00:57:47.891652 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.891563    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mlxmt\" (UniqueName: \"kubernetes.io/projected/304c1b0f-f5f9-4c4c-8041-ac384ad21516-kube-api-access-mlxmt\") pod \"node-ca-jrszh\" (UID: \"304c1b0f-f5f9-4c4c-8041-ac384ad21516\") " pod="openshift-image-registry/node-ca-jrszh"
Sep 11 00:57:47.979840 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.979803    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/37ee44c2-7e75-4918-bc66-636974fa9b94-node-log\") pod \"ovnkube-node-qsqkd\" (UID: \"37ee44c2-7e75-4918-bc66-636974fa9b94\") " pod="openshift-ovn-kubernetes/ovnkube-node-qsqkd"
Sep 11 00:57:47.979840 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.979842    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/920866ad-d7f7-46e4-8624-7ae8011a2b76-device-dir\") pod \"aws-ebs-csi-driver-node-t42lc\" (UID: \"920866ad-d7f7-46e4-8624-7ae8011a2b76\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t42lc"
Sep 11 00:57:47.980059 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.979862    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/920866ad-d7f7-46e4-8624-7ae8011a2b76-sys-fs\") pod \"aws-ebs-csi-driver-node-t42lc\" (UID: \"920866ad-d7f7-46e4-8624-7ae8011a2b76\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t42lc"
Sep 11 00:57:47.980059 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.979946    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/920866ad-d7f7-46e4-8624-7ae8011a2b76-sys-fs\") pod \"aws-ebs-csi-driver-node-t42lc\" (UID: \"920866ad-d7f7-46e4-8624-7ae8011a2b76\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t42lc"
Sep 11 00:57:47.980059 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.979946    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/920866ad-d7f7-46e4-8624-7ae8011a2b76-device-dir\") pod \"aws-ebs-csi-driver-node-t42lc\" (UID: \"920866ad-d7f7-46e4-8624-7ae8011a2b76\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t42lc"
Sep 11 00:57:47.980059 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.979979    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/37ee44c2-7e75-4918-bc66-636974fa9b94-node-log\") pod \"ovnkube-node-qsqkd\" (UID: \"37ee44c2-7e75-4918-bc66-636974fa9b94\") " pod="openshift-ovn-kubernetes/ovnkube-node-qsqkd"
Sep 11 00:57:47.980059 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.980042    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f27a49c0-52c5-42cf-9640-77fb72ef0998-etc-systemd\") pod \"tuned-m8lrq\" (UID: \"f27a49c0-52c5-42cf-9640-77fb72ef0998\") " pod="openshift-cluster-node-tuning-operator/tuned-m8lrq"
Sep 11 00:57:47.980312 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.980066    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/37ee44c2-7e75-4918-bc66-636974fa9b94-ovnkube-script-lib\") pod \"ovnkube-node-qsqkd\" (UID: \"37ee44c2-7e75-4918-bc66-636974fa9b94\") " pod="openshift-ovn-kubernetes/ovnkube-node-qsqkd"
Sep 11 00:57:47.980312 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.980086    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f27a49c0-52c5-42cf-9640-77fb72ef0998-sys\") pod \"tuned-m8lrq\" (UID: \"f27a49c0-52c5-42cf-9640-77fb72ef0998\") " pod="openshift-cluster-node-tuning-operator/tuned-m8lrq"
Sep 11 00:57:47.980312 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.980109    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/37ee44c2-7e75-4918-bc66-636974fa9b94-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qsqkd\" (UID: \"37ee44c2-7e75-4918-bc66-636974fa9b94\") " pod="openshift-ovn-kubernetes/ovnkube-node-qsqkd"
Sep 11 00:57:47.980312 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.980137    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/37ee44c2-7e75-4918-bc66-636974fa9b94-ovn-node-metrics-cert\") pod \"ovnkube-node-qsqkd\" (UID: \"37ee44c2-7e75-4918-bc66-636974fa9b94\") " pod="openshift-ovn-kubernetes/ovnkube-node-qsqkd"
Sep 11 00:57:47.980312 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.980166    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/37ee44c2-7e75-4918-bc66-636974fa9b94-host-kubelet\") pod \"ovnkube-node-qsqkd\" (UID: \"37ee44c2-7e75-4918-bc66-636974fa9b94\") " pod="openshift-ovn-kubernetes/ovnkube-node-qsqkd"
Sep 11 00:57:47.980312 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.980194    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/37ee44c2-7e75-4918-bc66-636974fa9b94-host-cni-bin\") pod \"ovnkube-node-qsqkd\" (UID: \"37ee44c2-7e75-4918-bc66-636974fa9b94\") " pod="openshift-ovn-kubernetes/ovnkube-node-qsqkd"
Sep 11 00:57:47.980312 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.980206    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f27a49c0-52c5-42cf-9640-77fb72ef0998-sys\") pod \"tuned-m8lrq\" (UID: \"f27a49c0-52c5-42cf-9640-77fb72ef0998\") " pod="openshift-cluster-node-tuning-operator/tuned-m8lrq"
Sep 11 00:57:47.980312 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.980207    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/37ee44c2-7e75-4918-bc66-636974fa9b94-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qsqkd\" (UID: \"37ee44c2-7e75-4918-bc66-636974fa9b94\") " pod="openshift-ovn-kubernetes/ovnkube-node-qsqkd"
Sep 11 00:57:47.980312 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.980217    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/920866ad-d7f7-46e4-8624-7ae8011a2b76-socket-dir\") pod \"aws-ebs-csi-driver-node-t42lc\" (UID: \"920866ad-d7f7-46e4-8624-7ae8011a2b76\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t42lc"
Sep 11 00:57:47.980312 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.980262    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f27a49c0-52c5-42cf-9640-77fb72ef0998-etc-systemd\") pod \"tuned-m8lrq\" (UID: \"f27a49c0-52c5-42cf-9640-77fb72ef0998\") " pod="openshift-cluster-node-tuning-operator/tuned-m8lrq"
Sep 11 00:57:47.980312 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.980284    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-856bx\" (UniqueName: \"kubernetes.io/projected/f2ed1cf4-6159-47f1-aff0-8915d23dc932-kube-api-access-856bx\") pod \"iptables-alerter-j862p\" (UID: \"f2ed1cf4-6159-47f1-aff0-8915d23dc932\") " pod="openshift-network-operator/iptables-alerter-j862p"
Sep 11 00:57:47.980897 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.980319    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f27a49c0-52c5-42cf-9640-77fb72ef0998-var-lib-kubelet\") pod \"tuned-m8lrq\" (UID: \"f27a49c0-52c5-42cf-9640-77fb72ef0998\") " pod="openshift-cluster-node-tuning-operator/tuned-m8lrq"
Sep 11 00:57:47.980897 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.980326    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/37ee44c2-7e75-4918-bc66-636974fa9b94-host-kubelet\") pod \"ovnkube-node-qsqkd\" (UID: \"37ee44c2-7e75-4918-bc66-636974fa9b94\") " pod="openshift-ovn-kubernetes/ovnkube-node-qsqkd"
Sep 11 00:57:47.980897 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.980338    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/920866ad-d7f7-46e4-8624-7ae8011a2b76-socket-dir\") pod \"aws-ebs-csi-driver-node-t42lc\" (UID: \"920866ad-d7f7-46e4-8624-7ae8011a2b76\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t42lc"
Sep 11 00:57:47.980897 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.980380    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f27a49c0-52c5-42cf-9640-77fb72ef0998-var-lib-kubelet\") pod \"tuned-m8lrq\" (UID: \"f27a49c0-52c5-42cf-9640-77fb72ef0998\") " pod="openshift-cluster-node-tuning-operator/tuned-m8lrq"
Sep 11 00:57:47.980897 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.980382    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/37ee44c2-7e75-4918-bc66-636974fa9b94-host-cni-bin\") pod \"ovnkube-node-qsqkd\" (UID: \"37ee44c2-7e75-4918-bc66-636974fa9b94\") " pod="openshift-ovn-kubernetes/ovnkube-node-qsqkd"
Sep 11 00:57:47.980897 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.980418    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f27a49c0-52c5-42cf-9640-77fb72ef0998-etc-modprobe-d\") pod \"tuned-m8lrq\" (UID: \"f27a49c0-52c5-42cf-9640-77fb72ef0998\") " pod="openshift-cluster-node-tuning-operator/tuned-m8lrq"
Sep 11 00:57:47.980897 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.980447    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f27a49c0-52c5-42cf-9640-77fb72ef0998-etc-sysconfig\") pod \"tuned-m8lrq\" (UID: \"f27a49c0-52c5-42cf-9640-77fb72ef0998\") " pod="openshift-cluster-node-tuning-operator/tuned-m8lrq"
Sep 11 00:57:47.980897 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.980474    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f27a49c0-52c5-42cf-9640-77fb72ef0998-etc-kubernetes\") pod \"tuned-m8lrq\" (UID: \"f27a49c0-52c5-42cf-9640-77fb72ef0998\") " pod="openshift-cluster-node-tuning-operator/tuned-m8lrq"
Sep 11 00:57:47.980897 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.980500    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/37ee44c2-7e75-4918-bc66-636974fa9b94-systemd-units\") pod \"ovnkube-node-qsqkd\" (UID: \"37ee44c2-7e75-4918-bc66-636974fa9b94\") " pod="openshift-ovn-kubernetes/ovnkube-node-qsqkd"
Sep 11 00:57:47.980897 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.980528    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/37ee44c2-7e75-4918-bc66-636974fa9b94-host-run-netns\") pod \"ovnkube-node-qsqkd\" (UID: \"37ee44c2-7e75-4918-bc66-636974fa9b94\") " pod="openshift-ovn-kubernetes/ovnkube-node-qsqkd"
Sep 11 00:57:47.980897 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.980562    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f27a49c0-52c5-42cf-9640-77fb72ef0998-etc-sysconfig\") pod \"tuned-m8lrq\" (UID: \"f27a49c0-52c5-42cf-9640-77fb72ef0998\") " pod="openshift-cluster-node-tuning-operator/tuned-m8lrq"
Sep 11 00:57:47.980897 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.980567    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f27a49c0-52c5-42cf-9640-77fb72ef0998-etc-sysctl-d\") pod \"tuned-m8lrq\" (UID: \"f27a49c0-52c5-42cf-9640-77fb72ef0998\") " pod="openshift-cluster-node-tuning-operator/tuned-m8lrq"
Sep 11 00:57:47.980897 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.980594    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/37ee44c2-7e75-4918-bc66-636974fa9b94-run-openvswitch\") pod \"ovnkube-node-qsqkd\" (UID: \"37ee44c2-7e75-4918-bc66-636974fa9b94\") " pod="openshift-ovn-kubernetes/ovnkube-node-qsqkd"
Sep 11 00:57:47.980897 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.980621    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/37ee44c2-7e75-4918-bc66-636974fa9b94-host-run-ovn-kubernetes\") pod \"ovnkube-node-qsqkd\" (UID: \"37ee44c2-7e75-4918-bc66-636974fa9b94\") " pod="openshift-ovn-kubernetes/ovnkube-node-qsqkd"
Sep 11 00:57:47.980897 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.980649    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vr7z\" (UniqueName: \"kubernetes.io/projected/920866ad-d7f7-46e4-8624-7ae8011a2b76-kube-api-access-5vr7z\") pod \"aws-ebs-csi-driver-node-t42lc\" (UID: \"920866ad-d7f7-46e4-8624-7ae8011a2b76\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t42lc"
Sep 11 00:57:47.980897 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.980655    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f27a49c0-52c5-42cf-9640-77fb72ef0998-etc-modprobe-d\") pod \"tuned-m8lrq\" (UID: \"f27a49c0-52c5-42cf-9640-77fb72ef0998\") " pod="openshift-cluster-node-tuning-operator/tuned-m8lrq"
Sep 11 00:57:47.980897 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.980687    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/37ee44c2-7e75-4918-bc66-636974fa9b94-host-slash\") pod \"ovnkube-node-qsqkd\" (UID: \"37ee44c2-7e75-4918-bc66-636974fa9b94\") " pod="openshift-ovn-kubernetes/ovnkube-node-qsqkd"
Sep 11 00:57:47.981886 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.980715    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/37ee44c2-7e75-4918-bc66-636974fa9b94-host-cni-netd\") pod \"ovnkube-node-qsqkd\" (UID: \"37ee44c2-7e75-4918-bc66-636974fa9b94\") " pod="openshift-ovn-kubernetes/ovnkube-node-qsqkd"
Sep 11 00:57:47.981886 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.980717    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/37ee44c2-7e75-4918-bc66-636974fa9b94-ovnkube-script-lib\") pod \"ovnkube-node-qsqkd\" (UID: \"37ee44c2-7e75-4918-bc66-636974fa9b94\") " pod="openshift-ovn-kubernetes/ovnkube-node-qsqkd"
Sep 11 00:57:47.981886 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.980740    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/37ee44c2-7e75-4918-bc66-636974fa9b94-env-overrides\") pod \"ovnkube-node-qsqkd\" (UID: \"37ee44c2-7e75-4918-bc66-636974fa9b94\") " pod="openshift-ovn-kubernetes/ovnkube-node-qsqkd"
Sep 11 00:57:47.981886 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.980765    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/920866ad-d7f7-46e4-8624-7ae8011a2b76-etc-selinux\") pod \"aws-ebs-csi-driver-node-t42lc\" (UID: \"920866ad-d7f7-46e4-8624-7ae8011a2b76\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t42lc"
Sep 11 00:57:47.981886 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.980779    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/37ee44c2-7e75-4918-bc66-636974fa9b94-host-slash\") pod \"ovnkube-node-qsqkd\" (UID: \"37ee44c2-7e75-4918-bc66-636974fa9b94\") " pod="openshift-ovn-kubernetes/ovnkube-node-qsqkd"
Sep 11 00:57:47.981886 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.980792    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f27a49c0-52c5-42cf-9640-77fb72ef0998-tmp\") pod \"tuned-m8lrq\" (UID: \"f27a49c0-52c5-42cf-9640-77fb72ef0998\") " pod="openshift-cluster-node-tuning-operator/tuned-m8lrq"
Sep 11 00:57:47.981886 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.980818    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/37ee44c2-7e75-4918-bc66-636974fa9b94-ovnkube-config\") pod \"ovnkube-node-qsqkd\" (UID: \"37ee44c2-7e75-4918-bc66-636974fa9b94\") " pod="openshift-ovn-kubernetes/ovnkube-node-qsqkd"
Sep 11 00:57:47.981886 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.980849    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f27a49c0-52c5-42cf-9640-77fb72ef0998-etc-sysctl-conf\") pod \"tuned-m8lrq\" (UID: \"f27a49c0-52c5-42cf-9640-77fb72ef0998\") " pod="openshift-cluster-node-tuning-operator/tuned-m8lrq"
Sep 11 00:57:47.981886 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.980874    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/bbc61763-126c-46a6-8607-9f1071004770-konnectivity-ca\") pod \"konnectivity-agent-st6p9\" (UID: \"bbc61763-126c-46a6-8607-9f1071004770\") " pod="kube-system/konnectivity-agent-st6p9"
Sep 11 00:57:47.981886 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.980898    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f27a49c0-52c5-42cf-9640-77fb72ef0998-host\") pod \"tuned-m8lrq\" (UID: \"f27a49c0-52c5-42cf-9640-77fb72ef0998\") " pod="openshift-cluster-node-tuning-operator/tuned-m8lrq"
Sep 11 00:57:47.981886 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.980921    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/37ee44c2-7e75-4918-bc66-636974fa9b94-etc-openvswitch\") pod \"ovnkube-node-qsqkd\" (UID: \"37ee44c2-7e75-4918-bc66-636974fa9b94\") " pod="openshift-ovn-kubernetes/ovnkube-node-qsqkd"
Sep 11 00:57:47.981886 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.980926    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f27a49c0-52c5-42cf-9640-77fb72ef0998-etc-kubernetes\") pod \"tuned-m8lrq\" (UID: \"f27a49c0-52c5-42cf-9640-77fb72ef0998\") " pod="openshift-cluster-node-tuning-operator/tuned-m8lrq"
Sep 11 00:57:47.981886 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.980966    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/37ee44c2-7e75-4918-bc66-636974fa9b94-log-socket\") pod \"ovnkube-node-qsqkd\" (UID: \"37ee44c2-7e75-4918-bc66-636974fa9b94\") " pod="openshift-ovn-kubernetes/ovnkube-node-qsqkd"
Sep 11 00:57:47.981886 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.980992    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f27a49c0-52c5-42cf-9640-77fb72ef0998-run\") pod \"tuned-m8lrq\" (UID: \"f27a49c0-52c5-42cf-9640-77fb72ef0998\") " pod="openshift-cluster-node-tuning-operator/tuned-m8lrq"
Sep 11 00:57:47.981886 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.980998    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/37ee44c2-7e75-4918-bc66-636974fa9b94-host-cni-netd\") pod \"ovnkube-node-qsqkd\" (UID: \"37ee44c2-7e75-4918-bc66-636974fa9b94\") " pod="openshift-ovn-kubernetes/ovnkube-node-qsqkd"
Sep 11 00:57:47.981886 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.981004    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/37ee44c2-7e75-4918-bc66-636974fa9b94-run-openvswitch\") pod \"ovnkube-node-qsqkd\" (UID: \"37ee44c2-7e75-4918-bc66-636974fa9b94\") " pod="openshift-ovn-kubernetes/ovnkube-node-qsqkd"
Sep 11 00:57:47.981886 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.981032    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9l8xf\" (UniqueName: \"kubernetes.io/projected/f27a49c0-52c5-42cf-9640-77fb72ef0998-kube-api-access-9l8xf\") pod \"tuned-m8lrq\" (UID: \"f27a49c0-52c5-42cf-9640-77fb72ef0998\") " pod="openshift-cluster-node-tuning-operator/tuned-m8lrq"
Sep 11 00:57:47.981886 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.981037    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f27a49c0-52c5-42cf-9640-77fb72ef0998-etc-sysctl-d\") pod \"tuned-m8lrq\" (UID: \"f27a49c0-52c5-42cf-9640-77fb72ef0998\") " pod="openshift-cluster-node-tuning-operator/tuned-m8lrq"
Sep 11 00:57:47.982866 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.981045    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/37ee44c2-7e75-4918-bc66-636974fa9b94-host-run-ovn-kubernetes\") pod \"ovnkube-node-qsqkd\" (UID: \"37ee44c2-7e75-4918-bc66-636974fa9b94\") " pod="openshift-ovn-kubernetes/ovnkube-node-qsqkd"
Sep 11 00:57:47.982866 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.981059    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f2ed1cf4-6159-47f1-aff0-8915d23dc932-iptables-alerter-script\") pod \"iptables-alerter-j862p\" (UID: \"f2ed1cf4-6159-47f1-aff0-8915d23dc932\") " pod="openshift-network-operator/iptables-alerter-j862p"
Sep 11 00:57:47.982866 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.981070    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/37ee44c2-7e75-4918-bc66-636974fa9b94-systemd-units\") pod \"ovnkube-node-qsqkd\" (UID: \"37ee44c2-7e75-4918-bc66-636974fa9b94\") " pod="openshift-ovn-kubernetes/ovnkube-node-qsqkd"
Sep 11 00:57:47.982866 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.981087    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f27a49c0-52c5-42cf-9640-77fb72ef0998-host\") pod \"tuned-m8lrq\" (UID: \"f27a49c0-52c5-42cf-9640-77fb72ef0998\") " pod="openshift-cluster-node-tuning-operator/tuned-m8lrq"
Sep 11 00:57:47.982866 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.981097    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/37ee44c2-7e75-4918-bc66-636974fa9b94-run-systemd\") pod \"ovnkube-node-qsqkd\" (UID: \"37ee44c2-7e75-4918-bc66-636974fa9b94\") " pod="openshift-ovn-kubernetes/ovnkube-node-qsqkd"
Sep 11 00:57:47.982866 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.981113    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/37ee44c2-7e75-4918-bc66-636974fa9b94-log-socket\") pod \"ovnkube-node-qsqkd\" (UID: \"37ee44c2-7e75-4918-bc66-636974fa9b94\") " pod="openshift-ovn-kubernetes/ovnkube-node-qsqkd"
Sep 11 00:57:47.982866 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.981127    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/37ee44c2-7e75-4918-bc66-636974fa9b94-var-lib-openvswitch\") pod \"ovnkube-node-qsqkd\" (UID: \"37ee44c2-7e75-4918-bc66-636974fa9b94\") " pod="openshift-ovn-kubernetes/ovnkube-node-qsqkd"
Sep 11 00:57:47.982866 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.981155    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/bbc61763-126c-46a6-8607-9f1071004770-agent-certs\") pod \"konnectivity-agent-st6p9\" (UID: \"bbc61763-126c-46a6-8607-9f1071004770\") " pod="kube-system/konnectivity-agent-st6p9"
Sep 11 00:57:47.982866 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.981158    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/37ee44c2-7e75-4918-bc66-636974fa9b94-env-overrides\") pod \"ovnkube-node-qsqkd\" (UID: \"37ee44c2-7e75-4918-bc66-636974fa9b94\") " pod="openshift-ovn-kubernetes/ovnkube-node-qsqkd"
Sep 11 00:57:47.982866 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.981182    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/920866ad-d7f7-46e4-8624-7ae8011a2b76-kubelet-dir\") pod \"aws-ebs-csi-driver-node-t42lc\" (UID: \"920866ad-d7f7-46e4-8624-7ae8011a2b76\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t42lc"
Sep 11 00:57:47.982866 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.981208    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f27a49c0-52c5-42cf-9640-77fb72ef0998-lib-modules\") pod \"tuned-m8lrq\" (UID: \"f27a49c0-52c5-42cf-9640-77fb72ef0998\") " pod="openshift-cluster-node-tuning-operator/tuned-m8lrq"
Sep 11 00:57:47.982866 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.981218    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f27a49c0-52c5-42cf-9640-77fb72ef0998-run\") pod \"tuned-m8lrq\" (UID: \"f27a49c0-52c5-42cf-9640-77fb72ef0998\") " pod="openshift-cluster-node-tuning-operator/tuned-m8lrq"
Sep 11 00:57:47.982866 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.981235    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f27a49c0-52c5-42cf-9640-77fb72ef0998-etc-tuned\") pod \"tuned-m8lrq\" (UID: \"f27a49c0-52c5-42cf-9640-77fb72ef0998\") " pod="openshift-cluster-node-tuning-operator/tuned-m8lrq"
Sep 11 00:57:47.982866 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.981257    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f2ed1cf4-6159-47f1-aff0-8915d23dc932-host-slash\") pod \"iptables-alerter-j862p\" (UID: \"f2ed1cf4-6159-47f1-aff0-8915d23dc932\") " pod="openshift-network-operator/iptables-alerter-j862p"
Sep 11 00:57:47.982866 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.981280    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/37ee44c2-7e75-4918-bc66-636974fa9b94-run-ovn\") pod \"ovnkube-node-qsqkd\" (UID: \"37ee44c2-7e75-4918-bc66-636974fa9b94\") " pod="openshift-ovn-kubernetes/ovnkube-node-qsqkd"
Sep 11 00:57:47.982866 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.981302    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpqdb\" (UniqueName: \"kubernetes.io/projected/37ee44c2-7e75-4918-bc66-636974fa9b94-kube-api-access-jpqdb\") pod \"ovnkube-node-qsqkd\" (UID: \"37ee44c2-7e75-4918-bc66-636974fa9b94\") " pod="openshift-ovn-kubernetes/ovnkube-node-qsqkd"
Sep 11 00:57:47.982866 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.981324    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/920866ad-d7f7-46e4-8624-7ae8011a2b76-registration-dir\") pod \"aws-ebs-csi-driver-node-t42lc\" (UID: \"920866ad-d7f7-46e4-8624-7ae8011a2b76\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t42lc"
Sep 11 00:57:47.983752 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.981345    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f27a49c0-52c5-42cf-9640-77fb72ef0998-etc-sysctl-conf\") pod \"tuned-m8lrq\" (UID: \"f27a49c0-52c5-42cf-9640-77fb72ef0998\") " pod="openshift-cluster-node-tuning-operator/tuned-m8lrq"
Sep 11 00:57:47.983752 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.981439    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/920866ad-d7f7-46e4-8624-7ae8011a2b76-registration-dir\") pod \"aws-ebs-csi-driver-node-t42lc\" (UID: \"920866ad-d7f7-46e4-8624-7ae8011a2b76\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t42lc"
Sep 11 00:57:47.983752 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.981440    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/37ee44c2-7e75-4918-bc66-636974fa9b94-etc-openvswitch\") pod \"ovnkube-node-qsqkd\" (UID: \"37ee44c2-7e75-4918-bc66-636974fa9b94\") " pod="openshift-ovn-kubernetes/ovnkube-node-qsqkd"
Sep 11 00:57:47.983752 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.980922    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/920866ad-d7f7-46e4-8624-7ae8011a2b76-etc-selinux\") pod \"aws-ebs-csi-driver-node-t42lc\" (UID: \"920866ad-d7f7-46e4-8624-7ae8011a2b76\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t42lc"
Sep 11 00:57:47.983752 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.981483    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/37ee44c2-7e75-4918-bc66-636974fa9b94-host-run-netns\") pod \"ovnkube-node-qsqkd\" (UID: \"37ee44c2-7e75-4918-bc66-636974fa9b94\") " pod="openshift-ovn-kubernetes/ovnkube-node-qsqkd"
Sep 11 00:57:47.983752 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.981521    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/37ee44c2-7e75-4918-bc66-636974fa9b94-run-systemd\") pod \"ovnkube-node-qsqkd\" (UID: \"37ee44c2-7e75-4918-bc66-636974fa9b94\") " pod="openshift-ovn-kubernetes/ovnkube-node-qsqkd"
Sep 11 00:57:47.983752 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.981523    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f2ed1cf4-6159-47f1-aff0-8915d23dc932-iptables-alerter-script\") pod \"iptables-alerter-j862p\" (UID: \"f2ed1cf4-6159-47f1-aff0-8915d23dc932\") " pod="openshift-network-operator/iptables-alerter-j862p"
Sep 11 00:57:47.983752 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.981521    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/37ee44c2-7e75-4918-bc66-636974fa9b94-var-lib-openvswitch\") pod \"ovnkube-node-qsqkd\" (UID: \"37ee44c2-7e75-4918-bc66-636974fa9b94\") " pod="openshift-ovn-kubernetes/ovnkube-node-qsqkd"
Sep 11 00:57:47.983752 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.981575    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/920866ad-d7f7-46e4-8624-7ae8011a2b76-kubelet-dir\") pod \"aws-ebs-csi-driver-node-t42lc\" (UID: \"920866ad-d7f7-46e4-8624-7ae8011a2b76\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t42lc"
Sep 11 00:57:47.983752 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.981586    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/37ee44c2-7e75-4918-bc66-636974fa9b94-run-ovn\") pod \"ovnkube-node-qsqkd\" (UID: \"37ee44c2-7e75-4918-bc66-636974fa9b94\") " pod="openshift-ovn-kubernetes/ovnkube-node-qsqkd"
Sep 11 00:57:47.983752 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.981633    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f27a49c0-52c5-42cf-9640-77fb72ef0998-lib-modules\") pod \"tuned-m8lrq\" (UID: \"f27a49c0-52c5-42cf-9640-77fb72ef0998\") " pod="openshift-cluster-node-tuning-operator/tuned-m8lrq"
Sep 11 00:57:47.983752 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.981662    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/37ee44c2-7e75-4918-bc66-636974fa9b94-ovnkube-config\") pod \"ovnkube-node-qsqkd\" (UID: \"37ee44c2-7e75-4918-bc66-636974fa9b94\") " pod="openshift-ovn-kubernetes/ovnkube-node-qsqkd"
Sep 11 00:57:47.983752 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.981683    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f2ed1cf4-6159-47f1-aff0-8915d23dc932-host-slash\") pod \"iptables-alerter-j862p\" (UID: \"f2ed1cf4-6159-47f1-aff0-8915d23dc932\") " pod="openshift-network-operator/iptables-alerter-j862p"
Sep 11 00:57:47.983752 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.982205    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/bbc61763-126c-46a6-8607-9f1071004770-konnectivity-ca\") pod \"konnectivity-agent-st6p9\" (UID: \"bbc61763-126c-46a6-8607-9f1071004770\") " pod="kube-system/konnectivity-agent-st6p9"
Sep 11 00:57:47.984485 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.983816    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f27a49c0-52c5-42cf-9640-77fb72ef0998-tmp\") pod \"tuned-m8lrq\" (UID: \"f27a49c0-52c5-42cf-9640-77fb72ef0998\") " pod="openshift-cluster-node-tuning-operator/tuned-m8lrq"
Sep 11 00:57:47.985421 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.985301    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/bbc61763-126c-46a6-8607-9f1071004770-agent-certs\") pod \"konnectivity-agent-st6p9\" (UID: \"bbc61763-126c-46a6-8607-9f1071004770\") " pod="kube-system/konnectivity-agent-st6p9"
Sep 11 00:57:47.985790 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.985769    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/37ee44c2-7e75-4918-bc66-636974fa9b94-ovn-node-metrics-cert\") pod \"ovnkube-node-qsqkd\" (UID: \"37ee44c2-7e75-4918-bc66-636974fa9b94\") " pod="openshift-ovn-kubernetes/ovnkube-node-qsqkd"
Sep 11 00:57:47.986729 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.986703    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f27a49c0-52c5-42cf-9640-77fb72ef0998-etc-tuned\") pod \"tuned-m8lrq\" (UID: \"f27a49c0-52c5-42cf-9640-77fb72ef0998\") " pod="openshift-cluster-node-tuning-operator/tuned-m8lrq"
Sep 11 00:57:47.991574 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.991477    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vr7z\" (UniqueName: \"kubernetes.io/projected/920866ad-d7f7-46e4-8624-7ae8011a2b76-kube-api-access-5vr7z\") pod \"aws-ebs-csi-driver-node-t42lc\" (UID: \"920866ad-d7f7-46e4-8624-7ae8011a2b76\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t42lc"
Sep 11 00:57:47.996103 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.996079    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9l8xf\" (UniqueName: \"kubernetes.io/projected/f27a49c0-52c5-42cf-9640-77fb72ef0998-kube-api-access-9l8xf\") pod \"tuned-m8lrq\" (UID: \"f27a49c0-52c5-42cf-9640-77fb72ef0998\") " pod="openshift-cluster-node-tuning-operator/tuned-m8lrq"
Sep 11 00:57:47.996520 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.996501    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-856bx\" (UniqueName: \"kubernetes.io/projected/f2ed1cf4-6159-47f1-aff0-8915d23dc932-kube-api-access-856bx\") pod \"iptables-alerter-j862p\" (UID: \"f2ed1cf4-6159-47f1-aff0-8915d23dc932\") " pod="openshift-network-operator/iptables-alerter-j862p"
Sep 11 00:57:47.998123 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:47.998102    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpqdb\" (UniqueName: \"kubernetes.io/projected/37ee44c2-7e75-4918-bc66-636974fa9b94-kube-api-access-jpqdb\") pod \"ovnkube-node-qsqkd\" (UID: \"37ee44c2-7e75-4918-bc66-636974fa9b94\") " pod="openshift-ovn-kubernetes/ovnkube-node-qsqkd"
Sep 11 00:57:48.068068 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:48.068037    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-2dvjn"
Sep 11 00:57:48.081051 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:48.081023    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-jrszh"
Sep 11 00:57:48.095011 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:48.094988    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-st6p9"
Sep 11 00:57:48.104027 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:48.103998    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-8vxxt"
Sep 11 00:57:48.112495 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:48.112472    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-8bg6q"
Sep 11 00:57:48.121370 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:48.121350    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t42lc"
Sep 11 00:57:48.129272 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:48.129241    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-m8lrq"
Sep 11 00:57:48.138099 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:48.138068    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-qsqkd"
Sep 11 00:57:48.144997 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:48.144974    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-j862p"
Sep 11 00:57:48.234086 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:48.234054    2569 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 00:57:48.384480 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:48.384404    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85c82c9f-52d0-49cd-afd9-b791628e1a50-metrics-certs\") pod \"network-metrics-daemon-r8n96\" (UID: \"85c82c9f-52d0-49cd-afd9-b791628e1a50\") " pod="openshift-multus/network-metrics-daemon-r8n96"
Sep 11 00:57:48.384480 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:48.384477    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d5d763bb-6142-409f-8155-7143a6d91dc6-original-pull-secret\") pod \"global-pull-secret-syncer-qgmsz\" (UID: \"d5d763bb-6142-409f-8155-7143a6d91dc6\") " pod="kube-system/global-pull-secret-syncer-qgmsz"
Sep 11 00:57:48.384672 ip-10-0-129-222 kubenswrapper[2569]: E0911 00:57:48.384567    2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 00:57:48.384672 ip-10-0-129-222 kubenswrapper[2569]: E0911 00:57:48.384572    2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 00:57:48.384672 ip-10-0-129-222 kubenswrapper[2569]: E0911 00:57:48.384626    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d5d763bb-6142-409f-8155-7143a6d91dc6-original-pull-secret podName:d5d763bb-6142-409f-8155-7143a6d91dc6 nodeName:}" failed. No retries permitted until 2026-09-11 00:57:49.384611047 +0000 UTC m=+4.035985295 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d5d763bb-6142-409f-8155-7143a6d91dc6-original-pull-secret") pod "global-pull-secret-syncer-qgmsz" (UID: "d5d763bb-6142-409f-8155-7143a6d91dc6") : object "kube-system"/"original-pull-secret" not registered
Sep 11 00:57:48.384672 ip-10-0-129-222 kubenswrapper[2569]: E0911 00:57:48.384642    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85c82c9f-52d0-49cd-afd9-b791628e1a50-metrics-certs podName:85c82c9f-52d0-49cd-afd9-b791628e1a50 nodeName:}" failed. No retries permitted until 2026-09-11 00:57:49.384634147 +0000 UTC m=+4.036008374 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/85c82c9f-52d0-49cd-afd9-b791628e1a50-metrics-certs") pod "network-metrics-daemon-r8n96" (UID: "85c82c9f-52d0-49cd-afd9-b791628e1a50") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 00:57:48.485472 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:48.485443    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwg5s\" (UniqueName: \"kubernetes.io/projected/239b25bd-4932-4b5b-b286-45d3e2ff9f1c-kube-api-access-qwg5s\") pod \"network-check-target-bbbq6\" (UID: \"239b25bd-4932-4b5b-b286-45d3e2ff9f1c\") " pod="openshift-network-diagnostics/network-check-target-bbbq6"
Sep 11 00:57:48.485618 ip-10-0-129-222 kubenswrapper[2569]: E0911 00:57:48.485553    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 00:57:48.485618 ip-10-0-129-222 kubenswrapper[2569]: E0911 00:57:48.485575    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 00:57:48.485618 ip-10-0-129-222 kubenswrapper[2569]: E0911 00:57:48.485588    2569 projected.go:196] Error preparing data for projected volume kube-api-access-qwg5s for pod openshift-network-diagnostics/network-check-target-bbbq6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 00:57:48.485761 ip-10-0-129-222 kubenswrapper[2569]: E0911 00:57:48.485648    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/239b25bd-4932-4b5b-b286-45d3e2ff9f1c-kube-api-access-qwg5s podName:239b25bd-4932-4b5b-b286-45d3e2ff9f1c nodeName:}" failed. No retries permitted until 2026-09-11 00:57:49.485629562 +0000 UTC m=+4.137003789 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-qwg5s" (UniqueName: "kubernetes.io/projected/239b25bd-4932-4b5b-b286-45d3e2ff9f1c-kube-api-access-qwg5s") pod "network-check-target-bbbq6" (UID: "239b25bd-4932-4b5b-b286-45d3e2ff9f1c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 00:57:48.820974 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:48.820891    2569 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-10 00:52:47 +0000 UTC" deadline="2028-04-18 11:26:56.609038755 +0000 UTC"
Sep 11 00:57:48.820974 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:48.820928    2569 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14050h29m7.788115938s"
Sep 11 00:57:49.099539 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:49.099504    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7486f1f_9947_4405_9186_66dfa5621253.slice/crio-ea5bb0b734af8a06b7d445121cb9e9a3297a0809c9b9c20dc1b1edb0363b0fbb WatchSource:0}: Error finding container ea5bb0b734af8a06b7d445121cb9e9a3297a0809c9b9c20dc1b1edb0363b0fbb: Status 404 returned error can't find the container with id ea5bb0b734af8a06b7d445121cb9e9a3297a0809c9b9c20dc1b1edb0363b0fbb
Sep 11 00:57:49.100140 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:49.100110    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2ed1cf4_6159_47f1_aff0_8915d23dc932.slice/crio-f5e2f9cc565acedbdb6b7801c140e92b94d7e183ab6c1631b6f2369c2c05d148 WatchSource:0}: Error finding container f5e2f9cc565acedbdb6b7801c140e92b94d7e183ab6c1631b6f2369c2c05d148: Status 404 returned error can't find the container with id f5e2f9cc565acedbdb6b7801c140e92b94d7e183ab6c1631b6f2369c2c05d148
Sep 11 00:57:49.104261 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:49.104230    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37ee44c2_7e75_4918_bc66_636974fa9b94.slice/crio-afe3b288255867000ea5e0dd1177936054ac74dd3d733bbe9f74a763478de96e WatchSource:0}: Error finding container afe3b288255867000ea5e0dd1177936054ac74dd3d733bbe9f74a763478de96e: Status 404 returned error can't find the container with id afe3b288255867000ea5e0dd1177936054ac74dd3d733bbe9f74a763478de96e
Sep 11 00:57:49.109133 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:49.108942    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod304c1b0f_f5f9_4c4c_8041_ac384ad21516.slice/crio-48868a7fb277631d588bca1d13dc93bb90955983162b78d9aa9e48fb8213e27c WatchSource:0}: Error finding container 48868a7fb277631d588bca1d13dc93bb90955983162b78d9aa9e48fb8213e27c: Status 404 returned error can't find the container with id 48868a7fb277631d588bca1d13dc93bb90955983162b78d9aa9e48fb8213e27c
Sep 11 00:57:49.111416 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:49.111398    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbda35fc9_c767_4b21_9509_994a67ffdeaf.slice/crio-003612b8e2379e0a0bdd1399394d866139dab82eff74d8f74a32c4e1823b395d WatchSource:0}: Error finding container 003612b8e2379e0a0bdd1399394d866139dab82eff74d8f74a32c4e1823b395d: Status 404 returned error can't find the container with id 003612b8e2379e0a0bdd1399394d866139dab82eff74d8f74a32c4e1823b395d
Sep 11 00:57:49.112548 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:49.112518    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf27a49c0_52c5_42cf_9640_77fb72ef0998.slice/crio-c0a8a22e81a09ea8dcbc7c35db78b0284a39ea452fa48165aed9e25b3a457f32 WatchSource:0}: Error finding container c0a8a22e81a09ea8dcbc7c35db78b0284a39ea452fa48165aed9e25b3a457f32: Status 404 returned error can't find the container with id c0a8a22e81a09ea8dcbc7c35db78b0284a39ea452fa48165aed9e25b3a457f32
Sep 11 00:57:49.113531 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:49.113497    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod799942b1_a6d8_4f5a_8d17_e7b2761b68b7.slice/crio-61938ea8fc919e052daa0c9e8e38af0e32e0d006a32aaf57b738aae383b57f3e WatchSource:0}: Error finding container 61938ea8fc919e052daa0c9e8e38af0e32e0d006a32aaf57b738aae383b57f3e: Status 404 returned error can't find the container with id 61938ea8fc919e052daa0c9e8e38af0e32e0d006a32aaf57b738aae383b57f3e
Sep 11 00:57:49.114457 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:49.114434    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod920866ad_d7f7_46e4_8624_7ae8011a2b76.slice/crio-b62fe0d16256fb75a9b5c4fa2a681fa61266272b2e268d5f4ba989e8d03cc040 WatchSource:0}: Error finding container b62fe0d16256fb75a9b5c4fa2a681fa61266272b2e268d5f4ba989e8d03cc040: Status 404 returned error can't find the container with id b62fe0d16256fb75a9b5c4fa2a681fa61266272b2e268d5f4ba989e8d03cc040
Sep 11 00:57:49.116107 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:57:49.115685    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbc61763_126c_46a6_8607_9f1071004770.slice/crio-465fbd35e7048abc4a55516aeebe45ed1ef3c47d09b574cde5a42c187fcce6f3 WatchSource:0}: Error finding container 465fbd35e7048abc4a55516aeebe45ed1ef3c47d09b574cde5a42c187fcce6f3: Status 404 returned error can't find the container with id 465fbd35e7048abc4a55516aeebe45ed1ef3c47d09b574cde5a42c187fcce6f3
Sep 11 00:57:49.392241 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:49.392209    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85c82c9f-52d0-49cd-afd9-b791628e1a50-metrics-certs\") pod \"network-metrics-daemon-r8n96\" (UID: \"85c82c9f-52d0-49cd-afd9-b791628e1a50\") " pod="openshift-multus/network-metrics-daemon-r8n96"
Sep 11 00:57:49.392337 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:49.392265    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d5d763bb-6142-409f-8155-7143a6d91dc6-original-pull-secret\") pod \"global-pull-secret-syncer-qgmsz\" (UID: \"d5d763bb-6142-409f-8155-7143a6d91dc6\") " pod="kube-system/global-pull-secret-syncer-qgmsz"
Sep 11 00:57:49.392374 ip-10-0-129-222 kubenswrapper[2569]: E0911 00:57:49.392360    2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 00:57:49.392430 ip-10-0-129-222 kubenswrapper[2569]: E0911 00:57:49.392418    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d5d763bb-6142-409f-8155-7143a6d91dc6-original-pull-secret podName:d5d763bb-6142-409f-8155-7143a6d91dc6 nodeName:}" failed. No retries permitted until 2026-09-11 00:57:51.392399058 +0000 UTC m=+6.043773301 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d5d763bb-6142-409f-8155-7143a6d91dc6-original-pull-secret") pod "global-pull-secret-syncer-qgmsz" (UID: "d5d763bb-6142-409f-8155-7143a6d91dc6") : object "kube-system"/"original-pull-secret" not registered
Sep 11 00:57:49.392483 ip-10-0-129-222 kubenswrapper[2569]: E0911 00:57:49.392361    2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 00:57:49.392528 ip-10-0-129-222 kubenswrapper[2569]: E0911 00:57:49.392497    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85c82c9f-52d0-49cd-afd9-b791628e1a50-metrics-certs podName:85c82c9f-52d0-49cd-afd9-b791628e1a50 nodeName:}" failed. No retries permitted until 2026-09-11 00:57:51.392479985 +0000 UTC m=+6.043854215 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/85c82c9f-52d0-49cd-afd9-b791628e1a50-metrics-certs") pod "network-metrics-daemon-r8n96" (UID: "85c82c9f-52d0-49cd-afd9-b791628e1a50") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 00:57:49.492706 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:49.492678    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwg5s\" (UniqueName: \"kubernetes.io/projected/239b25bd-4932-4b5b-b286-45d3e2ff9f1c-kube-api-access-qwg5s\") pod \"network-check-target-bbbq6\" (UID: \"239b25bd-4932-4b5b-b286-45d3e2ff9f1c\") " pod="openshift-network-diagnostics/network-check-target-bbbq6"
Sep 11 00:57:49.492850 ip-10-0-129-222 kubenswrapper[2569]: E0911 00:57:49.492831    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 00:57:49.492922 ip-10-0-129-222 kubenswrapper[2569]: E0911 00:57:49.492854    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 00:57:49.492922 ip-10-0-129-222 kubenswrapper[2569]: E0911 00:57:49.492864    2569 projected.go:196] Error preparing data for projected volume kube-api-access-qwg5s for pod openshift-network-diagnostics/network-check-target-bbbq6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 00:57:49.492922 ip-10-0-129-222 kubenswrapper[2569]: E0911 00:57:49.492921    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/239b25bd-4932-4b5b-b286-45d3e2ff9f1c-kube-api-access-qwg5s podName:239b25bd-4932-4b5b-b286-45d3e2ff9f1c nodeName:}" failed. No retries permitted until 2026-09-11 00:57:51.492908137 +0000 UTC m=+6.144282364 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-qwg5s" (UniqueName: "kubernetes.io/projected/239b25bd-4932-4b5b-b286-45d3e2ff9f1c-kube-api-access-qwg5s") pod "network-check-target-bbbq6" (UID: "239b25bd-4932-4b5b-b286-45d3e2ff9f1c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 00:57:49.821902 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:49.821803    2569 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-10 00:52:47 +0000 UTC" deadline="2028-06-21 22:51:11.169900532 +0000 UTC"
Sep 11 00:57:49.821902 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:49.821843    2569 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15597h53m21.348062361s"
Sep 11 00:57:49.871275 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:49.871127    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qgmsz"
Sep 11 00:57:49.871275 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:49.871176    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bbbq6"
Sep 11 00:57:49.871529 ip-10-0-129-222 kubenswrapper[2569]: E0911 00:57:49.871289    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qgmsz" podUID="d5d763bb-6142-409f-8155-7143a6d91dc6"
Sep 11 00:57:49.871529 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:49.871394    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r8n96"
Sep 11 00:57:49.871632 ip-10-0-129-222 kubenswrapper[2569]: E0911 00:57:49.871560    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r8n96" podUID="85c82c9f-52d0-49cd-afd9-b791628e1a50"
Sep 11 00:57:49.873144 ip-10-0-129-222 kubenswrapper[2569]: E0911 00:57:49.871674    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bbbq6" podUID="239b25bd-4932-4b5b-b286-45d3e2ff9f1c"
Sep 11 00:57:49.887900 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:49.887783    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-2dvjn" event={"ID":"c7486f1f-9947-4405-9186-66dfa5621253","Type":"ContainerStarted","Data":"ea5bb0b734af8a06b7d445121cb9e9a3297a0809c9b9c20dc1b1edb0363b0fbb"}
Sep 11 00:57:49.893593 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:49.893526    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-m8lrq" event={"ID":"f27a49c0-52c5-42cf-9640-77fb72ef0998","Type":"ContainerStarted","Data":"c0a8a22e81a09ea8dcbc7c35db78b0284a39ea452fa48165aed9e25b3a457f32"}
Sep 11 00:57:49.903117 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:49.903087    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-st6p9" event={"ID":"bbc61763-126c-46a6-8607-9f1071004770","Type":"ContainerStarted","Data":"465fbd35e7048abc4a55516aeebe45ed1ef3c47d09b574cde5a42c187fcce6f3"}
Sep 11 00:57:49.905770 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:49.905745    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t42lc" event={"ID":"920866ad-d7f7-46e4-8624-7ae8011a2b76","Type":"ContainerStarted","Data":"b62fe0d16256fb75a9b5c4fa2a681fa61266272b2e268d5f4ba989e8d03cc040"}
Sep 11 00:57:49.912476 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:49.912408    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8bg6q" event={"ID":"799942b1-a6d8-4f5a-8d17-e7b2761b68b7","Type":"ContainerStarted","Data":"61938ea8fc919e052daa0c9e8e38af0e32e0d006a32aaf57b738aae383b57f3e"}
Sep 11 00:57:49.920751 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:49.920724    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-8vxxt" event={"ID":"bda35fc9-c767-4b21-9509-994a67ffdeaf","Type":"ContainerStarted","Data":"003612b8e2379e0a0bdd1399394d866139dab82eff74d8f74a32c4e1823b395d"}
Sep 11 00:57:49.925291 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:49.925257    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qsqkd" event={"ID":"37ee44c2-7e75-4918-bc66-636974fa9b94","Type":"ContainerStarted","Data":"afe3b288255867000ea5e0dd1177936054ac74dd3d733bbe9f74a763478de96e"}
Sep 11 00:57:49.934275 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:49.934248    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-222.ec2.internal" event={"ID":"bef6176fe69d54950289897ea3150444","Type":"ContainerStarted","Data":"fa4b1f074003df195252e7939751073a06710e4a861270c4a56413cf2d3bc19b"}
Sep 11 00:57:49.962512 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:49.954204    2569 generic.go:356] "Generic (PLEG): container finished" podID="c00e6e1a91842597c59d08786ac60421" containerID="0adafd8ff0fd5f1446e4dc741a4de690a31417afcd797aec74a35427e5e2b00a" exitCode=0
Sep 11 00:57:49.963229 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:49.963210    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-222.ec2.internal" event={"ID":"c00e6e1a91842597c59d08786ac60421","Type":"ContainerDied","Data":"0adafd8ff0fd5f1446e4dc741a4de690a31417afcd797aec74a35427e5e2b00a"}
Sep 11 00:57:49.963634 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:49.963562    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-222.ec2.internal" podStartSLOduration=3.963545375 podStartE2EDuration="3.963545375s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 00:57:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 00:57:49.962799757 +0000 UTC m=+4.614174008" watchObservedRunningTime="2026-09-11 00:57:49.963545375 +0000 UTC m=+4.614919629"
Sep 11 00:57:49.972892 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:49.972817    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-jrszh" event={"ID":"304c1b0f-f5f9-4c4c-8041-ac384ad21516","Type":"ContainerStarted","Data":"48868a7fb277631d588bca1d13dc93bb90955983162b78d9aa9e48fb8213e27c"}
Sep 11 00:57:49.981149 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:49.980972    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-j862p" event={"ID":"f2ed1cf4-6159-47f1-aff0-8915d23dc932","Type":"ContainerStarted","Data":"f5e2f9cc565acedbdb6b7801c140e92b94d7e183ab6c1631b6f2369c2c05d148"}
Sep 11 00:57:50.996499 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:50.996152    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-222.ec2.internal" event={"ID":"c00e6e1a91842597c59d08786ac60421","Type":"ContainerStarted","Data":"be3b2859f4f355733b6509d7b4a5f6b2202555b2b0912bb90a3b042114b5650c"}
Sep 11 00:57:51.015510 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:51.015460    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-222.ec2.internal" podStartSLOduration=5.015440884 podStartE2EDuration="5.015440884s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 00:57:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 00:57:51.014463072 +0000 UTC m=+5.665837322" watchObservedRunningTime="2026-09-11 00:57:51.015440884 +0000 UTC m=+5.666815135"
Sep 11 00:57:51.406981 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:51.406901    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d5d763bb-6142-409f-8155-7143a6d91dc6-original-pull-secret\") pod \"global-pull-secret-syncer-qgmsz\" (UID: \"d5d763bb-6142-409f-8155-7143a6d91dc6\") " pod="kube-system/global-pull-secret-syncer-qgmsz"
Sep 11 00:57:51.407131 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:51.406964    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85c82c9f-52d0-49cd-afd9-b791628e1a50-metrics-certs\") pod \"network-metrics-daemon-r8n96\" (UID: \"85c82c9f-52d0-49cd-afd9-b791628e1a50\") " pod="openshift-multus/network-metrics-daemon-r8n96"
Sep 11 00:57:51.407131 ip-10-0-129-222 kubenswrapper[2569]: E0911 00:57:51.407106    2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 00:57:51.407238 ip-10-0-129-222 kubenswrapper[2569]: E0911 00:57:51.407169    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85c82c9f-52d0-49cd-afd9-b791628e1a50-metrics-certs podName:85c82c9f-52d0-49cd-afd9-b791628e1a50 nodeName:}" failed. No retries permitted until 2026-09-11 00:57:55.407151106 +0000 UTC m=+10.058525335 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/85c82c9f-52d0-49cd-afd9-b791628e1a50-metrics-certs") pod "network-metrics-daemon-r8n96" (UID: "85c82c9f-52d0-49cd-afd9-b791628e1a50") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 00:57:51.407715 ip-10-0-129-222 kubenswrapper[2569]: E0911 00:57:51.407671    2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 00:57:51.407818 ip-10-0-129-222 kubenswrapper[2569]: E0911 00:57:51.407724    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d5d763bb-6142-409f-8155-7143a6d91dc6-original-pull-secret podName:d5d763bb-6142-409f-8155-7143a6d91dc6 nodeName:}" failed. No retries permitted until 2026-09-11 00:57:55.407710464 +0000 UTC m=+10.059084695 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d5d763bb-6142-409f-8155-7143a6d91dc6-original-pull-secret") pod "global-pull-secret-syncer-qgmsz" (UID: "d5d763bb-6142-409f-8155-7143a6d91dc6") : object "kube-system"/"original-pull-secret" not registered
Sep 11 00:57:51.508355 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:51.508190    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwg5s\" (UniqueName: \"kubernetes.io/projected/239b25bd-4932-4b5b-b286-45d3e2ff9f1c-kube-api-access-qwg5s\") pod \"network-check-target-bbbq6\" (UID: \"239b25bd-4932-4b5b-b286-45d3e2ff9f1c\") " pod="openshift-network-diagnostics/network-check-target-bbbq6"
Sep 11 00:57:51.508550 ip-10-0-129-222 kubenswrapper[2569]: E0911 00:57:51.508447    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 00:57:51.508550 ip-10-0-129-222 kubenswrapper[2569]: E0911 00:57:51.508470    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 00:57:51.508550 ip-10-0-129-222 kubenswrapper[2569]: E0911 00:57:51.508482    2569 projected.go:196] Error preparing data for projected volume kube-api-access-qwg5s for pod openshift-network-diagnostics/network-check-target-bbbq6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 00:57:51.508550 ip-10-0-129-222 kubenswrapper[2569]: E0911 00:57:51.508543    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/239b25bd-4932-4b5b-b286-45d3e2ff9f1c-kube-api-access-qwg5s podName:239b25bd-4932-4b5b-b286-45d3e2ff9f1c nodeName:}" failed. No retries permitted until 2026-09-11 00:57:55.508525826 +0000 UTC m=+10.159900062 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-qwg5s" (UniqueName: "kubernetes.io/projected/239b25bd-4932-4b5b-b286-45d3e2ff9f1c-kube-api-access-qwg5s") pod "network-check-target-bbbq6" (UID: "239b25bd-4932-4b5b-b286-45d3e2ff9f1c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 00:57:51.867290 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:51.864276    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qgmsz"
Sep 11 00:57:51.867290 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:51.864340    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r8n96"
Sep 11 00:57:51.867290 ip-10-0-129-222 kubenswrapper[2569]: E0911 00:57:51.864452    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r8n96" podUID="85c82c9f-52d0-49cd-afd9-b791628e1a50"
Sep 11 00:57:51.867290 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:51.864820    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bbbq6"
Sep 11 00:57:51.867290 ip-10-0-129-222 kubenswrapper[2569]: E0911 00:57:51.864926    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bbbq6" podUID="239b25bd-4932-4b5b-b286-45d3e2ff9f1c"
Sep 11 00:57:51.867290 ip-10-0-129-222 kubenswrapper[2569]: E0911 00:57:51.867187    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qgmsz" podUID="d5d763bb-6142-409f-8155-7143a6d91dc6"
Sep 11 00:57:53.867652 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:53.866798    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qgmsz"
Sep 11 00:57:53.867652 ip-10-0-129-222 kubenswrapper[2569]: E0911 00:57:53.866907    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qgmsz" podUID="d5d763bb-6142-409f-8155-7143a6d91dc6"
Sep 11 00:57:53.867652 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:53.867280    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r8n96"
Sep 11 00:57:53.867652 ip-10-0-129-222 kubenswrapper[2569]: E0911 00:57:53.867383    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r8n96" podUID="85c82c9f-52d0-49cd-afd9-b791628e1a50"
Sep 11 00:57:53.867652 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:53.867434    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bbbq6"
Sep 11 00:57:53.867652 ip-10-0-129-222 kubenswrapper[2569]: E0911 00:57:53.867568    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bbbq6" podUID="239b25bd-4932-4b5b-b286-45d3e2ff9f1c"
Sep 11 00:57:55.438103 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:55.438061    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d5d763bb-6142-409f-8155-7143a6d91dc6-original-pull-secret\") pod \"global-pull-secret-syncer-qgmsz\" (UID: \"d5d763bb-6142-409f-8155-7143a6d91dc6\") " pod="kube-system/global-pull-secret-syncer-qgmsz"
Sep 11 00:57:55.438558 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:55.438115    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85c82c9f-52d0-49cd-afd9-b791628e1a50-metrics-certs\") pod \"network-metrics-daemon-r8n96\" (UID: \"85c82c9f-52d0-49cd-afd9-b791628e1a50\") " pod="openshift-multus/network-metrics-daemon-r8n96"
Sep 11 00:57:55.438558 ip-10-0-129-222 kubenswrapper[2569]: E0911 00:57:55.438265    2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 00:57:55.438558 ip-10-0-129-222 kubenswrapper[2569]: E0911 00:57:55.438337    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85c82c9f-52d0-49cd-afd9-b791628e1a50-metrics-certs podName:85c82c9f-52d0-49cd-afd9-b791628e1a50 nodeName:}" failed. No retries permitted until 2026-09-11 00:58:03.438320115 +0000 UTC m=+18.089694344 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/85c82c9f-52d0-49cd-afd9-b791628e1a50-metrics-certs") pod "network-metrics-daemon-r8n96" (UID: "85c82c9f-52d0-49cd-afd9-b791628e1a50") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 00:57:55.438558 ip-10-0-129-222 kubenswrapper[2569]: E0911 00:57:55.438267    2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 00:57:55.438558 ip-10-0-129-222 kubenswrapper[2569]: E0911 00:57:55.438406    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d5d763bb-6142-409f-8155-7143a6d91dc6-original-pull-secret podName:d5d763bb-6142-409f-8155-7143a6d91dc6 nodeName:}" failed. No retries permitted until 2026-09-11 00:58:03.438391288 +0000 UTC m=+18.089765519 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d5d763bb-6142-409f-8155-7143a6d91dc6-original-pull-secret") pod "global-pull-secret-syncer-qgmsz" (UID: "d5d763bb-6142-409f-8155-7143a6d91dc6") : object "kube-system"/"original-pull-secret" not registered
Sep 11 00:57:55.539476 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:55.539444    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwg5s\" (UniqueName: \"kubernetes.io/projected/239b25bd-4932-4b5b-b286-45d3e2ff9f1c-kube-api-access-qwg5s\") pod \"network-check-target-bbbq6\" (UID: \"239b25bd-4932-4b5b-b286-45d3e2ff9f1c\") " pod="openshift-network-diagnostics/network-check-target-bbbq6"
Sep 11 00:57:55.539647 ip-10-0-129-222 kubenswrapper[2569]: E0911 00:57:55.539630    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 00:57:55.539714 ip-10-0-129-222 kubenswrapper[2569]: E0911 00:57:55.539653    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 00:57:55.539714 ip-10-0-129-222 kubenswrapper[2569]: E0911 00:57:55.539666    2569 projected.go:196] Error preparing data for projected volume kube-api-access-qwg5s for pod openshift-network-diagnostics/network-check-target-bbbq6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 00:57:55.539815 ip-10-0-129-222 kubenswrapper[2569]: E0911 00:57:55.539732    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/239b25bd-4932-4b5b-b286-45d3e2ff9f1c-kube-api-access-qwg5s podName:239b25bd-4932-4b5b-b286-45d3e2ff9f1c nodeName:}" failed. No retries permitted until 2026-09-11 00:58:03.539711443 +0000 UTC m=+18.191085685 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-qwg5s" (UniqueName: "kubernetes.io/projected/239b25bd-4932-4b5b-b286-45d3e2ff9f1c-kube-api-access-qwg5s") pod "network-check-target-bbbq6" (UID: "239b25bd-4932-4b5b-b286-45d3e2ff9f1c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 00:57:55.865982 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:55.865900    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qgmsz"
Sep 11 00:57:55.866137 ip-10-0-129-222 kubenswrapper[2569]: E0911 00:57:55.866013    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qgmsz" podUID="d5d763bb-6142-409f-8155-7143a6d91dc6"
Sep 11 00:57:55.866529 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:55.866511    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r8n96"
Sep 11 00:57:55.866637 ip-10-0-129-222 kubenswrapper[2569]: E0911 00:57:55.866616    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r8n96" podUID="85c82c9f-52d0-49cd-afd9-b791628e1a50"
Sep 11 00:57:55.866700 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:55.866667    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bbbq6"
Sep 11 00:57:55.866754 ip-10-0-129-222 kubenswrapper[2569]: E0911 00:57:55.866740    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bbbq6" podUID="239b25bd-4932-4b5b-b286-45d3e2ff9f1c"
Sep 11 00:57:57.865115 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:57.865030    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r8n96"
Sep 11 00:57:57.865569 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:57.865030    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qgmsz"
Sep 11 00:57:57.865569 ip-10-0-129-222 kubenswrapper[2569]: E0911 00:57:57.865167    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r8n96" podUID="85c82c9f-52d0-49cd-afd9-b791628e1a50"
Sep 11 00:57:57.865569 ip-10-0-129-222 kubenswrapper[2569]: E0911 00:57:57.865237    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qgmsz" podUID="d5d763bb-6142-409f-8155-7143a6d91dc6"
Sep 11 00:57:57.865569 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:57.865030    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bbbq6"
Sep 11 00:57:57.865569 ip-10-0-129-222 kubenswrapper[2569]: E0911 00:57:57.865321    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bbbq6" podUID="239b25bd-4932-4b5b-b286-45d3e2ff9f1c"
Sep 11 00:57:59.864297 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:59.864262    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r8n96"
Sep 11 00:57:59.864748 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:59.864316    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bbbq6"
Sep 11 00:57:59.864748 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:57:59.864269    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qgmsz"
Sep 11 00:57:59.864748 ip-10-0-129-222 kubenswrapper[2569]: E0911 00:57:59.864412    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r8n96" podUID="85c82c9f-52d0-49cd-afd9-b791628e1a50"
Sep 11 00:57:59.864748 ip-10-0-129-222 kubenswrapper[2569]: E0911 00:57:59.864507    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qgmsz" podUID="d5d763bb-6142-409f-8155-7143a6d91dc6"
Sep 11 00:57:59.864748 ip-10-0-129-222 kubenswrapper[2569]: E0911 00:57:59.864583    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bbbq6" podUID="239b25bd-4932-4b5b-b286-45d3e2ff9f1c"
Sep 11 00:58:01.864940 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:01.864906    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bbbq6"
Sep 11 00:58:01.864940 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:01.864935    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qgmsz"
Sep 11 00:58:01.865420 ip-10-0-129-222 kubenswrapper[2569]: E0911 00:58:01.865034    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bbbq6" podUID="239b25bd-4932-4b5b-b286-45d3e2ff9f1c"
Sep 11 00:58:01.865420 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:01.865113    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r8n96"
Sep 11 00:58:01.865420 ip-10-0-129-222 kubenswrapper[2569]: E0911 00:58:01.865113    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qgmsz" podUID="d5d763bb-6142-409f-8155-7143a6d91dc6"
Sep 11 00:58:01.865420 ip-10-0-129-222 kubenswrapper[2569]: E0911 00:58:01.865216    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r8n96" podUID="85c82c9f-52d0-49cd-afd9-b791628e1a50"
Sep 11 00:58:03.499984 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:03.499947    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85c82c9f-52d0-49cd-afd9-b791628e1a50-metrics-certs\") pod \"network-metrics-daemon-r8n96\" (UID: \"85c82c9f-52d0-49cd-afd9-b791628e1a50\") " pod="openshift-multus/network-metrics-daemon-r8n96"
Sep 11 00:58:03.500444 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:03.500028    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d5d763bb-6142-409f-8155-7143a6d91dc6-original-pull-secret\") pod \"global-pull-secret-syncer-qgmsz\" (UID: \"d5d763bb-6142-409f-8155-7143a6d91dc6\") " pod="kube-system/global-pull-secret-syncer-qgmsz"
Sep 11 00:58:03.500444 ip-10-0-129-222 kubenswrapper[2569]: E0911 00:58:03.500165    2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 00:58:03.500444 ip-10-0-129-222 kubenswrapper[2569]: E0911 00:58:03.500227    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85c82c9f-52d0-49cd-afd9-b791628e1a50-metrics-certs podName:85c82c9f-52d0-49cd-afd9-b791628e1a50 nodeName:}" failed. No retries permitted until 2026-09-11 00:58:19.500211244 +0000 UTC m=+34.151585475 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/85c82c9f-52d0-49cd-afd9-b791628e1a50-metrics-certs") pod "network-metrics-daemon-r8n96" (UID: "85c82c9f-52d0-49cd-afd9-b791628e1a50") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 00:58:03.500444 ip-10-0-129-222 kubenswrapper[2569]: E0911 00:58:03.500168    2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 00:58:03.500444 ip-10-0-129-222 kubenswrapper[2569]: E0911 00:58:03.500332    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d5d763bb-6142-409f-8155-7143a6d91dc6-original-pull-secret podName:d5d763bb-6142-409f-8155-7143a6d91dc6 nodeName:}" failed. No retries permitted until 2026-09-11 00:58:19.50031239 +0000 UTC m=+34.151686631 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d5d763bb-6142-409f-8155-7143a6d91dc6-original-pull-secret") pod "global-pull-secret-syncer-qgmsz" (UID: "d5d763bb-6142-409f-8155-7143a6d91dc6") : object "kube-system"/"original-pull-secret" not registered
Sep 11 00:58:03.600560 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:03.600529    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwg5s\" (UniqueName: \"kubernetes.io/projected/239b25bd-4932-4b5b-b286-45d3e2ff9f1c-kube-api-access-qwg5s\") pod \"network-check-target-bbbq6\" (UID: \"239b25bd-4932-4b5b-b286-45d3e2ff9f1c\") " pod="openshift-network-diagnostics/network-check-target-bbbq6"
Sep 11 00:58:03.600745 ip-10-0-129-222 kubenswrapper[2569]: E0911 00:58:03.600648    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 00:58:03.600745 ip-10-0-129-222 kubenswrapper[2569]: E0911 00:58:03.600661    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 00:58:03.600745 ip-10-0-129-222 kubenswrapper[2569]: E0911 00:58:03.600670    2569 projected.go:196] Error preparing data for projected volume kube-api-access-qwg5s for pod openshift-network-diagnostics/network-check-target-bbbq6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 00:58:03.600745 ip-10-0-129-222 kubenswrapper[2569]: E0911 00:58:03.600722    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/239b25bd-4932-4b5b-b286-45d3e2ff9f1c-kube-api-access-qwg5s podName:239b25bd-4932-4b5b-b286-45d3e2ff9f1c nodeName:}" failed. No retries permitted until 2026-09-11 00:58:19.600710073 +0000 UTC m=+34.252084299 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-qwg5s" (UniqueName: "kubernetes.io/projected/239b25bd-4932-4b5b-b286-45d3e2ff9f1c-kube-api-access-qwg5s") pod "network-check-target-bbbq6" (UID: "239b25bd-4932-4b5b-b286-45d3e2ff9f1c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 00:58:03.864920 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:03.864827    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qgmsz"
Sep 11 00:58:03.864920 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:03.864852    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r8n96"
Sep 11 00:58:03.864920 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:03.864852    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bbbq6"
Sep 11 00:58:03.865156 ip-10-0-129-222 kubenswrapper[2569]: E0911 00:58:03.864958    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qgmsz" podUID="d5d763bb-6142-409f-8155-7143a6d91dc6"
Sep 11 00:58:03.865156 ip-10-0-129-222 kubenswrapper[2569]: E0911 00:58:03.865098    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r8n96" podUID="85c82c9f-52d0-49cd-afd9-b791628e1a50"
Sep 11 00:58:03.865253 ip-10-0-129-222 kubenswrapper[2569]: E0911 00:58:03.865188    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bbbq6" podUID="239b25bd-4932-4b5b-b286-45d3e2ff9f1c"
Sep 11 00:58:05.865377 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:05.865346    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qgmsz"
Sep 11 00:58:05.865805 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:05.865444    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bbbq6"
Sep 11 00:58:05.865805 ip-10-0-129-222 kubenswrapper[2569]: E0911 00:58:05.865476    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qgmsz" podUID="d5d763bb-6142-409f-8155-7143a6d91dc6"
Sep 11 00:58:05.865805 ip-10-0-129-222 kubenswrapper[2569]: E0911 00:58:05.865520    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bbbq6" podUID="239b25bd-4932-4b5b-b286-45d3e2ff9f1c"
Sep 11 00:58:05.865805 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:05.865560    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r8n96"
Sep 11 00:58:05.865805 ip-10-0-129-222 kubenswrapper[2569]: E0911 00:58:05.865642    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r8n96" podUID="85c82c9f-52d0-49cd-afd9-b791628e1a50"
Sep 11 00:58:07.028857 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:07.028427    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-jrszh" event={"ID":"304c1b0f-f5f9-4c4c-8041-ac384ad21516","Type":"ContainerStarted","Data":"0c37fab99164992a5b0c991ae757616a4a7f98878768a5c40f05885db5fcbc32"}
Sep 11 00:58:07.029935 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:07.029902    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-2dvjn" event={"ID":"c7486f1f-9947-4405-9186-66dfa5621253","Type":"ContainerStarted","Data":"7de8d11bd6c2dd4717c73e7bc7c2694cf990d12ffa4f715e05dac7ffba3f0b88"}
Sep 11 00:58:07.031506 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:07.031483    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-m8lrq" event={"ID":"f27a49c0-52c5-42cf-9640-77fb72ef0998","Type":"ContainerStarted","Data":"577cc99ad73c7e053226b89ce8387c827edc819f45ea85e65f52f577ee52ffed"}
Sep 11 00:58:07.033145 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:07.033118    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-st6p9" event={"ID":"bbc61763-126c-46a6-8607-9f1071004770","Type":"ContainerStarted","Data":"8cc5ebb0acdeba3fb58d255f97357ab6740b8e1825ae8e0e67701c674ccd0dac"}
Sep 11 00:58:07.034638 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:07.034617    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t42lc" event={"ID":"920866ad-d7f7-46e4-8624-7ae8011a2b76","Type":"ContainerStarted","Data":"eefc7ed7530c8ff7b381d6986d7c1d6dc775eb861e0c1158ded5a91f91ea6214"}
Sep 11 00:58:07.036309 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:07.036286    2569 generic.go:356] "Generic (PLEG): container finished" podID="799942b1-a6d8-4f5a-8d17-e7b2761b68b7" containerID="ee4f6d040a1f5b2c85b0d014e79a0e4240b3e9cdb1d59989115f69651295ba47" exitCode=0
Sep 11 00:58:07.036418 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:07.036369    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8bg6q" event={"ID":"799942b1-a6d8-4f5a-8d17-e7b2761b68b7","Type":"ContainerDied","Data":"ee4f6d040a1f5b2c85b0d014e79a0e4240b3e9cdb1d59989115f69651295ba47"}
Sep 11 00:58:07.038164 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:07.038141    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-8vxxt" event={"ID":"bda35fc9-c767-4b21-9509-994a67ffdeaf","Type":"ContainerStarted","Data":"01260e4319f93e2df16c7570016adbd142b0bb2b3be66b3e74145d8df17b1454"}
Sep 11 00:58:07.041310 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:07.041292    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qsqkd" event={"ID":"37ee44c2-7e75-4918-bc66-636974fa9b94","Type":"ContainerStarted","Data":"5c32d9e835a0f0b88a1a8d2d15a9849c43576530a9f98174d31b591213fa9e45"}
Sep 11 00:58:07.041399 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:07.041344    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qsqkd" event={"ID":"37ee44c2-7e75-4918-bc66-636974fa9b94","Type":"ContainerStarted","Data":"9a967d667149c82835001ea4b5cbf4851216229f48696b5707ed410eacca44e8"}
Sep 11 00:58:07.041399 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:07.041373    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qsqkd" event={"ID":"37ee44c2-7e75-4918-bc66-636974fa9b94","Type":"ContainerStarted","Data":"cf4b6e052d8a8577f76c61e0625fd4d90412c05a6ead9f8c3d88ef3380c9e881"}
Sep 11 00:58:07.041465 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:07.041402    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qsqkd" event={"ID":"37ee44c2-7e75-4918-bc66-636974fa9b94","Type":"ContainerStarted","Data":"0ce38f9fed3fa83c388cf11d391c5f8094bf894ea9d6c236960611a31ed5a86f"}
Sep 11 00:58:07.041465 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:07.041428    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qsqkd" event={"ID":"37ee44c2-7e75-4918-bc66-636974fa9b94","Type":"ContainerStarted","Data":"fcd4e67943f87574305dfb8d429ff6a72e50b3530fa1ced23710dcb64449db7e"}
Sep 11 00:58:07.041465 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:07.041454    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qsqkd" event={"ID":"37ee44c2-7e75-4918-bc66-636974fa9b94","Type":"ContainerStarted","Data":"55c5c8bc7ad652137e3944ab2fd8817409de5f9c0c78dc600e21846c580f27ab"}
Sep 11 00:58:07.061744 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:07.061704    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-jrszh" podStartSLOduration=3.837081819 podStartE2EDuration="21.061689059s" totalImagesPullingTime="17.22460724s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 00:57:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 00:58:07.061135181 +0000 UTC m=+21.712509433" watchObservedRunningTime="2026-09-11 00:58:07.061689059 +0000 UTC m=+21.713063308"
Sep 11 00:58:07.089084 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:07.087267    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-m8lrq" podStartSLOduration=3.841838435 podStartE2EDuration="21.087249025s" totalImagesPullingTime="17.24541059s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 00:57:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 00:58:07.086672694 +0000 UTC m=+21.738046943" watchObservedRunningTime="2026-09-11 00:58:07.087249025 +0000 UTC m=+21.738623278"
Sep 11 00:58:07.100508 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:07.100465    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-2dvjn" podStartSLOduration=3.865959069 podStartE2EDuration="21.100448085s" totalImagesPullingTime="17.234489016s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 00:57:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 00:58:07.100065714 +0000 UTC m=+21.751439963" watchObservedRunningTime="2026-09-11 00:58:07.100448085 +0000 UTC m=+21.751822336"
Sep 11 00:58:07.120230 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:07.120192    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-st6p9" podStartSLOduration=7.685481019 podStartE2EDuration="21.12017702s" totalImagesPullingTime="13.434696001s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 00:57:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 00:58:07.119664716 +0000 UTC m=+21.771038965" watchObservedRunningTime="2026-09-11 00:58:07.12017702 +0000 UTC m=+21.771551271"
Sep 11 00:58:07.140228 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:07.140185    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-8vxxt" podStartSLOduration=3.842798923 podStartE2EDuration="21.140169397s" totalImagesPullingTime="17.297370474s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 00:57:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 00:58:07.139731854 +0000 UTC m=+21.791106116" watchObservedRunningTime="2026-09-11 00:58:07.140169397 +0000 UTC m=+21.791543678"
Sep 11 00:58:07.864336 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:07.864306    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bbbq6"
Sep 11 00:58:07.864336 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:07.864321    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qgmsz"
Sep 11 00:58:07.864551 ip-10-0-129-222 kubenswrapper[2569]: E0911 00:58:07.864394    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bbbq6" podUID="239b25bd-4932-4b5b-b286-45d3e2ff9f1c"
Sep 11 00:58:07.864551 ip-10-0-129-222 kubenswrapper[2569]: E0911 00:58:07.864456    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qgmsz" podUID="d5d763bb-6142-409f-8155-7143a6d91dc6"
Sep 11 00:58:07.864551 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:07.864494    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r8n96"
Sep 11 00:58:07.864692 ip-10-0-129-222 kubenswrapper[2569]: E0911 00:58:07.864567    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r8n96" podUID="85c82c9f-52d0-49cd-afd9-b791628e1a50"
Sep 11 00:58:07.972959 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:07.972939    2569 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 11 00:58:08.045780 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:08.045707    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t42lc" event={"ID":"920866ad-d7f7-46e4-8624-7ae8011a2b76","Type":"ContainerStarted","Data":"d04f99e9c24f6780d8680e719d8d8747ad78b13078df8a5cc5d5b74adccaaf53"}
Sep 11 00:58:08.047476 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:08.047448    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-j862p" event={"ID":"f2ed1cf4-6159-47f1-aff0-8915d23dc932","Type":"ContainerStarted","Data":"fdaa30f1c9d1672426136afbfa2193c7e75b0b265fdfabf16c81e70338d1471f"}
Sep 11 00:58:08.062165 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:08.062125    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-j862p" podStartSLOduration=4.8079737940000005 podStartE2EDuration="22.062111778s" totalImagesPullingTime="17.254137984s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 00:57:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 00:58:08.06155468 +0000 UTC m=+22.712928928" watchObservedRunningTime="2026-09-11 00:58:08.062111778 +0000 UTC m=+22.713486026"
Sep 11 00:58:08.825468 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:08.825350    2569 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-11T00:58:07.972954951Z","UUID":"9e1cb937-5912-4da8-8981-8663e1308404","Handler":null,"Name":"","Endpoint":""}
Sep 11 00:58:08.827254 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:08.827230    2569 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 11 00:58:08.827409 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:08.827276    2569 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 11 00:58:09.131080 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:09.130993    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-st6p9"
Sep 11 00:58:09.131736 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:09.131712    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-st6p9"
Sep 11 00:58:09.864930 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:09.864905    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bbbq6"
Sep 11 00:58:09.865110 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:09.864913    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r8n96"
Sep 11 00:58:09.865110 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:09.864992    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qgmsz"
Sep 11 00:58:09.865110 ip-10-0-129-222 kubenswrapper[2569]: E0911 00:58:09.864992    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bbbq6" podUID="239b25bd-4932-4b5b-b286-45d3e2ff9f1c"
Sep 11 00:58:09.865272 ip-10-0-129-222 kubenswrapper[2569]: E0911 00:58:09.865107    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r8n96" podUID="85c82c9f-52d0-49cd-afd9-b791628e1a50"
Sep 11 00:58:09.865272 ip-10-0-129-222 kubenswrapper[2569]: E0911 00:58:09.865198    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qgmsz" podUID="d5d763bb-6142-409f-8155-7143a6d91dc6"
Sep 11 00:58:10.053884 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:10.053838    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t42lc" event={"ID":"920866ad-d7f7-46e4-8624-7ae8011a2b76","Type":"ContainerStarted","Data":"ab699c4ccfc40d1e3c2ba923ee5a0d5cafaac5437676c21378a7e8eea7e34f53"}
Sep 11 00:58:10.057610 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:10.057578    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qsqkd" event={"ID":"37ee44c2-7e75-4918-bc66-636974fa9b94","Type":"ContainerStarted","Data":"7ba09f0e5ee9110ccbcc7f14ffdf1b9e615e0fba655867e2a7520c65402a7c58"}
Sep 11 00:58:10.057767 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:10.057675    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-st6p9"
Sep 11 00:58:10.058301 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:10.058280    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-st6p9"
Sep 11 00:58:10.070930 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:10.070893    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t42lc" podStartSLOduration=4.312140861 podStartE2EDuration="24.070880529s" totalImagesPullingTime="19.758739668s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 00:57:46 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 00:58:10.070316183 +0000 UTC m=+24.721690436" watchObservedRunningTime="2026-09-11 00:58:10.070880529 +0000 UTC m=+24.722254769"
Sep 11 00:58:11.865064 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:11.864841    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qgmsz"
Sep 11 00:58:11.865812 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:11.864849    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bbbq6"
Sep 11 00:58:11.865812 ip-10-0-129-222 kubenswrapper[2569]: E0911 00:58:11.865154    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qgmsz" podUID="d5d763bb-6142-409f-8155-7143a6d91dc6"
Sep 11 00:58:11.865812 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:11.864853    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r8n96"
Sep 11 00:58:11.865812 ip-10-0-129-222 kubenswrapper[2569]: E0911 00:58:11.865240    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bbbq6" podUID="239b25bd-4932-4b5b-b286-45d3e2ff9f1c"
Sep 11 00:58:11.865812 ip-10-0-129-222 kubenswrapper[2569]: E0911 00:58:11.865312    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r8n96" podUID="85c82c9f-52d0-49cd-afd9-b791628e1a50"
Sep 11 00:58:12.066403 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:12.066272    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qsqkd" event={"ID":"37ee44c2-7e75-4918-bc66-636974fa9b94","Type":"ContainerStarted","Data":"02585eedea76afee688e5ba4698919145cda132adb0c30e1503276a1a5a4fb3a"}
Sep 11 00:58:12.110308 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:12.110260    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-qsqkd" podStartSLOduration=8.804232611 podStartE2EDuration="26.110245636s" totalImagesPullingTime="17.306013025s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 00:57:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 00:58:12.104497018 +0000 UTC m=+26.755871267" watchObservedRunningTime="2026-09-11 00:58:12.110245636 +0000 UTC m=+26.761619884"
Sep 11 00:58:13.069398 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:13.069368    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-qsqkd"
Sep 11 00:58:13.070013 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:13.069451    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-qsqkd"
Sep 11 00:58:13.070013 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:13.069501    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-qsqkd"
Sep 11 00:58:13.083834 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:13.083810    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qsqkd"
Sep 11 00:58:13.083981 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:13.083959    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qsqkd"
Sep 11 00:58:13.420201 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:13.420175    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-r8n96"]
Sep 11 00:58:13.420326 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:13.420313    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r8n96"
Sep 11 00:58:13.420466 ip-10-0-129-222 kubenswrapper[2569]: E0911 00:58:13.420430    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r8n96" podUID="85c82c9f-52d0-49cd-afd9-b791628e1a50"
Sep 11 00:58:13.423562 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:13.423530    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-bbbq6"]
Sep 11 00:58:13.423660 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:13.423646    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bbbq6"
Sep 11 00:58:13.423779 ip-10-0-129-222 kubenswrapper[2569]: E0911 00:58:13.423742    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bbbq6" podUID="239b25bd-4932-4b5b-b286-45d3e2ff9f1c"
Sep 11 00:58:13.424241 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:13.424218    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-qgmsz"]
Sep 11 00:58:13.424341 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:13.424303    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qgmsz"
Sep 11 00:58:13.424410 ip-10-0-129-222 kubenswrapper[2569]: E0911 00:58:13.424390    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qgmsz" podUID="d5d763bb-6142-409f-8155-7143a6d91dc6"
Sep 11 00:58:14.072713 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:14.072511    2569 generic.go:356] "Generic (PLEG): container finished" podID="799942b1-a6d8-4f5a-8d17-e7b2761b68b7" containerID="fc83925b447b586b3fffae64bde2799a77f6e9b531af0d5a6c87272a168697b2" exitCode=0
Sep 11 00:58:14.073222 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:14.072596    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8bg6q" event={"ID":"799942b1-a6d8-4f5a-8d17-e7b2761b68b7","Type":"ContainerDied","Data":"fc83925b447b586b3fffae64bde2799a77f6e9b531af0d5a6c87272a168697b2"}
Sep 11 00:58:14.864619 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:14.864586    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bbbq6"
Sep 11 00:58:14.864800 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:14.864634    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r8n96"
Sep 11 00:58:14.864800 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:14.864586    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qgmsz"
Sep 11 00:58:14.864800 ip-10-0-129-222 kubenswrapper[2569]: E0911 00:58:14.864712    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bbbq6" podUID="239b25bd-4932-4b5b-b286-45d3e2ff9f1c"
Sep 11 00:58:14.865000 ip-10-0-129-222 kubenswrapper[2569]: E0911 00:58:14.864807    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r8n96" podUID="85c82c9f-52d0-49cd-afd9-b791628e1a50"
Sep 11 00:58:14.865000 ip-10-0-129-222 kubenswrapper[2569]: E0911 00:58:14.864915    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qgmsz" podUID="d5d763bb-6142-409f-8155-7143a6d91dc6"
Sep 11 00:58:15.076334 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:15.076305    2569 generic.go:356] "Generic (PLEG): container finished" podID="799942b1-a6d8-4f5a-8d17-e7b2761b68b7" containerID="5d70166598afaa4b0986ff96bcb43aa4287042f433c5bc4246a731d589e08c17" exitCode=0
Sep 11 00:58:15.076637 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:15.076384    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8bg6q" event={"ID":"799942b1-a6d8-4f5a-8d17-e7b2761b68b7","Type":"ContainerDied","Data":"5d70166598afaa4b0986ff96bcb43aa4287042f433c5bc4246a731d589e08c17"}
Sep 11 00:58:16.081515 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:16.081487    2569 generic.go:356] "Generic (PLEG): container finished" podID="799942b1-a6d8-4f5a-8d17-e7b2761b68b7" containerID="2df9f17df8f0ce530d24123c129caf406c4ebb7e5e230dee0af7ce7d0da153ed" exitCode=0
Sep 11 00:58:16.081920 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:16.081531    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8bg6q" event={"ID":"799942b1-a6d8-4f5a-8d17-e7b2761b68b7","Type":"ContainerDied","Data":"2df9f17df8f0ce530d24123c129caf406c4ebb7e5e230dee0af7ce7d0da153ed"}
Sep 11 00:58:16.864303 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:16.864269    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bbbq6"
Sep 11 00:58:16.864481 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:16.864412    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qgmsz"
Sep 11 00:58:16.864481 ip-10-0-129-222 kubenswrapper[2569]: E0911 00:58:16.864419    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bbbq6" podUID="239b25bd-4932-4b5b-b286-45d3e2ff9f1c"
Sep 11 00:58:16.864481 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:16.864446    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r8n96"
Sep 11 00:58:16.864640 ip-10-0-129-222 kubenswrapper[2569]: E0911 00:58:16.864527    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qgmsz" podUID="d5d763bb-6142-409f-8155-7143a6d91dc6"
Sep 11 00:58:16.864640 ip-10-0-129-222 kubenswrapper[2569]: E0911 00:58:16.864600    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r8n96" podUID="85c82c9f-52d0-49cd-afd9-b791628e1a50"
Sep 11 00:58:18.864724 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:18.864691    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bbbq6"
Sep 11 00:58:18.865191 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:18.864696    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qgmsz"
Sep 11 00:58:18.865191 ip-10-0-129-222 kubenswrapper[2569]: E0911 00:58:18.864826    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bbbq6" podUID="239b25bd-4932-4b5b-b286-45d3e2ff9f1c"
Sep 11 00:58:18.865191 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:18.864700    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r8n96"
Sep 11 00:58:18.865191 ip-10-0-129-222 kubenswrapper[2569]: E0911 00:58:18.864960    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qgmsz" podUID="d5d763bb-6142-409f-8155-7143a6d91dc6"
Sep 11 00:58:18.865191 ip-10-0-129-222 kubenswrapper[2569]: E0911 00:58:18.865133    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r8n96" podUID="85c82c9f-52d0-49cd-afd9-b791628e1a50"
Sep 11 00:58:19.186975 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:19.186903    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-222.ec2.internal" event="NodeReady"
Sep 11 00:58:19.187134 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:19.187017    2569 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 11 00:58:19.222247 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:19.222221    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6f984758b5-srjjf"]
Sep 11 00:58:19.283120 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:19.283090    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-c2ph7"]
Sep 11 00:58:19.283265 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:19.283214    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6f984758b5-srjjf"
Sep 11 00:58:19.285559 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:19.285517    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-db2bf\""
Sep 11 00:58:19.285702 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:19.285565    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 11 00:58:19.285763 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:19.285728    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 11 00:58:19.285807 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:19.285764    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 11 00:58:19.292072 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:19.292049    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 11 00:58:19.295526 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:19.295511    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-6q5zl"]
Sep 11 00:58:19.295627 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:19.295614    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-c2ph7"
Sep 11 00:58:19.297931 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:19.297911    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-w7vvh\""
Sep 11 00:58:19.298040 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:19.297918    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 11 00:58:19.298040 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:19.297955    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 11 00:58:19.298156 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:19.298044    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 11 00:58:19.312296 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:19.312276    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6f984758b5-srjjf"]
Sep 11 00:58:19.312399 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:19.312313    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-c2ph7"]
Sep 11 00:58:19.312399 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:19.312380    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-6q5zl"
Sep 11 00:58:19.312399 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:19.312398    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-6q5zl"]
Sep 11 00:58:19.314728 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:19.314711    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-q9kdp\""
Sep 11 00:58:19.314728 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:19.314718    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 11 00:58:19.314990 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:19.314974    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 11 00:58:19.416648 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:19.416619    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v64zn\" (UniqueName: \"kubernetes.io/projected/75b89b12-20de-4813-b407-ee6836d0a777-kube-api-access-v64zn\") pod \"image-registry-6f984758b5-srjjf\" (UID: \"75b89b12-20de-4813-b407-ee6836d0a777\") " pod="openshift-image-registry/image-registry-6f984758b5-srjjf"
Sep 11 00:58:19.416766 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:19.416652    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f31a4ea6-4a42-40a1-bffa-7d59ce62a268-cert\") pod \"ingress-canary-c2ph7\" (UID: \"f31a4ea6-4a42-40a1-bffa-7d59ce62a268\") " pod="openshift-ingress-canary/ingress-canary-c2ph7"
Sep 11 00:58:19.416766 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:19.416675    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxtdf\" (UniqueName: \"kubernetes.io/projected/f31a4ea6-4a42-40a1-bffa-7d59ce62a268-kube-api-access-gxtdf\") pod \"ingress-canary-c2ph7\" (UID: \"f31a4ea6-4a42-40a1-bffa-7d59ce62a268\") " pod="openshift-ingress-canary/ingress-canary-c2ph7"
Sep 11 00:58:19.416766 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:19.416702    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d10c25a9-1b18-43a7-94f6-ae86285efcec-config-volume\") pod \"dns-default-6q5zl\" (UID: \"d10c25a9-1b18-43a7-94f6-ae86285efcec\") " pod="openshift-dns/dns-default-6q5zl"
Sep 11 00:58:19.416863 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:19.416787    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d10c25a9-1b18-43a7-94f6-ae86285efcec-metrics-tls\") pod \"dns-default-6q5zl\" (UID: \"d10c25a9-1b18-43a7-94f6-ae86285efcec\") " pod="openshift-dns/dns-default-6q5zl"
Sep 11 00:58:19.416863 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:19.416824    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d10c25a9-1b18-43a7-94f6-ae86285efcec-tmp-dir\") pod \"dns-default-6q5zl\" (UID: \"d10c25a9-1b18-43a7-94f6-ae86285efcec\") " pod="openshift-dns/dns-default-6q5zl"
Sep 11 00:58:19.416863 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:19.416840    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/75b89b12-20de-4813-b407-ee6836d0a777-registry-tls\") pod \"image-registry-6f984758b5-srjjf\" (UID: \"75b89b12-20de-4813-b407-ee6836d0a777\") " pod="openshift-image-registry/image-registry-6f984758b5-srjjf"
Sep 11 00:58:19.416863 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:19.416856    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/75b89b12-20de-4813-b407-ee6836d0a777-installation-pull-secrets\") pod \"image-registry-6f984758b5-srjjf\" (UID: \"75b89b12-20de-4813-b407-ee6836d0a777\") " pod="openshift-image-registry/image-registry-6f984758b5-srjjf"
Sep 11 00:58:19.417004 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:19.416888    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/75b89b12-20de-4813-b407-ee6836d0a777-ca-trust-extracted\") pod \"image-registry-6f984758b5-srjjf\" (UID: \"75b89b12-20de-4813-b407-ee6836d0a777\") " pod="openshift-image-registry/image-registry-6f984758b5-srjjf"
Sep 11 00:58:19.417004 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:19.416903    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/75b89b12-20de-4813-b407-ee6836d0a777-trusted-ca\") pod \"image-registry-6f984758b5-srjjf\" (UID: \"75b89b12-20de-4813-b407-ee6836d0a777\") " pod="openshift-image-registry/image-registry-6f984758b5-srjjf"
Sep 11 00:58:19.417004 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:19.416917    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/75b89b12-20de-4813-b407-ee6836d0a777-bound-sa-token\") pod \"image-registry-6f984758b5-srjjf\" (UID: \"75b89b12-20de-4813-b407-ee6836d0a777\") " pod="openshift-image-registry/image-registry-6f984758b5-srjjf"
Sep 11 00:58:19.417004 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:19.416969    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtqxk\" (UniqueName: \"kubernetes.io/projected/d10c25a9-1b18-43a7-94f6-ae86285efcec-kube-api-access-vtqxk\") pod \"dns-default-6q5zl\" (UID: \"d10c25a9-1b18-43a7-94f6-ae86285efcec\") " pod="openshift-dns/dns-default-6q5zl"
Sep 11 00:58:19.417004 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:19.416994    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/75b89b12-20de-4813-b407-ee6836d0a777-registry-certificates\") pod \"image-registry-6f984758b5-srjjf\" (UID: \"75b89b12-20de-4813-b407-ee6836d0a777\") " pod="openshift-image-registry/image-registry-6f984758b5-srjjf"
Sep 11 00:58:19.417149 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:19.417027    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/75b89b12-20de-4813-b407-ee6836d0a777-image-registry-private-configuration\") pod \"image-registry-6f984758b5-srjjf\" (UID: \"75b89b12-20de-4813-b407-ee6836d0a777\") " pod="openshift-image-registry/image-registry-6f984758b5-srjjf"
Sep 11 00:58:19.517910 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:19.517827    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f31a4ea6-4a42-40a1-bffa-7d59ce62a268-cert\") pod \"ingress-canary-c2ph7\" (UID: \"f31a4ea6-4a42-40a1-bffa-7d59ce62a268\") " pod="openshift-ingress-canary/ingress-canary-c2ph7"
Sep 11 00:58:19.517910 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:19.517864    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxtdf\" (UniqueName: \"kubernetes.io/projected/f31a4ea6-4a42-40a1-bffa-7d59ce62a268-kube-api-access-gxtdf\") pod \"ingress-canary-c2ph7\" (UID: \"f31a4ea6-4a42-40a1-bffa-7d59ce62a268\") " pod="openshift-ingress-canary/ingress-canary-c2ph7"
Sep 11 00:58:19.518104 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:19.517914    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d10c25a9-1b18-43a7-94f6-ae86285efcec-config-volume\") pod \"dns-default-6q5zl\" (UID: \"d10c25a9-1b18-43a7-94f6-ae86285efcec\") " pod="openshift-dns/dns-default-6q5zl"
Sep 11 00:58:19.518104 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:19.517954    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85c82c9f-52d0-49cd-afd9-b791628e1a50-metrics-certs\") pod \"network-metrics-daemon-r8n96\" (UID: \"85c82c9f-52d0-49cd-afd9-b791628e1a50\") " pod="openshift-multus/network-metrics-daemon-r8n96"
Sep 11 00:58:19.518104 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:19.517991    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d10c25a9-1b18-43a7-94f6-ae86285efcec-metrics-tls\") pod \"dns-default-6q5zl\" (UID: \"d10c25a9-1b18-43a7-94f6-ae86285efcec\") " pod="openshift-dns/dns-default-6q5zl"
Sep 11 00:58:19.518104 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:19.518029    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d10c25a9-1b18-43a7-94f6-ae86285efcec-tmp-dir\") pod \"dns-default-6q5zl\" (UID: \"d10c25a9-1b18-43a7-94f6-ae86285efcec\") " pod="openshift-dns/dns-default-6q5zl"
Sep 11 00:58:19.518104 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:19.518050    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/75b89b12-20de-4813-b407-ee6836d0a777-registry-tls\") pod \"image-registry-6f984758b5-srjjf\" (UID: \"75b89b12-20de-4813-b407-ee6836d0a777\") " pod="openshift-image-registry/image-registry-6f984758b5-srjjf"
Sep 11 00:58:19.518104 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:19.518074    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/75b89b12-20de-4813-b407-ee6836d0a777-installation-pull-secrets\") pod \"image-registry-6f984758b5-srjjf\" (UID: \"75b89b12-20de-4813-b407-ee6836d0a777\") " pod="openshift-image-registry/image-registry-6f984758b5-srjjf"
Sep 11 00:58:19.518104 ip-10-0-129-222 kubenswrapper[2569]: E0911 00:58:19.518082    2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 00:58:19.518104 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:19.518099    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/75b89b12-20de-4813-b407-ee6836d0a777-ca-trust-extracted\") pod \"image-registry-6f984758b5-srjjf\" (UID: \"75b89b12-20de-4813-b407-ee6836d0a777\") " pod="openshift-image-registry/image-registry-6f984758b5-srjjf"
Sep 11 00:58:19.518496 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:19.518121    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/75b89b12-20de-4813-b407-ee6836d0a777-trusted-ca\") pod \"image-registry-6f984758b5-srjjf\" (UID: \"75b89b12-20de-4813-b407-ee6836d0a777\") " pod="openshift-image-registry/image-registry-6f984758b5-srjjf"
Sep 11 00:58:19.518496 ip-10-0-129-222 kubenswrapper[2569]: E0911 00:58:19.518151    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85c82c9f-52d0-49cd-afd9-b791628e1a50-metrics-certs podName:85c82c9f-52d0-49cd-afd9-b791628e1a50 nodeName:}" failed. No retries permitted until 2026-09-11 00:58:51.518133621 +0000 UTC m=+66.169507848 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/85c82c9f-52d0-49cd-afd9-b791628e1a50-metrics-certs") pod "network-metrics-daemon-r8n96" (UID: "85c82c9f-52d0-49cd-afd9-b791628e1a50") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 00:58:19.518496 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:19.518175    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/75b89b12-20de-4813-b407-ee6836d0a777-bound-sa-token\") pod \"image-registry-6f984758b5-srjjf\" (UID: \"75b89b12-20de-4813-b407-ee6836d0a777\") " pod="openshift-image-registry/image-registry-6f984758b5-srjjf"
Sep 11 00:58:19.518496 ip-10-0-129-222 kubenswrapper[2569]: E0911 00:58:19.518217    2569 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 11 00:58:19.518496 ip-10-0-129-222 kubenswrapper[2569]: E0911 00:58:19.518231    2569 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6f984758b5-srjjf: secret "image-registry-tls" not found
Sep 11 00:58:19.518496 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:19.518235    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtqxk\" (UniqueName: \"kubernetes.io/projected/d10c25a9-1b18-43a7-94f6-ae86285efcec-kube-api-access-vtqxk\") pod \"dns-default-6q5zl\" (UID: \"d10c25a9-1b18-43a7-94f6-ae86285efcec\") " pod="openshift-dns/dns-default-6q5zl"
Sep 11 00:58:19.518496 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:19.518263    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/75b89b12-20de-4813-b407-ee6836d0a777-registry-certificates\") pod \"image-registry-6f984758b5-srjjf\" (UID: \"75b89b12-20de-4813-b407-ee6836d0a777\") " pod="openshift-image-registry/image-registry-6f984758b5-srjjf"
Sep 11 00:58:19.518496 ip-10-0-129-222 kubenswrapper[2569]: E0911 00:58:19.518269    2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 00:58:19.518496 ip-10-0-129-222 kubenswrapper[2569]: E0911 00:58:19.518273    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/75b89b12-20de-4813-b407-ee6836d0a777-registry-tls podName:75b89b12-20de-4813-b407-ee6836d0a777 nodeName:}" failed. No retries permitted until 2026-09-11 00:58:20.018260254 +0000 UTC m=+34.669634505 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/75b89b12-20de-4813-b407-ee6836d0a777-registry-tls") pod "image-registry-6f984758b5-srjjf" (UID: "75b89b12-20de-4813-b407-ee6836d0a777") : secret "image-registry-tls" not found
Sep 11 00:58:19.518496 ip-10-0-129-222 kubenswrapper[2569]: E0911 00:58:19.518364    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d10c25a9-1b18-43a7-94f6-ae86285efcec-metrics-tls podName:d10c25a9-1b18-43a7-94f6-ae86285efcec nodeName:}" failed. No retries permitted until 2026-09-11 00:58:20.018347714 +0000 UTC m=+34.669721944 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d10c25a9-1b18-43a7-94f6-ae86285efcec-metrics-tls") pod "dns-default-6q5zl" (UID: "d10c25a9-1b18-43a7-94f6-ae86285efcec") : secret "dns-default-metrics-tls" not found
Sep 11 00:58:19.518496 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:19.518401    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d5d763bb-6142-409f-8155-7143a6d91dc6-original-pull-secret\") pod \"global-pull-secret-syncer-qgmsz\" (UID: \"d5d763bb-6142-409f-8155-7143a6d91dc6\") " pod="kube-system/global-pull-secret-syncer-qgmsz"
Sep 11 00:58:19.518496 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:19.518433    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/75b89b12-20de-4813-b407-ee6836d0a777-image-registry-private-configuration\") pod \"image-registry-6f984758b5-srjjf\" (UID: \"75b89b12-20de-4813-b407-ee6836d0a777\") " pod="openshift-image-registry/image-registry-6f984758b5-srjjf"
Sep 11 00:58:19.518496 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:19.518464    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v64zn\" (UniqueName: \"kubernetes.io/projected/75b89b12-20de-4813-b407-ee6836d0a777-kube-api-access-v64zn\") pod \"image-registry-6f984758b5-srjjf\" (UID: \"75b89b12-20de-4813-b407-ee6836d0a777\") " pod="openshift-image-registry/image-registry-6f984758b5-srjjf"
Sep 11 00:58:19.518987 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:19.518538    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d10c25a9-1b18-43a7-94f6-ae86285efcec-tmp-dir\") pod \"dns-default-6q5zl\" (UID: \"d10c25a9-1b18-43a7-94f6-ae86285efcec\") " pod="openshift-dns/dns-default-6q5zl"
Sep 11 00:58:19.518987 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:19.518750    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d10c25a9-1b18-43a7-94f6-ae86285efcec-config-volume\") pod \"dns-default-6q5zl\" (UID: \"d10c25a9-1b18-43a7-94f6-ae86285efcec\") " pod="openshift-dns/dns-default-6q5zl"
Sep 11 00:58:19.518987 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:19.518908    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/75b89b12-20de-4813-b407-ee6836d0a777-ca-trust-extracted\") pod \"image-registry-6f984758b5-srjjf\" (UID: \"75b89b12-20de-4813-b407-ee6836d0a777\") " pod="openshift-image-registry/image-registry-6f984758b5-srjjf"
Sep 11 00:58:19.519125 ip-10-0-129-222 kubenswrapper[2569]: E0911 00:58:19.519106    2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 00:58:19.519196 ip-10-0-129-222 kubenswrapper[2569]: E0911 00:58:19.519184    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d5d763bb-6142-409f-8155-7143a6d91dc6-original-pull-secret podName:d5d763bb-6142-409f-8155-7143a6d91dc6 nodeName:}" failed. No retries permitted until 2026-09-11 00:58:51.519168056 +0000 UTC m=+66.170542303 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d5d763bb-6142-409f-8155-7143a6d91dc6-original-pull-secret") pod "global-pull-secret-syncer-qgmsz" (UID: "d5d763bb-6142-409f-8155-7143a6d91dc6") : object "kube-system"/"original-pull-secret" not registered
Sep 11 00:58:19.519421 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:19.519399    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/75b89b12-20de-4813-b407-ee6836d0a777-registry-certificates\") pod \"image-registry-6f984758b5-srjjf\" (UID: \"75b89b12-20de-4813-b407-ee6836d0a777\") " pod="openshift-image-registry/image-registry-6f984758b5-srjjf"
Sep 11 00:58:19.520024 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:19.519860    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/75b89b12-20de-4813-b407-ee6836d0a777-trusted-ca\") pod \"image-registry-6f984758b5-srjjf\" (UID: \"75b89b12-20de-4813-b407-ee6836d0a777\") " pod="openshift-image-registry/image-registry-6f984758b5-srjjf"
Sep 11 00:58:19.522563 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:19.522541    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/75b89b12-20de-4813-b407-ee6836d0a777-installation-pull-secrets\") pod \"image-registry-6f984758b5-srjjf\" (UID: \"75b89b12-20de-4813-b407-ee6836d0a777\") " pod="openshift-image-registry/image-registry-6f984758b5-srjjf"
Sep 11 00:58:19.522668 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:19.522561    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/75b89b12-20de-4813-b407-ee6836d0a777-image-registry-private-configuration\") pod \"image-registry-6f984758b5-srjjf\" (UID: \"75b89b12-20de-4813-b407-ee6836d0a777\") " pod="openshift-image-registry/image-registry-6f984758b5-srjjf"
Sep 11 00:58:19.523038 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:19.523017    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f31a4ea6-4a42-40a1-bffa-7d59ce62a268-cert\") pod \"ingress-canary-c2ph7\" (UID: \"f31a4ea6-4a42-40a1-bffa-7d59ce62a268\") " pod="openshift-ingress-canary/ingress-canary-c2ph7"
Sep 11 00:58:19.526852 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:19.526832    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtqxk\" (UniqueName: \"kubernetes.io/projected/d10c25a9-1b18-43a7-94f6-ae86285efcec-kube-api-access-vtqxk\") pod \"dns-default-6q5zl\" (UID: \"d10c25a9-1b18-43a7-94f6-ae86285efcec\") " pod="openshift-dns/dns-default-6q5zl"
Sep 11 00:58:19.527222 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:19.527200    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxtdf\" (UniqueName: \"kubernetes.io/projected/f31a4ea6-4a42-40a1-bffa-7d59ce62a268-kube-api-access-gxtdf\") pod \"ingress-canary-c2ph7\" (UID: \"f31a4ea6-4a42-40a1-bffa-7d59ce62a268\") " pod="openshift-ingress-canary/ingress-canary-c2ph7"
Sep 11 00:58:19.527296 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:19.527231    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v64zn\" (UniqueName: \"kubernetes.io/projected/75b89b12-20de-4813-b407-ee6836d0a777-kube-api-access-v64zn\") pod \"image-registry-6f984758b5-srjjf\" (UID: \"75b89b12-20de-4813-b407-ee6836d0a777\") " pod="openshift-image-registry/image-registry-6f984758b5-srjjf"
Sep 11 00:58:19.527549 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:19.527518    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/75b89b12-20de-4813-b407-ee6836d0a777-bound-sa-token\") pod \"image-registry-6f984758b5-srjjf\" (UID: \"75b89b12-20de-4813-b407-ee6836d0a777\") " pod="openshift-image-registry/image-registry-6f984758b5-srjjf"
Sep 11 00:58:19.604985 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:19.604954    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-c2ph7"
Sep 11 00:58:19.619419 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:19.619393    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwg5s\" (UniqueName: \"kubernetes.io/projected/239b25bd-4932-4b5b-b286-45d3e2ff9f1c-kube-api-access-qwg5s\") pod \"network-check-target-bbbq6\" (UID: \"239b25bd-4932-4b5b-b286-45d3e2ff9f1c\") " pod="openshift-network-diagnostics/network-check-target-bbbq6"
Sep 11 00:58:19.619575 ip-10-0-129-222 kubenswrapper[2569]: E0911 00:58:19.619558    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 00:58:19.619629 ip-10-0-129-222 kubenswrapper[2569]: E0911 00:58:19.619582    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 00:58:19.619629 ip-10-0-129-222 kubenswrapper[2569]: E0911 00:58:19.619593    2569 projected.go:196] Error preparing data for projected volume kube-api-access-qwg5s for pod openshift-network-diagnostics/network-check-target-bbbq6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 00:58:19.619691 ip-10-0-129-222 kubenswrapper[2569]: E0911 00:58:19.619645    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/239b25bd-4932-4b5b-b286-45d3e2ff9f1c-kube-api-access-qwg5s podName:239b25bd-4932-4b5b-b286-45d3e2ff9f1c nodeName:}" failed. No retries permitted until 2026-09-11 00:58:51.619629416 +0000 UTC m=+66.271003644 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-qwg5s" (UniqueName: "kubernetes.io/projected/239b25bd-4932-4b5b-b286-45d3e2ff9f1c-kube-api-access-qwg5s") pod "network-check-target-bbbq6" (UID: "239b25bd-4932-4b5b-b286-45d3e2ff9f1c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 00:58:20.022075 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:20.022043    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d10c25a9-1b18-43a7-94f6-ae86285efcec-metrics-tls\") pod \"dns-default-6q5zl\" (UID: \"d10c25a9-1b18-43a7-94f6-ae86285efcec\") " pod="openshift-dns/dns-default-6q5zl"
Sep 11 00:58:20.022579 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:20.022086    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/75b89b12-20de-4813-b407-ee6836d0a777-registry-tls\") pod \"image-registry-6f984758b5-srjjf\" (UID: \"75b89b12-20de-4813-b407-ee6836d0a777\") " pod="openshift-image-registry/image-registry-6f984758b5-srjjf"
Sep 11 00:58:20.022579 ip-10-0-129-222 kubenswrapper[2569]: E0911 00:58:20.022182    2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 00:58:20.022579 ip-10-0-129-222 kubenswrapper[2569]: E0911 00:58:20.022248    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d10c25a9-1b18-43a7-94f6-ae86285efcec-metrics-tls podName:d10c25a9-1b18-43a7-94f6-ae86285efcec nodeName:}" failed. No retries permitted until 2026-09-11 00:58:21.02223359 +0000 UTC m=+35.673607818 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d10c25a9-1b18-43a7-94f6-ae86285efcec-metrics-tls") pod "dns-default-6q5zl" (UID: "d10c25a9-1b18-43a7-94f6-ae86285efcec") : secret "dns-default-metrics-tls" not found
Sep 11 00:58:20.022579 ip-10-0-129-222 kubenswrapper[2569]: E0911 00:58:20.022185    2569 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 11 00:58:20.022579 ip-10-0-129-222 kubenswrapper[2569]: E0911 00:58:20.022294    2569 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6f984758b5-srjjf: secret "image-registry-tls" not found
Sep 11 00:58:20.022579 ip-10-0-129-222 kubenswrapper[2569]: E0911 00:58:20.022343    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/75b89b12-20de-4813-b407-ee6836d0a777-registry-tls podName:75b89b12-20de-4813-b407-ee6836d0a777 nodeName:}" failed. No retries permitted until 2026-09-11 00:58:21.022328286 +0000 UTC m=+35.673702518 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/75b89b12-20de-4813-b407-ee6836d0a777-registry-tls") pod "image-registry-6f984758b5-srjjf" (UID: "75b89b12-20de-4813-b407-ee6836d0a777") : secret "image-registry-tls" not found
Sep 11 00:58:20.864770 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:20.864687    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r8n96"
Sep 11 00:58:20.864770 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:20.864745    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bbbq6"
Sep 11 00:58:20.865131 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:20.864855    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qgmsz"
Sep 11 00:58:20.868094 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:20.867866    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-lb8gv\""
Sep 11 00:58:20.868094 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:20.867954    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 11 00:58:20.868094 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:20.867965    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-jwmm5\""
Sep 11 00:58:20.868094 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:20.867977    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 11 00:58:20.868094 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:20.867959    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 11 00:58:20.868094 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:20.868005    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 11 00:58:21.030333 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:21.030313    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d10c25a9-1b18-43a7-94f6-ae86285efcec-metrics-tls\") pod \"dns-default-6q5zl\" (UID: \"d10c25a9-1b18-43a7-94f6-ae86285efcec\") " pod="openshift-dns/dns-default-6q5zl"
Sep 11 00:58:21.030801 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:21.030363    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/75b89b12-20de-4813-b407-ee6836d0a777-registry-tls\") pod \"image-registry-6f984758b5-srjjf\" (UID: \"75b89b12-20de-4813-b407-ee6836d0a777\") " pod="openshift-image-registry/image-registry-6f984758b5-srjjf"
Sep 11 00:58:21.030801 ip-10-0-129-222 kubenswrapper[2569]: E0911 00:58:21.030475    2569 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 11 00:58:21.030801 ip-10-0-129-222 kubenswrapper[2569]: E0911 00:58:21.030486    2569 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6f984758b5-srjjf: secret "image-registry-tls" not found
Sep 11 00:58:21.030801 ip-10-0-129-222 kubenswrapper[2569]: E0911 00:58:21.030484    2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 00:58:21.030801 ip-10-0-129-222 kubenswrapper[2569]: E0911 00:58:21.030553    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/75b89b12-20de-4813-b407-ee6836d0a777-registry-tls podName:75b89b12-20de-4813-b407-ee6836d0a777 nodeName:}" failed. No retries permitted until 2026-09-11 00:58:23.030537376 +0000 UTC m=+37.681911603 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/75b89b12-20de-4813-b407-ee6836d0a777-registry-tls") pod "image-registry-6f984758b5-srjjf" (UID: "75b89b12-20de-4813-b407-ee6836d0a777") : secret "image-registry-tls" not found
Sep 11 00:58:21.030801 ip-10-0-129-222 kubenswrapper[2569]: E0911 00:58:21.030566    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d10c25a9-1b18-43a7-94f6-ae86285efcec-metrics-tls podName:d10c25a9-1b18-43a7-94f6-ae86285efcec nodeName:}" failed. No retries permitted until 2026-09-11 00:58:23.030559493 +0000 UTC m=+37.681933720 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d10c25a9-1b18-43a7-94f6-ae86285efcec-metrics-tls") pod "dns-default-6q5zl" (UID: "d10c25a9-1b18-43a7-94f6-ae86285efcec") : secret "dns-default-metrics-tls" not found
Sep 11 00:58:21.316022 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:21.315989    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-c2ph7"]
Sep 11 00:58:21.320732 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:58:21.320700    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf31a4ea6_4a42_40a1_bffa_7d59ce62a268.slice/crio-aa7ee703167bf2a009a627babcd306be772cf1554912a6ace3da18924d1423e3 WatchSource:0}: Error finding container aa7ee703167bf2a009a627babcd306be772cf1554912a6ace3da18924d1423e3: Status 404 returned error can't find the container with id aa7ee703167bf2a009a627babcd306be772cf1554912a6ace3da18924d1423e3
Sep 11 00:58:22.101535 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:22.101285    2569 generic.go:356] "Generic (PLEG): container finished" podID="799942b1-a6d8-4f5a-8d17-e7b2761b68b7" containerID="33348a3c603c149d95a142ffa5e6fb11f32b525f2f555c125d87cb2ceb739e82" exitCode=0
Sep 11 00:58:22.102297 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:22.101373    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8bg6q" event={"ID":"799942b1-a6d8-4f5a-8d17-e7b2761b68b7","Type":"ContainerDied","Data":"33348a3c603c149d95a142ffa5e6fb11f32b525f2f555c125d87cb2ceb739e82"}
Sep 11 00:58:22.103012 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:22.102956    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-c2ph7" event={"ID":"f31a4ea6-4a42-40a1-bffa-7d59ce62a268","Type":"ContainerStarted","Data":"aa7ee703167bf2a009a627babcd306be772cf1554912a6ace3da18924d1423e3"}
Sep 11 00:58:23.045117 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:23.045078    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d10c25a9-1b18-43a7-94f6-ae86285efcec-metrics-tls\") pod \"dns-default-6q5zl\" (UID: \"d10c25a9-1b18-43a7-94f6-ae86285efcec\") " pod="openshift-dns/dns-default-6q5zl"
Sep 11 00:58:23.045273 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:23.045128    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/75b89b12-20de-4813-b407-ee6836d0a777-registry-tls\") pod \"image-registry-6f984758b5-srjjf\" (UID: \"75b89b12-20de-4813-b407-ee6836d0a777\") " pod="openshift-image-registry/image-registry-6f984758b5-srjjf"
Sep 11 00:58:23.045273 ip-10-0-129-222 kubenswrapper[2569]: E0911 00:58:23.045221    2569 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 11 00:58:23.045273 ip-10-0-129-222 kubenswrapper[2569]: E0911 00:58:23.045233    2569 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6f984758b5-srjjf: secret "image-registry-tls" not found
Sep 11 00:58:23.045370 ip-10-0-129-222 kubenswrapper[2569]: E0911 00:58:23.045276    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/75b89b12-20de-4813-b407-ee6836d0a777-registry-tls podName:75b89b12-20de-4813-b407-ee6836d0a777 nodeName:}" failed. No retries permitted until 2026-09-11 00:58:27.045264025 +0000 UTC m=+41.696638252 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/75b89b12-20de-4813-b407-ee6836d0a777-registry-tls") pod "image-registry-6f984758b5-srjjf" (UID: "75b89b12-20de-4813-b407-ee6836d0a777") : secret "image-registry-tls" not found
Sep 11 00:58:23.045370 ip-10-0-129-222 kubenswrapper[2569]: E0911 00:58:23.045220    2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 00:58:23.045370 ip-10-0-129-222 kubenswrapper[2569]: E0911 00:58:23.045344    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d10c25a9-1b18-43a7-94f6-ae86285efcec-metrics-tls podName:d10c25a9-1b18-43a7-94f6-ae86285efcec nodeName:}" failed. No retries permitted until 2026-09-11 00:58:27.045330716 +0000 UTC m=+41.696704943 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d10c25a9-1b18-43a7-94f6-ae86285efcec-metrics-tls") pod "dns-default-6q5zl" (UID: "d10c25a9-1b18-43a7-94f6-ae86285efcec") : secret "dns-default-metrics-tls" not found
Sep 11 00:58:23.109215 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:23.109136    2569 generic.go:356] "Generic (PLEG): container finished" podID="799942b1-a6d8-4f5a-8d17-e7b2761b68b7" containerID="907ec351b00024058ca7066b64caea81d5585fdbb9e1840df689514e148ccf5b" exitCode=0
Sep 11 00:58:23.109215 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:23.109201    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8bg6q" event={"ID":"799942b1-a6d8-4f5a-8d17-e7b2761b68b7","Type":"ContainerDied","Data":"907ec351b00024058ca7066b64caea81d5585fdbb9e1840df689514e148ccf5b"}
Sep 11 00:58:24.115147 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:24.115082    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8bg6q" event={"ID":"799942b1-a6d8-4f5a-8d17-e7b2761b68b7","Type":"ContainerStarted","Data":"c254adba9b566082eca15c4396bb16fcb555899a7774d5e80f5355c06be0051a"}
Sep 11 00:58:24.136379 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:24.136227    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-8bg6q" podStartSLOduration=7.769452755 podStartE2EDuration="38.136210262s" totalImagesPullingTime="30.366757507s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 00:57:46 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 00:58:24.134813284 +0000 UTC m=+38.786187523" watchObservedRunningTime="2026-09-11 00:58:24.136210262 +0000 UTC m=+38.787584512"
Sep 11 00:58:25.118686 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:25.118649    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-c2ph7" event={"ID":"f31a4ea6-4a42-40a1-bffa-7d59ce62a268","Type":"ContainerStarted","Data":"101e35d5fc558c586ae6a4c78f1e908de4d84480e1539a92493f4053445b186c"}
Sep 11 00:58:25.134559 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:25.134512    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-c2ph7" podStartSLOduration=3.218387937 podStartE2EDuration="6.134496198s" totalImagesPullingTime="2.916108261s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 00:58:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 00:58:25.13386399 +0000 UTC m=+39.785238240" watchObservedRunningTime="2026-09-11 00:58:25.134496198 +0000 UTC m=+39.785870446"
Sep 11 00:58:27.076001 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:27.075951    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d10c25a9-1b18-43a7-94f6-ae86285efcec-metrics-tls\") pod \"dns-default-6q5zl\" (UID: \"d10c25a9-1b18-43a7-94f6-ae86285efcec\") " pod="openshift-dns/dns-default-6q5zl"
Sep 11 00:58:27.076469 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:27.076016    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/75b89b12-20de-4813-b407-ee6836d0a777-registry-tls\") pod \"image-registry-6f984758b5-srjjf\" (UID: \"75b89b12-20de-4813-b407-ee6836d0a777\") " pod="openshift-image-registry/image-registry-6f984758b5-srjjf"
Sep 11 00:58:27.076469 ip-10-0-129-222 kubenswrapper[2569]: E0911 00:58:27.076101    2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 00:58:27.076469 ip-10-0-129-222 kubenswrapper[2569]: E0911 00:58:27.076163    2569 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 11 00:58:27.076469 ip-10-0-129-222 kubenswrapper[2569]: E0911 00:58:27.076180    2569 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6f984758b5-srjjf: secret "image-registry-tls" not found
Sep 11 00:58:27.076469 ip-10-0-129-222 kubenswrapper[2569]: E0911 00:58:27.076169    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d10c25a9-1b18-43a7-94f6-ae86285efcec-metrics-tls podName:d10c25a9-1b18-43a7-94f6-ae86285efcec nodeName:}" failed. No retries permitted until 2026-09-11 00:58:35.076153945 +0000 UTC m=+49.727528177 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d10c25a9-1b18-43a7-94f6-ae86285efcec-metrics-tls") pod "dns-default-6q5zl" (UID: "d10c25a9-1b18-43a7-94f6-ae86285efcec") : secret "dns-default-metrics-tls" not found
Sep 11 00:58:27.076469 ip-10-0-129-222 kubenswrapper[2569]: E0911 00:58:27.076248    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/75b89b12-20de-4813-b407-ee6836d0a777-registry-tls podName:75b89b12-20de-4813-b407-ee6836d0a777 nodeName:}" failed. No retries permitted until 2026-09-11 00:58:35.07622247 +0000 UTC m=+49.727596698 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/75b89b12-20de-4813-b407-ee6836d0a777-registry-tls") pod "image-registry-6f984758b5-srjjf" (UID: "75b89b12-20de-4813-b407-ee6836d0a777") : secret "image-registry-tls" not found
Sep 11 00:58:35.129731 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:35.129695    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d10c25a9-1b18-43a7-94f6-ae86285efcec-metrics-tls\") pod \"dns-default-6q5zl\" (UID: \"d10c25a9-1b18-43a7-94f6-ae86285efcec\") " pod="openshift-dns/dns-default-6q5zl"
Sep 11 00:58:35.130225 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:35.129741    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/75b89b12-20de-4813-b407-ee6836d0a777-registry-tls\") pod \"image-registry-6f984758b5-srjjf\" (UID: \"75b89b12-20de-4813-b407-ee6836d0a777\") " pod="openshift-image-registry/image-registry-6f984758b5-srjjf"
Sep 11 00:58:35.130225 ip-10-0-129-222 kubenswrapper[2569]: E0911 00:58:35.129844    2569 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 11 00:58:35.130225 ip-10-0-129-222 kubenswrapper[2569]: E0911 00:58:35.129855    2569 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6f984758b5-srjjf: secret "image-registry-tls" not found
Sep 11 00:58:35.130225 ip-10-0-129-222 kubenswrapper[2569]: E0911 00:58:35.129857    2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 00:58:35.130225 ip-10-0-129-222 kubenswrapper[2569]: E0911 00:58:35.129926    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/75b89b12-20de-4813-b407-ee6836d0a777-registry-tls podName:75b89b12-20de-4813-b407-ee6836d0a777 nodeName:}" failed. No retries permitted until 2026-09-11 00:58:51.129914177 +0000 UTC m=+65.781288403 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/75b89b12-20de-4813-b407-ee6836d0a777-registry-tls") pod "image-registry-6f984758b5-srjjf" (UID: "75b89b12-20de-4813-b407-ee6836d0a777") : secret "image-registry-tls" not found
Sep 11 00:58:35.130225 ip-10-0-129-222 kubenswrapper[2569]: E0911 00:58:35.129956    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d10c25a9-1b18-43a7-94f6-ae86285efcec-metrics-tls podName:d10c25a9-1b18-43a7-94f6-ae86285efcec nodeName:}" failed. No retries permitted until 2026-09-11 00:58:51.129937965 +0000 UTC m=+65.781312210 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d10c25a9-1b18-43a7-94f6-ae86285efcec-metrics-tls") pod "dns-default-6q5zl" (UID: "d10c25a9-1b18-43a7-94f6-ae86285efcec") : secret "dns-default-metrics-tls" not found
Sep 11 00:58:37.324148 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:37.324114    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-8s6fn"]
Sep 11 00:58:37.336788 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:37.336764    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f8b768455-8s6fn"
Sep 11 00:58:37.341265 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:37.341242    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-8s6fn"]
Sep 11 00:58:37.342967 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:37.342943    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-w2529\""
Sep 11 00:58:37.429820 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:37.429788    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-8vpg2"]
Sep 11 00:58:37.446740 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:37.446709    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbprv\" (UniqueName: \"kubernetes.io/projected/09d01f73-edc9-4544-bc72-71b393412b8c-kube-api-access-kbprv\") pod \"network-check-source-6f8b768455-8s6fn\" (UID: \"09d01f73-edc9-4544-bc72-71b393412b8c\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-8s6fn"
Sep 11 00:58:37.452428 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:37.452401    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-bb2rz"]
Sep 11 00:58:37.452555 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:37.452534    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-8vpg2"
Sep 11 00:58:37.455128 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:37.455104    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Sep 11 00:58:37.455246 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:37.455156    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Sep 11 00:58:37.455502 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:37.455482    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Sep 11 00:58:37.455585 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:37.455567    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Sep 11 00:58:37.455824 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:37.455807    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-96d7w\""
Sep 11 00:58:37.470862 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:37.470841    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-8vpg2"]
Sep 11 00:58:37.470862 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:37.470856    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-bb2rz"
Sep 11 00:58:37.471034 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:37.470905    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-bb2rz"]
Sep 11 00:58:37.472959 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:37.472943    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Sep 11 00:58:37.473144 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:37.473123    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Sep 11 00:58:37.473252 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:37.473192    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-rmr8h\""
Sep 11 00:58:37.473252 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:37.473205    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Sep 11 00:58:37.547550 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:37.547521    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/dfc463c1-6efd-4f61-ba7a-9353d13b0b55-serving-cert\") pod \"service-ca-operator-cf7c776b7-8vpg2\" (UID: \"dfc463c1-6efd-4f61-ba7a-9353d13b0b55\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-8vpg2"
Sep 11 00:58:37.547664 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:37.547643    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbprv\" (UniqueName: \"kubernetes.io/projected/09d01f73-edc9-4544-bc72-71b393412b8c-kube-api-access-kbprv\") pod \"network-check-source-6f8b768455-8s6fn\" (UID: \"09d01f73-edc9-4544-bc72-71b393412b8c\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-8s6fn"
Sep 11 00:58:37.547719 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:37.547667    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/dfc463c1-6efd-4f61-ba7a-9353d13b0b55-config\") pod \"service-ca-operator-cf7c776b7-8vpg2\" (UID: \"dfc463c1-6efd-4f61-ba7a-9353d13b0b55\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-8vpg2"
Sep 11 00:58:37.547719 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:37.547701    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gtsm2\" (UniqueName: \"kubernetes.io/projected/dfc463c1-6efd-4f61-ba7a-9353d13b0b55-kube-api-access-gtsm2\") pod \"service-ca-operator-cf7c776b7-8vpg2\" (UID: \"dfc463c1-6efd-4f61-ba7a-9353d13b0b55\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-8vpg2"
Sep 11 00:58:37.556757 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:37.556725    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbprv\" (UniqueName: \"kubernetes.io/projected/09d01f73-edc9-4544-bc72-71b393412b8c-kube-api-access-kbprv\") pod \"network-check-source-6f8b768455-8s6fn\" (UID: \"09d01f73-edc9-4544-bc72-71b393412b8c\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-8s6fn"
Sep 11 00:58:37.646653 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:37.646592    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f8b768455-8s6fn"
Sep 11 00:58:37.648879 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:37.648852    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/dfc463c1-6efd-4f61-ba7a-9353d13b0b55-serving-cert\") pod \"service-ca-operator-cf7c776b7-8vpg2\" (UID: \"dfc463c1-6efd-4f61-ba7a-9353d13b0b55\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-8vpg2"
Sep 11 00:58:37.648945 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:37.648900    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rf2df\" (UniqueName: \"kubernetes.io/projected/7cdfae7d-135a-4f4f-b8f5-99cda86837fb-kube-api-access-rf2df\") pod \"cluster-samples-operator-86499c89c4-bb2rz\" (UID: \"7cdfae7d-135a-4f4f-b8f5-99cda86837fb\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-bb2rz"
Sep 11 00:58:37.648986 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:37.648957    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/dfc463c1-6efd-4f61-ba7a-9353d13b0b55-config\") pod \"service-ca-operator-cf7c776b7-8vpg2\" (UID: \"dfc463c1-6efd-4f61-ba7a-9353d13b0b55\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-8vpg2"
Sep 11 00:58:37.648986 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:37.648976    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7cdfae7d-135a-4f4f-b8f5-99cda86837fb-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-bb2rz\" (UID: \"7cdfae7d-135a-4f4f-b8f5-99cda86837fb\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-bb2rz"
Sep 11 00:58:37.649047 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:37.648997    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gtsm2\" (UniqueName: \"kubernetes.io/projected/dfc463c1-6efd-4f61-ba7a-9353d13b0b55-kube-api-access-gtsm2\") pod \"service-ca-operator-cf7c776b7-8vpg2\" (UID: \"dfc463c1-6efd-4f61-ba7a-9353d13b0b55\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-8vpg2"
Sep 11 00:58:37.649476 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:37.649449    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/dfc463c1-6efd-4f61-ba7a-9353d13b0b55-config\") pod \"service-ca-operator-cf7c776b7-8vpg2\" (UID: \"dfc463c1-6efd-4f61-ba7a-9353d13b0b55\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-8vpg2"
Sep 11 00:58:37.650851 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:37.650827    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/dfc463c1-6efd-4f61-ba7a-9353d13b0b55-serving-cert\") pod \"service-ca-operator-cf7c776b7-8vpg2\" (UID: \"dfc463c1-6efd-4f61-ba7a-9353d13b0b55\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-8vpg2"
Sep 11 00:58:37.656536 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:37.656513    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gtsm2\" (UniqueName: \"kubernetes.io/projected/dfc463c1-6efd-4f61-ba7a-9353d13b0b55-kube-api-access-gtsm2\") pod \"service-ca-operator-cf7c776b7-8vpg2\" (UID: \"dfc463c1-6efd-4f61-ba7a-9353d13b0b55\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-8vpg2"
Sep 11 00:58:37.749491 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:37.749459    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rf2df\" (UniqueName: \"kubernetes.io/projected/7cdfae7d-135a-4f4f-b8f5-99cda86837fb-kube-api-access-rf2df\") pod \"cluster-samples-operator-86499c89c4-bb2rz\" (UID: \"7cdfae7d-135a-4f4f-b8f5-99cda86837fb\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-bb2rz"
Sep 11 00:58:37.749613 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:37.749542    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7cdfae7d-135a-4f4f-b8f5-99cda86837fb-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-bb2rz\" (UID: \"7cdfae7d-135a-4f4f-b8f5-99cda86837fb\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-bb2rz"
Sep 11 00:58:37.749654 ip-10-0-129-222 kubenswrapper[2569]: E0911 00:58:37.749625    2569 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 11 00:58:37.749693 ip-10-0-129-222 kubenswrapper[2569]: E0911 00:58:37.749681    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7cdfae7d-135a-4f4f-b8f5-99cda86837fb-samples-operator-tls podName:7cdfae7d-135a-4f4f-b8f5-99cda86837fb nodeName:}" failed. No retries permitted until 2026-09-11 00:58:38.249667131 +0000 UTC m=+52.901041357 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/7cdfae7d-135a-4f4f-b8f5-99cda86837fb-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-bb2rz" (UID: "7cdfae7d-135a-4f4f-b8f5-99cda86837fb") : secret "samples-operator-tls" not found
Sep 11 00:58:37.759030 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:37.759003    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rf2df\" (UniqueName: \"kubernetes.io/projected/7cdfae7d-135a-4f4f-b8f5-99cda86837fb-kube-api-access-rf2df\") pod \"cluster-samples-operator-86499c89c4-bb2rz\" (UID: \"7cdfae7d-135a-4f4f-b8f5-99cda86837fb\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-bb2rz"
Sep 11 00:58:37.761766 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:37.761752    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-8vpg2"
Sep 11 00:58:37.930013 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:37.929939    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-8s6fn"]
Sep 11 00:58:37.932591 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:58:37.932565    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09d01f73_edc9_4544_bc72_71b393412b8c.slice/crio-5b5b08a7807c60fe788ae664615ef77020b6513547bb3c9053e2d1318afe5fff WatchSource:0}: Error finding container 5b5b08a7807c60fe788ae664615ef77020b6513547bb3c9053e2d1318afe5fff: Status 404 returned error can't find the container with id 5b5b08a7807c60fe788ae664615ef77020b6513547bb3c9053e2d1318afe5fff
Sep 11 00:58:38.050658 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:38.050632    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-8vpg2"]
Sep 11 00:58:38.053767 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:58:38.053739    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfc463c1_6efd_4f61_ba7a_9353d13b0b55.slice/crio-e519596ac67b05f0af85b2fd48fda8768f29623276ffa0e9328e18cee521829f WatchSource:0}: Error finding container e519596ac67b05f0af85b2fd48fda8768f29623276ffa0e9328e18cee521829f: Status 404 returned error can't find the container with id e519596ac67b05f0af85b2fd48fda8768f29623276ffa0e9328e18cee521829f
Sep 11 00:58:38.150410 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:38.150374    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-8vpg2" event={"ID":"dfc463c1-6efd-4f61-ba7a-9353d13b0b55","Type":"ContainerStarted","Data":"e519596ac67b05f0af85b2fd48fda8768f29623276ffa0e9328e18cee521829f"}
Sep 11 00:58:38.151387 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:38.151358    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f8b768455-8s6fn" event={"ID":"09d01f73-edc9-4544-bc72-71b393412b8c","Type":"ContainerStarted","Data":"5b5b08a7807c60fe788ae664615ef77020b6513547bb3c9053e2d1318afe5fff"}
Sep 11 00:58:38.253490 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:38.253424    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7cdfae7d-135a-4f4f-b8f5-99cda86837fb-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-bb2rz\" (UID: \"7cdfae7d-135a-4f4f-b8f5-99cda86837fb\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-bb2rz"
Sep 11 00:58:38.253620 ip-10-0-129-222 kubenswrapper[2569]: E0911 00:58:38.253587    2569 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 11 00:58:38.253679 ip-10-0-129-222 kubenswrapper[2569]: E0911 00:58:38.253661    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7cdfae7d-135a-4f4f-b8f5-99cda86837fb-samples-operator-tls podName:7cdfae7d-135a-4f4f-b8f5-99cda86837fb nodeName:}" failed. No retries permitted until 2026-09-11 00:58:39.253641529 +0000 UTC m=+53.905015761 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/7cdfae7d-135a-4f4f-b8f5-99cda86837fb-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-bb2rz" (UID: "7cdfae7d-135a-4f4f-b8f5-99cda86837fb") : secret "samples-operator-tls" not found
Sep 11 00:58:39.261998 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:39.261954    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7cdfae7d-135a-4f4f-b8f5-99cda86837fb-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-bb2rz\" (UID: \"7cdfae7d-135a-4f4f-b8f5-99cda86837fb\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-bb2rz"
Sep 11 00:58:39.262577 ip-10-0-129-222 kubenswrapper[2569]: E0911 00:58:39.262137    2569 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 11 00:58:39.262577 ip-10-0-129-222 kubenswrapper[2569]: E0911 00:58:39.262220    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7cdfae7d-135a-4f4f-b8f5-99cda86837fb-samples-operator-tls podName:7cdfae7d-135a-4f4f-b8f5-99cda86837fb nodeName:}" failed. No retries permitted until 2026-09-11 00:58:41.262198744 +0000 UTC m=+55.913572974 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/7cdfae7d-135a-4f4f-b8f5-99cda86837fb-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-bb2rz" (UID: "7cdfae7d-135a-4f4f-b8f5-99cda86837fb") : secret "samples-operator-tls" not found
Sep 11 00:58:40.158524 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:40.158487    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-8vpg2" event={"ID":"dfc463c1-6efd-4f61-ba7a-9353d13b0b55","Type":"ContainerStarted","Data":"2a608bc2ef81bca14ac3b6bb8b66a166f71c040fc018c5079209c29eb768f74c"}
Sep 11 00:58:40.174932 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:40.174887    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-8vpg2" podStartSLOduration=1.21794288 podStartE2EDuration="3.17485903s" totalImagesPullingTime="1.95691615s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 00:58:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 00:58:40.174071644 +0000 UTC m=+54.825445894" watchObservedRunningTime="2026-09-11 00:58:40.17485903 +0000 UTC m=+54.826233279"
Sep 11 00:58:41.278154 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:41.278111    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7cdfae7d-135a-4f4f-b8f5-99cda86837fb-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-bb2rz\" (UID: \"7cdfae7d-135a-4f4f-b8f5-99cda86837fb\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-bb2rz"
Sep 11 00:58:41.278600 ip-10-0-129-222 kubenswrapper[2569]: E0911 00:58:41.278235    2569 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 11 00:58:41.278600 ip-10-0-129-222 kubenswrapper[2569]: E0911 00:58:41.278311    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7cdfae7d-135a-4f4f-b8f5-99cda86837fb-samples-operator-tls podName:7cdfae7d-135a-4f4f-b8f5-99cda86837fb nodeName:}" failed. No retries permitted until 2026-09-11 00:58:45.278291892 +0000 UTC m=+59.929666125 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/7cdfae7d-135a-4f4f-b8f5-99cda86837fb-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-bb2rz" (UID: "7cdfae7d-135a-4f4f-b8f5-99cda86837fb") : secret "samples-operator-tls" not found
Sep 11 00:58:42.164834 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:42.164797    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f8b768455-8s6fn" event={"ID":"09d01f73-edc9-4544-bc72-71b393412b8c","Type":"ContainerStarted","Data":"02624cbffdf67e59fe3e357b98019b6ee0449362e8eac32912e68f062d4b7e03"}
Sep 11 00:58:42.181144 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:42.181099    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f8b768455-8s6fn" podStartSLOduration=2.001861069 podStartE2EDuration="5.18108257s" totalImagesPullingTime="3.179221501s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 00:58:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 00:58:42.180634 +0000 UTC m=+56.832008248" watchObservedRunningTime="2026-09-11 00:58:42.18108257 +0000 UTC m=+56.832456841"
Sep 11 00:58:42.675887 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:42.675848    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-bt7tj"]
Sep 11 00:58:42.677734 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:42.677718    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-bt7tj"
Sep 11 00:58:42.680009 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:42.679982    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-r8cj7\""
Sep 11 00:58:42.680181 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:42.680166    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Sep 11 00:58:42.680851 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:42.680829    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Sep 11 00:58:42.688952 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:42.688925    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-bt7tj"]
Sep 11 00:58:42.789965 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:42.789942    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xccf4\" (UniqueName: \"kubernetes.io/projected/aa2b1bf9-4803-4325-89bf-fdb1389d3ecb-kube-api-access-xccf4\") pod \"migrator-8b4d697c5-bt7tj\" (UID: \"aa2b1bf9-4803-4325-89bf-fdb1389d3ecb\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-bt7tj"
Sep 11 00:58:42.890923 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:42.890896    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xccf4\" (UniqueName: \"kubernetes.io/projected/aa2b1bf9-4803-4325-89bf-fdb1389d3ecb-kube-api-access-xccf4\") pod \"migrator-8b4d697c5-bt7tj\" (UID: \"aa2b1bf9-4803-4325-89bf-fdb1389d3ecb\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-bt7tj"
Sep 11 00:58:42.899180 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:42.899154    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xccf4\" (UniqueName: \"kubernetes.io/projected/aa2b1bf9-4803-4325-89bf-fdb1389d3ecb-kube-api-access-xccf4\") pod \"migrator-8b4d697c5-bt7tj\" (UID: \"aa2b1bf9-4803-4325-89bf-fdb1389d3ecb\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-bt7tj"
Sep 11 00:58:42.986026 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:42.985965    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-bt7tj"
Sep 11 00:58:43.242056 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:43.241988    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-bmdx6"]
Sep 11 00:58:43.244255 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:43.244238    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-bmdx6"
Sep 11 00:58:43.246700 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:43.246678    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 11 00:58:43.247063 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:43.247041    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 11 00:58:43.247182 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:43.247075    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-swjh6\""
Sep 11 00:58:43.247182 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:43.247048    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 11 00:58:43.247182 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:43.247162    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 11 00:58:43.260068 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:43.260044    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-bt7tj"]
Sep 11 00:58:43.260748 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:43.260729    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-bmdx6"]
Sep 11 00:58:43.263066 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:58:43.263040    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa2b1bf9_4803_4325_89bf_fdb1389d3ecb.slice/crio-1c201eaf24d66963d28912a1d8c667294dfd7cc86f79284aa5aeb741975090b6 WatchSource:0}: Error finding container 1c201eaf24d66963d28912a1d8c667294dfd7cc86f79284aa5aeb741975090b6: Status 404 returned error can't find the container with id 1c201eaf24d66963d28912a1d8c667294dfd7cc86f79284aa5aeb741975090b6
Sep 11 00:58:43.394738 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:43.394710    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ef9445dc-e523-489f-8112-04141c08da1d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bmdx6\" (UID: \"ef9445dc-e523-489f-8112-04141c08da1d\") " pod="openshift-insights/insights-runtime-extractor-bmdx6"
Sep 11 00:58:43.394738 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:43.394742    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ef9445dc-e523-489f-8112-04141c08da1d-crio-socket\") pod \"insights-runtime-extractor-bmdx6\" (UID: \"ef9445dc-e523-489f-8112-04141c08da1d\") " pod="openshift-insights/insights-runtime-extractor-bmdx6"
Sep 11 00:58:43.394936 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:43.394759    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ef9445dc-e523-489f-8112-04141c08da1d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bmdx6\" (UID: \"ef9445dc-e523-489f-8112-04141c08da1d\") " pod="openshift-insights/insights-runtime-extractor-bmdx6"
Sep 11 00:58:43.394936 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:43.394832    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ef9445dc-e523-489f-8112-04141c08da1d-data-volume\") pod \"insights-runtime-extractor-bmdx6\" (UID: \"ef9445dc-e523-489f-8112-04141c08da1d\") " pod="openshift-insights/insights-runtime-extractor-bmdx6"
Sep 11 00:58:43.394936 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:43.394848    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnnz2\" (UniqueName: \"kubernetes.io/projected/ef9445dc-e523-489f-8112-04141c08da1d-kube-api-access-gnnz2\") pod \"insights-runtime-extractor-bmdx6\" (UID: \"ef9445dc-e523-489f-8112-04141c08da1d\") " pod="openshift-insights/insights-runtime-extractor-bmdx6"
Sep 11 00:58:43.496241 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:43.496163    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ef9445dc-e523-489f-8112-04141c08da1d-data-volume\") pod \"insights-runtime-extractor-bmdx6\" (UID: \"ef9445dc-e523-489f-8112-04141c08da1d\") " pod="openshift-insights/insights-runtime-extractor-bmdx6"
Sep 11 00:58:43.496241 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:43.496208    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnnz2\" (UniqueName: \"kubernetes.io/projected/ef9445dc-e523-489f-8112-04141c08da1d-kube-api-access-gnnz2\") pod \"insights-runtime-extractor-bmdx6\" (UID: \"ef9445dc-e523-489f-8112-04141c08da1d\") " pod="openshift-insights/insights-runtime-extractor-bmdx6"
Sep 11 00:58:43.496398 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:43.496385    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ef9445dc-e523-489f-8112-04141c08da1d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bmdx6\" (UID: \"ef9445dc-e523-489f-8112-04141c08da1d\") " pod="openshift-insights/insights-runtime-extractor-bmdx6"
Sep 11 00:58:43.496433 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:43.496412    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ef9445dc-e523-489f-8112-04141c08da1d-crio-socket\") pod \"insights-runtime-extractor-bmdx6\" (UID: \"ef9445dc-e523-489f-8112-04141c08da1d\") " pod="openshift-insights/insights-runtime-extractor-bmdx6"
Sep 11 00:58:43.496433 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:43.496429    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ef9445dc-e523-489f-8112-04141c08da1d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bmdx6\" (UID: \"ef9445dc-e523-489f-8112-04141c08da1d\") " pod="openshift-insights/insights-runtime-extractor-bmdx6"
Sep 11 00:58:43.496493 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:43.496478    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ef9445dc-e523-489f-8112-04141c08da1d-data-volume\") pod \"insights-runtime-extractor-bmdx6\" (UID: \"ef9445dc-e523-489f-8112-04141c08da1d\") " pod="openshift-insights/insights-runtime-extractor-bmdx6"
Sep 11 00:58:43.496545 ip-10-0-129-222 kubenswrapper[2569]: E0911 00:58:43.496529    2569 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 00:58:43.496600 ip-10-0-129-222 kubenswrapper[2569]: E0911 00:58:43.496591    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef9445dc-e523-489f-8112-04141c08da1d-insights-runtime-extractor-tls podName:ef9445dc-e523-489f-8112-04141c08da1d nodeName:}" failed. No retries permitted until 2026-09-11 00:58:43.996577175 +0000 UTC m=+58.647951405 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ef9445dc-e523-489f-8112-04141c08da1d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bmdx6" (UID: "ef9445dc-e523-489f-8112-04141c08da1d") : secret "insights-runtime-extractor-tls" not found
Sep 11 00:58:43.496641 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:43.496599    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ef9445dc-e523-489f-8112-04141c08da1d-crio-socket\") pod \"insights-runtime-extractor-bmdx6\" (UID: \"ef9445dc-e523-489f-8112-04141c08da1d\") " pod="openshift-insights/insights-runtime-extractor-bmdx6"
Sep 11 00:58:43.496641 ip-10-0-129-222 kubenswrapper[2569]: E0911 00:58:43.496529    2569 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 11 00:58:43.496699 ip-10-0-129-222 kubenswrapper[2569]: E0911 00:58:43.496674    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ef9445dc-e523-489f-8112-04141c08da1d-kube-rbac-proxy-cm podName:ef9445dc-e523-489f-8112-04141c08da1d nodeName:}" failed. No retries permitted until 2026-09-11 00:58:43.996661474 +0000 UTC m=+58.648035720 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ef9445dc-e523-489f-8112-04141c08da1d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-bmdx6" (UID: "ef9445dc-e523-489f-8112-04141c08da1d") : configmap "kube-rbac-proxy" not found
Sep 11 00:58:43.509124 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:43.509098    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnnz2\" (UniqueName: \"kubernetes.io/projected/ef9445dc-e523-489f-8112-04141c08da1d-kube-api-access-gnnz2\") pod \"insights-runtime-extractor-bmdx6\" (UID: \"ef9445dc-e523-489f-8112-04141c08da1d\") " pod="openshift-insights/insights-runtime-extractor-bmdx6"
Sep 11 00:58:44.001243 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:44.001209    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ef9445dc-e523-489f-8112-04141c08da1d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bmdx6\" (UID: \"ef9445dc-e523-489f-8112-04141c08da1d\") " pod="openshift-insights/insights-runtime-extractor-bmdx6"
Sep 11 00:58:44.001243 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:44.001244    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ef9445dc-e523-489f-8112-04141c08da1d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bmdx6\" (UID: \"ef9445dc-e523-489f-8112-04141c08da1d\") " pod="openshift-insights/insights-runtime-extractor-bmdx6"
Sep 11 00:58:44.001631 ip-10-0-129-222 kubenswrapper[2569]: E0911 00:58:44.001352    2569 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 11 00:58:44.001631 ip-10-0-129-222 kubenswrapper[2569]: E0911 00:58:44.001354    2569 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 00:58:44.001631 ip-10-0-129-222 kubenswrapper[2569]: E0911 00:58:44.001403    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ef9445dc-e523-489f-8112-04141c08da1d-kube-rbac-proxy-cm podName:ef9445dc-e523-489f-8112-04141c08da1d nodeName:}" failed. No retries permitted until 2026-09-11 00:58:45.00139083 +0000 UTC m=+59.652765057 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ef9445dc-e523-489f-8112-04141c08da1d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-bmdx6" (UID: "ef9445dc-e523-489f-8112-04141c08da1d") : configmap "kube-rbac-proxy" not found
Sep 11 00:58:44.001631 ip-10-0-129-222 kubenswrapper[2569]: E0911 00:58:44.001416    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef9445dc-e523-489f-8112-04141c08da1d-insights-runtime-extractor-tls podName:ef9445dc-e523-489f-8112-04141c08da1d nodeName:}" failed. No retries permitted until 2026-09-11 00:58:45.001409743 +0000 UTC m=+59.652783970 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ef9445dc-e523-489f-8112-04141c08da1d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bmdx6" (UID: "ef9445dc-e523-489f-8112-04141c08da1d") : secret "insights-runtime-extractor-tls" not found
Sep 11 00:58:44.174555 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:44.174509    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-bt7tj" event={"ID":"aa2b1bf9-4803-4325-89bf-fdb1389d3ecb","Type":"ContainerStarted","Data":"1c201eaf24d66963d28912a1d8c667294dfd7cc86f79284aa5aeb741975090b6"}
Sep 11 00:58:45.009703 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:45.009631    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ef9445dc-e523-489f-8112-04141c08da1d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bmdx6\" (UID: \"ef9445dc-e523-489f-8112-04141c08da1d\") " pod="openshift-insights/insights-runtime-extractor-bmdx6"
Sep 11 00:58:45.010095 ip-10-0-129-222 kubenswrapper[2569]: E0911 00:58:45.009754    2569 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 11 00:58:45.010095 ip-10-0-129-222 kubenswrapper[2569]: E0911 00:58:45.009806    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ef9445dc-e523-489f-8112-04141c08da1d-kube-rbac-proxy-cm podName:ef9445dc-e523-489f-8112-04141c08da1d nodeName:}" failed. No retries permitted until 2026-09-11 00:58:47.009791707 +0000 UTC m=+61.661165935 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ef9445dc-e523-489f-8112-04141c08da1d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-bmdx6" (UID: "ef9445dc-e523-489f-8112-04141c08da1d") : configmap "kube-rbac-proxy" not found
Sep 11 00:58:45.010095 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:45.009848    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ef9445dc-e523-489f-8112-04141c08da1d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bmdx6\" (UID: \"ef9445dc-e523-489f-8112-04141c08da1d\") " pod="openshift-insights/insights-runtime-extractor-bmdx6"
Sep 11 00:58:45.010095 ip-10-0-129-222 kubenswrapper[2569]: E0911 00:58:45.009935    2569 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 00:58:45.010095 ip-10-0-129-222 kubenswrapper[2569]: E0911 00:58:45.009966    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef9445dc-e523-489f-8112-04141c08da1d-insights-runtime-extractor-tls podName:ef9445dc-e523-489f-8112-04141c08da1d nodeName:}" failed. No retries permitted until 2026-09-11 00:58:47.009958276 +0000 UTC m=+61.661332504 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ef9445dc-e523-489f-8112-04141c08da1d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bmdx6" (UID: "ef9445dc-e523-489f-8112-04141c08da1d") : secret "insights-runtime-extractor-tls" not found
Sep 11 00:58:45.091412 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:45.091388    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qsqkd"
Sep 11 00:58:45.179021 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:45.178986    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-bt7tj" event={"ID":"aa2b1bf9-4803-4325-89bf-fdb1389d3ecb","Type":"ContainerStarted","Data":"7edfa1facdaf54599dd3e3cd0d4e4d2e5f30bbde97eb3dcfd6b689102c88d294"}
Sep 11 00:58:45.179164 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:45.179037    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-bt7tj" event={"ID":"aa2b1bf9-4803-4325-89bf-fdb1389d3ecb","Type":"ContainerStarted","Data":"2ba75f0c8df981429cd31c53bea30ff65a60d4cb365a87065eac52f6ef3d7796"}
Sep 11 00:58:45.196279 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:45.196233    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-bt7tj" podStartSLOduration=1.764534276 podStartE2EDuration="3.19621745s" totalImagesPullingTime="1.431683174s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 00:58:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 00:58:45.195566418 +0000 UTC m=+59.846940667" watchObservedRunningTime="2026-09-11 00:58:45.19621745 +0000 UTC m=+59.847591691"
Sep 11 00:58:45.311912 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:45.311863    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7cdfae7d-135a-4f4f-b8f5-99cda86837fb-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-bb2rz\" (UID: \"7cdfae7d-135a-4f4f-b8f5-99cda86837fb\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-bb2rz"
Sep 11 00:58:45.312045 ip-10-0-129-222 kubenswrapper[2569]: E0911 00:58:45.311995    2569 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 11 00:58:45.312087 ip-10-0-129-222 kubenswrapper[2569]: E0911 00:58:45.312053    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7cdfae7d-135a-4f4f-b8f5-99cda86837fb-samples-operator-tls podName:7cdfae7d-135a-4f4f-b8f5-99cda86837fb nodeName:}" failed. No retries permitted until 2026-09-11 00:58:53.312039862 +0000 UTC m=+67.963414088 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/7cdfae7d-135a-4f4f-b8f5-99cda86837fb-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-bb2rz" (UID: "7cdfae7d-135a-4f4f-b8f5-99cda86837fb") : secret "samples-operator-tls" not found
Sep 11 00:58:45.812047 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:45.812023    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-2dvjn_c7486f1f-9947-4405-9186-66dfa5621253/dns-node-resolver/0.log"
Sep 11 00:58:47.013549 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:47.013521    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-jrszh_304c1b0f-f5f9-4c4c-8041-ac384ad21516/node-ca/0.log"
Sep 11 00:58:47.025034 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:47.025006    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ef9445dc-e523-489f-8112-04141c08da1d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bmdx6\" (UID: \"ef9445dc-e523-489f-8112-04141c08da1d\") " pod="openshift-insights/insights-runtime-extractor-bmdx6"
Sep 11 00:58:47.025140 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:47.025044    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ef9445dc-e523-489f-8112-04141c08da1d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bmdx6\" (UID: \"ef9445dc-e523-489f-8112-04141c08da1d\") " pod="openshift-insights/insights-runtime-extractor-bmdx6"
Sep 11 00:58:47.025140 ip-10-0-129-222 kubenswrapper[2569]: E0911 00:58:47.025132    2569 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 11 00:58:47.025225 ip-10-0-129-222 kubenswrapper[2569]: E0911 00:58:47.025155    2569 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 00:58:47.025225 ip-10-0-129-222 kubenswrapper[2569]: E0911 00:58:47.025206    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ef9445dc-e523-489f-8112-04141c08da1d-kube-rbac-proxy-cm podName:ef9445dc-e523-489f-8112-04141c08da1d nodeName:}" failed. No retries permitted until 2026-09-11 00:58:51.02518799 +0000 UTC m=+65.676562223 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ef9445dc-e523-489f-8112-04141c08da1d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-bmdx6" (UID: "ef9445dc-e523-489f-8112-04141c08da1d") : configmap "kube-rbac-proxy" not found
Sep 11 00:58:47.025225 ip-10-0-129-222 kubenswrapper[2569]: E0911 00:58:47.025220    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef9445dc-e523-489f-8112-04141c08da1d-insights-runtime-extractor-tls podName:ef9445dc-e523-489f-8112-04141c08da1d nodeName:}" failed. No retries permitted until 2026-09-11 00:58:51.025213859 +0000 UTC m=+65.676588086 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ef9445dc-e523-489f-8112-04141c08da1d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bmdx6" (UID: "ef9445dc-e523-489f-8112-04141c08da1d") : secret "insights-runtime-extractor-tls" not found
Sep 11 00:58:47.812159 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:47.812105    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-c2ph7_f31a4ea6-4a42-40a1-bffa-7d59ce62a268/serve-healthcheck-canary/0.log"
Sep 11 00:58:48.211812 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:48.211725    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-8b4d697c5-bt7tj_aa2b1bf9-4803-4325-89bf-fdb1389d3ecb/migrator/0.log"
Sep 11 00:58:48.411117 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:48.411090    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-8b4d697c5-bt7tj_aa2b1bf9-4803-4325-89bf-fdb1389d3ecb/graceful-termination/0.log"
Sep 11 00:58:51.055464 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:51.055426    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ef9445dc-e523-489f-8112-04141c08da1d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bmdx6\" (UID: \"ef9445dc-e523-489f-8112-04141c08da1d\") " pod="openshift-insights/insights-runtime-extractor-bmdx6"
Sep 11 00:58:51.055846 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:51.055483    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ef9445dc-e523-489f-8112-04141c08da1d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bmdx6\" (UID: \"ef9445dc-e523-489f-8112-04141c08da1d\") " pod="openshift-insights/insights-runtime-extractor-bmdx6"
Sep 11 00:58:51.055846 ip-10-0-129-222 kubenswrapper[2569]: E0911 00:58:51.055564    2569 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 00:58:51.055846 ip-10-0-129-222 kubenswrapper[2569]: E0911 00:58:51.055634    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef9445dc-e523-489f-8112-04141c08da1d-insights-runtime-extractor-tls podName:ef9445dc-e523-489f-8112-04141c08da1d nodeName:}" failed. No retries permitted until 2026-09-11 00:58:59.055621221 +0000 UTC m=+73.706995455 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ef9445dc-e523-489f-8112-04141c08da1d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bmdx6" (UID: "ef9445dc-e523-489f-8112-04141c08da1d") : secret "insights-runtime-extractor-tls" not found
Sep 11 00:58:51.055846 ip-10-0-129-222 kubenswrapper[2569]: E0911 00:58:51.055569    2569 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 11 00:58:51.055846 ip-10-0-129-222 kubenswrapper[2569]: E0911 00:58:51.055698    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ef9445dc-e523-489f-8112-04141c08da1d-kube-rbac-proxy-cm podName:ef9445dc-e523-489f-8112-04141c08da1d nodeName:}" failed. No retries permitted until 2026-09-11 00:58:59.055687007 +0000 UTC m=+73.707061233 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ef9445dc-e523-489f-8112-04141c08da1d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-bmdx6" (UID: "ef9445dc-e523-489f-8112-04141c08da1d") : configmap "kube-rbac-proxy" not found
Sep 11 00:58:51.156140 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:51.156107    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/75b89b12-20de-4813-b407-ee6836d0a777-registry-tls\") pod \"image-registry-6f984758b5-srjjf\" (UID: \"75b89b12-20de-4813-b407-ee6836d0a777\") " pod="openshift-image-registry/image-registry-6f984758b5-srjjf"
Sep 11 00:58:51.156294 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:51.156220    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d10c25a9-1b18-43a7-94f6-ae86285efcec-metrics-tls\") pod \"dns-default-6q5zl\" (UID: \"d10c25a9-1b18-43a7-94f6-ae86285efcec\") " pod="openshift-dns/dns-default-6q5zl"
Sep 11 00:58:51.156294 ip-10-0-129-222 kubenswrapper[2569]: E0911 00:58:51.156260    2569 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 11 00:58:51.156294 ip-10-0-129-222 kubenswrapper[2569]: E0911 00:58:51.156281    2569 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6f984758b5-srjjf: secret "image-registry-tls" not found
Sep 11 00:58:51.156399 ip-10-0-129-222 kubenswrapper[2569]: E0911 00:58:51.156335    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/75b89b12-20de-4813-b407-ee6836d0a777-registry-tls podName:75b89b12-20de-4813-b407-ee6836d0a777 nodeName:}" failed. No retries permitted until 2026-09-11 00:59:23.156320746 +0000 UTC m=+97.807694973 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/75b89b12-20de-4813-b407-ee6836d0a777-registry-tls") pod "image-registry-6f984758b5-srjjf" (UID: "75b89b12-20de-4813-b407-ee6836d0a777") : secret "image-registry-tls" not found
Sep 11 00:58:51.158403 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:51.158385    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d10c25a9-1b18-43a7-94f6-ae86285efcec-metrics-tls\") pod \"dns-default-6q5zl\" (UID: \"d10c25a9-1b18-43a7-94f6-ae86285efcec\") " pod="openshift-dns/dns-default-6q5zl"
Sep 11 00:58:51.422913 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:51.422828    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-q9kdp\""
Sep 11 00:58:51.431079 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:51.431060    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-6q5zl"
Sep 11 00:58:51.559453 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:51.559421    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85c82c9f-52d0-49cd-afd9-b791628e1a50-metrics-certs\") pod \"network-metrics-daemon-r8n96\" (UID: \"85c82c9f-52d0-49cd-afd9-b791628e1a50\") " pod="openshift-multus/network-metrics-daemon-r8n96"
Sep 11 00:58:51.559601 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:51.559514    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d5d763bb-6142-409f-8155-7143a6d91dc6-original-pull-secret\") pod \"global-pull-secret-syncer-qgmsz\" (UID: \"d5d763bb-6142-409f-8155-7143a6d91dc6\") " pod="kube-system/global-pull-secret-syncer-qgmsz"
Sep 11 00:58:51.561907 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:51.561867    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 11 00:58:51.562011 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:51.561914    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 11 00:58:51.569746 ip-10-0-129-222 kubenswrapper[2569]: E0911 00:58:51.569727    2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 11 00:58:51.569800 ip-10-0-129-222 kubenswrapper[2569]: E0911 00:58:51.569785    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85c82c9f-52d0-49cd-afd9-b791628e1a50-metrics-certs podName:85c82c9f-52d0-49cd-afd9-b791628e1a50 nodeName:}" failed. No retries permitted until 2026-09-11 00:59:55.56977123 +0000 UTC m=+130.221145457 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/85c82c9f-52d0-49cd-afd9-b791628e1a50-metrics-certs") pod "network-metrics-daemon-r8n96" (UID: "85c82c9f-52d0-49cd-afd9-b791628e1a50") : secret "metrics-daemon-secret" not found
Sep 11 00:58:51.571740 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:51.571723    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d5d763bb-6142-409f-8155-7143a6d91dc6-original-pull-secret\") pod \"global-pull-secret-syncer-qgmsz\" (UID: \"d5d763bb-6142-409f-8155-7143a6d91dc6\") " pod="kube-system/global-pull-secret-syncer-qgmsz"
Sep 11 00:58:51.660554 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:51.660528    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwg5s\" (UniqueName: \"kubernetes.io/projected/239b25bd-4932-4b5b-b286-45d3e2ff9f1c-kube-api-access-qwg5s\") pod \"network-check-target-bbbq6\" (UID: \"239b25bd-4932-4b5b-b286-45d3e2ff9f1c\") " pod="openshift-network-diagnostics/network-check-target-bbbq6"
Sep 11 00:58:51.662919 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:51.662898    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwg5s\" (UniqueName: \"kubernetes.io/projected/239b25bd-4932-4b5b-b286-45d3e2ff9f1c-kube-api-access-qwg5s\") pod \"network-check-target-bbbq6\" (UID: \"239b25bd-4932-4b5b-b286-45d3e2ff9f1c\") " pod="openshift-network-diagnostics/network-check-target-bbbq6"
Sep 11 00:58:51.737713 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:51.737651    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-6q5zl"]
Sep 11 00:58:51.740717 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:58:51.740694    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd10c25a9_1b18_43a7_94f6_ae86285efcec.slice/crio-f897c2fc7dfba2749cf35abc612a812bf8ea6ca83dda446dc3413436acce5057 WatchSource:0}: Error finding container f897c2fc7dfba2749cf35abc612a812bf8ea6ca83dda446dc3413436acce5057: Status 404 returned error can't find the container with id f897c2fc7dfba2749cf35abc612a812bf8ea6ca83dda446dc3413436acce5057
Sep 11 00:58:51.782660 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:51.782636    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qgmsz"
Sep 11 00:58:51.791108 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:51.791083    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-jwmm5\""
Sep 11 00:58:51.798880 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:51.798855    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bbbq6"
Sep 11 00:58:52.081316 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:52.081284    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-qgmsz"]
Sep 11 00:58:52.084134 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:58:52.084108    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5d763bb_6142_409f_8155_7143a6d91dc6.slice/crio-c769f54482355abe247e689720cdc9d9771fca483599b477674cea5a5669a77e WatchSource:0}: Error finding container c769f54482355abe247e689720cdc9d9771fca483599b477674cea5a5669a77e: Status 404 returned error can't find the container with id c769f54482355abe247e689720cdc9d9771fca483599b477674cea5a5669a77e
Sep 11 00:58:52.101555 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:52.101355    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-bbbq6"]
Sep 11 00:58:52.104586 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:58:52.104562    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod239b25bd_4932_4b5b_b286_45d3e2ff9f1c.slice/crio-4b900615b51f0f54f9ba039b298e378675abf68b2eb4aeea312ca63e53927ad6 WatchSource:0}: Error finding container 4b900615b51f0f54f9ba039b298e378675abf68b2eb4aeea312ca63e53927ad6: Status 404 returned error can't find the container with id 4b900615b51f0f54f9ba039b298e378675abf68b2eb4aeea312ca63e53927ad6
Sep 11 00:58:52.196735 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:52.196709    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-6q5zl" event={"ID":"d10c25a9-1b18-43a7-94f6-ae86285efcec","Type":"ContainerStarted","Data":"f897c2fc7dfba2749cf35abc612a812bf8ea6ca83dda446dc3413436acce5057"}
Sep 11 00:58:52.197847 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:52.197818    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-bbbq6" event={"ID":"239b25bd-4932-4b5b-b286-45d3e2ff9f1c","Type":"ContainerStarted","Data":"4b900615b51f0f54f9ba039b298e378675abf68b2eb4aeea312ca63e53927ad6"}
Sep 11 00:58:52.198969 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:52.198937    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-qgmsz" event={"ID":"d5d763bb-6142-409f-8155-7143a6d91dc6","Type":"ContainerStarted","Data":"c769f54482355abe247e689720cdc9d9771fca483599b477674cea5a5669a77e"}
Sep 11 00:58:53.205022 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:53.204987    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-bbbq6" event={"ID":"239b25bd-4932-4b5b-b286-45d3e2ff9f1c","Type":"ContainerStarted","Data":"875753f05a1f3be968516be031e7d28dffcc21602f9477a451d63c06f9bc96b9"}
Sep 11 00:58:53.205597 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:53.205148    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-bbbq6"
Sep 11 00:58:53.221605 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:53.221545    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-bbbq6" podStartSLOduration=67.221524869 podStartE2EDuration="1m7.221524869s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 00:57:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 00:58:53.219959946 +0000 UTC m=+67.871334197" watchObservedRunningTime="2026-09-11 00:58:53.221524869 +0000 UTC m=+67.872899118"
Sep 11 00:58:53.374076 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:53.374028    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7cdfae7d-135a-4f4f-b8f5-99cda86837fb-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-bb2rz\" (UID: \"7cdfae7d-135a-4f4f-b8f5-99cda86837fb\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-bb2rz"
Sep 11 00:58:53.377110 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:53.377078    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7cdfae7d-135a-4f4f-b8f5-99cda86837fb-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-bb2rz\" (UID: \"7cdfae7d-135a-4f4f-b8f5-99cda86837fb\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-bb2rz"
Sep 11 00:58:53.381138 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:53.381112    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-rmr8h\""
Sep 11 00:58:53.389205 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:53.389178    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-bb2rz"
Sep 11 00:58:53.813564 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:53.813534    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-bb2rz"]
Sep 11 00:58:54.210845 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:54.210806    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-bb2rz" event={"ID":"7cdfae7d-135a-4f4f-b8f5-99cda86837fb","Type":"ContainerStarted","Data":"1fb10ca39169994d8fccba6381a39bf7af905cdceb0e274cdf0faf10516abb3a"}
Sep 11 00:58:54.212928 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:54.212900    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-6q5zl" event={"ID":"d10c25a9-1b18-43a7-94f6-ae86285efcec","Type":"ContainerStarted","Data":"c7e3dd86ce3af9b0aa275744eae5c295f76ae8b292425b2758dc35824d42d5b6"}
Sep 11 00:58:54.213072 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:54.212951    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-6q5zl" event={"ID":"d10c25a9-1b18-43a7-94f6-ae86285efcec","Type":"ContainerStarted","Data":"b8b2a86d4466ea8833475a161916485f42d1c27739219e0a1402ab99f1a10240"}
Sep 11 00:58:54.231468 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:54.231352    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-6q5zl" podStartSLOduration=33.381983576 podStartE2EDuration="35.231335678s" totalImagesPullingTime="1.849352102s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 00:58:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 00:58:54.230532641 +0000 UTC m=+68.881906892" watchObservedRunningTime="2026-09-11 00:58:54.231335678 +0000 UTC m=+68.882709928"
Sep 11 00:58:55.217143 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:55.217108    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-6q5zl"
Sep 11 00:58:57.224924 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:57.224885    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-qgmsz" event={"ID":"d5d763bb-6142-409f-8155-7143a6d91dc6","Type":"ContainerStarted","Data":"519774be54c2f4d5742c96c39bf87a06527fc5ebc1ab190e750d295685184adb"}
Sep 11 00:58:57.226749 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:57.226727    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-bb2rz" event={"ID":"7cdfae7d-135a-4f4f-b8f5-99cda86837fb","Type":"ContainerStarted","Data":"9e1d92aab653b730b9454a760552943406d4119621217a1ed0b80a6b997b11b1"}
Sep 11 00:58:57.226887 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:57.226767    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-bb2rz" event={"ID":"7cdfae7d-135a-4f4f-b8f5-99cda86837fb","Type":"ContainerStarted","Data":"fc8e197d3b98056463c8eb0c676ca9317ad34b6dc43cb5f4cd26a7e488f9df66"}
Sep 11 00:58:57.239950 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:57.239907    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-qgmsz" podStartSLOduration=66.62176963 podStartE2EDuration="1m11.239895124s" totalImagesPullingTime="4.618125494s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 00:57:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 00:58:57.238493089 +0000 UTC m=+71.889867351" watchObservedRunningTime="2026-09-11 00:58:57.239895124 +0000 UTC m=+71.891269424"
Sep 11 00:58:57.253746 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:57.253709    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-bb2rz" podStartSLOduration=17.407963132 podStartE2EDuration="20.253699484s" totalImagesPullingTime="2.845736352s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 00:58:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 00:58:57.252830605 +0000 UTC m=+71.904204854" watchObservedRunningTime="2026-09-11 00:58:57.253699484 +0000 UTC m=+71.905073732"
Sep 11 00:58:59.118266 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:59.118232    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ef9445dc-e523-489f-8112-04141c08da1d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bmdx6\" (UID: \"ef9445dc-e523-489f-8112-04141c08da1d\") " pod="openshift-insights/insights-runtime-extractor-bmdx6"
Sep 11 00:58:59.118266 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:59.118269    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ef9445dc-e523-489f-8112-04141c08da1d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bmdx6\" (UID: \"ef9445dc-e523-489f-8112-04141c08da1d\") " pod="openshift-insights/insights-runtime-extractor-bmdx6"
Sep 11 00:58:59.118735 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:59.118719    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ef9445dc-e523-489f-8112-04141c08da1d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bmdx6\" (UID: \"ef9445dc-e523-489f-8112-04141c08da1d\") " pod="openshift-insights/insights-runtime-extractor-bmdx6"
Sep 11 00:58:59.120473 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:59.120456    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ef9445dc-e523-489f-8112-04141c08da1d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bmdx6\" (UID: \"ef9445dc-e523-489f-8112-04141c08da1d\") " pod="openshift-insights/insights-runtime-extractor-bmdx6"
Sep 11 00:58:59.155232 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:59.155208    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-swjh6\""
Sep 11 00:58:59.163932 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:59.163914    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-bmdx6"
Sep 11 00:58:59.434322 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:58:59.434246    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-bmdx6"]
Sep 11 00:58:59.437538 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:58:59.437511    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef9445dc_e523_489f_8112_04141c08da1d.slice/crio-f87204225f13427b0f5c94ca3a1f96d524e451cb688161f84448fd99bf69fdc9 WatchSource:0}: Error finding container f87204225f13427b0f5c94ca3a1f96d524e451cb688161f84448fd99bf69fdc9: Status 404 returned error can't find the container with id f87204225f13427b0f5c94ca3a1f96d524e451cb688161f84448fd99bf69fdc9
Sep 11 00:59:00.237585 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:00.237555    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bmdx6" event={"ID":"ef9445dc-e523-489f-8112-04141c08da1d","Type":"ContainerStarted","Data":"735de078e01ed27a49457f2f0b3c707119b98e20fb87ae7eaafccf7f4aace0ff"}
Sep 11 00:59:00.237924 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:00.237609    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bmdx6" event={"ID":"ef9445dc-e523-489f-8112-04141c08da1d","Type":"ContainerStarted","Data":"7559893d7d56bcc7877b4fd4bf6a41d2f937bcd08b83e218b31f3309ad67fb9f"}
Sep 11 00:59:00.237924 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:00.237629    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bmdx6" event={"ID":"ef9445dc-e523-489f-8112-04141c08da1d","Type":"ContainerStarted","Data":"f87204225f13427b0f5c94ca3a1f96d524e451cb688161f84448fd99bf69fdc9"}
Sep 11 00:59:02.246055 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:02.246017    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bmdx6" event={"ID":"ef9445dc-e523-489f-8112-04141c08da1d","Type":"ContainerStarted","Data":"da39217c4f2c1363f3ca520c90426f5f5d9ad18ffc9bc192878a98e67a78d9f2"}
Sep 11 00:59:02.263625 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:02.263576    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-bmdx6" podStartSLOduration=17.295044963 podStartE2EDuration="19.263563711s" totalImagesPullingTime="1.968518748s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 00:58:43 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 00:59:02.262728307 +0000 UTC m=+76.914102558" watchObservedRunningTime="2026-09-11 00:59:02.263563711 +0000 UTC m=+76.914937959"
Sep 11 00:59:05.222310 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:05.222278    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-6q5zl"
Sep 11 00:59:06.755099 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:06.755065    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7cc7bc476c-pscjt"]
Sep 11 00:59:06.758417 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:06.758387    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7cc7bc476c-pscjt"
Sep 11 00:59:06.762269 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:06.762246    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 11 00:59:06.763230 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:06.763212    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Sep 11 00:59:06.763348 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:06.763233    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 11 00:59:06.763400 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:06.763371    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 11 00:59:06.772950 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:06.772923    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7cc7bc476c-pscjt"]
Sep 11 00:59:06.874988 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:06.874962    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66wnq\" (UniqueName: \"kubernetes.io/projected/2f36dfe0-196b-4c56-9d8f-0d3722108688-kube-api-access-66wnq\") pod \"klusterlet-addon-workmgr-7cc7bc476c-pscjt\" (UID: \"2f36dfe0-196b-4c56-9d8f-0d3722108688\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7cc7bc476c-pscjt"
Sep 11 00:59:06.875124 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:06.875013    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2f36dfe0-196b-4c56-9d8f-0d3722108688-tmp\") pod \"klusterlet-addon-workmgr-7cc7bc476c-pscjt\" (UID: \"2f36dfe0-196b-4c56-9d8f-0d3722108688\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7cc7bc476c-pscjt"
Sep 11 00:59:06.875124 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:06.875040    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/2f36dfe0-196b-4c56-9d8f-0d3722108688-klusterlet-config\") pod \"klusterlet-addon-workmgr-7cc7bc476c-pscjt\" (UID: \"2f36dfe0-196b-4c56-9d8f-0d3722108688\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7cc7bc476c-pscjt"
Sep 11 00:59:06.891239 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:06.891214    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-76f99584f4-j6gxg"]
Sep 11 00:59:06.894218 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:06.894204    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-76f99584f4-j6gxg"
Sep 11 00:59:06.897951 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:06.897934    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 11 00:59:06.898060 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:06.898012    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 11 00:59:06.898399 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:06.898385    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-qjkh8\""
Sep 11 00:59:06.913294 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:06.913271    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-76f99584f4-j6gxg"]
Sep 11 00:59:06.975801 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:06.975772    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66wnq\" (UniqueName: \"kubernetes.io/projected/2f36dfe0-196b-4c56-9d8f-0d3722108688-kube-api-access-66wnq\") pod \"klusterlet-addon-workmgr-7cc7bc476c-pscjt\" (UID: \"2f36dfe0-196b-4c56-9d8f-0d3722108688\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7cc7bc476c-pscjt"
Sep 11 00:59:06.975969 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:06.975820    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2f36dfe0-196b-4c56-9d8f-0d3722108688-tmp\") pod \"klusterlet-addon-workmgr-7cc7bc476c-pscjt\" (UID: \"2f36dfe0-196b-4c56-9d8f-0d3722108688\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7cc7bc476c-pscjt"
Sep 11 00:59:06.975969 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:06.975846    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/2f36dfe0-196b-4c56-9d8f-0d3722108688-klusterlet-config\") pod \"klusterlet-addon-workmgr-7cc7bc476c-pscjt\" (UID: \"2f36dfe0-196b-4c56-9d8f-0d3722108688\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7cc7bc476c-pscjt"
Sep 11 00:59:06.976097 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:06.975973    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h66rd\" (UniqueName: \"kubernetes.io/projected/722ada5b-0930-4b9b-8752-49be36c404b4-kube-api-access-h66rd\") pod \"downloads-76f99584f4-j6gxg\" (UID: \"722ada5b-0930-4b9b-8752-49be36c404b4\") " pod="openshift-console/downloads-76f99584f4-j6gxg"
Sep 11 00:59:06.976276 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:06.976255    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2f36dfe0-196b-4c56-9d8f-0d3722108688-tmp\") pod \"klusterlet-addon-workmgr-7cc7bc476c-pscjt\" (UID: \"2f36dfe0-196b-4c56-9d8f-0d3722108688\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7cc7bc476c-pscjt"
Sep 11 00:59:06.978294 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:06.978277    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/2f36dfe0-196b-4c56-9d8f-0d3722108688-klusterlet-config\") pod \"klusterlet-addon-workmgr-7cc7bc476c-pscjt\" (UID: \"2f36dfe0-196b-4c56-9d8f-0d3722108688\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7cc7bc476c-pscjt"
Sep 11 00:59:06.985999 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:06.985982    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-66wnq\" (UniqueName: \"kubernetes.io/projected/2f36dfe0-196b-4c56-9d8f-0d3722108688-kube-api-access-66wnq\") pod \"klusterlet-addon-workmgr-7cc7bc476c-pscjt\" (UID: \"2f36dfe0-196b-4c56-9d8f-0d3722108688\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7cc7bc476c-pscjt"
Sep 11 00:59:07.068067 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:07.068042    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7cc7bc476c-pscjt"
Sep 11 00:59:07.076390 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:07.076366    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h66rd\" (UniqueName: \"kubernetes.io/projected/722ada5b-0930-4b9b-8752-49be36c404b4-kube-api-access-h66rd\") pod \"downloads-76f99584f4-j6gxg\" (UID: \"722ada5b-0930-4b9b-8752-49be36c404b4\") " pod="openshift-console/downloads-76f99584f4-j6gxg"
Sep 11 00:59:07.084592 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:07.084569    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h66rd\" (UniqueName: \"kubernetes.io/projected/722ada5b-0930-4b9b-8752-49be36c404b4-kube-api-access-h66rd\") pod \"downloads-76f99584f4-j6gxg\" (UID: \"722ada5b-0930-4b9b-8752-49be36c404b4\") " pod="openshift-console/downloads-76f99584f4-j6gxg"
Sep 11 00:59:07.202580 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:07.202531    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-76f99584f4-j6gxg"
Sep 11 00:59:07.334374 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:07.334309    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7cc7bc476c-pscjt"]
Sep 11 00:59:07.337420 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:59:07.337392    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f36dfe0_196b_4c56_9d8f_0d3722108688.slice/crio-54dd843e48f7c3f8b2af8b94f9c596e3e56453fc7e97ba259b5bddb9bf3cc65d WatchSource:0}: Error finding container 54dd843e48f7c3f8b2af8b94f9c596e3e56453fc7e97ba259b5bddb9bf3cc65d: Status 404 returned error can't find the container with id 54dd843e48f7c3f8b2af8b94f9c596e3e56453fc7e97ba259b5bddb9bf3cc65d
Sep 11 00:59:07.470812 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:07.470792    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-76f99584f4-j6gxg"]
Sep 11 00:59:07.473054 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:59:07.473036    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod722ada5b_0930_4b9b_8752_49be36c404b4.slice/crio-5728910933dfa10261f571a892cfca52fd1cb1ae551728f08ff75dbc2dd92743 WatchSource:0}: Error finding container 5728910933dfa10261f571a892cfca52fd1cb1ae551728f08ff75dbc2dd92743: Status 404 returned error can't find the container with id 5728910933dfa10261f571a892cfca52fd1cb1ae551728f08ff75dbc2dd92743
Sep 11 00:59:08.268475 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:08.268333    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7cc7bc476c-pscjt" event={"ID":"2f36dfe0-196b-4c56-9d8f-0d3722108688","Type":"ContainerStarted","Data":"54dd843e48f7c3f8b2af8b94f9c596e3e56453fc7e97ba259b5bddb9bf3cc65d"}
Sep 11 00:59:08.270594 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:08.270559    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-76f99584f4-j6gxg" event={"ID":"722ada5b-0930-4b9b-8752-49be36c404b4","Type":"ContainerStarted","Data":"5728910933dfa10261f571a892cfca52fd1cb1ae551728f08ff75dbc2dd92743"}
Sep 11 00:59:12.289755 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:12.289712    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7cc7bc476c-pscjt" event={"ID":"2f36dfe0-196b-4c56-9d8f-0d3722108688","Type":"ContainerStarted","Data":"d8234f90990820c8f50d68bb1eb47b6b911c53f47be1317722789fefa26164d1"}
Sep 11 00:59:12.290221 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:12.290065    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7cc7bc476c-pscjt"
Sep 11 00:59:12.291783 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:12.291760    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7cc7bc476c-pscjt"
Sep 11 00:59:12.306476 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:12.306430    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7cc7bc476c-pscjt" podStartSLOduration=2.193444845 podStartE2EDuration="6.306414693s" totalImagesPullingTime="4.112969848s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 00:59:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 00:59:12.305792273 +0000 UTC m=+86.957166520" watchObservedRunningTime="2026-09-11 00:59:12.306414693 +0000 UTC m=+86.957788943"
Sep 11 00:59:12.921265 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:12.921234    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-ctjfd"]
Sep 11 00:59:12.924657 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:12.924633    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-ctjfd"
Sep 11 00:59:12.927007 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:12.926982    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Sep 11 00:59:12.927170 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:12.927151    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-kh24d\""
Sep 11 00:59:12.932090 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:12.932066    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-ctjfd"]
Sep 11 00:59:13.024287 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:13.024252    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/735598ab-461b-4145-824b-b197ee49967e-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-ctjfd\" (UID: \"735598ab-461b-4145-824b-b197ee49967e\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-ctjfd"
Sep 11 00:59:13.124812 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:13.124781    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/735598ab-461b-4145-824b-b197ee49967e-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-ctjfd\" (UID: \"735598ab-461b-4145-824b-b197ee49967e\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-ctjfd"
Sep 11 00:59:13.124997 ip-10-0-129-222 kubenswrapper[2569]: E0911 00:59:13.124922    2569 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found
Sep 11 00:59:13.124997 ip-10-0-129-222 kubenswrapper[2569]: E0911 00:59:13.124992    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/735598ab-461b-4145-824b-b197ee49967e-tls-certificates podName:735598ab-461b-4145-824b-b197ee49967e nodeName:}" failed. No retries permitted until 2026-09-11 00:59:13.62497381 +0000 UTC m=+88.276348041 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/735598ab-461b-4145-824b-b197ee49967e-tls-certificates") pod "prometheus-operator-admission-webhook-5864cd99d7-ctjfd" (UID: "735598ab-461b-4145-824b-b197ee49967e") : secret "prometheus-operator-admission-webhook-tls" not found
Sep 11 00:59:13.628972 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:13.628941    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/735598ab-461b-4145-824b-b197ee49967e-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-ctjfd\" (UID: \"735598ab-461b-4145-824b-b197ee49967e\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-ctjfd"
Sep 11 00:59:13.631812 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:13.631785    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/735598ab-461b-4145-824b-b197ee49967e-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-ctjfd\" (UID: \"735598ab-461b-4145-824b-b197ee49967e\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-ctjfd"
Sep 11 00:59:13.835674 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:13.835635    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-ctjfd"
Sep 11 00:59:14.110744 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:14.110714    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-ctjfd"]
Sep 11 00:59:14.114580 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:59:14.114553    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod735598ab_461b_4145_824b_b197ee49967e.slice/crio-f698ab4b0c51955ac1f64969636ececc6af54b22effd76c223925b68e4f664ac WatchSource:0}: Error finding container f698ab4b0c51955ac1f64969636ececc6af54b22effd76c223925b68e4f664ac: Status 404 returned error can't find the container with id f698ab4b0c51955ac1f64969636ececc6af54b22effd76c223925b68e4f664ac
Sep 11 00:59:14.300115 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:14.300028    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-ctjfd" event={"ID":"735598ab-461b-4145-824b-b197ee49967e","Type":"ContainerStarted","Data":"f698ab4b0c51955ac1f64969636ececc6af54b22effd76c223925b68e4f664ac"}
Sep 11 00:59:16.308655 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:16.308620    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-ctjfd" event={"ID":"735598ab-461b-4145-824b-b197ee49967e","Type":"ContainerStarted","Data":"a98f54d34aa5fb100747ba216a459e363ba0b38dc62ca45a24623dd4d090d165"}
Sep 11 00:59:16.309047 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:16.308820    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-ctjfd"
Sep 11 00:59:16.314857 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:16.314832    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-ctjfd"
Sep 11 00:59:16.323275 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:16.323236    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-ctjfd" podStartSLOduration=2.819742152 podStartE2EDuration="4.323219929s" totalImagesPullingTime="1.503477777s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 00:59:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 00:59:16.32237098 +0000 UTC m=+90.973745240" watchObservedRunningTime="2026-09-11 00:59:16.323219929 +0000 UTC m=+90.974594174"
Sep 11 00:59:21.344427 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:21.344390    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-zl6np"]
Sep 11 00:59:21.347195 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:21.347174    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-zl6np"
Sep 11 00:59:21.350793 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:21.350734    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Sep 11 00:59:21.350793 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:21.350748    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-nsrvp\""
Sep 11 00:59:21.351076 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:21.350833    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 11 00:59:21.351076 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:21.350836    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 11 00:59:21.351076 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:21.350893    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Sep 11 00:59:21.351076 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:21.350748    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 11 00:59:21.359675 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:21.359653    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-pqgn8"]
Sep 11 00:59:21.362042 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:21.361986    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-pqgn8"
Sep 11 00:59:21.363348 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:21.363325    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-zl6np"]
Sep 11 00:59:21.365050 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:21.365026    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 11 00:59:21.365691 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:21.365640    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 11 00:59:21.365691 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:21.365644    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-t5pv6\""
Sep 11 00:59:21.366383 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:21.366363    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 11 00:59:21.379110 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:21.379089    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-v9pqc"]
Sep 11 00:59:21.381255 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:21.381235    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-v9pqc"
Sep 11 00:59:21.383462 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:21.383442    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Sep 11 00:59:21.383573 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:21.383535    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-prgv5\""
Sep 11 00:59:21.383633 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:21.383593    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Sep 11 00:59:21.383910 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:21.383890    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Sep 11 00:59:21.393353 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:21.393330    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-v9pqc"]
Sep 11 00:59:21.393436 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:21.393400    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c9757f20-e580-4d80-aa5f-2f148ce4690d-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-zl6np\" (UID: \"c9757f20-e580-4d80-aa5f-2f148ce4690d\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-zl6np"
Sep 11 00:59:21.393490 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:21.393470    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xt474\" (UniqueName: \"kubernetes.io/projected/c9757f20-e580-4d80-aa5f-2f148ce4690d-kube-api-access-xt474\") pod \"openshift-state-metrics-67dcf8c979-zl6np\" (UID: \"c9757f20-e580-4d80-aa5f-2f148ce4690d\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-zl6np"
Sep 11 00:59:21.393536 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:21.393522    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c9757f20-e580-4d80-aa5f-2f148ce4690d-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-zl6np\" (UID: \"c9757f20-e580-4d80-aa5f-2f148ce4690d\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-zl6np"
Sep 11 00:59:21.393579 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:21.393545    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c9757f20-e580-4d80-aa5f-2f148ce4690d-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-zl6np\" (UID: \"c9757f20-e580-4d80-aa5f-2f148ce4690d\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-zl6np"
Sep 11 00:59:21.494615 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:21.494518    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/78562e6b-b985-4098-a024-2d2b21e73fe3-node-exporter-accelerators-collector-config\") pod \"node-exporter-pqgn8\" (UID: \"78562e6b-b985-4098-a024-2d2b21e73fe3\") " pod="openshift-monitoring/node-exporter-pqgn8"
Sep 11 00:59:21.494615 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:21.494565    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/78562e6b-b985-4098-a024-2d2b21e73fe3-node-exporter-wtmp\") pod \"node-exporter-pqgn8\" (UID: \"78562e6b-b985-4098-a024-2d2b21e73fe3\") " pod="openshift-monitoring/node-exporter-pqgn8"
Sep 11 00:59:21.494615 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:21.494605    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c9757f20-e580-4d80-aa5f-2f148ce4690d-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-zl6np\" (UID: \"c9757f20-e580-4d80-aa5f-2f148ce4690d\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-zl6np"
Sep 11 00:59:21.494939 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:21.494639    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/f7a5dad1-f163-4920-b8a4-3ab9e37de477-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-v9pqc\" (UID: \"f7a5dad1-f163-4920-b8a4-3ab9e37de477\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-v9pqc"
Sep 11 00:59:21.494939 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:21.494682    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/78562e6b-b985-4098-a024-2d2b21e73fe3-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-pqgn8\" (UID: \"78562e6b-b985-4098-a024-2d2b21e73fe3\") " pod="openshift-monitoring/node-exporter-pqgn8"
Sep 11 00:59:21.494939 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:21.494714    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vrq4\" (UniqueName: \"kubernetes.io/projected/78562e6b-b985-4098-a024-2d2b21e73fe3-kube-api-access-7vrq4\") pod \"node-exporter-pqgn8\" (UID: \"78562e6b-b985-4098-a024-2d2b21e73fe3\") " pod="openshift-monitoring/node-exporter-pqgn8"
Sep 11 00:59:21.494939 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:21.494736    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/78562e6b-b985-4098-a024-2d2b21e73fe3-sys\") pod \"node-exporter-pqgn8\" (UID: \"78562e6b-b985-4098-a024-2d2b21e73fe3\") " pod="openshift-monitoring/node-exporter-pqgn8"
Sep 11 00:59:21.494939 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:21.494762    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f7a5dad1-f163-4920-b8a4-3ab9e37de477-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-v9pqc\" (UID: \"f7a5dad1-f163-4920-b8a4-3ab9e37de477\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-v9pqc"
Sep 11 00:59:21.494939 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:21.494812    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xt474\" (UniqueName: \"kubernetes.io/projected/c9757f20-e580-4d80-aa5f-2f148ce4690d-kube-api-access-xt474\") pod \"openshift-state-metrics-67dcf8c979-zl6np\" (UID: \"c9757f20-e580-4d80-aa5f-2f148ce4690d\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-zl6np"
Sep 11 00:59:21.494939 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:21.494841    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/78562e6b-b985-4098-a024-2d2b21e73fe3-node-exporter-textfile\") pod \"node-exporter-pqgn8\" (UID: \"78562e6b-b985-4098-a024-2d2b21e73fe3\") " pod="openshift-monitoring/node-exporter-pqgn8"
Sep 11 00:59:21.494939 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:21.494867    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f7a5dad1-f163-4920-b8a4-3ab9e37de477-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-v9pqc\" (UID: \"f7a5dad1-f163-4920-b8a4-3ab9e37de477\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-v9pqc"
Sep 11 00:59:21.494939 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:21.494917    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c9757f20-e580-4d80-aa5f-2f148ce4690d-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-zl6np\" (UID: \"c9757f20-e580-4d80-aa5f-2f148ce4690d\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-zl6np"
Sep 11 00:59:21.494939 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:21.494939    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c9757f20-e580-4d80-aa5f-2f148ce4690d-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-zl6np\" (UID: \"c9757f20-e580-4d80-aa5f-2f148ce4690d\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-zl6np"
Sep 11 00:59:21.495359 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:21.494974    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/f7a5dad1-f163-4920-b8a4-3ab9e37de477-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-v9pqc\" (UID: \"f7a5dad1-f163-4920-b8a4-3ab9e37de477\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-v9pqc"
Sep 11 00:59:21.495359 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:21.494999    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/78562e6b-b985-4098-a024-2d2b21e73fe3-metrics-client-ca\") pod \"node-exporter-pqgn8\" (UID: \"78562e6b-b985-4098-a024-2d2b21e73fe3\") " pod="openshift-monitoring/node-exporter-pqgn8"
Sep 11 00:59:21.495359 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:21.495024    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/78562e6b-b985-4098-a024-2d2b21e73fe3-root\") pod \"node-exporter-pqgn8\" (UID: \"78562e6b-b985-4098-a024-2d2b21e73fe3\") " pod="openshift-monitoring/node-exporter-pqgn8"
Sep 11 00:59:21.495359 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:21.495047    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xm9t6\" (UniqueName: \"kubernetes.io/projected/f7a5dad1-f163-4920-b8a4-3ab9e37de477-kube-api-access-xm9t6\") pod \"kube-state-metrics-59fbdd77d-v9pqc\" (UID: \"f7a5dad1-f163-4920-b8a4-3ab9e37de477\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-v9pqc"
Sep 11 00:59:21.495359 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:21.495072    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f7a5dad1-f163-4920-b8a4-3ab9e37de477-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-v9pqc\" (UID: \"f7a5dad1-f163-4920-b8a4-3ab9e37de477\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-v9pqc"
Sep 11 00:59:21.495359 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:21.495097    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/78562e6b-b985-4098-a024-2d2b21e73fe3-node-exporter-tls\") pod \"node-exporter-pqgn8\" (UID: \"78562e6b-b985-4098-a024-2d2b21e73fe3\") " pod="openshift-monitoring/node-exporter-pqgn8"
Sep 11 00:59:21.495359 ip-10-0-129-222 kubenswrapper[2569]: E0911 00:59:21.495098    2569 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found
Sep 11 00:59:21.495359 ip-10-0-129-222 kubenswrapper[2569]: E0911 00:59:21.495185    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c9757f20-e580-4d80-aa5f-2f148ce4690d-openshift-state-metrics-tls podName:c9757f20-e580-4d80-aa5f-2f148ce4690d nodeName:}" failed. No retries permitted until 2026-09-11 00:59:21.995163247 +0000 UTC m=+96.646537491 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/c9757f20-e580-4d80-aa5f-2f148ce4690d-openshift-state-metrics-tls") pod "openshift-state-metrics-67dcf8c979-zl6np" (UID: "c9757f20-e580-4d80-aa5f-2f148ce4690d") : secret "openshift-state-metrics-tls" not found
Sep 11 00:59:21.495801 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:21.495640    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c9757f20-e580-4d80-aa5f-2f148ce4690d-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-zl6np\" (UID: \"c9757f20-e580-4d80-aa5f-2f148ce4690d\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-zl6np"
Sep 11 00:59:21.497513 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:21.497486    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c9757f20-e580-4d80-aa5f-2f148ce4690d-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-zl6np\" (UID: \"c9757f20-e580-4d80-aa5f-2f148ce4690d\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-zl6np"
Sep 11 00:59:21.505473 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:21.505448    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xt474\" (UniqueName: \"kubernetes.io/projected/c9757f20-e580-4d80-aa5f-2f148ce4690d-kube-api-access-xt474\") pod \"openshift-state-metrics-67dcf8c979-zl6np\" (UID: \"c9757f20-e580-4d80-aa5f-2f148ce4690d\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-zl6np"
Sep 11 00:59:21.596026 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:21.595925    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vrq4\" (UniqueName: \"kubernetes.io/projected/78562e6b-b985-4098-a024-2d2b21e73fe3-kube-api-access-7vrq4\") pod \"node-exporter-pqgn8\" (UID: \"78562e6b-b985-4098-a024-2d2b21e73fe3\") " pod="openshift-monitoring/node-exporter-pqgn8"
Sep 11 00:59:21.596026 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:21.595972    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/78562e6b-b985-4098-a024-2d2b21e73fe3-sys\") pod \"node-exporter-pqgn8\" (UID: \"78562e6b-b985-4098-a024-2d2b21e73fe3\") " pod="openshift-monitoring/node-exporter-pqgn8"
Sep 11 00:59:21.596026 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:21.595996    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f7a5dad1-f163-4920-b8a4-3ab9e37de477-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-v9pqc\" (UID: \"f7a5dad1-f163-4920-b8a4-3ab9e37de477\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-v9pqc"
Sep 11 00:59:21.596275 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:21.596037    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/78562e6b-b985-4098-a024-2d2b21e73fe3-node-exporter-textfile\") pod \"node-exporter-pqgn8\" (UID: \"78562e6b-b985-4098-a024-2d2b21e73fe3\") " pod="openshift-monitoring/node-exporter-pqgn8"
Sep 11 00:59:21.596275 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:21.596065    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f7a5dad1-f163-4920-b8a4-3ab9e37de477-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-v9pqc\" (UID: \"f7a5dad1-f163-4920-b8a4-3ab9e37de477\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-v9pqc"
Sep 11 00:59:21.596275 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:21.596115    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/f7a5dad1-f163-4920-b8a4-3ab9e37de477-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-v9pqc\" (UID: \"f7a5dad1-f163-4920-b8a4-3ab9e37de477\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-v9pqc"
Sep 11 00:59:21.596275 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:21.596139    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/78562e6b-b985-4098-a024-2d2b21e73fe3-metrics-client-ca\") pod \"node-exporter-pqgn8\" (UID: \"78562e6b-b985-4098-a024-2d2b21e73fe3\") " pod="openshift-monitoring/node-exporter-pqgn8"
Sep 11 00:59:21.596275 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:21.596155    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/78562e6b-b985-4098-a024-2d2b21e73fe3-root\") pod \"node-exporter-pqgn8\" (UID: \"78562e6b-b985-4098-a024-2d2b21e73fe3\") " pod="openshift-monitoring/node-exporter-pqgn8"
Sep 11 00:59:21.596275 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:21.596171    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xm9t6\" (UniqueName: \"kubernetes.io/projected/f7a5dad1-f163-4920-b8a4-3ab9e37de477-kube-api-access-xm9t6\") pod \"kube-state-metrics-59fbdd77d-v9pqc\" (UID: \"f7a5dad1-f163-4920-b8a4-3ab9e37de477\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-v9pqc"
Sep 11 00:59:21.596275 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:21.596190    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f7a5dad1-f163-4920-b8a4-3ab9e37de477-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-v9pqc\" (UID: \"f7a5dad1-f163-4920-b8a4-3ab9e37de477\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-v9pqc"
Sep 11 00:59:21.596275 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:21.596212    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/78562e6b-b985-4098-a024-2d2b21e73fe3-node-exporter-tls\") pod \"node-exporter-pqgn8\" (UID: \"78562e6b-b985-4098-a024-2d2b21e73fe3\") " pod="openshift-monitoring/node-exporter-pqgn8"
Sep 11 00:59:21.596275 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:21.596237    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/78562e6b-b985-4098-a024-2d2b21e73fe3-node-exporter-accelerators-collector-config\") pod \"node-exporter-pqgn8\" (UID: \"78562e6b-b985-4098-a024-2d2b21e73fe3\") " pod="openshift-monitoring/node-exporter-pqgn8"
Sep 11 00:59:21.596275 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:21.596254    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/78562e6b-b985-4098-a024-2d2b21e73fe3-node-exporter-wtmp\") pod \"node-exporter-pqgn8\" (UID: \"78562e6b-b985-4098-a024-2d2b21e73fe3\") " pod="openshift-monitoring/node-exporter-pqgn8"
Sep 11 00:59:21.596275 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:21.596278    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/f7a5dad1-f163-4920-b8a4-3ab9e37de477-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-v9pqc\" (UID: \"f7a5dad1-f163-4920-b8a4-3ab9e37de477\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-v9pqc"
Sep 11 00:59:21.596768 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:21.596308    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/78562e6b-b985-4098-a024-2d2b21e73fe3-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-pqgn8\" (UID: \"78562e6b-b985-4098-a024-2d2b21e73fe3\") " pod="openshift-monitoring/node-exporter-pqgn8"
Sep 11 00:59:21.596768 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:21.596562    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/78562e6b-b985-4098-a024-2d2b21e73fe3-sys\") pod \"node-exporter-pqgn8\" (UID: \"78562e6b-b985-4098-a024-2d2b21e73fe3\") " pod="openshift-monitoring/node-exporter-pqgn8"
Sep 11 00:59:21.597370 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:21.596905    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/f7a5dad1-f163-4920-b8a4-3ab9e37de477-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-v9pqc\" (UID: \"f7a5dad1-f163-4920-b8a4-3ab9e37de477\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-v9pqc"
Sep 11 00:59:21.597370 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:21.596937    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/78562e6b-b985-4098-a024-2d2b21e73fe3-node-exporter-textfile\") pod \"node-exporter-pqgn8\" (UID: \"78562e6b-b985-4098-a024-2d2b21e73fe3\") " pod="openshift-monitoring/node-exporter-pqgn8"
Sep 11 00:59:21.597370 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:21.597087    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/78562e6b-b985-4098-a024-2d2b21e73fe3-node-exporter-wtmp\") pod \"node-exporter-pqgn8\" (UID: \"78562e6b-b985-4098-a024-2d2b21e73fe3\") " pod="openshift-monitoring/node-exporter-pqgn8"
Sep 11 00:59:21.597370 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:21.597273    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/78562e6b-b985-4098-a024-2d2b21e73fe3-root\") pod \"node-exporter-pqgn8\" (UID: \"78562e6b-b985-4098-a024-2d2b21e73fe3\") " pod="openshift-monitoring/node-exporter-pqgn8"
Sep 11 00:59:21.597370 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:21.597308    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/78562e6b-b985-4098-a024-2d2b21e73fe3-metrics-client-ca\") pod \"node-exporter-pqgn8\" (UID: \"78562e6b-b985-4098-a024-2d2b21e73fe3\") " pod="openshift-monitoring/node-exporter-pqgn8"
Sep 11 00:59:21.597370 ip-10-0-129-222 kubenswrapper[2569]: E0911 00:59:21.597365    2569 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found
Sep 11 00:59:21.597752 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:21.597417    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f7a5dad1-f163-4920-b8a4-3ab9e37de477-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-v9pqc\" (UID: \"f7a5dad1-f163-4920-b8a4-3ab9e37de477\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-v9pqc"
Sep 11 00:59:21.597752 ip-10-0-129-222 kubenswrapper[2569]: E0911 00:59:21.597442    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f7a5dad1-f163-4920-b8a4-3ab9e37de477-kube-state-metrics-tls podName:f7a5dad1-f163-4920-b8a4-3ab9e37de477 nodeName:}" failed. No retries permitted until 2026-09-11 00:59:22.097423891 +0000 UTC m=+96.748798124 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/f7a5dad1-f163-4920-b8a4-3ab9e37de477-kube-state-metrics-tls") pod "kube-state-metrics-59fbdd77d-v9pqc" (UID: "f7a5dad1-f163-4920-b8a4-3ab9e37de477") : secret "kube-state-metrics-tls" not found
Sep 11 00:59:21.597752 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:21.597583    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/78562e6b-b985-4098-a024-2d2b21e73fe3-node-exporter-accelerators-collector-config\") pod \"node-exporter-pqgn8\" (UID: \"78562e6b-b985-4098-a024-2d2b21e73fe3\") " pod="openshift-monitoring/node-exporter-pqgn8"
Sep 11 00:59:21.597943 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:21.597840    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/f7a5dad1-f163-4920-b8a4-3ab9e37de477-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-v9pqc\" (UID: \"f7a5dad1-f163-4920-b8a4-3ab9e37de477\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-v9pqc"
Sep 11 00:59:21.598976 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:21.598942    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/78562e6b-b985-4098-a024-2d2b21e73fe3-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-pqgn8\" (UID: \"78562e6b-b985-4098-a024-2d2b21e73fe3\") " pod="openshift-monitoring/node-exporter-pqgn8"
Sep 11 00:59:21.599390 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:21.599360    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f7a5dad1-f163-4920-b8a4-3ab9e37de477-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-v9pqc\" (UID: \"f7a5dad1-f163-4920-b8a4-3ab9e37de477\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-v9pqc"
Sep 11 00:59:21.599832 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:21.599799    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/78562e6b-b985-4098-a024-2d2b21e73fe3-node-exporter-tls\") pod \"node-exporter-pqgn8\" (UID: \"78562e6b-b985-4098-a024-2d2b21e73fe3\") " pod="openshift-monitoring/node-exporter-pqgn8"
Sep 11 00:59:21.605885 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:21.605812    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xm9t6\" (UniqueName: \"kubernetes.io/projected/f7a5dad1-f163-4920-b8a4-3ab9e37de477-kube-api-access-xm9t6\") pod \"kube-state-metrics-59fbdd77d-v9pqc\" (UID: \"f7a5dad1-f163-4920-b8a4-3ab9e37de477\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-v9pqc"
Sep 11 00:59:21.605885 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:21.605829    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vrq4\" (UniqueName: \"kubernetes.io/projected/78562e6b-b985-4098-a024-2d2b21e73fe3-kube-api-access-7vrq4\") pod \"node-exporter-pqgn8\" (UID: \"78562e6b-b985-4098-a024-2d2b21e73fe3\") " pod="openshift-monitoring/node-exporter-pqgn8"
Sep 11 00:59:21.672960 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:21.672928    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-pqgn8"
Sep 11 00:59:22.000053 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:21.999978    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c9757f20-e580-4d80-aa5f-2f148ce4690d-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-zl6np\" (UID: \"c9757f20-e580-4d80-aa5f-2f148ce4690d\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-zl6np"
Sep 11 00:59:22.002717 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:22.002691    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c9757f20-e580-4d80-aa5f-2f148ce4690d-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-zl6np\" (UID: \"c9757f20-e580-4d80-aa5f-2f148ce4690d\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-zl6np"
Sep 11 00:59:22.101000 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:22.100965    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f7a5dad1-f163-4920-b8a4-3ab9e37de477-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-v9pqc\" (UID: \"f7a5dad1-f163-4920-b8a4-3ab9e37de477\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-v9pqc"
Sep 11 00:59:22.101171 ip-10-0-129-222 kubenswrapper[2569]: E0911 00:59:22.101132    2569 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found
Sep 11 00:59:22.101243 ip-10-0-129-222 kubenswrapper[2569]: E0911 00:59:22.101212    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f7a5dad1-f163-4920-b8a4-3ab9e37de477-kube-state-metrics-tls podName:f7a5dad1-f163-4920-b8a4-3ab9e37de477 nodeName:}" failed. No retries permitted until 2026-09-11 00:59:23.101191107 +0000 UTC m=+97.752565355 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/f7a5dad1-f163-4920-b8a4-3ab9e37de477-kube-state-metrics-tls") pod "kube-state-metrics-59fbdd77d-v9pqc" (UID: "f7a5dad1-f163-4920-b8a4-3ab9e37de477") : secret "kube-state-metrics-tls" not found
Sep 11 00:59:22.258389 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:22.258307    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-zl6np"
Sep 11 00:59:22.419754 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:22.419692    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 11 00:59:22.422562 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:22.422539    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 11 00:59:22.424834 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:22.424813    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 11 00:59:22.424957 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:22.424866    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 11 00:59:22.424957 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:22.424900    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 11 00:59:22.425098 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:22.424892    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 11 00:59:22.425301 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:22.425270    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 11 00:59:22.425301 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:22.425278    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 11 00:59:22.425562 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:22.425542    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 11 00:59:22.425651 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:22.425576    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-dgkvd\""
Sep 11 00:59:22.425651 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:22.425606    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 11 00:59:22.425768 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:22.425692    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 11 00:59:22.440214 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:22.440184    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 11 00:59:22.503828 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:22.503798    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4e0aeff2-324c-44d3-87ed-6bd7237f2df0-web-config\") pod \"alertmanager-main-0\" (UID: \"4e0aeff2-324c-44d3-87ed-6bd7237f2df0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 00:59:22.504001 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:22.503867    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/4e0aeff2-324c-44d3-87ed-6bd7237f2df0-config-volume\") pod \"alertmanager-main-0\" (UID: \"4e0aeff2-324c-44d3-87ed-6bd7237f2df0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 00:59:22.504001 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:22.503913    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/4e0aeff2-324c-44d3-87ed-6bd7237f2df0-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"4e0aeff2-324c-44d3-87ed-6bd7237f2df0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 00:59:22.504001 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:22.503938    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4e0aeff2-324c-44d3-87ed-6bd7237f2df0-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"4e0aeff2-324c-44d3-87ed-6bd7237f2df0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 00:59:22.504001 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:22.503966    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4e0aeff2-324c-44d3-87ed-6bd7237f2df0-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"4e0aeff2-324c-44d3-87ed-6bd7237f2df0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 00:59:22.504220 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:22.503990    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4e0aeff2-324c-44d3-87ed-6bd7237f2df0-tls-assets\") pod \"alertmanager-main-0\" (UID: \"4e0aeff2-324c-44d3-87ed-6bd7237f2df0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 00:59:22.504220 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:22.504049    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4e0aeff2-324c-44d3-87ed-6bd7237f2df0-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"4e0aeff2-324c-44d3-87ed-6bd7237f2df0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 00:59:22.504220 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:22.504076    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/4e0aeff2-324c-44d3-87ed-6bd7237f2df0-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"4e0aeff2-324c-44d3-87ed-6bd7237f2df0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 00:59:22.504220 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:22.504125    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/4e0aeff2-324c-44d3-87ed-6bd7237f2df0-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"4e0aeff2-324c-44d3-87ed-6bd7237f2df0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 00:59:22.504220 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:22.504206    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jlx9t\" (UniqueName: \"kubernetes.io/projected/4e0aeff2-324c-44d3-87ed-6bd7237f2df0-kube-api-access-jlx9t\") pod \"alertmanager-main-0\" (UID: \"4e0aeff2-324c-44d3-87ed-6bd7237f2df0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 00:59:22.504424 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:22.504249    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4e0aeff2-324c-44d3-87ed-6bd7237f2df0-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"4e0aeff2-324c-44d3-87ed-6bd7237f2df0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 00:59:22.504424 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:22.504282    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4e0aeff2-324c-44d3-87ed-6bd7237f2df0-config-out\") pod \"alertmanager-main-0\" (UID: \"4e0aeff2-324c-44d3-87ed-6bd7237f2df0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 00:59:22.504424 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:22.504315    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/4e0aeff2-324c-44d3-87ed-6bd7237f2df0-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"4e0aeff2-324c-44d3-87ed-6bd7237f2df0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 00:59:22.605335 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:22.605297    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/4e0aeff2-324c-44d3-87ed-6bd7237f2df0-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"4e0aeff2-324c-44d3-87ed-6bd7237f2df0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 00:59:22.605500 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:22.605432    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jlx9t\" (UniqueName: \"kubernetes.io/projected/4e0aeff2-324c-44d3-87ed-6bd7237f2df0-kube-api-access-jlx9t\") pod \"alertmanager-main-0\" (UID: \"4e0aeff2-324c-44d3-87ed-6bd7237f2df0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 00:59:22.605500 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:22.605463    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4e0aeff2-324c-44d3-87ed-6bd7237f2df0-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"4e0aeff2-324c-44d3-87ed-6bd7237f2df0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 00:59:22.605500 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:22.605489    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4e0aeff2-324c-44d3-87ed-6bd7237f2df0-config-out\") pod \"alertmanager-main-0\" (UID: \"4e0aeff2-324c-44d3-87ed-6bd7237f2df0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 00:59:22.605658 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:22.605515    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/4e0aeff2-324c-44d3-87ed-6bd7237f2df0-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"4e0aeff2-324c-44d3-87ed-6bd7237f2df0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 00:59:22.605658 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:22.605569    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4e0aeff2-324c-44d3-87ed-6bd7237f2df0-web-config\") pod \"alertmanager-main-0\" (UID: \"4e0aeff2-324c-44d3-87ed-6bd7237f2df0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 00:59:22.605819 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:22.605655    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/4e0aeff2-324c-44d3-87ed-6bd7237f2df0-config-volume\") pod \"alertmanager-main-0\" (UID: \"4e0aeff2-324c-44d3-87ed-6bd7237f2df0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 00:59:22.605819 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:22.605682    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/4e0aeff2-324c-44d3-87ed-6bd7237f2df0-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"4e0aeff2-324c-44d3-87ed-6bd7237f2df0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 00:59:22.605819 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:22.605706    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4e0aeff2-324c-44d3-87ed-6bd7237f2df0-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"4e0aeff2-324c-44d3-87ed-6bd7237f2df0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 00:59:22.605819 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:22.605735    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4e0aeff2-324c-44d3-87ed-6bd7237f2df0-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"4e0aeff2-324c-44d3-87ed-6bd7237f2df0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 00:59:22.605819 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:22.605750    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/4e0aeff2-324c-44d3-87ed-6bd7237f2df0-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"4e0aeff2-324c-44d3-87ed-6bd7237f2df0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 00:59:22.605819 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:22.605763    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4e0aeff2-324c-44d3-87ed-6bd7237f2df0-tls-assets\") pod \"alertmanager-main-0\" (UID: \"4e0aeff2-324c-44d3-87ed-6bd7237f2df0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 00:59:22.605819 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:22.605818    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4e0aeff2-324c-44d3-87ed-6bd7237f2df0-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"4e0aeff2-324c-44d3-87ed-6bd7237f2df0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 00:59:22.606184 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:22.605844    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/4e0aeff2-324c-44d3-87ed-6bd7237f2df0-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"4e0aeff2-324c-44d3-87ed-6bd7237f2df0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 00:59:22.606571 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:22.606485    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4e0aeff2-324c-44d3-87ed-6bd7237f2df0-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"4e0aeff2-324c-44d3-87ed-6bd7237f2df0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 00:59:22.607614 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:22.607586    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4e0aeff2-324c-44d3-87ed-6bd7237f2df0-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"4e0aeff2-324c-44d3-87ed-6bd7237f2df0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 00:59:22.608600 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:22.608554    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4e0aeff2-324c-44d3-87ed-6bd7237f2df0-config-out\") pod \"alertmanager-main-0\" (UID: \"4e0aeff2-324c-44d3-87ed-6bd7237f2df0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 00:59:22.608908 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:22.608882    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/4e0aeff2-324c-44d3-87ed-6bd7237f2df0-config-volume\") pod \"alertmanager-main-0\" (UID: \"4e0aeff2-324c-44d3-87ed-6bd7237f2df0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 00:59:22.609001 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:22.608914    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4e0aeff2-324c-44d3-87ed-6bd7237f2df0-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"4e0aeff2-324c-44d3-87ed-6bd7237f2df0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 00:59:22.609001 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:22.608979    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/4e0aeff2-324c-44d3-87ed-6bd7237f2df0-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"4e0aeff2-324c-44d3-87ed-6bd7237f2df0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 00:59:22.609474 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:22.609367    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4e0aeff2-324c-44d3-87ed-6bd7237f2df0-tls-assets\") pod \"alertmanager-main-0\" (UID: \"4e0aeff2-324c-44d3-87ed-6bd7237f2df0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 00:59:22.609474 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:22.609405    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/4e0aeff2-324c-44d3-87ed-6bd7237f2df0-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"4e0aeff2-324c-44d3-87ed-6bd7237f2df0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 00:59:22.610263 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:22.610221    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4e0aeff2-324c-44d3-87ed-6bd7237f2df0-web-config\") pod \"alertmanager-main-0\" (UID: \"4e0aeff2-324c-44d3-87ed-6bd7237f2df0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 00:59:22.610862 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:22.610838    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/4e0aeff2-324c-44d3-87ed-6bd7237f2df0-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"4e0aeff2-324c-44d3-87ed-6bd7237f2df0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 00:59:22.611265 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:22.611247    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4e0aeff2-324c-44d3-87ed-6bd7237f2df0-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"4e0aeff2-324c-44d3-87ed-6bd7237f2df0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 00:59:22.615366 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:22.615341    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jlx9t\" (UniqueName: \"kubernetes.io/projected/4e0aeff2-324c-44d3-87ed-6bd7237f2df0-kube-api-access-jlx9t\") pod \"alertmanager-main-0\" (UID: \"4e0aeff2-324c-44d3-87ed-6bd7237f2df0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 00:59:22.733677 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:22.733644    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 11 00:59:23.108771 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:23.108740    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f7a5dad1-f163-4920-b8a4-3ab9e37de477-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-v9pqc\" (UID: \"f7a5dad1-f163-4920-b8a4-3ab9e37de477\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-v9pqc"
Sep 11 00:59:23.111163 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:23.111137    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f7a5dad1-f163-4920-b8a4-3ab9e37de477-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-v9pqc\" (UID: \"f7a5dad1-f163-4920-b8a4-3ab9e37de477\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-v9pqc"
Sep 11 00:59:23.191152 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:23.191118    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-v9pqc"
Sep 11 00:59:23.209966 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:23.209935    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/75b89b12-20de-4813-b407-ee6836d0a777-registry-tls\") pod \"image-registry-6f984758b5-srjjf\" (UID: \"75b89b12-20de-4813-b407-ee6836d0a777\") " pod="openshift-image-registry/image-registry-6f984758b5-srjjf"
Sep 11 00:59:23.212600 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:23.212581    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/75b89b12-20de-4813-b407-ee6836d0a777-registry-tls\") pod \"image-registry-6f984758b5-srjjf\" (UID: \"75b89b12-20de-4813-b407-ee6836d0a777\") " pod="openshift-image-registry/image-registry-6f984758b5-srjjf"
Sep 11 00:59:23.495977 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:23.495894    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-db2bf\""
Sep 11 00:59:23.504569 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:23.504533    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6f984758b5-srjjf"
Sep 11 00:59:24.043425 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:24.043275    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-v9pqc"]
Sep 11 00:59:24.059902 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:24.052988    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6f984758b5-srjjf"]
Sep 11 00:59:24.063020 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:24.062912    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 11 00:59:24.066004 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:59:24.065975    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e0aeff2_324c_44d3_87ed_6bd7237f2df0.slice/crio-465d6813deb4081df2ac68adcc028734fad98968d7e58c220e877adbd38a4614 WatchSource:0}: Error finding container 465d6813deb4081df2ac68adcc028734fad98968d7e58c220e877adbd38a4614: Status 404 returned error can't find the container with id 465d6813deb4081df2ac68adcc028734fad98968d7e58c220e877adbd38a4614
Sep 11 00:59:24.067695 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:24.067624    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-zl6np"]
Sep 11 00:59:24.067836 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:59:24.067794    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9757f20_e580_4d80_aa5f_2f148ce4690d.slice/crio-2b0471dd7b89a886e91f28c2988d7f6320d6978be2b41d4217a9d9903140e123 WatchSource:0}: Error finding container 2b0471dd7b89a886e91f28c2988d7f6320d6978be2b41d4217a9d9903140e123: Status 404 returned error can't find the container with id 2b0471dd7b89a886e91f28c2988d7f6320d6978be2b41d4217a9d9903140e123
Sep 11 00:59:24.218208 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:24.216594    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-bbbq6"
Sep 11 00:59:24.345256 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:24.345215    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-76f99584f4-j6gxg" event={"ID":"722ada5b-0930-4b9b-8752-49be36c404b4","Type":"ContainerStarted","Data":"f0e9eb7b6fbba675d7dea4ffbdc5d4f54df68a1e4bcdf0b71f7c42b8ee2a9639"}
Sep 11 00:59:24.348265 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:24.348205    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-76f99584f4-j6gxg"
Sep 11 00:59:24.349613 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:24.349558    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-pqgn8" event={"ID":"78562e6b-b985-4098-a024-2d2b21e73fe3","Type":"ContainerStarted","Data":"1b48b67cad08e9f8b1a80e6ebbd036ad0ea0801b08c8b962d4c87ad3842cbbd1"}
Sep 11 00:59:24.352444 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:24.352115    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6f984758b5-srjjf" event={"ID":"75b89b12-20de-4813-b407-ee6836d0a777","Type":"ContainerStarted","Data":"058e12f9811bbc1f7c8807f6b455d17631a597600b882ca2c3c4e0b232a31dd6"}
Sep 11 00:59:24.352444 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:24.352158    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6f984758b5-srjjf" event={"ID":"75b89b12-20de-4813-b407-ee6836d0a777","Type":"ContainerStarted","Data":"4946b8a1969eaf1fbae45d45d5d489bd622f3cb0dbda9e11c9236686cbfbe416"}
Sep 11 00:59:24.352731 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:24.352681    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6f984758b5-srjjf"
Sep 11 00:59:24.355565 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:24.355451    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-zl6np" event={"ID":"c9757f20-e580-4d80-aa5f-2f148ce4690d","Type":"ContainerStarted","Data":"818a0e2054bf3873c689157420089048ab81850bcb091acece567a9e333aad8c"}
Sep 11 00:59:24.355565 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:24.355502    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-zl6np" event={"ID":"c9757f20-e580-4d80-aa5f-2f148ce4690d","Type":"ContainerStarted","Data":"15d20e4e0bf4c2cd9e8142dbc061be1bbed1d6cfa734002a72894bee4233be42"}
Sep 11 00:59:24.355565 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:24.355524    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-zl6np" event={"ID":"c9757f20-e580-4d80-aa5f-2f148ce4690d","Type":"ContainerStarted","Data":"2b0471dd7b89a886e91f28c2988d7f6320d6978be2b41d4217a9d9903140e123"}
Sep 11 00:59:24.357150 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:24.357121    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-v9pqc" event={"ID":"f7a5dad1-f163-4920-b8a4-3ab9e37de477","Type":"ContainerStarted","Data":"9e50d2c35491a94e56a14bdfaa8fec98e64ee2d8cc463e693503e4b070f7e30d"}
Sep 11 00:59:24.358762 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:24.358602    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4e0aeff2-324c-44d3-87ed-6bd7237f2df0","Type":"ContainerStarted","Data":"465d6813deb4081df2ac68adcc028734fad98968d7e58c220e877adbd38a4614"}
Sep 11 00:59:24.359056 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:24.358997    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-76f99584f4-j6gxg"
Sep 11 00:59:24.365295 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:24.365246    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-76f99584f4-j6gxg" podStartSLOduration=1.9829400480000001 podStartE2EDuration="18.365228068s" totalImagesPullingTime="16.38228802s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 00:59:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 00:59:24.364983458 +0000 UTC m=+99.016357719" watchObservedRunningTime="2026-09-11 00:59:24.365228068 +0000 UTC m=+99.016602318"
Sep 11 00:59:24.408182 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:24.408122    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-6f984758b5-srjjf" podStartSLOduration=98.408099115 podStartE2EDuration="1m38.408099115s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 00:57:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 00:59:24.405909981 +0000 UTC m=+99.057284231" watchObservedRunningTime="2026-09-11 00:59:24.408099115 +0000 UTC m=+99.059473366"
Sep 11 00:59:26.607323 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:26.603514    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-6cd4587857-vvhjq"]
Sep 11 00:59:26.635970 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:26.635940    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-6cd4587857-vvhjq"
Sep 11 00:59:26.638352 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:26.638296    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-6cd4587857-vvhjq"]
Sep 11 00:59:26.638820 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:26.638793    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-mrpln\""
Sep 11 00:59:26.639147 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:26.638793    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Sep 11 00:59:26.639376 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:26.639351    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Sep 11 00:59:26.639478 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:26.639351    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Sep 11 00:59:26.639820 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:26.639792    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Sep 11 00:59:26.640503 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:26.640482    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Sep 11 00:59:26.645669 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:26.645651    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Sep 11 00:59:26.743203 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:26.743170    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9c2b3f67-8ddc-4cce-994c-364bf13799fa-metrics-client-ca\") pod \"telemeter-client-6cd4587857-vvhjq\" (UID: \"9c2b3f67-8ddc-4cce-994c-364bf13799fa\") " pod="openshift-monitoring/telemeter-client-6cd4587857-vvhjq"
Sep 11 00:59:26.743351 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:26.743213    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9c2b3f67-8ddc-4cce-994c-364bf13799fa-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6cd4587857-vvhjq\" (UID: \"9c2b3f67-8ddc-4cce-994c-364bf13799fa\") " pod="openshift-monitoring/telemeter-client-6cd4587857-vvhjq"
Sep 11 00:59:26.743351 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:26.743327    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/9c2b3f67-8ddc-4cce-994c-364bf13799fa-federate-client-tls\") pod \"telemeter-client-6cd4587857-vvhjq\" (UID: \"9c2b3f67-8ddc-4cce-994c-364bf13799fa\") " pod="openshift-monitoring/telemeter-client-6cd4587857-vvhjq"
Sep 11 00:59:26.743351 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:26.743349    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/9c2b3f67-8ddc-4cce-994c-364bf13799fa-secret-telemeter-client\") pod \"telemeter-client-6cd4587857-vvhjq\" (UID: \"9c2b3f67-8ddc-4cce-994c-364bf13799fa\") " pod="openshift-monitoring/telemeter-client-6cd4587857-vvhjq"
Sep 11 00:59:26.743495 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:26.743371    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9c2b3f67-8ddc-4cce-994c-364bf13799fa-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6cd4587857-vvhjq\" (UID: \"9c2b3f67-8ddc-4cce-994c-364bf13799fa\") " pod="openshift-monitoring/telemeter-client-6cd4587857-vvhjq"
Sep 11 00:59:26.743495 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:26.743462    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/9c2b3f67-8ddc-4cce-994c-364bf13799fa-telemeter-client-tls\") pod \"telemeter-client-6cd4587857-vvhjq\" (UID: \"9c2b3f67-8ddc-4cce-994c-364bf13799fa\") " pod="openshift-monitoring/telemeter-client-6cd4587857-vvhjq"
Sep 11 00:59:26.743578 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:26.743496    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjtfn\" (UniqueName: \"kubernetes.io/projected/9c2b3f67-8ddc-4cce-994c-364bf13799fa-kube-api-access-zjtfn\") pod \"telemeter-client-6cd4587857-vvhjq\" (UID: \"9c2b3f67-8ddc-4cce-994c-364bf13799fa\") " pod="openshift-monitoring/telemeter-client-6cd4587857-vvhjq"
Sep 11 00:59:26.743578 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:26.743521    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9c2b3f67-8ddc-4cce-994c-364bf13799fa-serving-certs-ca-bundle\") pod \"telemeter-client-6cd4587857-vvhjq\" (UID: \"9c2b3f67-8ddc-4cce-994c-364bf13799fa\") " pod="openshift-monitoring/telemeter-client-6cd4587857-vvhjq"
Sep 11 00:59:26.844499 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:26.844463    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/9c2b3f67-8ddc-4cce-994c-364bf13799fa-secret-telemeter-client\") pod \"telemeter-client-6cd4587857-vvhjq\" (UID: \"9c2b3f67-8ddc-4cce-994c-364bf13799fa\") " pod="openshift-monitoring/telemeter-client-6cd4587857-vvhjq"
Sep 11 00:59:26.844666 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:26.844508    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9c2b3f67-8ddc-4cce-994c-364bf13799fa-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6cd4587857-vvhjq\" (UID: \"9c2b3f67-8ddc-4cce-994c-364bf13799fa\") " pod="openshift-monitoring/telemeter-client-6cd4587857-vvhjq"
Sep 11 00:59:26.844666 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:26.844577    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/9c2b3f67-8ddc-4cce-994c-364bf13799fa-telemeter-client-tls\") pod \"telemeter-client-6cd4587857-vvhjq\" (UID: \"9c2b3f67-8ddc-4cce-994c-364bf13799fa\") " pod="openshift-monitoring/telemeter-client-6cd4587857-vvhjq"
Sep 11 00:59:26.844666 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:26.844617    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjtfn\" (UniqueName: \"kubernetes.io/projected/9c2b3f67-8ddc-4cce-994c-364bf13799fa-kube-api-access-zjtfn\") pod \"telemeter-client-6cd4587857-vvhjq\" (UID: \"9c2b3f67-8ddc-4cce-994c-364bf13799fa\") " pod="openshift-monitoring/telemeter-client-6cd4587857-vvhjq"
Sep 11 00:59:26.844666 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:26.844650    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9c2b3f67-8ddc-4cce-994c-364bf13799fa-serving-certs-ca-bundle\") pod \"telemeter-client-6cd4587857-vvhjq\" (UID: \"9c2b3f67-8ddc-4cce-994c-364bf13799fa\") " pod="openshift-monitoring/telemeter-client-6cd4587857-vvhjq"
Sep 11 00:59:26.844951 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:26.844709    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9c2b3f67-8ddc-4cce-994c-364bf13799fa-metrics-client-ca\") pod \"telemeter-client-6cd4587857-vvhjq\" (UID: \"9c2b3f67-8ddc-4cce-994c-364bf13799fa\") " pod="openshift-monitoring/telemeter-client-6cd4587857-vvhjq"
Sep 11 00:59:26.844951 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:26.844740    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9c2b3f67-8ddc-4cce-994c-364bf13799fa-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6cd4587857-vvhjq\" (UID: \"9c2b3f67-8ddc-4cce-994c-364bf13799fa\") " pod="openshift-monitoring/telemeter-client-6cd4587857-vvhjq"
Sep 11 00:59:26.844951 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:26.844838    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/9c2b3f67-8ddc-4cce-994c-364bf13799fa-federate-client-tls\") pod \"telemeter-client-6cd4587857-vvhjq\" (UID: \"9c2b3f67-8ddc-4cce-994c-364bf13799fa\") " pod="openshift-monitoring/telemeter-client-6cd4587857-vvhjq"
Sep 11 00:59:26.845533 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:26.845506    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9c2b3f67-8ddc-4cce-994c-364bf13799fa-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6cd4587857-vvhjq\" (UID: \"9c2b3f67-8ddc-4cce-994c-364bf13799fa\") " pod="openshift-monitoring/telemeter-client-6cd4587857-vvhjq"
Sep 11 00:59:26.845696 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:26.845619    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9c2b3f67-8ddc-4cce-994c-364bf13799fa-serving-certs-ca-bundle\") pod \"telemeter-client-6cd4587857-vvhjq\" (UID: \"9c2b3f67-8ddc-4cce-994c-364bf13799fa\") " pod="openshift-monitoring/telemeter-client-6cd4587857-vvhjq"
Sep 11 00:59:26.846530 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:26.846499    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9c2b3f67-8ddc-4cce-994c-364bf13799fa-metrics-client-ca\") pod \"telemeter-client-6cd4587857-vvhjq\" (UID: \"9c2b3f67-8ddc-4cce-994c-364bf13799fa\") " pod="openshift-monitoring/telemeter-client-6cd4587857-vvhjq"
Sep 11 00:59:26.847939 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:26.847891    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/9c2b3f67-8ddc-4cce-994c-364bf13799fa-federate-client-tls\") pod \"telemeter-client-6cd4587857-vvhjq\" (UID: \"9c2b3f67-8ddc-4cce-994c-364bf13799fa\") " pod="openshift-monitoring/telemeter-client-6cd4587857-vvhjq"
Sep 11 00:59:26.849154 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:26.849131    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/9c2b3f67-8ddc-4cce-994c-364bf13799fa-telemeter-client-tls\") pod \"telemeter-client-6cd4587857-vvhjq\" (UID: \"9c2b3f67-8ddc-4cce-994c-364bf13799fa\") " pod="openshift-monitoring/telemeter-client-6cd4587857-vvhjq"
Sep 11 00:59:26.849261 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:26.849215    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/9c2b3f67-8ddc-4cce-994c-364bf13799fa-secret-telemeter-client\") pod \"telemeter-client-6cd4587857-vvhjq\" (UID: \"9c2b3f67-8ddc-4cce-994c-364bf13799fa\") " pod="openshift-monitoring/telemeter-client-6cd4587857-vvhjq"
Sep 11 00:59:26.849487 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:26.849466    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9c2b3f67-8ddc-4cce-994c-364bf13799fa-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6cd4587857-vvhjq\" (UID: \"9c2b3f67-8ddc-4cce-994c-364bf13799fa\") " pod="openshift-monitoring/telemeter-client-6cd4587857-vvhjq"
Sep 11 00:59:26.854071 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:26.854051    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjtfn\" (UniqueName: \"kubernetes.io/projected/9c2b3f67-8ddc-4cce-994c-364bf13799fa-kube-api-access-zjtfn\") pod \"telemeter-client-6cd4587857-vvhjq\" (UID: \"9c2b3f67-8ddc-4cce-994c-364bf13799fa\") " pod="openshift-monitoring/telemeter-client-6cd4587857-vvhjq"
Sep 11 00:59:26.950392 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:26.950356    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-6cd4587857-vvhjq"
Sep 11 00:59:27.376621 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:27.376243    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-v9pqc" event={"ID":"f7a5dad1-f163-4920-b8a4-3ab9e37de477","Type":"ContainerStarted","Data":"554854a48bc830b802e0f9b1a7e9f45844622fb35c7a2dff3dfd4760133a4dbb"}
Sep 11 00:59:27.379354 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:27.378563    2569 generic.go:356] "Generic (PLEG): container finished" podID="4e0aeff2-324c-44d3-87ed-6bd7237f2df0" containerID="2aae85020e3b8e8b190b0f56277444eaafa2fffe0cb2c6300a7a1f3490838623" exitCode=0
Sep 11 00:59:27.379354 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:27.378683    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4e0aeff2-324c-44d3-87ed-6bd7237f2df0","Type":"ContainerDied","Data":"2aae85020e3b8e8b190b0f56277444eaafa2fffe0cb2c6300a7a1f3490838623"}
Sep 11 00:59:27.386902 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:27.384698    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-pqgn8" event={"ID":"78562e6b-b985-4098-a024-2d2b21e73fe3","Type":"ContainerStarted","Data":"a93d15e29405890b5e1b7fec120d37c5c84000e528479b42eac73a0eba7c069e"}
Sep 11 00:59:27.391011 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:27.390845    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-zl6np" event={"ID":"c9757f20-e580-4d80-aa5f-2f148ce4690d","Type":"ContainerStarted","Data":"1e7d7a6b9b6f9e98296a494d51f0958f0e020f92b277cb956950e32c49255366"}
Sep 11 00:59:27.403087 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:27.402999    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-6cd4587857-vvhjq"]
Sep 11 00:59:27.423738 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:27.423686    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-zl6np" podStartSLOduration=3.524458707 podStartE2EDuration="6.423663683s" totalImagesPullingTime="2.899204976s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 00:59:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 00:59:27.421563842 +0000 UTC m=+102.072938091" watchObservedRunningTime="2026-09-11 00:59:27.423663683 +0000 UTC m=+102.075037933"
Sep 11 00:59:27.604179 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:27.604138    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 11 00:59:27.623411 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:27.623376    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 00:59:27.626484 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:27.626458    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 11 00:59:27.626601 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:27.626464    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 11 00:59:27.627116 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:27.627013    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 11 00:59:27.627235 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:27.627208    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 11 00:59:27.627402 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:27.627382    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 11 00:59:27.628433 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:27.628412    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 11 00:59:27.635602 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:27.635435    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-8d4cx\""
Sep 11 00:59:27.637508 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:27.637071    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 11 00:59:27.637508 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:27.637104    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 11 00:59:27.637508 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:27.637144    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 11 00:59:27.637508 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:27.637264    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 11 00:59:27.637508 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:27.637347    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 11 00:59:27.637795 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:27.637651    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 11 00:59:27.652190 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:27.652015    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-4fdskhji27lft\""
Sep 11 00:59:27.657551 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:27.657531    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 11 00:59:27.666894 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:27.660569    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 11 00:59:27.666894 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:27.661355    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 11 00:59:27.669861 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:27.669834    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 11 00:59:27.753735 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:27.753696    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f30e94a8-4e99-4971-a70c-c96786257fb9-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"f30e94a8-4e99-4971-a70c-c96786257fb9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 00:59:27.753930 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:27.753751    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f30e94a8-4e99-4971-a70c-c96786257fb9-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"f30e94a8-4e99-4971-a70c-c96786257fb9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 00:59:27.753930 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:27.753794    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/f30e94a8-4e99-4971-a70c-c96786257fb9-config\") pod \"prometheus-k8s-0\" (UID: \"f30e94a8-4e99-4971-a70c-c96786257fb9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 00:59:27.753930 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:27.753821    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f30e94a8-4e99-4971-a70c-c96786257fb9-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f30e94a8-4e99-4971-a70c-c96786257fb9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 00:59:27.753930 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:27.753846    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f30e94a8-4e99-4971-a70c-c96786257fb9-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"f30e94a8-4e99-4971-a70c-c96786257fb9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 00:59:27.754143 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:27.753926    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8c8v8\" (UniqueName: \"kubernetes.io/projected/f30e94a8-4e99-4971-a70c-c96786257fb9-kube-api-access-8c8v8\") pod \"prometheus-k8s-0\" (UID: \"f30e94a8-4e99-4971-a70c-c96786257fb9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 00:59:27.754143 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:27.754013    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f30e94a8-4e99-4971-a70c-c96786257fb9-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"f30e94a8-4e99-4971-a70c-c96786257fb9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 00:59:27.754143 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:27.754052    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f30e94a8-4e99-4971-a70c-c96786257fb9-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"f30e94a8-4e99-4971-a70c-c96786257fb9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 00:59:27.754143 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:27.754086    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f30e94a8-4e99-4971-a70c-c96786257fb9-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"f30e94a8-4e99-4971-a70c-c96786257fb9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 00:59:27.754340 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:27.754145    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f30e94a8-4e99-4971-a70c-c96786257fb9-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f30e94a8-4e99-4971-a70c-c96786257fb9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 00:59:27.754340 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:27.754171    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f30e94a8-4e99-4971-a70c-c96786257fb9-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f30e94a8-4e99-4971-a70c-c96786257fb9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 00:59:27.754340 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:27.754197    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f30e94a8-4e99-4971-a70c-c96786257fb9-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"f30e94a8-4e99-4971-a70c-c96786257fb9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 00:59:27.754340 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:27.754214    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f30e94a8-4e99-4971-a70c-c96786257fb9-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"f30e94a8-4e99-4971-a70c-c96786257fb9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 00:59:27.754340 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:27.754228    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f30e94a8-4e99-4971-a70c-c96786257fb9-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"f30e94a8-4e99-4971-a70c-c96786257fb9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 00:59:27.754340 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:27.754249    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f30e94a8-4e99-4971-a70c-c96786257fb9-web-config\") pod \"prometheus-k8s-0\" (UID: \"f30e94a8-4e99-4971-a70c-c96786257fb9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 00:59:27.754615 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:27.754338    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f30e94a8-4e99-4971-a70c-c96786257fb9-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"f30e94a8-4e99-4971-a70c-c96786257fb9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 00:59:27.754615 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:27.754382    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f30e94a8-4e99-4971-a70c-c96786257fb9-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"f30e94a8-4e99-4971-a70c-c96786257fb9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 00:59:27.754615 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:27.754417    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f30e94a8-4e99-4971-a70c-c96786257fb9-config-out\") pod \"prometheus-k8s-0\" (UID: \"f30e94a8-4e99-4971-a70c-c96786257fb9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 00:59:27.754615 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:27.754448    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f30e94a8-4e99-4971-a70c-c96786257fb9-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"f30e94a8-4e99-4971-a70c-c96786257fb9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 00:59:27.754615 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:27.754521    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f30e94a8-4e99-4971-a70c-c96786257fb9-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"f30e94a8-4e99-4971-a70c-c96786257fb9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 00:59:27.855540 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:27.855509    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f30e94a8-4e99-4971-a70c-c96786257fb9-config-out\") pod \"prometheus-k8s-0\" (UID: \"f30e94a8-4e99-4971-a70c-c96786257fb9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 00:59:27.855767 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:27.855748    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f30e94a8-4e99-4971-a70c-c96786257fb9-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"f30e94a8-4e99-4971-a70c-c96786257fb9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 00:59:27.855946 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:27.855929    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f30e94a8-4e99-4971-a70c-c96786257fb9-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"f30e94a8-4e99-4971-a70c-c96786257fb9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 00:59:27.856115 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:27.856099    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f30e94a8-4e99-4971-a70c-c96786257fb9-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"f30e94a8-4e99-4971-a70c-c96786257fb9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 00:59:27.856672 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:27.856652    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f30e94a8-4e99-4971-a70c-c96786257fb9-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"f30e94a8-4e99-4971-a70c-c96786257fb9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 00:59:27.856816 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:27.856799    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/f30e94a8-4e99-4971-a70c-c96786257fb9-config\") pod \"prometheus-k8s-0\" (UID: \"f30e94a8-4e99-4971-a70c-c96786257fb9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 00:59:27.856967 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:27.856944    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f30e94a8-4e99-4971-a70c-c96786257fb9-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f30e94a8-4e99-4971-a70c-c96786257fb9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 00:59:27.857086 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:27.857072    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f30e94a8-4e99-4971-a70c-c96786257fb9-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"f30e94a8-4e99-4971-a70c-c96786257fb9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 00:59:27.857213 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:27.857199    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8c8v8\" (UniqueName: \"kubernetes.io/projected/f30e94a8-4e99-4971-a70c-c96786257fb9-kube-api-access-8c8v8\") pod \"prometheus-k8s-0\" (UID: \"f30e94a8-4e99-4971-a70c-c96786257fb9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 00:59:27.858077 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:27.856587    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f30e94a8-4e99-4971-a70c-c96786257fb9-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"f30e94a8-4e99-4971-a70c-c96786257fb9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 00:59:27.858077 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:27.857357    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f30e94a8-4e99-4971-a70c-c96786257fb9-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"f30e94a8-4e99-4971-a70c-c96786257fb9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 00:59:27.858077 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:27.857396    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f30e94a8-4e99-4971-a70c-c96786257fb9-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"f30e94a8-4e99-4971-a70c-c96786257fb9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 00:59:27.858077 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:27.857445    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f30e94a8-4e99-4971-a70c-c96786257fb9-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"f30e94a8-4e99-4971-a70c-c96786257fb9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 00:59:27.858077 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:27.857487    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f30e94a8-4e99-4971-a70c-c96786257fb9-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f30e94a8-4e99-4971-a70c-c96786257fb9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 00:59:27.858077 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:27.857521    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f30e94a8-4e99-4971-a70c-c96786257fb9-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f30e94a8-4e99-4971-a70c-c96786257fb9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 00:59:27.858077 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:27.857554    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f30e94a8-4e99-4971-a70c-c96786257fb9-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"f30e94a8-4e99-4971-a70c-c96786257fb9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 00:59:27.858077 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:27.857583    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f30e94a8-4e99-4971-a70c-c96786257fb9-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"f30e94a8-4e99-4971-a70c-c96786257fb9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 00:59:27.858077 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:27.857612    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f30e94a8-4e99-4971-a70c-c96786257fb9-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"f30e94a8-4e99-4971-a70c-c96786257fb9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 00:59:27.858077 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:27.857614    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f30e94a8-4e99-4971-a70c-c96786257fb9-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"f30e94a8-4e99-4971-a70c-c96786257fb9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 00:59:27.858077 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:27.857638    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f30e94a8-4e99-4971-a70c-c96786257fb9-web-config\") pod \"prometheus-k8s-0\" (UID: \"f30e94a8-4e99-4971-a70c-c96786257fb9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 00:59:27.858077 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:27.857675    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f30e94a8-4e99-4971-a70c-c96786257fb9-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"f30e94a8-4e99-4971-a70c-c96786257fb9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 00:59:27.858077 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:27.857706    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f30e94a8-4e99-4971-a70c-c96786257fb9-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"f30e94a8-4e99-4971-a70c-c96786257fb9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 00:59:27.858748 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:27.858354    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f30e94a8-4e99-4971-a70c-c96786257fb9-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f30e94a8-4e99-4971-a70c-c96786257fb9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 00:59:27.858748 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:27.858379    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f30e94a8-4e99-4971-a70c-c96786257fb9-config-out\") pod \"prometheus-k8s-0\" (UID: \"f30e94a8-4e99-4971-a70c-c96786257fb9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 00:59:27.858748 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:27.858665    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f30e94a8-4e99-4971-a70c-c96786257fb9-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"f30e94a8-4e99-4971-a70c-c96786257fb9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 00:59:27.858993 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:27.858969    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f30e94a8-4e99-4971-a70c-c96786257fb9-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"f30e94a8-4e99-4971-a70c-c96786257fb9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 00:59:27.859127 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:27.859067    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f30e94a8-4e99-4971-a70c-c96786257fb9-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f30e94a8-4e99-4971-a70c-c96786257fb9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 00:59:27.859534 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:27.859221    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f30e94a8-4e99-4971-a70c-c96786257fb9-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"f30e94a8-4e99-4971-a70c-c96786257fb9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 00:59:27.859920 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:27.859849    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f30e94a8-4e99-4971-a70c-c96786257fb9-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"f30e94a8-4e99-4971-a70c-c96786257fb9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 00:59:27.860668 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:27.860641    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f30e94a8-4e99-4971-a70c-c96786257fb9-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f30e94a8-4e99-4971-a70c-c96786257fb9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 00:59:27.861575 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:27.861547    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f30e94a8-4e99-4971-a70c-c96786257fb9-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"f30e94a8-4e99-4971-a70c-c96786257fb9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 00:59:27.862280 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:27.862196    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f30e94a8-4e99-4971-a70c-c96786257fb9-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"f30e94a8-4e99-4971-a70c-c96786257fb9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 00:59:27.862280 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:27.862271    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f30e94a8-4e99-4971-a70c-c96786257fb9-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"f30e94a8-4e99-4971-a70c-c96786257fb9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 00:59:27.862822 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:27.862652    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f30e94a8-4e99-4971-a70c-c96786257fb9-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"f30e94a8-4e99-4971-a70c-c96786257fb9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 00:59:27.863461 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:27.863413    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/f30e94a8-4e99-4971-a70c-c96786257fb9-config\") pod \"prometheus-k8s-0\" (UID: \"f30e94a8-4e99-4971-a70c-c96786257fb9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 00:59:27.863662 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:27.863617    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f30e94a8-4e99-4971-a70c-c96786257fb9-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"f30e94a8-4e99-4971-a70c-c96786257fb9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 00:59:27.864134 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:27.864056    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f30e94a8-4e99-4971-a70c-c96786257fb9-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"f30e94a8-4e99-4971-a70c-c96786257fb9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 00:59:27.864713 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:27.864687    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f30e94a8-4e99-4971-a70c-c96786257fb9-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"f30e94a8-4e99-4971-a70c-c96786257fb9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 00:59:27.864951 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:27.864904    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f30e94a8-4e99-4971-a70c-c96786257fb9-web-config\") pod \"prometheus-k8s-0\" (UID: \"f30e94a8-4e99-4971-a70c-c96786257fb9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 00:59:27.867042 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:27.867009    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8c8v8\" (UniqueName: \"kubernetes.io/projected/f30e94a8-4e99-4971-a70c-c96786257fb9-kube-api-access-8c8v8\") pod \"prometheus-k8s-0\" (UID: \"f30e94a8-4e99-4971-a70c-c96786257fb9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 00:59:27.936059 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:27.935991    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 00:59:28.257228 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:28.256774    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 11 00:59:28.262773 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:59:28.262733    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf30e94a8_4e99_4971_a70c_c96786257fb9.slice/crio-e685db21a0d283d79b1a998acfb0e1c77897ac68d5dd68145d6944d7038bf0f3 WatchSource:0}: Error finding container e685db21a0d283d79b1a998acfb0e1c77897ac68d5dd68145d6944d7038bf0f3: Status 404 returned error can't find the container with id e685db21a0d283d79b1a998acfb0e1c77897ac68d5dd68145d6944d7038bf0f3
Sep 11 00:59:28.397777 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:28.397732    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-pqgn8" event={"ID":"78562e6b-b985-4098-a024-2d2b21e73fe3","Type":"ContainerDied","Data":"a93d15e29405890b5e1b7fec120d37c5c84000e528479b42eac73a0eba7c069e"}
Sep 11 00:59:28.398136 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:28.398038    2569 generic.go:356] "Generic (PLEG): container finished" podID="78562e6b-b985-4098-a024-2d2b21e73fe3" containerID="a93d15e29405890b5e1b7fec120d37c5c84000e528479b42eac73a0eba7c069e" exitCode=0
Sep 11 00:59:28.400051 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:28.399945    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f30e94a8-4e99-4971-a70c-c96786257fb9","Type":"ContainerStarted","Data":"e685db21a0d283d79b1a998acfb0e1c77897ac68d5dd68145d6944d7038bf0f3"}
Sep 11 00:59:28.402546 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:28.402475    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6cd4587857-vvhjq" event={"ID":"9c2b3f67-8ddc-4cce-994c-364bf13799fa","Type":"ContainerStarted","Data":"0ec25a5f53d47ee3558503d05f5476648e9cad2db4f4fc5d01739137d0a12177"}
Sep 11 00:59:28.405115 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:28.404974    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-v9pqc" event={"ID":"f7a5dad1-f163-4920-b8a4-3ab9e37de477","Type":"ContainerStarted","Data":"77338bc9ef1945993c4278691eab47907c12305ba4ab627755276fffa06f220f"}
Sep 11 00:59:28.405115 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:28.405021    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-v9pqc" event={"ID":"f7a5dad1-f163-4920-b8a4-3ab9e37de477","Type":"ContainerStarted","Data":"e03127c5e989facaa0714e5583f61c5c4f70f008b58142dbcb8471c2e8d4191f"}
Sep 11 00:59:28.444138 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:28.441802    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-v9pqc" podStartSLOduration=4.394727465 podStartE2EDuration="7.441782483s" totalImagesPullingTime="3.047055018s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 00:59:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 00:59:28.440440262 +0000 UTC m=+103.091814512" watchObservedRunningTime="2026-09-11 00:59:28.441782483 +0000 UTC m=+103.093156733"
Sep 11 00:59:28.971375 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:28.971333    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6f984758b5-srjjf"]
Sep 11 00:59:29.413491 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:29.413364    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-pqgn8" event={"ID":"78562e6b-b985-4098-a024-2d2b21e73fe3","Type":"ContainerStarted","Data":"c164c5b177ca670cde82a698136153df8fddbed9ce732ed33595c8425ec8adf7"}
Sep 11 00:59:29.413904 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:29.413465    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-pqgn8" event={"ID":"78562e6b-b985-4098-a024-2d2b21e73fe3","Type":"ContainerStarted","Data":"f354b1b69a387981603717f50e231ba06ea4b495145df508d327b642bcb73a92"}
Sep 11 00:59:29.416424 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:29.416397    2569 generic.go:356] "Generic (PLEG): container finished" podID="f30e94a8-4e99-4971-a70c-c96786257fb9" containerID="ddf849c406b4ea503fc952e9610f603d1f0cf62f73d17d6e927fabfcd59cbac2" exitCode=0
Sep 11 00:59:29.416642 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:29.416564    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f30e94a8-4e99-4971-a70c-c96786257fb9","Type":"ContainerDied","Data":"ddf849c406b4ea503fc952e9610f603d1f0cf62f73d17d6e927fabfcd59cbac2"}
Sep 11 00:59:29.440940 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:29.440884    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-pqgn8" podStartSLOduration=4.951402086 podStartE2EDuration="8.440850152s" totalImagesPullingTime="3.359684271s" totalInitContainerRuntime="129.763795ms" isStatefulPod=true podCreationTimestamp="2026-09-11 00:59:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 00:59:29.438805658 +0000 UTC m=+104.090179920" watchObservedRunningTime="2026-09-11 00:59:29.440850152 +0000 UTC m=+104.092224403"
Sep 11 00:59:31.437051 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:31.436970    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6cd4587857-vvhjq" event={"ID":"9c2b3f67-8ddc-4cce-994c-364bf13799fa","Type":"ContainerStarted","Data":"f3b7cfac3321c994d3822f69c45a9d18ab0afb6f4c520d761b3e140189b00f87"}
Sep 11 00:59:31.437051 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:31.437044    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6cd4587857-vvhjq" event={"ID":"9c2b3f67-8ddc-4cce-994c-364bf13799fa","Type":"ContainerStarted","Data":"dd5910354e03a24971f6eb6433669f5046d3cdbadae42ba648ca6d273d039600"}
Sep 11 00:59:31.437584 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:31.437074    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6cd4587857-vvhjq" event={"ID":"9c2b3f67-8ddc-4cce-994c-364bf13799fa","Type":"ContainerStarted","Data":"521c38f7257beb6da4a3e76f5edae5028b6be222dd2cb231b950dcec831c8783"}
Sep 11 00:59:31.450048 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:31.450011    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4e0aeff2-324c-44d3-87ed-6bd7237f2df0","Type":"ContainerStarted","Data":"2f0872639447bd04f69a24870b358f71aa162d9b8c31a51eeede435e1cb16804"}
Sep 11 00:59:31.450210 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:31.450122    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4e0aeff2-324c-44d3-87ed-6bd7237f2df0","Type":"ContainerStarted","Data":"c10c594e1c733c7ec1757920c0d1e26db7459a312218aeba65cef53c10fad0a6"}
Sep 11 00:59:31.450210 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:31.450152    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4e0aeff2-324c-44d3-87ed-6bd7237f2df0","Type":"ContainerStarted","Data":"b04795c23ac48b15a5ec412a4d8149f31882c60c24bf8186e487401c3582a1f0"}
Sep 11 00:59:31.450210 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:31.450187    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4e0aeff2-324c-44d3-87ed-6bd7237f2df0","Type":"ContainerStarted","Data":"548ad5207ec4dc685f6c4bf27d5e8ce2429e33b5476a98543f9d5f0f9e06b077"}
Sep 11 00:59:31.450358 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:31.450256    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4e0aeff2-324c-44d3-87ed-6bd7237f2df0","Type":"ContainerStarted","Data":"5f5c1cf91412b31f1548b369df4fca9309874c2626970d85af8b90c60a95610c"}
Sep 11 00:59:31.465960 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:31.465905    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-6cd4587857-vvhjq" podStartSLOduration=2.455477999 podStartE2EDuration="5.465886371s" totalImagesPullingTime="3.010408372s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 00:59:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 00:59:31.464536571 +0000 UTC m=+106.115910832" watchObservedRunningTime="2026-09-11 00:59:31.465886371 +0000 UTC m=+106.117260619"
Sep 11 00:59:34.466987 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:34.466948    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f30e94a8-4e99-4971-a70c-c96786257fb9","Type":"ContainerStarted","Data":"459b83d39c5911e91e42b108b13ba7c89048120502fc187c131203b02bcc8221"}
Sep 11 00:59:34.467415 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:34.467039    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f30e94a8-4e99-4971-a70c-c96786257fb9","Type":"ContainerStarted","Data":"70b9a3ea10dce6c2bed553500bb5b79c263915ec13b318a96fe58a8e0a30f3d2"}
Sep 11 00:59:34.470415 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:34.470380    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4e0aeff2-324c-44d3-87ed-6bd7237f2df0","Type":"ContainerStarted","Data":"6d29af2022cb3c11d7c59e5cc833a628020ea0eb69875c5f01d36b865c2759b0"}
Sep 11 00:59:34.496178 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:34.496098    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.886628499 podStartE2EDuration="12.496082236s" totalImagesPullingTime="8.575976124s" totalInitContainerRuntime="33.477613ms" isStatefulPod=false podCreationTimestamp="2026-09-11 00:59:22 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 00:59:34.495013695 +0000 UTC m=+109.146387943" watchObservedRunningTime="2026-09-11 00:59:34.496082236 +0000 UTC m=+109.147456485"
Sep 11 00:59:36.482130 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:36.482099    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f30e94a8-4e99-4971-a70c-c96786257fb9","Type":"ContainerStarted","Data":"cfd1a641939c265b5b10ae08171f6ecd522672448650c3ea904e57b25d48312a"}
Sep 11 00:59:37.490105 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:37.490068    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f30e94a8-4e99-4971-a70c-c96786257fb9","Type":"ContainerStarted","Data":"2f9b94fe105a2aaf78b3a9fd89cbd9f8d9363da40047bc2838f029a503508ad4"}
Sep 11 00:59:37.490559 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:37.490129    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f30e94a8-4e99-4971-a70c-c96786257fb9","Type":"ContainerStarted","Data":"2decc0a7d51ac5cc9481b62aa7f9562096b9271ba7378f40c8e63d8b885e7bb6"}
Sep 11 00:59:37.490559 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:37.490156    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f30e94a8-4e99-4971-a70c-c96786257fb9","Type":"ContainerStarted","Data":"b55d63f1960c065d1dc6e816cd3f143579d8220f2a9ff6d4780e71abbdb41582"}
Sep 11 00:59:37.519758 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:37.519699    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.846982793 podStartE2EDuration="10.51967962s" totalImagesPullingTime="6.645116355s" totalInitContainerRuntime="27.580472ms" isStatefulPod=false podCreationTimestamp="2026-09-11 00:59:27 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 00:59:37.518098782 +0000 UTC m=+112.169473054" watchObservedRunningTime="2026-09-11 00:59:37.51967962 +0000 UTC m=+112.171053870"
Sep 11 00:59:37.936606 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:37.936572    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 00:59:46.524191 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:46.524156    2569 generic.go:356] "Generic (PLEG): container finished" podID="dfc463c1-6efd-4f61-ba7a-9353d13b0b55" containerID="2a608bc2ef81bca14ac3b6bb8b66a166f71c040fc018c5079209c29eb768f74c" exitCode=0
Sep 11 00:59:46.524592 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:46.524212    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-8vpg2" event={"ID":"dfc463c1-6efd-4f61-ba7a-9353d13b0b55","Type":"ContainerDied","Data":"2a608bc2ef81bca14ac3b6bb8b66a166f71c040fc018c5079209c29eb768f74c"}
Sep 11 00:59:46.524592 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:46.524513    2569 scope.go:117] "RemoveContainer" containerID="2a608bc2ef81bca14ac3b6bb8b66a166f71c040fc018c5079209c29eb768f74c"
Sep 11 00:59:47.530009 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:47.529975    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-8vpg2" event={"ID":"dfc463c1-6efd-4f61-ba7a-9353d13b0b55","Type":"ContainerStarted","Data":"b5487db7c44909f99dbbc87260390f729aea8055bcb5e83be97e63e0f20636e3"}
Sep 11 00:59:48.977322 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:48.977289    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6f984758b5-srjjf"
Sep 11 00:59:54.003231 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:54.003190    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-6f984758b5-srjjf" podUID="75b89b12-20de-4813-b407-ee6836d0a777" containerName="registry" containerID="cri-o://058e12f9811bbc1f7c8807f6b455d17631a597600b882ca2c3c4e0b232a31dd6" gracePeriod=30
Sep 11 00:59:54.281497 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:54.281432    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6f984758b5-srjjf"
Sep 11 00:59:54.299377 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:54.299345    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/75b89b12-20de-4813-b407-ee6836d0a777-registry-tls\") pod \"75b89b12-20de-4813-b407-ee6836d0a777\" (UID: \"75b89b12-20de-4813-b407-ee6836d0a777\") "
Sep 11 00:59:54.299521 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:54.299390    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/75b89b12-20de-4813-b407-ee6836d0a777-trusted-ca\") pod \"75b89b12-20de-4813-b407-ee6836d0a777\" (UID: \"75b89b12-20de-4813-b407-ee6836d0a777\") "
Sep 11 00:59:54.299521 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:54.299421    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/75b89b12-20de-4813-b407-ee6836d0a777-registry-certificates\") pod \"75b89b12-20de-4813-b407-ee6836d0a777\" (UID: \"75b89b12-20de-4813-b407-ee6836d0a777\") "
Sep 11 00:59:54.299521 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:54.299450    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v64zn\" (UniqueName: \"kubernetes.io/projected/75b89b12-20de-4813-b407-ee6836d0a777-kube-api-access-v64zn\") pod \"75b89b12-20de-4813-b407-ee6836d0a777\" (UID: \"75b89b12-20de-4813-b407-ee6836d0a777\") "
Sep 11 00:59:54.299521 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:54.299488    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/75b89b12-20de-4813-b407-ee6836d0a777-image-registry-private-configuration\") pod \"75b89b12-20de-4813-b407-ee6836d0a777\" (UID: \"75b89b12-20de-4813-b407-ee6836d0a777\") "
Sep 11 00:59:54.299744 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:54.299556    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/75b89b12-20de-4813-b407-ee6836d0a777-installation-pull-secrets\") pod \"75b89b12-20de-4813-b407-ee6836d0a777\" (UID: \"75b89b12-20de-4813-b407-ee6836d0a777\") "
Sep 11 00:59:54.299744 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:54.299597    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/75b89b12-20de-4813-b407-ee6836d0a777-bound-sa-token\") pod \"75b89b12-20de-4813-b407-ee6836d0a777\" (UID: \"75b89b12-20de-4813-b407-ee6836d0a777\") "
Sep 11 00:59:54.299744 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:54.299650    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/75b89b12-20de-4813-b407-ee6836d0a777-ca-trust-extracted\") pod \"75b89b12-20de-4813-b407-ee6836d0a777\" (UID: \"75b89b12-20de-4813-b407-ee6836d0a777\") "
Sep 11 00:59:54.300388 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:54.300099    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75b89b12-20de-4813-b407-ee6836d0a777-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "75b89b12-20de-4813-b407-ee6836d0a777" (UID: "75b89b12-20de-4813-b407-ee6836d0a777"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 00:59:54.300547 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:54.300525    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75b89b12-20de-4813-b407-ee6836d0a777-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "75b89b12-20de-4813-b407-ee6836d0a777" (UID: "75b89b12-20de-4813-b407-ee6836d0a777"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 00:59:54.302829 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:54.302788    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75b89b12-20de-4813-b407-ee6836d0a777-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "75b89b12-20de-4813-b407-ee6836d0a777" (UID: "75b89b12-20de-4813-b407-ee6836d0a777"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 00:59:54.303544 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:54.303481    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75b89b12-20de-4813-b407-ee6836d0a777-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "75b89b12-20de-4813-b407-ee6836d0a777" (UID: "75b89b12-20de-4813-b407-ee6836d0a777"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 00:59:54.303644 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:54.303622    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75b89b12-20de-4813-b407-ee6836d0a777-kube-api-access-v64zn" (OuterVolumeSpecName: "kube-api-access-v64zn") pod "75b89b12-20de-4813-b407-ee6836d0a777" (UID: "75b89b12-20de-4813-b407-ee6836d0a777"). InnerVolumeSpecName "kube-api-access-v64zn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 00:59:54.303703 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:54.303670    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75b89b12-20de-4813-b407-ee6836d0a777-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "75b89b12-20de-4813-b407-ee6836d0a777" (UID: "75b89b12-20de-4813-b407-ee6836d0a777"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 00:59:54.304206 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:54.304177    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75b89b12-20de-4813-b407-ee6836d0a777-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "75b89b12-20de-4813-b407-ee6836d0a777" (UID: "75b89b12-20de-4813-b407-ee6836d0a777"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 00:59:54.311547 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:54.311518    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75b89b12-20de-4813-b407-ee6836d0a777-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "75b89b12-20de-4813-b407-ee6836d0a777" (UID: "75b89b12-20de-4813-b407-ee6836d0a777"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 00:59:54.401252 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:54.401217    2569 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/75b89b12-20de-4813-b407-ee6836d0a777-installation-pull-secrets\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 00:59:54.401252 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:54.401245    2569 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/75b89b12-20de-4813-b407-ee6836d0a777-bound-sa-token\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 00:59:54.401252 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:54.401254    2569 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/75b89b12-20de-4813-b407-ee6836d0a777-ca-trust-extracted\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 00:59:54.401446 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:54.401263    2569 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/75b89b12-20de-4813-b407-ee6836d0a777-registry-tls\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 00:59:54.401446 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:54.401271    2569 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/75b89b12-20de-4813-b407-ee6836d0a777-trusted-ca\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 00:59:54.401446 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:54.401279    2569 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/75b89b12-20de-4813-b407-ee6836d0a777-registry-certificates\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 00:59:54.401446 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:54.401286    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v64zn\" (UniqueName: \"kubernetes.io/projected/75b89b12-20de-4813-b407-ee6836d0a777-kube-api-access-v64zn\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 00:59:54.401446 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:54.401296    2569 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/75b89b12-20de-4813-b407-ee6836d0a777-image-registry-private-configuration\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 00:59:54.557141 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:54.557110    2569 generic.go:356] "Generic (PLEG): container finished" podID="75b89b12-20de-4813-b407-ee6836d0a777" containerID="058e12f9811bbc1f7c8807f6b455d17631a597600b882ca2c3c4e0b232a31dd6" exitCode=0
Sep 11 00:59:54.557262 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:54.557155    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6f984758b5-srjjf"
Sep 11 00:59:54.557262 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:54.557158    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6f984758b5-srjjf" event={"ID":"75b89b12-20de-4813-b407-ee6836d0a777","Type":"ContainerDied","Data":"058e12f9811bbc1f7c8807f6b455d17631a597600b882ca2c3c4e0b232a31dd6"}
Sep 11 00:59:54.557262 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:54.557199    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6f984758b5-srjjf" event={"ID":"75b89b12-20de-4813-b407-ee6836d0a777","Type":"ContainerDied","Data":"4946b8a1969eaf1fbae45d45d5d489bd622f3cb0dbda9e11c9236686cbfbe416"}
Sep 11 00:59:54.557262 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:54.557219    2569 scope.go:117] "RemoveContainer" containerID="058e12f9811bbc1f7c8807f6b455d17631a597600b882ca2c3c4e0b232a31dd6"
Sep 11 00:59:54.566516 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:54.566498    2569 scope.go:117] "RemoveContainer" containerID="058e12f9811bbc1f7c8807f6b455d17631a597600b882ca2c3c4e0b232a31dd6"
Sep 11 00:59:54.566791 ip-10-0-129-222 kubenswrapper[2569]: E0911 00:59:54.566770    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"058e12f9811bbc1f7c8807f6b455d17631a597600b882ca2c3c4e0b232a31dd6\": container with ID starting with 058e12f9811bbc1f7c8807f6b455d17631a597600b882ca2c3c4e0b232a31dd6 not found: ID does not exist" containerID="058e12f9811bbc1f7c8807f6b455d17631a597600b882ca2c3c4e0b232a31dd6"
Sep 11 00:59:54.566849 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:54.566799    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"058e12f9811bbc1f7c8807f6b455d17631a597600b882ca2c3c4e0b232a31dd6"} err="failed to get container status \"058e12f9811bbc1f7c8807f6b455d17631a597600b882ca2c3c4e0b232a31dd6\": rpc error: code = NotFound desc = could not find container \"058e12f9811bbc1f7c8807f6b455d17631a597600b882ca2c3c4e0b232a31dd6\": container with ID starting with 058e12f9811bbc1f7c8807f6b455d17631a597600b882ca2c3c4e0b232a31dd6 not found: ID does not exist"
Sep 11 00:59:54.577526 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:54.577504    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6f984758b5-srjjf"]
Sep 11 00:59:54.583312 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:54.583291    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-6f984758b5-srjjf"]
Sep 11 00:59:55.611803 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:55.611767    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85c82c9f-52d0-49cd-afd9-b791628e1a50-metrics-certs\") pod \"network-metrics-daemon-r8n96\" (UID: \"85c82c9f-52d0-49cd-afd9-b791628e1a50\") " pod="openshift-multus/network-metrics-daemon-r8n96"
Sep 11 00:59:55.614135 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:55.614111    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85c82c9f-52d0-49cd-afd9-b791628e1a50-metrics-certs\") pod \"network-metrics-daemon-r8n96\" (UID: \"85c82c9f-52d0-49cd-afd9-b791628e1a50\") " pod="openshift-multus/network-metrics-daemon-r8n96"
Sep 11 00:59:55.678390 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:55.678361    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-lb8gv\""
Sep 11 00:59:55.686201 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:55.686182    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r8n96"
Sep 11 00:59:55.869903 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:55.869815    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="75b89b12-20de-4813-b407-ee6836d0a777" path="/var/lib/kubelet/pods/75b89b12-20de-4813-b407-ee6836d0a777/volumes"
Sep 11 00:59:55.954944 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:55.954810    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-r8n96"]
Sep 11 00:59:55.957359 ip-10-0-129-222 kubenswrapper[2569]: W0911 00:59:55.957334    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85c82c9f_52d0_49cd_afd9_b791628e1a50.slice/crio-228de7fe171817986a9486a910fdcdc4db534ef5eaf194ad4d27452a1278610a WatchSource:0}: Error finding container 228de7fe171817986a9486a910fdcdc4db534ef5eaf194ad4d27452a1278610a: Status 404 returned error can't find the container with id 228de7fe171817986a9486a910fdcdc4db534ef5eaf194ad4d27452a1278610a
Sep 11 00:59:56.569015 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:56.568979    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-r8n96" event={"ID":"85c82c9f-52d0-49cd-afd9-b791628e1a50","Type":"ContainerStarted","Data":"228de7fe171817986a9486a910fdcdc4db534ef5eaf194ad4d27452a1278610a"}
Sep 11 00:59:57.574847 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:57.574814    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-r8n96" event={"ID":"85c82c9f-52d0-49cd-afd9-b791628e1a50","Type":"ContainerStarted","Data":"7ac2eb239a10f46b5d2b5f56015270ad5cc322d86118bebbcaf2d63a75b1d114"}
Sep 11 00:59:57.575230 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:57.574879    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-r8n96" event={"ID":"85c82c9f-52d0-49cd-afd9-b791628e1a50","Type":"ContainerStarted","Data":"a508ab140ff216f911a3c6af4a62f3aa9fbe51a96b058a372d469776f3629db9"}
Sep 11 00:59:57.591263 ip-10-0-129-222 kubenswrapper[2569]: I0911 00:59:57.591208    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-r8n96" podStartSLOduration=130.653728861 podStartE2EDuration="2m11.591191113s" totalImagesPullingTime="937.462252ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 00:57:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 00:59:57.589365884 +0000 UTC m=+132.240740133" watchObservedRunningTime="2026-09-11 00:59:57.591191113 +0000 UTC m=+132.242565362"
Sep 11 01:00:27.936735 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:27.936693    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 01:00:27.960697 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:27.960672    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 01:00:28.726643 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:28.726619    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 01:00:41.691391 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:41.691304    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 11 01:00:41.691960 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:41.691803    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="4e0aeff2-324c-44d3-87ed-6bd7237f2df0" containerName="alertmanager" containerID="cri-o://5f5c1cf91412b31f1548b369df4fca9309874c2626970d85af8b90c60a95610c" gracePeriod=120
Sep 11 01:00:41.691960 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:41.691820    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="4e0aeff2-324c-44d3-87ed-6bd7237f2df0" containerName="kube-rbac-proxy-metric" containerID="cri-o://2f0872639447bd04f69a24870b358f71aa162d9b8c31a51eeede435e1cb16804" gracePeriod=120
Sep 11 01:00:41.691960 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:41.691819    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="4e0aeff2-324c-44d3-87ed-6bd7237f2df0" containerName="kube-rbac-proxy-web" containerID="cri-o://b04795c23ac48b15a5ec412a4d8149f31882c60c24bf8186e487401c3582a1f0" gracePeriod=120
Sep 11 01:00:41.691960 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:41.691837    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="4e0aeff2-324c-44d3-87ed-6bd7237f2df0" containerName="config-reloader" containerID="cri-o://548ad5207ec4dc685f6c4bf27d5e8ce2429e33b5476a98543f9d5f0f9e06b077" gracePeriod=120
Sep 11 01:00:41.691960 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:41.691894    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="4e0aeff2-324c-44d3-87ed-6bd7237f2df0" containerName="kube-rbac-proxy" containerID="cri-o://c10c594e1c733c7ec1757920c0d1e26db7459a312218aeba65cef53c10fad0a6" gracePeriod=120
Sep 11 01:00:41.691960 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:41.691912    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="4e0aeff2-324c-44d3-87ed-6bd7237f2df0" containerName="prom-label-proxy" containerID="cri-o://6d29af2022cb3c11d7c59e5cc833a628020ea0eb69875c5f01d36b865c2759b0" gracePeriod=120
Sep 11 01:00:42.777026 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:42.776994    2569 generic.go:356] "Generic (PLEG): container finished" podID="4e0aeff2-324c-44d3-87ed-6bd7237f2df0" containerID="6d29af2022cb3c11d7c59e5cc833a628020ea0eb69875c5f01d36b865c2759b0" exitCode=0
Sep 11 01:00:42.777026 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:42.777019    2569 generic.go:356] "Generic (PLEG): container finished" podID="4e0aeff2-324c-44d3-87ed-6bd7237f2df0" containerID="c10c594e1c733c7ec1757920c0d1e26db7459a312218aeba65cef53c10fad0a6" exitCode=0
Sep 11 01:00:42.777026 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:42.777026    2569 generic.go:356] "Generic (PLEG): container finished" podID="4e0aeff2-324c-44d3-87ed-6bd7237f2df0" containerID="548ad5207ec4dc685f6c4bf27d5e8ce2429e33b5476a98543f9d5f0f9e06b077" exitCode=0
Sep 11 01:00:42.777026 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:42.777032    2569 generic.go:356] "Generic (PLEG): container finished" podID="4e0aeff2-324c-44d3-87ed-6bd7237f2df0" containerID="5f5c1cf91412b31f1548b369df4fca9309874c2626970d85af8b90c60a95610c" exitCode=0
Sep 11 01:00:42.777420 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:42.777060    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4e0aeff2-324c-44d3-87ed-6bd7237f2df0","Type":"ContainerDied","Data":"6d29af2022cb3c11d7c59e5cc833a628020ea0eb69875c5f01d36b865c2759b0"}
Sep 11 01:00:42.777420 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:42.777121    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4e0aeff2-324c-44d3-87ed-6bd7237f2df0","Type":"ContainerDied","Data":"c10c594e1c733c7ec1757920c0d1e26db7459a312218aeba65cef53c10fad0a6"}
Sep 11 01:00:42.777420 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:42.777149    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4e0aeff2-324c-44d3-87ed-6bd7237f2df0","Type":"ContainerDied","Data":"548ad5207ec4dc685f6c4bf27d5e8ce2429e33b5476a98543f9d5f0f9e06b077"}
Sep 11 01:00:42.777420 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:42.777173    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4e0aeff2-324c-44d3-87ed-6bd7237f2df0","Type":"ContainerDied","Data":"5f5c1cf91412b31f1548b369df4fca9309874c2626970d85af8b90c60a95610c"}
Sep 11 01:00:42.926906 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:42.926863    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 11 01:00:42.994568 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:42.994535    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4e0aeff2-324c-44d3-87ed-6bd7237f2df0-alertmanager-trusted-ca-bundle\") pod \"4e0aeff2-324c-44d3-87ed-6bd7237f2df0\" (UID: \"4e0aeff2-324c-44d3-87ed-6bd7237f2df0\") "
Sep 11 01:00:42.994727 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:42.994573    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/4e0aeff2-324c-44d3-87ed-6bd7237f2df0-cluster-tls-config\") pod \"4e0aeff2-324c-44d3-87ed-6bd7237f2df0\" (UID: \"4e0aeff2-324c-44d3-87ed-6bd7237f2df0\") "
Sep 11 01:00:42.994727 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:42.994603    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/4e0aeff2-324c-44d3-87ed-6bd7237f2df0-alertmanager-main-db\") pod \"4e0aeff2-324c-44d3-87ed-6bd7237f2df0\" (UID: \"4e0aeff2-324c-44d3-87ed-6bd7237f2df0\") "
Sep 11 01:00:42.994727 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:42.994627    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/4e0aeff2-324c-44d3-87ed-6bd7237f2df0-config-volume\") pod \"4e0aeff2-324c-44d3-87ed-6bd7237f2df0\" (UID: \"4e0aeff2-324c-44d3-87ed-6bd7237f2df0\") "
Sep 11 01:00:42.994727 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:42.994645    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jlx9t\" (UniqueName: \"kubernetes.io/projected/4e0aeff2-324c-44d3-87ed-6bd7237f2df0-kube-api-access-jlx9t\") pod \"4e0aeff2-324c-44d3-87ed-6bd7237f2df0\" (UID: \"4e0aeff2-324c-44d3-87ed-6bd7237f2df0\") "
Sep 11 01:00:42.994727 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:42.994664    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4e0aeff2-324c-44d3-87ed-6bd7237f2df0-config-out\") pod \"4e0aeff2-324c-44d3-87ed-6bd7237f2df0\" (UID: \"4e0aeff2-324c-44d3-87ed-6bd7237f2df0\") "
Sep 11 01:00:42.994727 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:42.994692    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4e0aeff2-324c-44d3-87ed-6bd7237f2df0-secret-alertmanager-kube-rbac-proxy-web\") pod \"4e0aeff2-324c-44d3-87ed-6bd7237f2df0\" (UID: \"4e0aeff2-324c-44d3-87ed-6bd7237f2df0\") "
Sep 11 01:00:42.994727 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:42.994714    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4e0aeff2-324c-44d3-87ed-6bd7237f2df0-metrics-client-ca\") pod \"4e0aeff2-324c-44d3-87ed-6bd7237f2df0\" (UID: \"4e0aeff2-324c-44d3-87ed-6bd7237f2df0\") "
Sep 11 01:00:42.995112 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:42.994772    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4e0aeff2-324c-44d3-87ed-6bd7237f2df0-secret-alertmanager-kube-rbac-proxy\") pod \"4e0aeff2-324c-44d3-87ed-6bd7237f2df0\" (UID: \"4e0aeff2-324c-44d3-87ed-6bd7237f2df0\") "
Sep 11 01:00:42.995112 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:42.994803    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/4e0aeff2-324c-44d3-87ed-6bd7237f2df0-secret-alertmanager-main-tls\") pod \"4e0aeff2-324c-44d3-87ed-6bd7237f2df0\" (UID: \"4e0aeff2-324c-44d3-87ed-6bd7237f2df0\") "
Sep 11 01:00:42.995112 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:42.994820    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/4e0aeff2-324c-44d3-87ed-6bd7237f2df0-secret-alertmanager-kube-rbac-proxy-metric\") pod \"4e0aeff2-324c-44d3-87ed-6bd7237f2df0\" (UID: \"4e0aeff2-324c-44d3-87ed-6bd7237f2df0\") "
Sep 11 01:00:42.995112 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:42.994859    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4e0aeff2-324c-44d3-87ed-6bd7237f2df0-web-config\") pod \"4e0aeff2-324c-44d3-87ed-6bd7237f2df0\" (UID: \"4e0aeff2-324c-44d3-87ed-6bd7237f2df0\") "
Sep 11 01:00:42.995112 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:42.994900    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4e0aeff2-324c-44d3-87ed-6bd7237f2df0-tls-assets\") pod \"4e0aeff2-324c-44d3-87ed-6bd7237f2df0\" (UID: \"4e0aeff2-324c-44d3-87ed-6bd7237f2df0\") "
Sep 11 01:00:42.995356 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:42.995004    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e0aeff2-324c-44d3-87ed-6bd7237f2df0-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "4e0aeff2-324c-44d3-87ed-6bd7237f2df0" (UID: "4e0aeff2-324c-44d3-87ed-6bd7237f2df0"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 01:00:42.995356 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:42.994973    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e0aeff2-324c-44d3-87ed-6bd7237f2df0-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "4e0aeff2-324c-44d3-87ed-6bd7237f2df0" (UID: "4e0aeff2-324c-44d3-87ed-6bd7237f2df0"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 01:00:42.995462 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:42.995350    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e0aeff2-324c-44d3-87ed-6bd7237f2df0-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "4e0aeff2-324c-44d3-87ed-6bd7237f2df0" (UID: "4e0aeff2-324c-44d3-87ed-6bd7237f2df0"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 01:00:42.998585 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:42.998511    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e0aeff2-324c-44d3-87ed-6bd7237f2df0-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "4e0aeff2-324c-44d3-87ed-6bd7237f2df0" (UID: "4e0aeff2-324c-44d3-87ed-6bd7237f2df0"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 01:00:42.998585 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:42.998551    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e0aeff2-324c-44d3-87ed-6bd7237f2df0-config-volume" (OuterVolumeSpecName: "config-volume") pod "4e0aeff2-324c-44d3-87ed-6bd7237f2df0" (UID: "4e0aeff2-324c-44d3-87ed-6bd7237f2df0"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 01:00:42.998585 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:42.998579    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e0aeff2-324c-44d3-87ed-6bd7237f2df0-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "4e0aeff2-324c-44d3-87ed-6bd7237f2df0" (UID: "4e0aeff2-324c-44d3-87ed-6bd7237f2df0"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 01:00:42.998826 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:42.998661    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e0aeff2-324c-44d3-87ed-6bd7237f2df0-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "4e0aeff2-324c-44d3-87ed-6bd7237f2df0" (UID: "4e0aeff2-324c-44d3-87ed-6bd7237f2df0"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 01:00:42.998826 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:42.998785    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e0aeff2-324c-44d3-87ed-6bd7237f2df0-kube-api-access-jlx9t" (OuterVolumeSpecName: "kube-api-access-jlx9t") pod "4e0aeff2-324c-44d3-87ed-6bd7237f2df0" (UID: "4e0aeff2-324c-44d3-87ed-6bd7237f2df0"). InnerVolumeSpecName "kube-api-access-jlx9t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 01:00:42.999680 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:42.999593    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e0aeff2-324c-44d3-87ed-6bd7237f2df0-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "4e0aeff2-324c-44d3-87ed-6bd7237f2df0" (UID: "4e0aeff2-324c-44d3-87ed-6bd7237f2df0"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 01:00:42.999680 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:42.999647    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e0aeff2-324c-44d3-87ed-6bd7237f2df0-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "4e0aeff2-324c-44d3-87ed-6bd7237f2df0" (UID: "4e0aeff2-324c-44d3-87ed-6bd7237f2df0"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 01:00:43.000136 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:43.000114    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e0aeff2-324c-44d3-87ed-6bd7237f2df0-config-out" (OuterVolumeSpecName: "config-out") pod "4e0aeff2-324c-44d3-87ed-6bd7237f2df0" (UID: "4e0aeff2-324c-44d3-87ed-6bd7237f2df0"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 01:00:43.002363 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:43.002332    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e0aeff2-324c-44d3-87ed-6bd7237f2df0-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "4e0aeff2-324c-44d3-87ed-6bd7237f2df0" (UID: "4e0aeff2-324c-44d3-87ed-6bd7237f2df0"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 01:00:43.010267 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:43.010230    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e0aeff2-324c-44d3-87ed-6bd7237f2df0-web-config" (OuterVolumeSpecName: "web-config") pod "4e0aeff2-324c-44d3-87ed-6bd7237f2df0" (UID: "4e0aeff2-324c-44d3-87ed-6bd7237f2df0"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 01:00:43.095628 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:43.095605    2569 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/4e0aeff2-324c-44d3-87ed-6bd7237f2df0-cluster-tls-config\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:00:43.095628 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:43.095630    2569 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/4e0aeff2-324c-44d3-87ed-6bd7237f2df0-alertmanager-main-db\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:00:43.095799 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:43.095641    2569 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/4e0aeff2-324c-44d3-87ed-6bd7237f2df0-config-volume\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:00:43.095799 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:43.095649    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jlx9t\" (UniqueName: \"kubernetes.io/projected/4e0aeff2-324c-44d3-87ed-6bd7237f2df0-kube-api-access-jlx9t\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:00:43.095799 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:43.095657    2569 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4e0aeff2-324c-44d3-87ed-6bd7237f2df0-config-out\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:00:43.095799 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:43.095667    2569 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4e0aeff2-324c-44d3-87ed-6bd7237f2df0-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:00:43.095799 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:43.095675    2569 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4e0aeff2-324c-44d3-87ed-6bd7237f2df0-metrics-client-ca\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:00:43.095799 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:43.095684    2569 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4e0aeff2-324c-44d3-87ed-6bd7237f2df0-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:00:43.095799 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:43.095693    2569 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/4e0aeff2-324c-44d3-87ed-6bd7237f2df0-secret-alertmanager-main-tls\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:00:43.095799 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:43.095701    2569 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/4e0aeff2-324c-44d3-87ed-6bd7237f2df0-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:00:43.095799 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:43.095709    2569 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4e0aeff2-324c-44d3-87ed-6bd7237f2df0-web-config\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:00:43.095799 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:43.095717    2569 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4e0aeff2-324c-44d3-87ed-6bd7237f2df0-tls-assets\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:00:43.095799 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:43.095725    2569 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4e0aeff2-324c-44d3-87ed-6bd7237f2df0-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:00:43.784046 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:43.784017    2569 generic.go:356] "Generic (PLEG): container finished" podID="4e0aeff2-324c-44d3-87ed-6bd7237f2df0" containerID="2f0872639447bd04f69a24870b358f71aa162d9b8c31a51eeede435e1cb16804" exitCode=0
Sep 11 01:00:43.784046 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:43.784040    2569 generic.go:356] "Generic (PLEG): container finished" podID="4e0aeff2-324c-44d3-87ed-6bd7237f2df0" containerID="b04795c23ac48b15a5ec412a4d8149f31882c60c24bf8186e487401c3582a1f0" exitCode=0
Sep 11 01:00:43.784481 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:43.784107    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 11 01:00:43.784481 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:43.784102    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4e0aeff2-324c-44d3-87ed-6bd7237f2df0","Type":"ContainerDied","Data":"2f0872639447bd04f69a24870b358f71aa162d9b8c31a51eeede435e1cb16804"}
Sep 11 01:00:43.784481 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:43.784254    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4e0aeff2-324c-44d3-87ed-6bd7237f2df0","Type":"ContainerDied","Data":"b04795c23ac48b15a5ec412a4d8149f31882c60c24bf8186e487401c3582a1f0"}
Sep 11 01:00:43.784481 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:43.784283    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4e0aeff2-324c-44d3-87ed-6bd7237f2df0","Type":"ContainerDied","Data":"465d6813deb4081df2ac68adcc028734fad98968d7e58c220e877adbd38a4614"}
Sep 11 01:00:43.784481 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:43.784313    2569 scope.go:117] "RemoveContainer" containerID="6d29af2022cb3c11d7c59e5cc833a628020ea0eb69875c5f01d36b865c2759b0"
Sep 11 01:00:43.792087 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:43.792069    2569 scope.go:117] "RemoveContainer" containerID="2f0872639447bd04f69a24870b358f71aa162d9b8c31a51eeede435e1cb16804"
Sep 11 01:00:43.798979 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:43.798961    2569 scope.go:117] "RemoveContainer" containerID="c10c594e1c733c7ec1757920c0d1e26db7459a312218aeba65cef53c10fad0a6"
Sep 11 01:00:43.806017 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:43.805812    2569 scope.go:117] "RemoveContainer" containerID="b04795c23ac48b15a5ec412a4d8149f31882c60c24bf8186e487401c3582a1f0"
Sep 11 01:00:43.807525 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:43.807509    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 11 01:00:43.811100 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:43.811079    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 11 01:00:43.813760 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:43.813746    2569 scope.go:117] "RemoveContainer" containerID="548ad5207ec4dc685f6c4bf27d5e8ce2429e33b5476a98543f9d5f0f9e06b077"
Sep 11 01:00:43.820224 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:43.820209    2569 scope.go:117] "RemoveContainer" containerID="5f5c1cf91412b31f1548b369df4fca9309874c2626970d85af8b90c60a95610c"
Sep 11 01:00:43.826937 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:43.826920    2569 scope.go:117] "RemoveContainer" containerID="2aae85020e3b8e8b190b0f56277444eaafa2fffe0cb2c6300a7a1f3490838623"
Sep 11 01:00:43.833769 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:43.833754    2569 scope.go:117] "RemoveContainer" containerID="6d29af2022cb3c11d7c59e5cc833a628020ea0eb69875c5f01d36b865c2759b0"
Sep 11 01:00:43.834066 ip-10-0-129-222 kubenswrapper[2569]: E0911 01:00:43.834046    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d29af2022cb3c11d7c59e5cc833a628020ea0eb69875c5f01d36b865c2759b0\": container with ID starting with 6d29af2022cb3c11d7c59e5cc833a628020ea0eb69875c5f01d36b865c2759b0 not found: ID does not exist" containerID="6d29af2022cb3c11d7c59e5cc833a628020ea0eb69875c5f01d36b865c2759b0"
Sep 11 01:00:43.834151 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:43.834070    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d29af2022cb3c11d7c59e5cc833a628020ea0eb69875c5f01d36b865c2759b0"} err="failed to get container status \"6d29af2022cb3c11d7c59e5cc833a628020ea0eb69875c5f01d36b865c2759b0\": rpc error: code = NotFound desc = could not find container \"6d29af2022cb3c11d7c59e5cc833a628020ea0eb69875c5f01d36b865c2759b0\": container with ID starting with 6d29af2022cb3c11d7c59e5cc833a628020ea0eb69875c5f01d36b865c2759b0 not found: ID does not exist"
Sep 11 01:00:43.834151 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:43.834088    2569 scope.go:117] "RemoveContainer" containerID="2f0872639447bd04f69a24870b358f71aa162d9b8c31a51eeede435e1cb16804"
Sep 11 01:00:43.834349 ip-10-0-129-222 kubenswrapper[2569]: E0911 01:00:43.834331    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f0872639447bd04f69a24870b358f71aa162d9b8c31a51eeede435e1cb16804\": container with ID starting with 2f0872639447bd04f69a24870b358f71aa162d9b8c31a51eeede435e1cb16804 not found: ID does not exist" containerID="2f0872639447bd04f69a24870b358f71aa162d9b8c31a51eeede435e1cb16804"
Sep 11 01:00:43.834392 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:43.834359    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f0872639447bd04f69a24870b358f71aa162d9b8c31a51eeede435e1cb16804"} err="failed to get container status \"2f0872639447bd04f69a24870b358f71aa162d9b8c31a51eeede435e1cb16804\": rpc error: code = NotFound desc = could not find container \"2f0872639447bd04f69a24870b358f71aa162d9b8c31a51eeede435e1cb16804\": container with ID starting with 2f0872639447bd04f69a24870b358f71aa162d9b8c31a51eeede435e1cb16804 not found: ID does not exist"
Sep 11 01:00:43.834392 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:43.834385    2569 scope.go:117] "RemoveContainer" containerID="c10c594e1c733c7ec1757920c0d1e26db7459a312218aeba65cef53c10fad0a6"
Sep 11 01:00:43.834608 ip-10-0-129-222 kubenswrapper[2569]: E0911 01:00:43.834593    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c10c594e1c733c7ec1757920c0d1e26db7459a312218aeba65cef53c10fad0a6\": container with ID starting with c10c594e1c733c7ec1757920c0d1e26db7459a312218aeba65cef53c10fad0a6 not found: ID does not exist" containerID="c10c594e1c733c7ec1757920c0d1e26db7459a312218aeba65cef53c10fad0a6"
Sep 11 01:00:43.834649 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:43.834611    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c10c594e1c733c7ec1757920c0d1e26db7459a312218aeba65cef53c10fad0a6"} err="failed to get container status \"c10c594e1c733c7ec1757920c0d1e26db7459a312218aeba65cef53c10fad0a6\": rpc error: code = NotFound desc = could not find container \"c10c594e1c733c7ec1757920c0d1e26db7459a312218aeba65cef53c10fad0a6\": container with ID starting with c10c594e1c733c7ec1757920c0d1e26db7459a312218aeba65cef53c10fad0a6 not found: ID does not exist"
Sep 11 01:00:43.834649 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:43.834625    2569 scope.go:117] "RemoveContainer" containerID="b04795c23ac48b15a5ec412a4d8149f31882c60c24bf8186e487401c3582a1f0"
Sep 11 01:00:43.834828 ip-10-0-129-222 kubenswrapper[2569]: E0911 01:00:43.834814    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b04795c23ac48b15a5ec412a4d8149f31882c60c24bf8186e487401c3582a1f0\": container with ID starting with b04795c23ac48b15a5ec412a4d8149f31882c60c24bf8186e487401c3582a1f0 not found: ID does not exist" containerID="b04795c23ac48b15a5ec412a4d8149f31882c60c24bf8186e487401c3582a1f0"
Sep 11 01:00:43.834890 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:43.834831    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b04795c23ac48b15a5ec412a4d8149f31882c60c24bf8186e487401c3582a1f0"} err="failed to get container status \"b04795c23ac48b15a5ec412a4d8149f31882c60c24bf8186e487401c3582a1f0\": rpc error: code = NotFound desc = could not find container \"b04795c23ac48b15a5ec412a4d8149f31882c60c24bf8186e487401c3582a1f0\": container with ID starting with b04795c23ac48b15a5ec412a4d8149f31882c60c24bf8186e487401c3582a1f0 not found: ID does not exist"
Sep 11 01:00:43.834890 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:43.834844    2569 scope.go:117] "RemoveContainer" containerID="548ad5207ec4dc685f6c4bf27d5e8ce2429e33b5476a98543f9d5f0f9e06b077"
Sep 11 01:00:43.835068 ip-10-0-129-222 kubenswrapper[2569]: E0911 01:00:43.835053    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"548ad5207ec4dc685f6c4bf27d5e8ce2429e33b5476a98543f9d5f0f9e06b077\": container with ID starting with 548ad5207ec4dc685f6c4bf27d5e8ce2429e33b5476a98543f9d5f0f9e06b077 not found: ID does not exist" containerID="548ad5207ec4dc685f6c4bf27d5e8ce2429e33b5476a98543f9d5f0f9e06b077"
Sep 11 01:00:43.835110 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:43.835071    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"548ad5207ec4dc685f6c4bf27d5e8ce2429e33b5476a98543f9d5f0f9e06b077"} err="failed to get container status \"548ad5207ec4dc685f6c4bf27d5e8ce2429e33b5476a98543f9d5f0f9e06b077\": rpc error: code = NotFound desc = could not find container \"548ad5207ec4dc685f6c4bf27d5e8ce2429e33b5476a98543f9d5f0f9e06b077\": container with ID starting with 548ad5207ec4dc685f6c4bf27d5e8ce2429e33b5476a98543f9d5f0f9e06b077 not found: ID does not exist"
Sep 11 01:00:43.835110 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:43.835083    2569 scope.go:117] "RemoveContainer" containerID="5f5c1cf91412b31f1548b369df4fca9309874c2626970d85af8b90c60a95610c"
Sep 11 01:00:43.835269 ip-10-0-129-222 kubenswrapper[2569]: E0911 01:00:43.835254    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f5c1cf91412b31f1548b369df4fca9309874c2626970d85af8b90c60a95610c\": container with ID starting with 5f5c1cf91412b31f1548b369df4fca9309874c2626970d85af8b90c60a95610c not found: ID does not exist" containerID="5f5c1cf91412b31f1548b369df4fca9309874c2626970d85af8b90c60a95610c"
Sep 11 01:00:43.835307 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:43.835270    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f5c1cf91412b31f1548b369df4fca9309874c2626970d85af8b90c60a95610c"} err="failed to get container status \"5f5c1cf91412b31f1548b369df4fca9309874c2626970d85af8b90c60a95610c\": rpc error: code = NotFound desc = could not find container \"5f5c1cf91412b31f1548b369df4fca9309874c2626970d85af8b90c60a95610c\": container with ID starting with 5f5c1cf91412b31f1548b369df4fca9309874c2626970d85af8b90c60a95610c not found: ID does not exist"
Sep 11 01:00:43.835307 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:43.835281    2569 scope.go:117] "RemoveContainer" containerID="2aae85020e3b8e8b190b0f56277444eaafa2fffe0cb2c6300a7a1f3490838623"
Sep 11 01:00:43.835521 ip-10-0-129-222 kubenswrapper[2569]: E0911 01:00:43.835501    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2aae85020e3b8e8b190b0f56277444eaafa2fffe0cb2c6300a7a1f3490838623\": container with ID starting with 2aae85020e3b8e8b190b0f56277444eaafa2fffe0cb2c6300a7a1f3490838623 not found: ID does not exist" containerID="2aae85020e3b8e8b190b0f56277444eaafa2fffe0cb2c6300a7a1f3490838623"
Sep 11 01:00:43.835592 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:43.835528    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2aae85020e3b8e8b190b0f56277444eaafa2fffe0cb2c6300a7a1f3490838623"} err="failed to get container status \"2aae85020e3b8e8b190b0f56277444eaafa2fffe0cb2c6300a7a1f3490838623\": rpc error: code = NotFound desc = could not find container \"2aae85020e3b8e8b190b0f56277444eaafa2fffe0cb2c6300a7a1f3490838623\": container with ID starting with 2aae85020e3b8e8b190b0f56277444eaafa2fffe0cb2c6300a7a1f3490838623 not found: ID does not exist"
Sep 11 01:00:43.835592 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:43.835549    2569 scope.go:117] "RemoveContainer" containerID="6d29af2022cb3c11d7c59e5cc833a628020ea0eb69875c5f01d36b865c2759b0"
Sep 11 01:00:43.836050 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:43.835919    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d29af2022cb3c11d7c59e5cc833a628020ea0eb69875c5f01d36b865c2759b0"} err="failed to get container status \"6d29af2022cb3c11d7c59e5cc833a628020ea0eb69875c5f01d36b865c2759b0\": rpc error: code = NotFound desc = could not find container \"6d29af2022cb3c11d7c59e5cc833a628020ea0eb69875c5f01d36b865c2759b0\": container with ID starting with 6d29af2022cb3c11d7c59e5cc833a628020ea0eb69875c5f01d36b865c2759b0 not found: ID does not exist"
Sep 11 01:00:43.836050 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:43.835950    2569 scope.go:117] "RemoveContainer" containerID="2f0872639447bd04f69a24870b358f71aa162d9b8c31a51eeede435e1cb16804"
Sep 11 01:00:43.836291 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:43.836261    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f0872639447bd04f69a24870b358f71aa162d9b8c31a51eeede435e1cb16804"} err="failed to get container status \"2f0872639447bd04f69a24870b358f71aa162d9b8c31a51eeede435e1cb16804\": rpc error: code = NotFound desc = could not find container \"2f0872639447bd04f69a24870b358f71aa162d9b8c31a51eeede435e1cb16804\": container with ID starting with 2f0872639447bd04f69a24870b358f71aa162d9b8c31a51eeede435e1cb16804 not found: ID does not exist"
Sep 11 01:00:43.836291 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:43.836288    2569 scope.go:117] "RemoveContainer" containerID="c10c594e1c733c7ec1757920c0d1e26db7459a312218aeba65cef53c10fad0a6"
Sep 11 01:00:43.836911 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:43.836565    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c10c594e1c733c7ec1757920c0d1e26db7459a312218aeba65cef53c10fad0a6"} err="failed to get container status \"c10c594e1c733c7ec1757920c0d1e26db7459a312218aeba65cef53c10fad0a6\": rpc error: code = NotFound desc = could not find container \"c10c594e1c733c7ec1757920c0d1e26db7459a312218aeba65cef53c10fad0a6\": container with ID starting with c10c594e1c733c7ec1757920c0d1e26db7459a312218aeba65cef53c10fad0a6 not found: ID does not exist"
Sep 11 01:00:43.836911 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:43.836588    2569 scope.go:117] "RemoveContainer" containerID="b04795c23ac48b15a5ec412a4d8149f31882c60c24bf8186e487401c3582a1f0"
Sep 11 01:00:43.836911 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:43.836839    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b04795c23ac48b15a5ec412a4d8149f31882c60c24bf8186e487401c3582a1f0"} err="failed to get container status \"b04795c23ac48b15a5ec412a4d8149f31882c60c24bf8186e487401c3582a1f0\": rpc error: code = NotFound desc = could not find container \"b04795c23ac48b15a5ec412a4d8149f31882c60c24bf8186e487401c3582a1f0\": container with ID starting with b04795c23ac48b15a5ec412a4d8149f31882c60c24bf8186e487401c3582a1f0 not found: ID does not exist"
Sep 11 01:00:43.836911 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:43.836864    2569 scope.go:117] "RemoveContainer" containerID="548ad5207ec4dc685f6c4bf27d5e8ce2429e33b5476a98543f9d5f0f9e06b077"
Sep 11 01:00:43.837183 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:43.837164    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"548ad5207ec4dc685f6c4bf27d5e8ce2429e33b5476a98543f9d5f0f9e06b077"} err="failed to get container status \"548ad5207ec4dc685f6c4bf27d5e8ce2429e33b5476a98543f9d5f0f9e06b077\": rpc error: code = NotFound desc = could not find container \"548ad5207ec4dc685f6c4bf27d5e8ce2429e33b5476a98543f9d5f0f9e06b077\": container with ID starting with 548ad5207ec4dc685f6c4bf27d5e8ce2429e33b5476a98543f9d5f0f9e06b077 not found: ID does not exist"
Sep 11 01:00:43.837232 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:43.837192    2569 scope.go:117] "RemoveContainer" containerID="5f5c1cf91412b31f1548b369df4fca9309874c2626970d85af8b90c60a95610c"
Sep 11 01:00:43.837468 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:43.837448    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f5c1cf91412b31f1548b369df4fca9309874c2626970d85af8b90c60a95610c"} err="failed to get container status \"5f5c1cf91412b31f1548b369df4fca9309874c2626970d85af8b90c60a95610c\": rpc error: code = NotFound desc = could not find container \"5f5c1cf91412b31f1548b369df4fca9309874c2626970d85af8b90c60a95610c\": container with ID starting with 5f5c1cf91412b31f1548b369df4fca9309874c2626970d85af8b90c60a95610c not found: ID does not exist"
Sep 11 01:00:43.837555 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:43.837471    2569 scope.go:117] "RemoveContainer" containerID="2aae85020e3b8e8b190b0f56277444eaafa2fffe0cb2c6300a7a1f3490838623"
Sep 11 01:00:43.837743 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:43.837725    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 11 01:00:43.837804 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:43.837739    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2aae85020e3b8e8b190b0f56277444eaafa2fffe0cb2c6300a7a1f3490838623"} err="failed to get container status \"2aae85020e3b8e8b190b0f56277444eaafa2fffe0cb2c6300a7a1f3490838623\": rpc error: code = NotFound desc = could not find container \"2aae85020e3b8e8b190b0f56277444eaafa2fffe0cb2c6300a7a1f3490838623\": container with ID starting with 2aae85020e3b8e8b190b0f56277444eaafa2fffe0cb2c6300a7a1f3490838623 not found: ID does not exist"
Sep 11 01:00:43.838076 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:43.838062    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="75b89b12-20de-4813-b407-ee6836d0a777" containerName="registry"
Sep 11 01:00:43.838127 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:43.838077    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="75b89b12-20de-4813-b407-ee6836d0a777" containerName="registry"
Sep 11 01:00:43.838127 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:43.838091    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e0aeff2-324c-44d3-87ed-6bd7237f2df0" containerName="kube-rbac-proxy-web"
Sep 11 01:00:43.838127 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:43.838097    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e0aeff2-324c-44d3-87ed-6bd7237f2df0" containerName="kube-rbac-proxy-web"
Sep 11 01:00:43.838127 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:43.838109    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e0aeff2-324c-44d3-87ed-6bd7237f2df0" containerName="kube-rbac-proxy-metric"
Sep 11 01:00:43.838127 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:43.838114    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e0aeff2-324c-44d3-87ed-6bd7237f2df0" containerName="kube-rbac-proxy-metric"
Sep 11 01:00:43.838127 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:43.838126    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e0aeff2-324c-44d3-87ed-6bd7237f2df0" containerName="prom-label-proxy"
Sep 11 01:00:43.838306 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:43.838134    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e0aeff2-324c-44d3-87ed-6bd7237f2df0" containerName="prom-label-proxy"
Sep 11 01:00:43.838306 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:43.838143    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e0aeff2-324c-44d3-87ed-6bd7237f2df0" containerName="alertmanager"
Sep 11 01:00:43.838306 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:43.838148    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e0aeff2-324c-44d3-87ed-6bd7237f2df0" containerName="alertmanager"
Sep 11 01:00:43.838306 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:43.838155    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e0aeff2-324c-44d3-87ed-6bd7237f2df0" containerName="config-reloader"
Sep 11 01:00:43.838306 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:43.838160    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e0aeff2-324c-44d3-87ed-6bd7237f2df0" containerName="config-reloader"
Sep 11 01:00:43.838306 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:43.838166    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e0aeff2-324c-44d3-87ed-6bd7237f2df0" containerName="init-config-reloader"
Sep 11 01:00:43.838306 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:43.838170    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e0aeff2-324c-44d3-87ed-6bd7237f2df0" containerName="init-config-reloader"
Sep 11 01:00:43.838306 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:43.838177    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e0aeff2-324c-44d3-87ed-6bd7237f2df0" containerName="kube-rbac-proxy"
Sep 11 01:00:43.838306 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:43.838181    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e0aeff2-324c-44d3-87ed-6bd7237f2df0" containerName="kube-rbac-proxy"
Sep 11 01:00:43.838306 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:43.838229    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e0aeff2-324c-44d3-87ed-6bd7237f2df0" containerName="kube-rbac-proxy-web"
Sep 11 01:00:43.838306 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:43.838237    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e0aeff2-324c-44d3-87ed-6bd7237f2df0" containerName="kube-rbac-proxy-metric"
Sep 11 01:00:43.838306 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:43.838244    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e0aeff2-324c-44d3-87ed-6bd7237f2df0" containerName="kube-rbac-proxy"
Sep 11 01:00:43.838306 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:43.838250    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="75b89b12-20de-4813-b407-ee6836d0a777" containerName="registry"
Sep 11 01:00:43.838306 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:43.838256    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e0aeff2-324c-44d3-87ed-6bd7237f2df0" containerName="alertmanager"
Sep 11 01:00:43.838306 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:43.838269    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e0aeff2-324c-44d3-87ed-6bd7237f2df0" containerName="prom-label-proxy"
Sep 11 01:00:43.838306 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:43.838277    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e0aeff2-324c-44d3-87ed-6bd7237f2df0" containerName="config-reloader"
Sep 11 01:00:43.841728 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:43.841710    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 11 01:00:43.844105 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:43.844082    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 11 01:00:43.844246 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:43.844226    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 11 01:00:43.844311 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:43.844231    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 11 01:00:43.844311 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:43.844294    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 11 01:00:43.844416 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:43.844311    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 11 01:00:43.844416 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:43.844364    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 11 01:00:43.844416 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:43.844367    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-dgkvd\""
Sep 11 01:00:43.844562 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:43.844467    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 11 01:00:43.844707 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:43.844690    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 11 01:00:43.850489 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:43.850415    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 11 01:00:43.856510 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:43.856490    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 11 01:00:43.869061 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:43.869037    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4e0aeff2-324c-44d3-87ed-6bd7237f2df0" path="/var/lib/kubelet/pods/4e0aeff2-324c-44d3-87ed-6bd7237f2df0/volumes"
Sep 11 01:00:43.900548 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:43.900519    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/575eb6a2-71fc-481b-a5da-84ebbdef443f-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"575eb6a2-71fc-481b-a5da-84ebbdef443f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 01:00:43.900668 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:43.900552    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/575eb6a2-71fc-481b-a5da-84ebbdef443f-config-volume\") pod \"alertmanager-main-0\" (UID: \"575eb6a2-71fc-481b-a5da-84ebbdef443f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 01:00:43.900668 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:43.900576    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/575eb6a2-71fc-481b-a5da-84ebbdef443f-web-config\") pod \"alertmanager-main-0\" (UID: \"575eb6a2-71fc-481b-a5da-84ebbdef443f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 01:00:43.900668 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:43.900615    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkbvr\" (UniqueName: \"kubernetes.io/projected/575eb6a2-71fc-481b-a5da-84ebbdef443f-kube-api-access-zkbvr\") pod \"alertmanager-main-0\" (UID: \"575eb6a2-71fc-481b-a5da-84ebbdef443f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 01:00:43.900668 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:43.900641    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/575eb6a2-71fc-481b-a5da-84ebbdef443f-config-out\") pod \"alertmanager-main-0\" (UID: \"575eb6a2-71fc-481b-a5da-84ebbdef443f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 01:00:43.900821 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:43.900669    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/575eb6a2-71fc-481b-a5da-84ebbdef443f-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"575eb6a2-71fc-481b-a5da-84ebbdef443f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 01:00:43.900821 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:43.900732    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/575eb6a2-71fc-481b-a5da-84ebbdef443f-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"575eb6a2-71fc-481b-a5da-84ebbdef443f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 01:00:43.900821 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:43.900759    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/575eb6a2-71fc-481b-a5da-84ebbdef443f-tls-assets\") pod \"alertmanager-main-0\" (UID: \"575eb6a2-71fc-481b-a5da-84ebbdef443f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 01:00:43.900821 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:43.900792    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/575eb6a2-71fc-481b-a5da-84ebbdef443f-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"575eb6a2-71fc-481b-a5da-84ebbdef443f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 01:00:43.900821 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:43.900812    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/575eb6a2-71fc-481b-a5da-84ebbdef443f-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"575eb6a2-71fc-481b-a5da-84ebbdef443f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 01:00:43.901047 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:43.900834    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/575eb6a2-71fc-481b-a5da-84ebbdef443f-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"575eb6a2-71fc-481b-a5da-84ebbdef443f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 01:00:43.901047 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:43.900908    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/575eb6a2-71fc-481b-a5da-84ebbdef443f-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"575eb6a2-71fc-481b-a5da-84ebbdef443f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 01:00:43.901047 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:43.900947    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/575eb6a2-71fc-481b-a5da-84ebbdef443f-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"575eb6a2-71fc-481b-a5da-84ebbdef443f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 01:00:44.001620 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:44.001584    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/575eb6a2-71fc-481b-a5da-84ebbdef443f-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"575eb6a2-71fc-481b-a5da-84ebbdef443f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 01:00:44.001620 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:44.001626    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/575eb6a2-71fc-481b-a5da-84ebbdef443f-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"575eb6a2-71fc-481b-a5da-84ebbdef443f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 01:00:44.001815 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:44.001647    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/575eb6a2-71fc-481b-a5da-84ebbdef443f-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"575eb6a2-71fc-481b-a5da-84ebbdef443f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 01:00:44.001815 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:44.001670    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/575eb6a2-71fc-481b-a5da-84ebbdef443f-config-volume\") pod \"alertmanager-main-0\" (UID: \"575eb6a2-71fc-481b-a5da-84ebbdef443f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 01:00:44.001815 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:44.001692    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/575eb6a2-71fc-481b-a5da-84ebbdef443f-web-config\") pod \"alertmanager-main-0\" (UID: \"575eb6a2-71fc-481b-a5da-84ebbdef443f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 01:00:44.001815 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:44.001719    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkbvr\" (UniqueName: \"kubernetes.io/projected/575eb6a2-71fc-481b-a5da-84ebbdef443f-kube-api-access-zkbvr\") pod \"alertmanager-main-0\" (UID: \"575eb6a2-71fc-481b-a5da-84ebbdef443f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 01:00:44.001815 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:44.001746    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/575eb6a2-71fc-481b-a5da-84ebbdef443f-config-out\") pod \"alertmanager-main-0\" (UID: \"575eb6a2-71fc-481b-a5da-84ebbdef443f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 01:00:44.001815 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:44.001774    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/575eb6a2-71fc-481b-a5da-84ebbdef443f-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"575eb6a2-71fc-481b-a5da-84ebbdef443f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 01:00:44.001815 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:44.001811    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/575eb6a2-71fc-481b-a5da-84ebbdef443f-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"575eb6a2-71fc-481b-a5da-84ebbdef443f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 01:00:44.002168 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:44.001837    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/575eb6a2-71fc-481b-a5da-84ebbdef443f-tls-assets\") pod \"alertmanager-main-0\" (UID: \"575eb6a2-71fc-481b-a5da-84ebbdef443f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 01:00:44.002168 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:44.001887    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/575eb6a2-71fc-481b-a5da-84ebbdef443f-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"575eb6a2-71fc-481b-a5da-84ebbdef443f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 01:00:44.002168 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:44.001923    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/575eb6a2-71fc-481b-a5da-84ebbdef443f-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"575eb6a2-71fc-481b-a5da-84ebbdef443f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 01:00:44.002168 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:44.001955    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/575eb6a2-71fc-481b-a5da-84ebbdef443f-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"575eb6a2-71fc-481b-a5da-84ebbdef443f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 01:00:44.002168 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:44.002049    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/575eb6a2-71fc-481b-a5da-84ebbdef443f-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"575eb6a2-71fc-481b-a5da-84ebbdef443f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 01:00:44.002776 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:44.002750    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/575eb6a2-71fc-481b-a5da-84ebbdef443f-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"575eb6a2-71fc-481b-a5da-84ebbdef443f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 01:00:44.004605 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:44.004573    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/575eb6a2-71fc-481b-a5da-84ebbdef443f-config-out\") pod \"alertmanager-main-0\" (UID: \"575eb6a2-71fc-481b-a5da-84ebbdef443f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 01:00:44.005440 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:44.004797    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/575eb6a2-71fc-481b-a5da-84ebbdef443f-tls-assets\") pod \"alertmanager-main-0\" (UID: \"575eb6a2-71fc-481b-a5da-84ebbdef443f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 01:00:44.005440 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:44.004801    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/575eb6a2-71fc-481b-a5da-84ebbdef443f-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"575eb6a2-71fc-481b-a5da-84ebbdef443f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 01:00:44.005440 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:44.004958    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/575eb6a2-71fc-481b-a5da-84ebbdef443f-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"575eb6a2-71fc-481b-a5da-84ebbdef443f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 01:00:44.005440 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:44.005046    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/575eb6a2-71fc-481b-a5da-84ebbdef443f-web-config\") pod \"alertmanager-main-0\" (UID: \"575eb6a2-71fc-481b-a5da-84ebbdef443f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 01:00:44.005440 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:44.005391    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/575eb6a2-71fc-481b-a5da-84ebbdef443f-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"575eb6a2-71fc-481b-a5da-84ebbdef443f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 01:00:44.005440 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:44.005435    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/575eb6a2-71fc-481b-a5da-84ebbdef443f-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"575eb6a2-71fc-481b-a5da-84ebbdef443f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 01:00:44.005767 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:44.005463    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/575eb6a2-71fc-481b-a5da-84ebbdef443f-config-volume\") pod \"alertmanager-main-0\" (UID: \"575eb6a2-71fc-481b-a5da-84ebbdef443f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 01:00:44.005767 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:44.005471    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/575eb6a2-71fc-481b-a5da-84ebbdef443f-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"575eb6a2-71fc-481b-a5da-84ebbdef443f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 01:00:44.006559 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:44.006539    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/575eb6a2-71fc-481b-a5da-84ebbdef443f-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"575eb6a2-71fc-481b-a5da-84ebbdef443f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 01:00:44.009752 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:44.009729    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkbvr\" (UniqueName: \"kubernetes.io/projected/575eb6a2-71fc-481b-a5da-84ebbdef443f-kube-api-access-zkbvr\") pod \"alertmanager-main-0\" (UID: \"575eb6a2-71fc-481b-a5da-84ebbdef443f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 01:00:44.155017 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:44.154988    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 11 01:00:44.432477 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:44.432361    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 11 01:00:44.434148 ip-10-0-129-222 kubenswrapper[2569]: W0911 01:00:44.434122    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod575eb6a2_71fc_481b_a5da_84ebbdef443f.slice/crio-0654bc97162b59fc47c397e872cb8fbb37c3a68a6e9ef3146d729ee91c9193b4 WatchSource:0}: Error finding container 0654bc97162b59fc47c397e872cb8fbb37c3a68a6e9ef3146d729ee91c9193b4: Status 404 returned error can't find the container with id 0654bc97162b59fc47c397e872cb8fbb37c3a68a6e9ef3146d729ee91c9193b4
Sep 11 01:00:44.789159 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:44.789073    2569 generic.go:356] "Generic (PLEG): container finished" podID="575eb6a2-71fc-481b-a5da-84ebbdef443f" containerID="8022a10a25b0bff06b52d71d8a5194ced85892bf52b93bd35529ff6a9e84f2ac" exitCode=0
Sep 11 01:00:44.789159 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:44.789112    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"575eb6a2-71fc-481b-a5da-84ebbdef443f","Type":"ContainerDied","Data":"8022a10a25b0bff06b52d71d8a5194ced85892bf52b93bd35529ff6a9e84f2ac"}
Sep 11 01:00:44.789616 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:44.789194    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"575eb6a2-71fc-481b-a5da-84ebbdef443f","Type":"ContainerStarted","Data":"0654bc97162b59fc47c397e872cb8fbb37c3a68a6e9ef3146d729ee91c9193b4"}
Sep 11 01:00:45.797032 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:45.797002    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"575eb6a2-71fc-481b-a5da-84ebbdef443f","Type":"ContainerStarted","Data":"872a00c54ed899f679397a28fd8029f70014212cbae2923f5db02328b11c5a94"}
Sep 11 01:00:45.797384 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:45.797089    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"575eb6a2-71fc-481b-a5da-84ebbdef443f","Type":"ContainerStarted","Data":"ff7c85a66804a0837eea880e46276568eea8eef1baaa7e097e0cffe9c9941dfb"}
Sep 11 01:00:45.797384 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:45.797114    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"575eb6a2-71fc-481b-a5da-84ebbdef443f","Type":"ContainerStarted","Data":"b3fbfe08f21bdbf2255f0ddd0294afce1c7cbea8b6b4409165913f4597843aff"}
Sep 11 01:00:45.797384 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:45.797138    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"575eb6a2-71fc-481b-a5da-84ebbdef443f","Type":"ContainerStarted","Data":"6cd6e94a3085a22b5272ef4315c4da0f29c25eec54452a8940da83d1a00957fc"}
Sep 11 01:00:45.797384 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:45.797160    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"575eb6a2-71fc-481b-a5da-84ebbdef443f","Type":"ContainerStarted","Data":"678b805799e05a4a4c8f3883dfdfd417a1fd6449d80ecd5dbfcb9d052cabfd3c"}
Sep 11 01:00:45.797384 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:45.797182    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"575eb6a2-71fc-481b-a5da-84ebbdef443f","Type":"ContainerStarted","Data":"62d7bad559834dad741fb6540bc5a922d67a1d93a76313bdb6c409f430fefe4f"}
Sep 11 01:00:45.826346 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:45.826298    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.810970538 podStartE2EDuration="2.82628311s" totalImagesPullingTime="0s" totalInitContainerRuntime="15.312572ms" isStatefulPod=false podCreationTimestamp="2026-09-11 01:00:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 01:00:45.824562115 +0000 UTC m=+180.475936363" watchObservedRunningTime="2026-09-11 01:00:45.82628311 +0000 UTC m=+180.477657386"
Sep 11 01:00:45.948570 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:45.948538    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 11 01:00:45.948867 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:45.948831    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="f30e94a8-4e99-4971-a70c-c96786257fb9" containerName="kube-rbac-proxy" containerID="cri-o://2decc0a7d51ac5cc9481b62aa7f9562096b9271ba7378f40c8e63d8b885e7bb6" gracePeriod=600
Sep 11 01:00:45.949032 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:45.948883    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="f30e94a8-4e99-4971-a70c-c96786257fb9" containerName="thanos-sidecar" containerID="cri-o://cfd1a641939c265b5b10ae08171f6ecd522672448650c3ea904e57b25d48312a" gracePeriod=600
Sep 11 01:00:45.949032 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:45.948910    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="f30e94a8-4e99-4971-a70c-c96786257fb9" containerName="kube-rbac-proxy-thanos" containerID="cri-o://2f9b94fe105a2aaf78b3a9fd89cbd9f8d9363da40047bc2838f029a503508ad4" gracePeriod=600
Sep 11 01:00:45.949032 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:45.948896    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="f30e94a8-4e99-4971-a70c-c96786257fb9" containerName="config-reloader" containerID="cri-o://459b83d39c5911e91e42b108b13ba7c89048120502fc187c131203b02bcc8221" gracePeriod=600
Sep 11 01:00:45.949032 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:45.948822    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="f30e94a8-4e99-4971-a70c-c96786257fb9" containerName="prometheus" containerID="cri-o://70b9a3ea10dce6c2bed553500bb5b79c263915ec13b318a96fe58a8e0a30f3d2" gracePeriod=600
Sep 11 01:00:45.949032 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:45.948902    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="f30e94a8-4e99-4971-a70c-c96786257fb9" containerName="kube-rbac-proxy-web" containerID="cri-o://b55d63f1960c065d1dc6e816cd3f143579d8220f2a9ff6d4780e71abbdb41582" gracePeriod=600
Sep 11 01:00:46.804377 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:46.804348    2569 generic.go:356] "Generic (PLEG): container finished" podID="f30e94a8-4e99-4971-a70c-c96786257fb9" containerID="2f9b94fe105a2aaf78b3a9fd89cbd9f8d9363da40047bc2838f029a503508ad4" exitCode=0
Sep 11 01:00:46.804377 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:46.804372    2569 generic.go:356] "Generic (PLEG): container finished" podID="f30e94a8-4e99-4971-a70c-c96786257fb9" containerID="2decc0a7d51ac5cc9481b62aa7f9562096b9271ba7378f40c8e63d8b885e7bb6" exitCode=0
Sep 11 01:00:46.804377 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:46.804379    2569 generic.go:356] "Generic (PLEG): container finished" podID="f30e94a8-4e99-4971-a70c-c96786257fb9" containerID="cfd1a641939c265b5b10ae08171f6ecd522672448650c3ea904e57b25d48312a" exitCode=0
Sep 11 01:00:46.804755 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:46.804386    2569 generic.go:356] "Generic (PLEG): container finished" podID="f30e94a8-4e99-4971-a70c-c96786257fb9" containerID="459b83d39c5911e91e42b108b13ba7c89048120502fc187c131203b02bcc8221" exitCode=0
Sep 11 01:00:46.804755 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:46.804393    2569 generic.go:356] "Generic (PLEG): container finished" podID="f30e94a8-4e99-4971-a70c-c96786257fb9" containerID="70b9a3ea10dce6c2bed553500bb5b79c263915ec13b318a96fe58a8e0a30f3d2" exitCode=0
Sep 11 01:00:46.804755 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:46.804421    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f30e94a8-4e99-4971-a70c-c96786257fb9","Type":"ContainerDied","Data":"2f9b94fe105a2aaf78b3a9fd89cbd9f8d9363da40047bc2838f029a503508ad4"}
Sep 11 01:00:46.804755 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:46.804499    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f30e94a8-4e99-4971-a70c-c96786257fb9","Type":"ContainerDied","Data":"2decc0a7d51ac5cc9481b62aa7f9562096b9271ba7378f40c8e63d8b885e7bb6"}
Sep 11 01:00:46.804755 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:46.804528    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f30e94a8-4e99-4971-a70c-c96786257fb9","Type":"ContainerDied","Data":"cfd1a641939c265b5b10ae08171f6ecd522672448650c3ea904e57b25d48312a"}
Sep 11 01:00:46.804755 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:46.804561    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f30e94a8-4e99-4971-a70c-c96786257fb9","Type":"ContainerDied","Data":"459b83d39c5911e91e42b108b13ba7c89048120502fc187c131203b02bcc8221"}
Sep 11 01:00:46.804755 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:46.804590    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f30e94a8-4e99-4971-a70c-c96786257fb9","Type":"ContainerDied","Data":"70b9a3ea10dce6c2bed553500bb5b79c263915ec13b318a96fe58a8e0a30f3d2"}
Sep 11 01:00:47.187971 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:47.187946    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 01:00:47.224716 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:47.224688    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f30e94a8-4e99-4971-a70c-c96786257fb9-prometheus-k8s-rulefiles-2\") pod \"f30e94a8-4e99-4971-a70c-c96786257fb9\" (UID: \"f30e94a8-4e99-4971-a70c-c96786257fb9\") "
Sep 11 01:00:47.224863 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:47.224786    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f30e94a8-4e99-4971-a70c-c96786257fb9-prometheus-k8s-rulefiles-1\") pod \"f30e94a8-4e99-4971-a70c-c96786257fb9\" (UID: \"f30e94a8-4e99-4971-a70c-c96786257fb9\") "
Sep 11 01:00:47.224863 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:47.224817    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f30e94a8-4e99-4971-a70c-c96786257fb9-prometheus-k8s-db\") pod \"f30e94a8-4e99-4971-a70c-c96786257fb9\" (UID: \"f30e94a8-4e99-4971-a70c-c96786257fb9\") "
Sep 11 01:00:47.224863 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:47.224849    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/f30e94a8-4e99-4971-a70c-c96786257fb9-config\") pod \"f30e94a8-4e99-4971-a70c-c96786257fb9\" (UID: \"f30e94a8-4e99-4971-a70c-c96786257fb9\") "
Sep 11 01:00:47.225052 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:47.224866    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f30e94a8-4e99-4971-a70c-c96786257fb9-secret-metrics-client-certs\") pod \"f30e94a8-4e99-4971-a70c-c96786257fb9\" (UID: \"f30e94a8-4e99-4971-a70c-c96786257fb9\") "
Sep 11 01:00:47.225052 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:47.224910    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f30e94a8-4e99-4971-a70c-c96786257fb9-thanos-prometheus-http-client-file\") pod \"f30e94a8-4e99-4971-a70c-c96786257fb9\" (UID: \"f30e94a8-4e99-4971-a70c-c96786257fb9\") "
Sep 11 01:00:47.225052 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:47.224937    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f30e94a8-4e99-4971-a70c-c96786257fb9-config-out\") pod \"f30e94a8-4e99-4971-a70c-c96786257fb9\" (UID: \"f30e94a8-4e99-4971-a70c-c96786257fb9\") "
Sep 11 01:00:47.225052 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:47.224963    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f30e94a8-4e99-4971-a70c-c96786257fb9-tls-assets\") pod \"f30e94a8-4e99-4971-a70c-c96786257fb9\" (UID: \"f30e94a8-4e99-4971-a70c-c96786257fb9\") "
Sep 11 01:00:47.225052 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:47.224996    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f30e94a8-4e99-4971-a70c-c96786257fb9-configmap-serving-certs-ca-bundle\") pod \"f30e94a8-4e99-4971-a70c-c96786257fb9\" (UID: \"f30e94a8-4e99-4971-a70c-c96786257fb9\") "
Sep 11 01:00:47.225052 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:47.225029    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f30e94a8-4e99-4971-a70c-c96786257fb9-secret-prometheus-k8s-tls\") pod \"f30e94a8-4e99-4971-a70c-c96786257fb9\" (UID: \"f30e94a8-4e99-4971-a70c-c96786257fb9\") "
Sep 11 01:00:47.225052 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:47.225035    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f30e94a8-4e99-4971-a70c-c96786257fb9-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "f30e94a8-4e99-4971-a70c-c96786257fb9" (UID: "f30e94a8-4e99-4971-a70c-c96786257fb9"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 01:00:47.225052 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:47.225053    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f30e94a8-4e99-4971-a70c-c96786257fb9-web-config\") pod \"f30e94a8-4e99-4971-a70c-c96786257fb9\" (UID: \"f30e94a8-4e99-4971-a70c-c96786257fb9\") "
Sep 11 01:00:47.225454 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:47.225079    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f30e94a8-4e99-4971-a70c-c96786257fb9-prometheus-trusted-ca-bundle\") pod \"f30e94a8-4e99-4971-a70c-c96786257fb9\" (UID: \"f30e94a8-4e99-4971-a70c-c96786257fb9\") "
Sep 11 01:00:47.225454 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:47.225110    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f30e94a8-4e99-4971-a70c-c96786257fb9-configmap-metrics-client-ca\") pod \"f30e94a8-4e99-4971-a70c-c96786257fb9\" (UID: \"f30e94a8-4e99-4971-a70c-c96786257fb9\") "
Sep 11 01:00:47.225454 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:47.225133    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f30e94a8-4e99-4971-a70c-c96786257fb9-secret-kube-rbac-proxy\") pod \"f30e94a8-4e99-4971-a70c-c96786257fb9\" (UID: \"f30e94a8-4e99-4971-a70c-c96786257fb9\") "
Sep 11 01:00:47.225454 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:47.225162    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f30e94a8-4e99-4971-a70c-c96786257fb9-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"f30e94a8-4e99-4971-a70c-c96786257fb9\" (UID: \"f30e94a8-4e99-4971-a70c-c96786257fb9\") "
Sep 11 01:00:47.225454 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:47.225171    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f30e94a8-4e99-4971-a70c-c96786257fb9-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "f30e94a8-4e99-4971-a70c-c96786257fb9" (UID: "f30e94a8-4e99-4971-a70c-c96786257fb9"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 01:00:47.225454 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:47.225193    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f30e94a8-4e99-4971-a70c-c96786257fb9-prometheus-k8s-rulefiles-0\") pod \"f30e94a8-4e99-4971-a70c-c96786257fb9\" (UID: \"f30e94a8-4e99-4971-a70c-c96786257fb9\") "
Sep 11 01:00:47.225454 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:47.225230    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f30e94a8-4e99-4971-a70c-c96786257fb9-secret-grpc-tls\") pod \"f30e94a8-4e99-4971-a70c-c96786257fb9\" (UID: \"f30e94a8-4e99-4971-a70c-c96786257fb9\") "
Sep 11 01:00:47.225454 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:47.225268    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8c8v8\" (UniqueName: \"kubernetes.io/projected/f30e94a8-4e99-4971-a70c-c96786257fb9-kube-api-access-8c8v8\") pod \"f30e94a8-4e99-4971-a70c-c96786257fb9\" (UID: \"f30e94a8-4e99-4971-a70c-c96786257fb9\") "
Sep 11 01:00:47.225454 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:47.225298    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f30e94a8-4e99-4971-a70c-c96786257fb9-configmap-kubelet-serving-ca-bundle\") pod \"f30e94a8-4e99-4971-a70c-c96786257fb9\" (UID: \"f30e94a8-4e99-4971-a70c-c96786257fb9\") "
Sep 11 01:00:47.225454 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:47.225340    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f30e94a8-4e99-4971-a70c-c96786257fb9-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"f30e94a8-4e99-4971-a70c-c96786257fb9\" (UID: \"f30e94a8-4e99-4971-a70c-c96786257fb9\") "
Sep 11 01:00:47.232084 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:47.225468    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f30e94a8-4e99-4971-a70c-c96786257fb9-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "f30e94a8-4e99-4971-a70c-c96786257fb9" (UID: "f30e94a8-4e99-4971-a70c-c96786257fb9"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 01:00:47.232084 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:47.225621    2569 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f30e94a8-4e99-4971-a70c-c96786257fb9-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:00:47.232084 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:47.225640    2569 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f30e94a8-4e99-4971-a70c-c96786257fb9-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:00:47.232084 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:47.225654    2569 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f30e94a8-4e99-4971-a70c-c96786257fb9-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:00:47.232084 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:47.226450    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f30e94a8-4e99-4971-a70c-c96786257fb9-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "f30e94a8-4e99-4971-a70c-c96786257fb9" (UID: "f30e94a8-4e99-4971-a70c-c96786257fb9"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 01:00:47.232084 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:47.227999    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f30e94a8-4e99-4971-a70c-c96786257fb9-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "f30e94a8-4e99-4971-a70c-c96786257fb9" (UID: "f30e94a8-4e99-4971-a70c-c96786257fb9"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 01:00:47.232084 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:47.228087    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f30e94a8-4e99-4971-a70c-c96786257fb9-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "f30e94a8-4e99-4971-a70c-c96786257fb9" (UID: "f30e94a8-4e99-4971-a70c-c96786257fb9"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 01:00:47.232084 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:47.228328    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f30e94a8-4e99-4971-a70c-c96786257fb9-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "f30e94a8-4e99-4971-a70c-c96786257fb9" (UID: "f30e94a8-4e99-4971-a70c-c96786257fb9"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 01:00:47.232084 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:47.228469    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f30e94a8-4e99-4971-a70c-c96786257fb9-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "f30e94a8-4e99-4971-a70c-c96786257fb9" (UID: "f30e94a8-4e99-4971-a70c-c96786257fb9"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 01:00:47.232084 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:47.231778    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f30e94a8-4e99-4971-a70c-c96786257fb9-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "f30e94a8-4e99-4971-a70c-c96786257fb9" (UID: "f30e94a8-4e99-4971-a70c-c96786257fb9"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 01:00:47.232084 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:47.231836    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f30e94a8-4e99-4971-a70c-c96786257fb9-config" (OuterVolumeSpecName: "config") pod "f30e94a8-4e99-4971-a70c-c96786257fb9" (UID: "f30e94a8-4e99-4971-a70c-c96786257fb9"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 01:00:47.232084 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:47.231927    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f30e94a8-4e99-4971-a70c-c96786257fb9-config-out" (OuterVolumeSpecName: "config-out") pod "f30e94a8-4e99-4971-a70c-c96786257fb9" (UID: "f30e94a8-4e99-4971-a70c-c96786257fb9"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 01:00:47.232084 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:47.231962    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f30e94a8-4e99-4971-a70c-c96786257fb9-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "f30e94a8-4e99-4971-a70c-c96786257fb9" (UID: "f30e94a8-4e99-4971-a70c-c96786257fb9"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 01:00:47.232084 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:47.231986    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f30e94a8-4e99-4971-a70c-c96786257fb9-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "f30e94a8-4e99-4971-a70c-c96786257fb9" (UID: "f30e94a8-4e99-4971-a70c-c96786257fb9"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 01:00:47.232084 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:47.231998    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f30e94a8-4e99-4971-a70c-c96786257fb9-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "f30e94a8-4e99-4971-a70c-c96786257fb9" (UID: "f30e94a8-4e99-4971-a70c-c96786257fb9"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 01:00:47.232787 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:47.232075    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f30e94a8-4e99-4971-a70c-c96786257fb9-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "f30e94a8-4e99-4971-a70c-c96786257fb9" (UID: "f30e94a8-4e99-4971-a70c-c96786257fb9"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 01:00:47.232787 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:47.232382    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f30e94a8-4e99-4971-a70c-c96786257fb9-kube-api-access-8c8v8" (OuterVolumeSpecName: "kube-api-access-8c8v8") pod "f30e94a8-4e99-4971-a70c-c96786257fb9" (UID: "f30e94a8-4e99-4971-a70c-c96786257fb9"). InnerVolumeSpecName "kube-api-access-8c8v8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 01:00:47.232787 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:47.232472    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f30e94a8-4e99-4971-a70c-c96786257fb9-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "f30e94a8-4e99-4971-a70c-c96786257fb9" (UID: "f30e94a8-4e99-4971-a70c-c96786257fb9"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 01:00:47.232787 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:47.232741    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f30e94a8-4e99-4971-a70c-c96786257fb9-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "f30e94a8-4e99-4971-a70c-c96786257fb9" (UID: "f30e94a8-4e99-4971-a70c-c96786257fb9"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 01:00:47.233645 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:47.233612    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f30e94a8-4e99-4971-a70c-c96786257fb9-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "f30e94a8-4e99-4971-a70c-c96786257fb9" (UID: "f30e94a8-4e99-4971-a70c-c96786257fb9"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 01:00:47.243527 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:47.243502    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f30e94a8-4e99-4971-a70c-c96786257fb9-web-config" (OuterVolumeSpecName: "web-config") pod "f30e94a8-4e99-4971-a70c-c96786257fb9" (UID: "f30e94a8-4e99-4971-a70c-c96786257fb9"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 01:00:47.326634 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:47.326563    2569 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f30e94a8-4e99-4971-a70c-c96786257fb9-secret-prometheus-k8s-tls\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:00:47.326634 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:47.326587    2569 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f30e94a8-4e99-4971-a70c-c96786257fb9-web-config\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:00:47.326634 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:47.326596    2569 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f30e94a8-4e99-4971-a70c-c96786257fb9-prometheus-trusted-ca-bundle\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:00:47.326634 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:47.326605    2569 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f30e94a8-4e99-4971-a70c-c96786257fb9-configmap-metrics-client-ca\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:00:47.326634 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:47.326613    2569 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f30e94a8-4e99-4971-a70c-c96786257fb9-secret-kube-rbac-proxy\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:00:47.326634 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:47.326622    2569 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f30e94a8-4e99-4971-a70c-c96786257fb9-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:00:47.326634 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:47.326635    2569 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f30e94a8-4e99-4971-a70c-c96786257fb9-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:00:47.326952 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:47.326648    2569 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f30e94a8-4e99-4971-a70c-c96786257fb9-secret-grpc-tls\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:00:47.326952 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:47.326658    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8c8v8\" (UniqueName: \"kubernetes.io/projected/f30e94a8-4e99-4971-a70c-c96786257fb9-kube-api-access-8c8v8\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:00:47.326952 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:47.326667    2569 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f30e94a8-4e99-4971-a70c-c96786257fb9-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:00:47.326952 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:47.326676    2569 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f30e94a8-4e99-4971-a70c-c96786257fb9-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:00:47.326952 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:47.326687    2569 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f30e94a8-4e99-4971-a70c-c96786257fb9-prometheus-k8s-db\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:00:47.326952 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:47.326695    2569 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/f30e94a8-4e99-4971-a70c-c96786257fb9-config\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:00:47.326952 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:47.326702    2569 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f30e94a8-4e99-4971-a70c-c96786257fb9-secret-metrics-client-certs\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:00:47.326952 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:47.326710    2569 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f30e94a8-4e99-4971-a70c-c96786257fb9-thanos-prometheus-http-client-file\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:00:47.326952 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:47.326718    2569 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f30e94a8-4e99-4971-a70c-c96786257fb9-config-out\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:00:47.326952 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:47.326727    2569 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f30e94a8-4e99-4971-a70c-c96786257fb9-tls-assets\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:00:47.817227 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:47.817197    2569 generic.go:356] "Generic (PLEG): container finished" podID="f30e94a8-4e99-4971-a70c-c96786257fb9" containerID="b55d63f1960c065d1dc6e816cd3f143579d8220f2a9ff6d4780e71abbdb41582" exitCode=0
Sep 11 01:00:47.817586 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:47.817249    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f30e94a8-4e99-4971-a70c-c96786257fb9","Type":"ContainerDied","Data":"b55d63f1960c065d1dc6e816cd3f143579d8220f2a9ff6d4780e71abbdb41582"}
Sep 11 01:00:47.817586 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:47.817287    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 01:00:47.817586 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:47.817326    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f30e94a8-4e99-4971-a70c-c96786257fb9","Type":"ContainerDied","Data":"e685db21a0d283d79b1a998acfb0e1c77897ac68d5dd68145d6944d7038bf0f3"}
Sep 11 01:00:47.817586 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:47.817358    2569 scope.go:117] "RemoveContainer" containerID="2f9b94fe105a2aaf78b3a9fd89cbd9f8d9363da40047bc2838f029a503508ad4"
Sep 11 01:00:47.825253 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:47.825233    2569 scope.go:117] "RemoveContainer" containerID="2decc0a7d51ac5cc9481b62aa7f9562096b9271ba7378f40c8e63d8b885e7bb6"
Sep 11 01:00:47.832234 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:47.832220    2569 scope.go:117] "RemoveContainer" containerID="b55d63f1960c065d1dc6e816cd3f143579d8220f2a9ff6d4780e71abbdb41582"
Sep 11 01:00:47.838745 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:47.838723    2569 scope.go:117] "RemoveContainer" containerID="cfd1a641939c265b5b10ae08171f6ecd522672448650c3ea904e57b25d48312a"
Sep 11 01:00:47.841023 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:47.840726    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 11 01:00:47.844629 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:47.844604    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 11 01:00:47.846508 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:47.846491    2569 scope.go:117] "RemoveContainer" containerID="459b83d39c5911e91e42b108b13ba7c89048120502fc187c131203b02bcc8221"
Sep 11 01:00:47.853069 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:47.853050    2569 scope.go:117] "RemoveContainer" containerID="70b9a3ea10dce6c2bed553500bb5b79c263915ec13b318a96fe58a8e0a30f3d2"
Sep 11 01:00:47.860231 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:47.860213    2569 scope.go:117] "RemoveContainer" containerID="ddf849c406b4ea503fc952e9610f603d1f0cf62f73d17d6e927fabfcd59cbac2"
Sep 11 01:00:47.866480 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:47.866466    2569 scope.go:117] "RemoveContainer" containerID="2f9b94fe105a2aaf78b3a9fd89cbd9f8d9363da40047bc2838f029a503508ad4"
Sep 11 01:00:47.866782 ip-10-0-129-222 kubenswrapper[2569]: E0911 01:00:47.866755    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f9b94fe105a2aaf78b3a9fd89cbd9f8d9363da40047bc2838f029a503508ad4\": container with ID starting with 2f9b94fe105a2aaf78b3a9fd89cbd9f8d9363da40047bc2838f029a503508ad4 not found: ID does not exist" containerID="2f9b94fe105a2aaf78b3a9fd89cbd9f8d9363da40047bc2838f029a503508ad4"
Sep 11 01:00:47.866975 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:47.866791    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f9b94fe105a2aaf78b3a9fd89cbd9f8d9363da40047bc2838f029a503508ad4"} err="failed to get container status \"2f9b94fe105a2aaf78b3a9fd89cbd9f8d9363da40047bc2838f029a503508ad4\": rpc error: code = NotFound desc = could not find container \"2f9b94fe105a2aaf78b3a9fd89cbd9f8d9363da40047bc2838f029a503508ad4\": container with ID starting with 2f9b94fe105a2aaf78b3a9fd89cbd9f8d9363da40047bc2838f029a503508ad4 not found: ID does not exist"
Sep 11 01:00:47.866975 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:47.866823    2569 scope.go:117] "RemoveContainer" containerID="2decc0a7d51ac5cc9481b62aa7f9562096b9271ba7378f40c8e63d8b885e7bb6"
Sep 11 01:00:47.867434 ip-10-0-129-222 kubenswrapper[2569]: E0911 01:00:47.867128    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2decc0a7d51ac5cc9481b62aa7f9562096b9271ba7378f40c8e63d8b885e7bb6\": container with ID starting with 2decc0a7d51ac5cc9481b62aa7f9562096b9271ba7378f40c8e63d8b885e7bb6 not found: ID does not exist" containerID="2decc0a7d51ac5cc9481b62aa7f9562096b9271ba7378f40c8e63d8b885e7bb6"
Sep 11 01:00:47.867434 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:47.867155    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2decc0a7d51ac5cc9481b62aa7f9562096b9271ba7378f40c8e63d8b885e7bb6"} err="failed to get container status \"2decc0a7d51ac5cc9481b62aa7f9562096b9271ba7378f40c8e63d8b885e7bb6\": rpc error: code = NotFound desc = could not find container \"2decc0a7d51ac5cc9481b62aa7f9562096b9271ba7378f40c8e63d8b885e7bb6\": container with ID starting with 2decc0a7d51ac5cc9481b62aa7f9562096b9271ba7378f40c8e63d8b885e7bb6 not found: ID does not exist"
Sep 11 01:00:47.867434 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:47.867174    2569 scope.go:117] "RemoveContainer" containerID="b55d63f1960c065d1dc6e816cd3f143579d8220f2a9ff6d4780e71abbdb41582"
Sep 11 01:00:47.867621 ip-10-0-129-222 kubenswrapper[2569]: E0911 01:00:47.867482    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b55d63f1960c065d1dc6e816cd3f143579d8220f2a9ff6d4780e71abbdb41582\": container with ID starting with b55d63f1960c065d1dc6e816cd3f143579d8220f2a9ff6d4780e71abbdb41582 not found: ID does not exist" containerID="b55d63f1960c065d1dc6e816cd3f143579d8220f2a9ff6d4780e71abbdb41582"
Sep 11 01:00:47.867621 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:47.867514    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b55d63f1960c065d1dc6e816cd3f143579d8220f2a9ff6d4780e71abbdb41582"} err="failed to get container status \"b55d63f1960c065d1dc6e816cd3f143579d8220f2a9ff6d4780e71abbdb41582\": rpc error: code = NotFound desc = could not find container \"b55d63f1960c065d1dc6e816cd3f143579d8220f2a9ff6d4780e71abbdb41582\": container with ID starting with b55d63f1960c065d1dc6e816cd3f143579d8220f2a9ff6d4780e71abbdb41582 not found: ID does not exist"
Sep 11 01:00:47.867621 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:47.867537    2569 scope.go:117] "RemoveContainer" containerID="cfd1a641939c265b5b10ae08171f6ecd522672448650c3ea904e57b25d48312a"
Sep 11 01:00:47.870522 ip-10-0-129-222 kubenswrapper[2569]: E0911 01:00:47.870495    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cfd1a641939c265b5b10ae08171f6ecd522672448650c3ea904e57b25d48312a\": container with ID starting with cfd1a641939c265b5b10ae08171f6ecd522672448650c3ea904e57b25d48312a not found: ID does not exist" containerID="cfd1a641939c265b5b10ae08171f6ecd522672448650c3ea904e57b25d48312a"
Sep 11 01:00:47.870617 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:47.870530    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cfd1a641939c265b5b10ae08171f6ecd522672448650c3ea904e57b25d48312a"} err="failed to get container status \"cfd1a641939c265b5b10ae08171f6ecd522672448650c3ea904e57b25d48312a\": rpc error: code = NotFound desc = could not find container \"cfd1a641939c265b5b10ae08171f6ecd522672448650c3ea904e57b25d48312a\": container with ID starting with cfd1a641939c265b5b10ae08171f6ecd522672448650c3ea904e57b25d48312a not found: ID does not exist"
Sep 11 01:00:47.870617 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:47.870551    2569 scope.go:117] "RemoveContainer" containerID="459b83d39c5911e91e42b108b13ba7c89048120502fc187c131203b02bcc8221"
Sep 11 01:00:47.871167 ip-10-0-129-222 kubenswrapper[2569]: E0911 01:00:47.871141    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"459b83d39c5911e91e42b108b13ba7c89048120502fc187c131203b02bcc8221\": container with ID starting with 459b83d39c5911e91e42b108b13ba7c89048120502fc187c131203b02bcc8221 not found: ID does not exist" containerID="459b83d39c5911e91e42b108b13ba7c89048120502fc187c131203b02bcc8221"
Sep 11 01:00:47.871256 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:47.871169    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"459b83d39c5911e91e42b108b13ba7c89048120502fc187c131203b02bcc8221"} err="failed to get container status \"459b83d39c5911e91e42b108b13ba7c89048120502fc187c131203b02bcc8221\": rpc error: code = NotFound desc = could not find container \"459b83d39c5911e91e42b108b13ba7c89048120502fc187c131203b02bcc8221\": container with ID starting with 459b83d39c5911e91e42b108b13ba7c89048120502fc187c131203b02bcc8221 not found: ID does not exist"
Sep 11 01:00:47.871256 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:47.871189    2569 scope.go:117] "RemoveContainer" containerID="70b9a3ea10dce6c2bed553500bb5b79c263915ec13b318a96fe58a8e0a30f3d2"
Sep 11 01:00:47.871627 ip-10-0-129-222 kubenswrapper[2569]: E0911 01:00:47.871592    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70b9a3ea10dce6c2bed553500bb5b79c263915ec13b318a96fe58a8e0a30f3d2\": container with ID starting with 70b9a3ea10dce6c2bed553500bb5b79c263915ec13b318a96fe58a8e0a30f3d2 not found: ID does not exist" containerID="70b9a3ea10dce6c2bed553500bb5b79c263915ec13b318a96fe58a8e0a30f3d2"
Sep 11 01:00:47.871627 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:47.871621    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70b9a3ea10dce6c2bed553500bb5b79c263915ec13b318a96fe58a8e0a30f3d2"} err="failed to get container status \"70b9a3ea10dce6c2bed553500bb5b79c263915ec13b318a96fe58a8e0a30f3d2\": rpc error: code = NotFound desc = could not find container \"70b9a3ea10dce6c2bed553500bb5b79c263915ec13b318a96fe58a8e0a30f3d2\": container with ID starting with 70b9a3ea10dce6c2bed553500bb5b79c263915ec13b318a96fe58a8e0a30f3d2 not found: ID does not exist"
Sep 11 01:00:47.871766 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:47.871640    2569 scope.go:117] "RemoveContainer" containerID="ddf849c406b4ea503fc952e9610f603d1f0cf62f73d17d6e927fabfcd59cbac2"
Sep 11 01:00:47.872017 ip-10-0-129-222 kubenswrapper[2569]: E0911 01:00:47.872002    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ddf849c406b4ea503fc952e9610f603d1f0cf62f73d17d6e927fabfcd59cbac2\": container with ID starting with ddf849c406b4ea503fc952e9610f603d1f0cf62f73d17d6e927fabfcd59cbac2 not found: ID does not exist" containerID="ddf849c406b4ea503fc952e9610f603d1f0cf62f73d17d6e927fabfcd59cbac2"
Sep 11 01:00:47.872065 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:47.872021    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ddf849c406b4ea503fc952e9610f603d1f0cf62f73d17d6e927fabfcd59cbac2"} err="failed to get container status \"ddf849c406b4ea503fc952e9610f603d1f0cf62f73d17d6e927fabfcd59cbac2\": rpc error: code = NotFound desc = could not find container \"ddf849c406b4ea503fc952e9610f603d1f0cf62f73d17d6e927fabfcd59cbac2\": container with ID starting with ddf849c406b4ea503fc952e9610f603d1f0cf62f73d17d6e927fabfcd59cbac2 not found: ID does not exist"
Sep 11 01:00:47.872796 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:47.872779    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f30e94a8-4e99-4971-a70c-c96786257fb9" path="/var/lib/kubelet/pods/f30e94a8-4e99-4971-a70c-c96786257fb9/volumes"
Sep 11 01:00:47.873350 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:47.873336    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 11 01:00:47.873618 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:47.873607    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f30e94a8-4e99-4971-a70c-c96786257fb9" containerName="prometheus"
Sep 11 01:00:47.873653 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:47.873620    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="f30e94a8-4e99-4971-a70c-c96786257fb9" containerName="prometheus"
Sep 11 01:00:47.873653 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:47.873629    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f30e94a8-4e99-4971-a70c-c96786257fb9" containerName="thanos-sidecar"
Sep 11 01:00:47.873653 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:47.873635    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="f30e94a8-4e99-4971-a70c-c96786257fb9" containerName="thanos-sidecar"
Sep 11 01:00:47.873653 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:47.873641    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f30e94a8-4e99-4971-a70c-c96786257fb9" containerName="kube-rbac-proxy-web"
Sep 11 01:00:47.873653 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:47.873646    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="f30e94a8-4e99-4971-a70c-c96786257fb9" containerName="kube-rbac-proxy-web"
Sep 11 01:00:47.873653 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:47.873653    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f30e94a8-4e99-4971-a70c-c96786257fb9" containerName="init-config-reloader"
Sep 11 01:00:47.873819 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:47.873658    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="f30e94a8-4e99-4971-a70c-c96786257fb9" containerName="init-config-reloader"
Sep 11 01:00:47.873819 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:47.873664    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f30e94a8-4e99-4971-a70c-c96786257fb9" containerName="kube-rbac-proxy"
Sep 11 01:00:47.873819 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:47.873669    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="f30e94a8-4e99-4971-a70c-c96786257fb9" containerName="kube-rbac-proxy"
Sep 11 01:00:47.873819 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:47.873678    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f30e94a8-4e99-4971-a70c-c96786257fb9" containerName="config-reloader"
Sep 11 01:00:47.873819 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:47.873684    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="f30e94a8-4e99-4971-a70c-c96786257fb9" containerName="config-reloader"
Sep 11 01:00:47.873819 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:47.873693    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f30e94a8-4e99-4971-a70c-c96786257fb9" containerName="kube-rbac-proxy-thanos"
Sep 11 01:00:47.873819 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:47.873698    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="f30e94a8-4e99-4971-a70c-c96786257fb9" containerName="kube-rbac-proxy-thanos"
Sep 11 01:00:47.873819 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:47.873745    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="f30e94a8-4e99-4971-a70c-c96786257fb9" containerName="thanos-sidecar"
Sep 11 01:00:47.873819 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:47.873754    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="f30e94a8-4e99-4971-a70c-c96786257fb9" containerName="kube-rbac-proxy"
Sep 11 01:00:47.873819 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:47.873760    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="f30e94a8-4e99-4971-a70c-c96786257fb9" containerName="kube-rbac-proxy-thanos"
Sep 11 01:00:47.873819 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:47.873765    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="f30e94a8-4e99-4971-a70c-c96786257fb9" containerName="config-reloader"
Sep 11 01:00:47.873819 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:47.873772    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="f30e94a8-4e99-4971-a70c-c96786257fb9" containerName="prometheus"
Sep 11 01:00:47.873819 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:47.873778    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="f30e94a8-4e99-4971-a70c-c96786257fb9" containerName="kube-rbac-proxy-web"
Sep 11 01:00:47.877480 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:47.877465    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 01:00:47.879948 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:47.879928    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 11 01:00:47.880058 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:47.879946    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 11 01:00:47.880058 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:47.879955    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 11 01:00:47.880058 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:47.879977    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-4fdskhji27lft\""
Sep 11 01:00:47.880058 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:47.880003    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 11 01:00:47.880269 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:47.880064    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 11 01:00:47.880366 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:47.880348    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 11 01:00:47.880436 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:47.880351    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 11 01:00:47.880436 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:47.880409    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 11 01:00:47.880560 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:47.880543    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 11 01:00:47.880611 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:47.880558    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-8d4cx\""
Sep 11 01:00:47.880611 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:47.880548    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 11 01:00:47.880799 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:47.880779    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 11 01:00:47.880997 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:47.880978    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 11 01:00:47.881243 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:47.881225    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 11 01:00:47.883538 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:47.883516    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 11 01:00:47.886239 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:47.886218    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 11 01:00:47.893926 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:47.893899    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 11 01:00:47.931133 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:47.931103    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/06d240fd-ec52-46cd-a90d-40cd7ad8c7d9-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"06d240fd-ec52-46cd-a90d-40cd7ad8c7d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 01:00:47.931251 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:47.931160    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/06d240fd-ec52-46cd-a90d-40cd7ad8c7d9-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"06d240fd-ec52-46cd-a90d-40cd7ad8c7d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 01:00:47.931251 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:47.931189    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/06d240fd-ec52-46cd-a90d-40cd7ad8c7d9-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"06d240fd-ec52-46cd-a90d-40cd7ad8c7d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 01:00:47.931251 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:47.931223    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/06d240fd-ec52-46cd-a90d-40cd7ad8c7d9-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"06d240fd-ec52-46cd-a90d-40cd7ad8c7d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 01:00:47.931415 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:47.931253    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/06d240fd-ec52-46cd-a90d-40cd7ad8c7d9-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"06d240fd-ec52-46cd-a90d-40cd7ad8c7d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 01:00:47.931415 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:47.931279    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/06d240fd-ec52-46cd-a90d-40cd7ad8c7d9-config\") pod \"prometheus-k8s-0\" (UID: \"06d240fd-ec52-46cd-a90d-40cd7ad8c7d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 01:00:47.931415 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:47.931305    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/06d240fd-ec52-46cd-a90d-40cd7ad8c7d9-config-out\") pod \"prometheus-k8s-0\" (UID: \"06d240fd-ec52-46cd-a90d-40cd7ad8c7d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 01:00:47.931415 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:47.931336    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/06d240fd-ec52-46cd-a90d-40cd7ad8c7d9-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"06d240fd-ec52-46cd-a90d-40cd7ad8c7d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 01:00:47.931415 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:47.931370    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/06d240fd-ec52-46cd-a90d-40cd7ad8c7d9-web-config\") pod \"prometheus-k8s-0\" (UID: \"06d240fd-ec52-46cd-a90d-40cd7ad8c7d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 01:00:47.931415 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:47.931396    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/06d240fd-ec52-46cd-a90d-40cd7ad8c7d9-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"06d240fd-ec52-46cd-a90d-40cd7ad8c7d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 01:00:47.931714 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:47.931422    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/06d240fd-ec52-46cd-a90d-40cd7ad8c7d9-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"06d240fd-ec52-46cd-a90d-40cd7ad8c7d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 01:00:47.931714 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:47.931456    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/06d240fd-ec52-46cd-a90d-40cd7ad8c7d9-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"06d240fd-ec52-46cd-a90d-40cd7ad8c7d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 01:00:47.931714 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:47.931483    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/06d240fd-ec52-46cd-a90d-40cd7ad8c7d9-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"06d240fd-ec52-46cd-a90d-40cd7ad8c7d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 01:00:47.931714 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:47.931515    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/06d240fd-ec52-46cd-a90d-40cd7ad8c7d9-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"06d240fd-ec52-46cd-a90d-40cd7ad8c7d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 01:00:47.931714 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:47.931555    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/06d240fd-ec52-46cd-a90d-40cd7ad8c7d9-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"06d240fd-ec52-46cd-a90d-40cd7ad8c7d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 01:00:47.931714 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:47.931588    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/06d240fd-ec52-46cd-a90d-40cd7ad8c7d9-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"06d240fd-ec52-46cd-a90d-40cd7ad8c7d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 01:00:47.931714 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:47.931624    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/06d240fd-ec52-46cd-a90d-40cd7ad8c7d9-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"06d240fd-ec52-46cd-a90d-40cd7ad8c7d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 01:00:47.931714 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:47.931664    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q4zgg\" (UniqueName: \"kubernetes.io/projected/06d240fd-ec52-46cd-a90d-40cd7ad8c7d9-kube-api-access-q4zgg\") pod \"prometheus-k8s-0\" (UID: \"06d240fd-ec52-46cd-a90d-40cd7ad8c7d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 01:00:47.931714 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:47.931698    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/06d240fd-ec52-46cd-a90d-40cd7ad8c7d9-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"06d240fd-ec52-46cd-a90d-40cd7ad8c7d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 01:00:47.932033 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:47.931721    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/06d240fd-ec52-46cd-a90d-40cd7ad8c7d9-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"06d240fd-ec52-46cd-a90d-40cd7ad8c7d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 01:00:48.032896 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:48.032812    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/06d240fd-ec52-46cd-a90d-40cd7ad8c7d9-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"06d240fd-ec52-46cd-a90d-40cd7ad8c7d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 01:00:48.032896 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:48.032845    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/06d240fd-ec52-46cd-a90d-40cd7ad8c7d9-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"06d240fd-ec52-46cd-a90d-40cd7ad8c7d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 01:00:48.033071 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:48.032897    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/06d240fd-ec52-46cd-a90d-40cd7ad8c7d9-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"06d240fd-ec52-46cd-a90d-40cd7ad8c7d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 01:00:48.033071 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:48.032923    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/06d240fd-ec52-46cd-a90d-40cd7ad8c7d9-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"06d240fd-ec52-46cd-a90d-40cd7ad8c7d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 01:00:48.033071 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:48.032945    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/06d240fd-ec52-46cd-a90d-40cd7ad8c7d9-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"06d240fd-ec52-46cd-a90d-40cd7ad8c7d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 01:00:48.033071 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:48.032977    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/06d240fd-ec52-46cd-a90d-40cd7ad8c7d9-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"06d240fd-ec52-46cd-a90d-40cd7ad8c7d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 01:00:48.033071 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:48.033010    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/06d240fd-ec52-46cd-a90d-40cd7ad8c7d9-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"06d240fd-ec52-46cd-a90d-40cd7ad8c7d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 01:00:48.033071 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:48.033036    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/06d240fd-ec52-46cd-a90d-40cd7ad8c7d9-config\") pod \"prometheus-k8s-0\" (UID: \"06d240fd-ec52-46cd-a90d-40cd7ad8c7d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 01:00:48.033071 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:48.033061    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/06d240fd-ec52-46cd-a90d-40cd7ad8c7d9-config-out\") pod \"prometheus-k8s-0\" (UID: \"06d240fd-ec52-46cd-a90d-40cd7ad8c7d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 01:00:48.033410 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:48.033090    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/06d240fd-ec52-46cd-a90d-40cd7ad8c7d9-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"06d240fd-ec52-46cd-a90d-40cd7ad8c7d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 01:00:48.033410 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:48.033124    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/06d240fd-ec52-46cd-a90d-40cd7ad8c7d9-web-config\") pod \"prometheus-k8s-0\" (UID: \"06d240fd-ec52-46cd-a90d-40cd7ad8c7d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 01:00:48.033410 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:48.033148    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/06d240fd-ec52-46cd-a90d-40cd7ad8c7d9-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"06d240fd-ec52-46cd-a90d-40cd7ad8c7d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 01:00:48.033410 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:48.033178    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/06d240fd-ec52-46cd-a90d-40cd7ad8c7d9-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"06d240fd-ec52-46cd-a90d-40cd7ad8c7d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 01:00:48.033410 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:48.033207    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/06d240fd-ec52-46cd-a90d-40cd7ad8c7d9-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"06d240fd-ec52-46cd-a90d-40cd7ad8c7d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 01:00:48.033410 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:48.033234    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/06d240fd-ec52-46cd-a90d-40cd7ad8c7d9-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"06d240fd-ec52-46cd-a90d-40cd7ad8c7d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 01:00:48.033410 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:48.033273    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/06d240fd-ec52-46cd-a90d-40cd7ad8c7d9-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"06d240fd-ec52-46cd-a90d-40cd7ad8c7d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 01:00:48.033410 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:48.033302    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/06d240fd-ec52-46cd-a90d-40cd7ad8c7d9-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"06d240fd-ec52-46cd-a90d-40cd7ad8c7d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 01:00:48.033410 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:48.033337    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/06d240fd-ec52-46cd-a90d-40cd7ad8c7d9-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"06d240fd-ec52-46cd-a90d-40cd7ad8c7d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 01:00:48.033410 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:48.033370    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/06d240fd-ec52-46cd-a90d-40cd7ad8c7d9-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"06d240fd-ec52-46cd-a90d-40cd7ad8c7d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 01:00:48.033852 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:48.033421    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4zgg\" (UniqueName: \"kubernetes.io/projected/06d240fd-ec52-46cd-a90d-40cd7ad8c7d9-kube-api-access-q4zgg\") pod \"prometheus-k8s-0\" (UID: \"06d240fd-ec52-46cd-a90d-40cd7ad8c7d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 01:00:48.033946 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:48.033915    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/06d240fd-ec52-46cd-a90d-40cd7ad8c7d9-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"06d240fd-ec52-46cd-a90d-40cd7ad8c7d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 01:00:48.034324 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:48.034299    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/06d240fd-ec52-46cd-a90d-40cd7ad8c7d9-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"06d240fd-ec52-46cd-a90d-40cd7ad8c7d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 01:00:48.034324 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:48.034316    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/06d240fd-ec52-46cd-a90d-40cd7ad8c7d9-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"06d240fd-ec52-46cd-a90d-40cd7ad8c7d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 01:00:48.034457 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:48.034385    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/06d240fd-ec52-46cd-a90d-40cd7ad8c7d9-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"06d240fd-ec52-46cd-a90d-40cd7ad8c7d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 01:00:48.035089 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:48.035065    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/06d240fd-ec52-46cd-a90d-40cd7ad8c7d9-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"06d240fd-ec52-46cd-a90d-40cd7ad8c7d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 01:00:48.035544 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:48.035519    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/06d240fd-ec52-46cd-a90d-40cd7ad8c7d9-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"06d240fd-ec52-46cd-a90d-40cd7ad8c7d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 01:00:48.035894 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:48.035857    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/06d240fd-ec52-46cd-a90d-40cd7ad8c7d9-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"06d240fd-ec52-46cd-a90d-40cd7ad8c7d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 01:00:48.036982 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:48.036958    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/06d240fd-ec52-46cd-a90d-40cd7ad8c7d9-web-config\") pod \"prometheus-k8s-0\" (UID: \"06d240fd-ec52-46cd-a90d-40cd7ad8c7d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 01:00:48.037268 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:48.037243    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/06d240fd-ec52-46cd-a90d-40cd7ad8c7d9-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"06d240fd-ec52-46cd-a90d-40cd7ad8c7d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 01:00:48.037397 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:48.037372    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/06d240fd-ec52-46cd-a90d-40cd7ad8c7d9-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"06d240fd-ec52-46cd-a90d-40cd7ad8c7d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 01:00:48.037714 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:48.037686    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/06d240fd-ec52-46cd-a90d-40cd7ad8c7d9-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"06d240fd-ec52-46cd-a90d-40cd7ad8c7d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 01:00:48.037714 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:48.037702    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/06d240fd-ec52-46cd-a90d-40cd7ad8c7d9-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"06d240fd-ec52-46cd-a90d-40cd7ad8c7d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 01:00:48.038317 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:48.038278    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/06d240fd-ec52-46cd-a90d-40cd7ad8c7d9-config-out\") pod \"prometheus-k8s-0\" (UID: \"06d240fd-ec52-46cd-a90d-40cd7ad8c7d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 01:00:48.038317 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:48.038292    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/06d240fd-ec52-46cd-a90d-40cd7ad8c7d9-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"06d240fd-ec52-46cd-a90d-40cd7ad8c7d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 01:00:48.038484 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:48.038422    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/06d240fd-ec52-46cd-a90d-40cd7ad8c7d9-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"06d240fd-ec52-46cd-a90d-40cd7ad8c7d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 01:00:48.038547 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:48.038522    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/06d240fd-ec52-46cd-a90d-40cd7ad8c7d9-config\") pod \"prometheus-k8s-0\" (UID: \"06d240fd-ec52-46cd-a90d-40cd7ad8c7d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 01:00:48.038643 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:48.038621    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/06d240fd-ec52-46cd-a90d-40cd7ad8c7d9-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"06d240fd-ec52-46cd-a90d-40cd7ad8c7d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 01:00:48.040016 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:48.039996    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/06d240fd-ec52-46cd-a90d-40cd7ad8c7d9-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"06d240fd-ec52-46cd-a90d-40cd7ad8c7d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 01:00:48.040113 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:48.040071    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/06d240fd-ec52-46cd-a90d-40cd7ad8c7d9-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"06d240fd-ec52-46cd-a90d-40cd7ad8c7d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 01:00:48.041651 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:48.041631    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4zgg\" (UniqueName: \"kubernetes.io/projected/06d240fd-ec52-46cd-a90d-40cd7ad8c7d9-kube-api-access-q4zgg\") pod \"prometheus-k8s-0\" (UID: \"06d240fd-ec52-46cd-a90d-40cd7ad8c7d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 01:00:48.188409 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:48.188377    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 01:00:48.466441 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:48.466414    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 11 01:00:48.467724 ip-10-0-129-222 kubenswrapper[2569]: W0911 01:00:48.467696    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06d240fd_ec52_46cd_a90d_40cd7ad8c7d9.slice/crio-1395652b520aa4402a85d44ce8d4a1d1cf56284d45e88676d75da48a0f90f09d WatchSource:0}: Error finding container 1395652b520aa4402a85d44ce8d4a1d1cf56284d45e88676d75da48a0f90f09d: Status 404 returned error can't find the container with id 1395652b520aa4402a85d44ce8d4a1d1cf56284d45e88676d75da48a0f90f09d
Sep 11 01:00:48.824246 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:48.824212    2569 generic.go:356] "Generic (PLEG): container finished" podID="06d240fd-ec52-46cd-a90d-40cd7ad8c7d9" containerID="a7250597883a26fde67fd63a7bf1151908938ceae214296ad002e81beff9e03b" exitCode=0
Sep 11 01:00:48.824609 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:48.824252    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"06d240fd-ec52-46cd-a90d-40cd7ad8c7d9","Type":"ContainerDied","Data":"a7250597883a26fde67fd63a7bf1151908938ceae214296ad002e81beff9e03b"}
Sep 11 01:00:48.824609 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:48.824308    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"06d240fd-ec52-46cd-a90d-40cd7ad8c7d9","Type":"ContainerStarted","Data":"1395652b520aa4402a85d44ce8d4a1d1cf56284d45e88676d75da48a0f90f09d"}
Sep 11 01:00:49.832055 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:49.832021    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"06d240fd-ec52-46cd-a90d-40cd7ad8c7d9","Type":"ContainerStarted","Data":"033fd98b2c539d169f3d19f31bb56cf64149a4cf311938b9a8dc9d52209a4cb6"}
Sep 11 01:00:49.832431 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:49.832112    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"06d240fd-ec52-46cd-a90d-40cd7ad8c7d9","Type":"ContainerStarted","Data":"236a2a0ba3f7ed34cfee6fa32f0e07165edc10433f542dd8795df6d7815e77c0"}
Sep 11 01:00:49.832431 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:49.832141    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"06d240fd-ec52-46cd-a90d-40cd7ad8c7d9","Type":"ContainerStarted","Data":"203667796b90dab336ec3757915bb552677905d8a328b195fefacd001dcb1534"}
Sep 11 01:00:49.832431 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:49.832172    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"06d240fd-ec52-46cd-a90d-40cd7ad8c7d9","Type":"ContainerStarted","Data":"892fdaac2e4f6f25c5b8c1f0991e5732a1f9c8098cc1d1014a20c12e12cede50"}
Sep 11 01:00:49.832431 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:49.832212    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"06d240fd-ec52-46cd-a90d-40cd7ad8c7d9","Type":"ContainerStarted","Data":"51eb2a38061225bfc8ca54f20de806562e29ad6787f51616b07a827cf5ec6cd7"}
Sep 11 01:00:49.832431 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:49.832241    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"06d240fd-ec52-46cd-a90d-40cd7ad8c7d9","Type":"ContainerStarted","Data":"b88bd75008694261d4e2686eeba8a1eb429866339f8f5eb223772550376a994f"}
Sep 11 01:00:49.865889 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:49.865798    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.8426617849999998 podStartE2EDuration="2.865777903s" totalImagesPullingTime="0s" totalInitContainerRuntime="23.116118ms" isStatefulPod=false podCreationTimestamp="2026-09-11 01:00:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 01:00:49.863245889 +0000 UTC m=+184.514620137" watchObservedRunningTime="2026-09-11 01:00:49.865777903 +0000 UTC m=+184.517152153"
Sep 11 01:00:53.189179 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:00:53.189141    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 01:01:48.188580 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:01:48.188541    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 01:01:48.204046 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:01:48.204021    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 01:01:49.088095 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:01:49.088067    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 01:02:45.793256 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:02:45.793230    2569 kubelet.go:1648] "Image garbage collection succeeded"
Sep 11 01:03:09.086061 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:03:09.086029    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7c49b45946-25wxw"]
Sep 11 01:03:09.089950 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:03:09.089927    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7c49b45946-25wxw"
Sep 11 01:03:09.097254 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:03:09.097236    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 11 01:03:09.097387 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:03:09.097369    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 11 01:03:09.098113 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:03:09.098099    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 11 01:03:09.098570 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:03:09.098557    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 11 01:03:09.098900 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:03:09.098884    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-sqsm4\""
Sep 11 01:03:09.108662 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:03:09.108642    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7c49b45946-25wxw"]
Sep 11 01:03:09.125738 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:03:09.125715    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5ww2\" (UniqueName: \"kubernetes.io/projected/823b7670-1630-48fc-9963-fb96437d04b0-kube-api-access-z5ww2\") pod \"opendatahub-operator-controller-manager-7c49b45946-25wxw\" (UID: \"823b7670-1630-48fc-9963-fb96437d04b0\") " pod="opendatahub/opendatahub-operator-controller-manager-7c49b45946-25wxw"
Sep 11 01:03:09.125846 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:03:09.125757    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/823b7670-1630-48fc-9963-fb96437d04b0-webhook-cert\") pod \"opendatahub-operator-controller-manager-7c49b45946-25wxw\" (UID: \"823b7670-1630-48fc-9963-fb96437d04b0\") " pod="opendatahub/opendatahub-operator-controller-manager-7c49b45946-25wxw"
Sep 11 01:03:09.125846 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:03:09.125790    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/823b7670-1630-48fc-9963-fb96437d04b0-tmp\") pod \"opendatahub-operator-controller-manager-7c49b45946-25wxw\" (UID: \"823b7670-1630-48fc-9963-fb96437d04b0\") " pod="opendatahub/opendatahub-operator-controller-manager-7c49b45946-25wxw"
Sep 11 01:03:09.125846 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:03:09.125809    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/823b7670-1630-48fc-9963-fb96437d04b0-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7c49b45946-25wxw\" (UID: \"823b7670-1630-48fc-9963-fb96437d04b0\") " pod="opendatahub/opendatahub-operator-controller-manager-7c49b45946-25wxw"
Sep 11 01:03:09.125846 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:03:09.125835    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/823b7670-1630-48fc-9963-fb96437d04b0-manifests\") pod \"opendatahub-operator-controller-manager-7c49b45946-25wxw\" (UID: \"823b7670-1630-48fc-9963-fb96437d04b0\") " pod="opendatahub/opendatahub-operator-controller-manager-7c49b45946-25wxw"
Sep 11 01:03:09.226865 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:03:09.226837    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5ww2\" (UniqueName: \"kubernetes.io/projected/823b7670-1630-48fc-9963-fb96437d04b0-kube-api-access-z5ww2\") pod \"opendatahub-operator-controller-manager-7c49b45946-25wxw\" (UID: \"823b7670-1630-48fc-9963-fb96437d04b0\") " pod="opendatahub/opendatahub-operator-controller-manager-7c49b45946-25wxw"
Sep 11 01:03:09.226995 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:03:09.226901    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/823b7670-1630-48fc-9963-fb96437d04b0-webhook-cert\") pod \"opendatahub-operator-controller-manager-7c49b45946-25wxw\" (UID: \"823b7670-1630-48fc-9963-fb96437d04b0\") " pod="opendatahub/opendatahub-operator-controller-manager-7c49b45946-25wxw"
Sep 11 01:03:09.226995 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:03:09.226934    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/823b7670-1630-48fc-9963-fb96437d04b0-tmp\") pod \"opendatahub-operator-controller-manager-7c49b45946-25wxw\" (UID: \"823b7670-1630-48fc-9963-fb96437d04b0\") " pod="opendatahub/opendatahub-operator-controller-manager-7c49b45946-25wxw"
Sep 11 01:03:09.226995 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:03:09.226952    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/823b7670-1630-48fc-9963-fb96437d04b0-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7c49b45946-25wxw\" (UID: \"823b7670-1630-48fc-9963-fb96437d04b0\") " pod="opendatahub/opendatahub-operator-controller-manager-7c49b45946-25wxw"
Sep 11 01:03:09.226995 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:03:09.226984    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/823b7670-1630-48fc-9963-fb96437d04b0-manifests\") pod \"opendatahub-operator-controller-manager-7c49b45946-25wxw\" (UID: \"823b7670-1630-48fc-9963-fb96437d04b0\") " pod="opendatahub/opendatahub-operator-controller-manager-7c49b45946-25wxw"
Sep 11 01:03:09.227422 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:03:09.227393    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/823b7670-1630-48fc-9963-fb96437d04b0-manifests\") pod \"opendatahub-operator-controller-manager-7c49b45946-25wxw\" (UID: \"823b7670-1630-48fc-9963-fb96437d04b0\") " pod="opendatahub/opendatahub-operator-controller-manager-7c49b45946-25wxw"
Sep 11 01:03:09.229128 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:03:09.229102    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/823b7670-1630-48fc-9963-fb96437d04b0-tmp\") pod \"opendatahub-operator-controller-manager-7c49b45946-25wxw\" (UID: \"823b7670-1630-48fc-9963-fb96437d04b0\") " pod="opendatahub/opendatahub-operator-controller-manager-7c49b45946-25wxw"
Sep 11 01:03:09.229441 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:03:09.229420    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/823b7670-1630-48fc-9963-fb96437d04b0-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7c49b45946-25wxw\" (UID: \"823b7670-1630-48fc-9963-fb96437d04b0\") " pod="opendatahub/opendatahub-operator-controller-manager-7c49b45946-25wxw"
Sep 11 01:03:09.229500 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:03:09.229437    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/823b7670-1630-48fc-9963-fb96437d04b0-webhook-cert\") pod \"opendatahub-operator-controller-manager-7c49b45946-25wxw\" (UID: \"823b7670-1630-48fc-9963-fb96437d04b0\") " pod="opendatahub/opendatahub-operator-controller-manager-7c49b45946-25wxw"
Sep 11 01:03:09.235373 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:03:09.235350    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5ww2\" (UniqueName: \"kubernetes.io/projected/823b7670-1630-48fc-9963-fb96437d04b0-kube-api-access-z5ww2\") pod \"opendatahub-operator-controller-manager-7c49b45946-25wxw\" (UID: \"823b7670-1630-48fc-9963-fb96437d04b0\") " pod="opendatahub/opendatahub-operator-controller-manager-7c49b45946-25wxw"
Sep 11 01:03:09.399998 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:03:09.399907    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7c49b45946-25wxw"
Sep 11 01:03:09.686545 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:03:09.686525    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7c49b45946-25wxw"]
Sep 11 01:03:09.689686 ip-10-0-129-222 kubenswrapper[2569]: W0911 01:03:09.689660    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod823b7670_1630_48fc_9963_fb96437d04b0.slice/crio-b591c3860407e6ccdf85ece3f50be3afef5854c51ec9f0c4849340d102758430 WatchSource:0}: Error finding container b591c3860407e6ccdf85ece3f50be3afef5854c51ec9f0c4849340d102758430: Status 404 returned error can't find the container with id b591c3860407e6ccdf85ece3f50be3afef5854c51ec9f0c4849340d102758430
Sep 11 01:03:09.692083 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:03:09.692067    2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 01:03:10.392788 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:03:10.392740    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7c49b45946-25wxw" event={"ID":"823b7670-1630-48fc-9963-fb96437d04b0","Type":"ContainerStarted","Data":"b591c3860407e6ccdf85ece3f50be3afef5854c51ec9f0c4849340d102758430"}
Sep 11 01:03:13.409865 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:03:13.409827    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7c49b45946-25wxw" event={"ID":"823b7670-1630-48fc-9963-fb96437d04b0","Type":"ContainerStarted","Data":"5899aadeab3e6359e25cd5308d4fa3f4dc70c1cf1cf9cba25dea94680824742d"}
Sep 11 01:03:14.415681 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:03:14.415644    2569 generic.go:356] "Generic (PLEG): container finished" podID="823b7670-1630-48fc-9963-fb96437d04b0" containerID="5899aadeab3e6359e25cd5308d4fa3f4dc70c1cf1cf9cba25dea94680824742d" exitCode=0
Sep 11 01:03:14.416100 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:03:14.415733    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7c49b45946-25wxw" event={"ID":"823b7670-1630-48fc-9963-fb96437d04b0","Type":"ContainerDied","Data":"5899aadeab3e6359e25cd5308d4fa3f4dc70c1cf1cf9cba25dea94680824742d"}
Sep 11 01:03:15.421718 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:03:15.421683    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7c49b45946-25wxw" event={"ID":"823b7670-1630-48fc-9963-fb96437d04b0","Type":"ContainerStarted","Data":"163ae723735a264846924ea3fb43182686e4beef9a60abfb7178405b6c535492"}
Sep 11 01:03:15.422091 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:03:15.421795    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-7c49b45946-25wxw"
Sep 11 01:03:15.446070 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:03:15.446025    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-7c49b45946-25wxw" podStartSLOduration=2.057092239 podStartE2EDuration="6.446010249s" totalImagesPullingTime="3.378451149s" totalInitContainerRuntime="1.010466861s" isStatefulPod=false podCreationTimestamp="2026-09-11 01:03:09 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 01:03:15.443202244 +0000 UTC m=+330.094576493" watchObservedRunningTime="2026-09-11 01:03:15.446010249 +0000 UTC m=+330.097384497"
Sep 11 01:03:26.427775 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:03:26.427741    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-7c49b45946-25wxw"
Sep 11 01:03:28.691104 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:03:28.691070    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-6cff9449b4-vtxpl"]
Sep 11 01:03:28.700860 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:03:28.700832    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6cff9449b4-vtxpl"
Sep 11 01:03:28.704594 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:03:28.704574    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 11 01:03:28.706823 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:03:28.706803    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 11 01:03:28.707167 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:03:28.707141    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 11 01:03:28.707793 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:03:28.707774    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 11 01:03:28.708018 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:03:28.708000    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-zsbnv\""
Sep 11 01:03:28.708018 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:03:28.708020    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 11 01:03:28.748293 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:03:28.748266    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6cff9449b4-vtxpl"]
Sep 11 01:03:28.790322 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:03:28.790299    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/91d835d3-5954-4a73-970e-3f2c924b3daa-manager-config\") pod \"lws-controller-manager-6cff9449b4-vtxpl\" (UID: \"91d835d3-5954-4a73-970e-3f2c924b3daa\") " pod="openshift-lws-operator/lws-controller-manager-6cff9449b4-vtxpl"
Sep 11 01:03:28.790428 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:03:28.790329    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qnbg\" (UniqueName: \"kubernetes.io/projected/91d835d3-5954-4a73-970e-3f2c924b3daa-kube-api-access-6qnbg\") pod \"lws-controller-manager-6cff9449b4-vtxpl\" (UID: \"91d835d3-5954-4a73-970e-3f2c924b3daa\") " pod="openshift-lws-operator/lws-controller-manager-6cff9449b4-vtxpl"
Sep 11 01:03:28.790428 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:03:28.790358    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/91d835d3-5954-4a73-970e-3f2c924b3daa-cert\") pod \"lws-controller-manager-6cff9449b4-vtxpl\" (UID: \"91d835d3-5954-4a73-970e-3f2c924b3daa\") " pod="openshift-lws-operator/lws-controller-manager-6cff9449b4-vtxpl"
Sep 11 01:03:28.790428 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:03:28.790379    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/91d835d3-5954-4a73-970e-3f2c924b3daa-metrics-cert\") pod \"lws-controller-manager-6cff9449b4-vtxpl\" (UID: \"91d835d3-5954-4a73-970e-3f2c924b3daa\") " pod="openshift-lws-operator/lws-controller-manager-6cff9449b4-vtxpl"
Sep 11 01:03:28.891111 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:03:28.891087    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/91d835d3-5954-4a73-970e-3f2c924b3daa-manager-config\") pod \"lws-controller-manager-6cff9449b4-vtxpl\" (UID: \"91d835d3-5954-4a73-970e-3f2c924b3daa\") " pod="openshift-lws-operator/lws-controller-manager-6cff9449b4-vtxpl"
Sep 11 01:03:28.891265 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:03:28.891121    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qnbg\" (UniqueName: \"kubernetes.io/projected/91d835d3-5954-4a73-970e-3f2c924b3daa-kube-api-access-6qnbg\") pod \"lws-controller-manager-6cff9449b4-vtxpl\" (UID: \"91d835d3-5954-4a73-970e-3f2c924b3daa\") " pod="openshift-lws-operator/lws-controller-manager-6cff9449b4-vtxpl"
Sep 11 01:03:28.891265 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:03:28.891147    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/91d835d3-5954-4a73-970e-3f2c924b3daa-cert\") pod \"lws-controller-manager-6cff9449b4-vtxpl\" (UID: \"91d835d3-5954-4a73-970e-3f2c924b3daa\") " pod="openshift-lws-operator/lws-controller-manager-6cff9449b4-vtxpl"
Sep 11 01:03:28.891265 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:03:28.891166    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/91d835d3-5954-4a73-970e-3f2c924b3daa-metrics-cert\") pod \"lws-controller-manager-6cff9449b4-vtxpl\" (UID: \"91d835d3-5954-4a73-970e-3f2c924b3daa\") " pod="openshift-lws-operator/lws-controller-manager-6cff9449b4-vtxpl"
Sep 11 01:03:28.891834 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:03:28.891810    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/91d835d3-5954-4a73-970e-3f2c924b3daa-manager-config\") pod \"lws-controller-manager-6cff9449b4-vtxpl\" (UID: \"91d835d3-5954-4a73-970e-3f2c924b3daa\") " pod="openshift-lws-operator/lws-controller-manager-6cff9449b4-vtxpl"
Sep 11 01:03:28.893625 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:03:28.893603    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/91d835d3-5954-4a73-970e-3f2c924b3daa-metrics-cert\") pod \"lws-controller-manager-6cff9449b4-vtxpl\" (UID: \"91d835d3-5954-4a73-970e-3f2c924b3daa\") " pod="openshift-lws-operator/lws-controller-manager-6cff9449b4-vtxpl"
Sep 11 01:03:28.893708 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:03:28.893674    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/91d835d3-5954-4a73-970e-3f2c924b3daa-cert\") pod \"lws-controller-manager-6cff9449b4-vtxpl\" (UID: \"91d835d3-5954-4a73-970e-3f2c924b3daa\") " pod="openshift-lws-operator/lws-controller-manager-6cff9449b4-vtxpl"
Sep 11 01:03:28.932193 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:03:28.932168    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qnbg\" (UniqueName: \"kubernetes.io/projected/91d835d3-5954-4a73-970e-3f2c924b3daa-kube-api-access-6qnbg\") pod \"lws-controller-manager-6cff9449b4-vtxpl\" (UID: \"91d835d3-5954-4a73-970e-3f2c924b3daa\") " pod="openshift-lws-operator/lws-controller-manager-6cff9449b4-vtxpl"
Sep 11 01:03:29.009962 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:03:29.009866    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6cff9449b4-vtxpl"
Sep 11 01:03:29.263129 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:03:29.263063    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-5cc5ff74fb-8j6n9"]
Sep 11 01:03:29.267112 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:03:29.267088    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5cc5ff74fb-8j6n9"
Sep 11 01:03:29.277719 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:03:29.277699    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 11 01:03:29.278002 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:03:29.277988    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 11 01:03:29.278835 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:03:29.278820    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 11 01:03:29.279042 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:03:29.279029    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-5wgf8\""
Sep 11 01:03:29.287933 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:03:29.287917    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 11 01:03:29.291748 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:03:29.291728    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5cc5ff74fb-8j6n9"]
Sep 11 01:03:29.322302 ip-10-0-129-222 kubenswrapper[2569]: W0911 01:03:29.322281    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91d835d3_5954_4a73_970e_3f2c924b3daa.slice/crio-1e598de3246c2bc8acff29d9e51566ad8aa9280b92e206133a01f600f30f2c0b WatchSource:0}: Error finding container 1e598de3246c2bc8acff29d9e51566ad8aa9280b92e206133a01f600f30f2c0b: Status 404 returned error can't find the container with id 1e598de3246c2bc8acff29d9e51566ad8aa9280b92e206133a01f600f30f2c0b
Sep 11 01:03:29.329725 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:03:29.329702    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6cff9449b4-vtxpl"]
Sep 11 01:03:29.395798 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:03:29.395774    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b406e46a-719d-42d9-80b6-5706557ec9f8-tmp\") pod \"kube-auth-proxy-5cc5ff74fb-8j6n9\" (UID: \"b406e46a-719d-42d9-80b6-5706557ec9f8\") " pod="openshift-ingress/kube-auth-proxy-5cc5ff74fb-8j6n9"
Sep 11 01:03:29.395923 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:03:29.395802    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmkq7\" (UniqueName: \"kubernetes.io/projected/b406e46a-719d-42d9-80b6-5706557ec9f8-kube-api-access-dmkq7\") pod \"kube-auth-proxy-5cc5ff74fb-8j6n9\" (UID: \"b406e46a-719d-42d9-80b6-5706557ec9f8\") " pod="openshift-ingress/kube-auth-proxy-5cc5ff74fb-8j6n9"
Sep 11 01:03:29.395923 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:03:29.395835    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b406e46a-719d-42d9-80b6-5706557ec9f8-tls-certs\") pod \"kube-auth-proxy-5cc5ff74fb-8j6n9\" (UID: \"b406e46a-719d-42d9-80b6-5706557ec9f8\") " pod="openshift-ingress/kube-auth-proxy-5cc5ff74fb-8j6n9"
Sep 11 01:03:29.477175 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:03:29.477146    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6cff9449b4-vtxpl" event={"ID":"91d835d3-5954-4a73-970e-3f2c924b3daa","Type":"ContainerStarted","Data":"1e598de3246c2bc8acff29d9e51566ad8aa9280b92e206133a01f600f30f2c0b"}
Sep 11 01:03:29.496461 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:03:29.496432    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b406e46a-719d-42d9-80b6-5706557ec9f8-tmp\") pod \"kube-auth-proxy-5cc5ff74fb-8j6n9\" (UID: \"b406e46a-719d-42d9-80b6-5706557ec9f8\") " pod="openshift-ingress/kube-auth-proxy-5cc5ff74fb-8j6n9"
Sep 11 01:03:29.496570 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:03:29.496465    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmkq7\" (UniqueName: \"kubernetes.io/projected/b406e46a-719d-42d9-80b6-5706557ec9f8-kube-api-access-dmkq7\") pod \"kube-auth-proxy-5cc5ff74fb-8j6n9\" (UID: \"b406e46a-719d-42d9-80b6-5706557ec9f8\") " pod="openshift-ingress/kube-auth-proxy-5cc5ff74fb-8j6n9"
Sep 11 01:03:29.496570 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:03:29.496519    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b406e46a-719d-42d9-80b6-5706557ec9f8-tls-certs\") pod \"kube-auth-proxy-5cc5ff74fb-8j6n9\" (UID: \"b406e46a-719d-42d9-80b6-5706557ec9f8\") " pod="openshift-ingress/kube-auth-proxy-5cc5ff74fb-8j6n9"
Sep 11 01:03:29.498760 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:03:29.498739    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b406e46a-719d-42d9-80b6-5706557ec9f8-tmp\") pod \"kube-auth-proxy-5cc5ff74fb-8j6n9\" (UID: \"b406e46a-719d-42d9-80b6-5706557ec9f8\") " pod="openshift-ingress/kube-auth-proxy-5cc5ff74fb-8j6n9"
Sep 11 01:03:29.499130 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:03:29.499112    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b406e46a-719d-42d9-80b6-5706557ec9f8-tls-certs\") pod \"kube-auth-proxy-5cc5ff74fb-8j6n9\" (UID: \"b406e46a-719d-42d9-80b6-5706557ec9f8\") " pod="openshift-ingress/kube-auth-proxy-5cc5ff74fb-8j6n9"
Sep 11 01:03:29.513312 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:03:29.513265    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmkq7\" (UniqueName: \"kubernetes.io/projected/b406e46a-719d-42d9-80b6-5706557ec9f8-kube-api-access-dmkq7\") pod \"kube-auth-proxy-5cc5ff74fb-8j6n9\" (UID: \"b406e46a-719d-42d9-80b6-5706557ec9f8\") " pod="openshift-ingress/kube-auth-proxy-5cc5ff74fb-8j6n9"
Sep 11 01:03:29.576802 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:03:29.576763    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5cc5ff74fb-8j6n9"
Sep 11 01:03:29.869224 ip-10-0-129-222 kubenswrapper[2569]: W0911 01:03:29.869196    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb406e46a_719d_42d9_80b6_5706557ec9f8.slice/crio-adc78b9176a5b6cc1317839102b2ff0b135d4dac98d0d73bd20bcb84b002452e WatchSource:0}: Error finding container adc78b9176a5b6cc1317839102b2ff0b135d4dac98d0d73bd20bcb84b002452e: Status 404 returned error can't find the container with id adc78b9176a5b6cc1317839102b2ff0b135d4dac98d0d73bd20bcb84b002452e
Sep 11 01:03:29.870670 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:03:29.870643    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5cc5ff74fb-8j6n9"]
Sep 11 01:03:30.488671 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:03:30.488629    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5cc5ff74fb-8j6n9" event={"ID":"b406e46a-719d-42d9-80b6-5706557ec9f8","Type":"ContainerStarted","Data":"adc78b9176a5b6cc1317839102b2ff0b135d4dac98d0d73bd20bcb84b002452e"}
Sep 11 01:03:34.511710 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:03:34.511674    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5cc5ff74fb-8j6n9" event={"ID":"b406e46a-719d-42d9-80b6-5706557ec9f8","Type":"ContainerStarted","Data":"af5edcf86e5dbbef06f8ac4c7c83c6a637a13c58fed673a3d1ec7492d327285c"}
Sep 11 01:03:34.513260 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:03:34.513233    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6cff9449b4-vtxpl" event={"ID":"91d835d3-5954-4a73-970e-3f2c924b3daa","Type":"ContainerStarted","Data":"3e7c4ec7b78d0d29b645051e5f93ccd2e9fccc21d979b2543656ca990b4ab141"}
Sep 11 01:03:34.513402 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:03:34.513364    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-6cff9449b4-vtxpl"
Sep 11 01:03:34.527536 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:03:34.527493    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-5cc5ff74fb-8j6n9" podStartSLOduration=1.527480414 podStartE2EDuration="5.527481314s" totalImagesPullingTime="4.0000009s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 01:03:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 01:03:34.526255701 +0000 UTC m=+349.177629947" watchObservedRunningTime="2026-09-11 01:03:34.527481314 +0000 UTC m=+349.178855563"
Sep 11 01:03:34.540607 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:03:34.540557    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-6cff9449b4-vtxpl" podStartSLOduration=1.945713043 podStartE2EDuration="6.540539424s" totalImagesPullingTime="4.594826381s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 01:03:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 01:03:34.540072057 +0000 UTC m=+349.191446306" watchObservedRunningTime="2026-09-11 01:03:34.540539424 +0000 UTC m=+349.191913674"
Sep 11 01:03:45.519658 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:03:45.519580    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-6cff9449b4-vtxpl"
Sep 11 01:05:05.063817 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:05:05.063783    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-wmkfk"]
Sep 11 01:05:05.067259 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:05:05.067243    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-wmkfk"
Sep 11 01:05:05.069645 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:05:05.069617    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Sep 11 01:05:05.069645 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:05:05.069635    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 11 01:05:05.069815 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:05:05.069695    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-zs6x2\""
Sep 11 01:05:05.070394 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:05:05.070379    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 11 01:05:05.077083 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:05:05.077059    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-wmkfk"]
Sep 11 01:05:05.091171 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:05:05.091139    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-977xl\" (UniqueName: \"kubernetes.io/projected/61cd974e-1ee7-4b63-aa87-1d32dc4b2952-kube-api-access-977xl\") pod \"dns-operator-controller-manager-7b84cd7c9c-wmkfk\" (UID: \"61cd974e-1ee7-4b63-aa87-1d32dc4b2952\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-wmkfk"
Sep 11 01:05:05.192408 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:05:05.192381    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-977xl\" (UniqueName: \"kubernetes.io/projected/61cd974e-1ee7-4b63-aa87-1d32dc4b2952-kube-api-access-977xl\") pod \"dns-operator-controller-manager-7b84cd7c9c-wmkfk\" (UID: \"61cd974e-1ee7-4b63-aa87-1d32dc4b2952\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-wmkfk"
Sep 11 01:05:05.201340 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:05:05.201320    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-977xl\" (UniqueName: \"kubernetes.io/projected/61cd974e-1ee7-4b63-aa87-1d32dc4b2952-kube-api-access-977xl\") pod \"dns-operator-controller-manager-7b84cd7c9c-wmkfk\" (UID: \"61cd974e-1ee7-4b63-aa87-1d32dc4b2952\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-wmkfk"
Sep 11 01:05:05.378420 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:05:05.378338    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-wmkfk"
Sep 11 01:05:05.648613 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:05:05.648534    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-wmkfk"]
Sep 11 01:05:05.651822 ip-10-0-129-222 kubenswrapper[2569]: W0911 01:05:05.651791    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61cd974e_1ee7_4b63_aa87_1d32dc4b2952.slice/crio-0deb1e1d5af5c21f34726318a568e5a93bbc7470c73e3733d1cf30642c196f7c WatchSource:0}: Error finding container 0deb1e1d5af5c21f34726318a568e5a93bbc7470c73e3733d1cf30642c196f7c: Status 404 returned error can't find the container with id 0deb1e1d5af5c21f34726318a568e5a93bbc7470c73e3733d1cf30642c196f7c
Sep 11 01:05:05.903014 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:05:05.902938    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-wmkfk" event={"ID":"61cd974e-1ee7-4b63-aa87-1d32dc4b2952","Type":"ContainerStarted","Data":"0deb1e1d5af5c21f34726318a568e5a93bbc7470c73e3733d1cf30642c196f7c"}
Sep 11 01:05:08.919907 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:05:08.919859    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-wmkfk" event={"ID":"61cd974e-1ee7-4b63-aa87-1d32dc4b2952","Type":"ContainerStarted","Data":"8faf490e9a7964bfe82e35db9aaae3b89a721e96334be7052d91e59dda7bcbc1"}
Sep 11 01:05:08.920311 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:05:08.919989    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-wmkfk"
Sep 11 01:05:08.939622 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:05:08.939572    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-wmkfk" podStartSLOduration=1.3290102529999999 podStartE2EDuration="3.939553111s" totalImagesPullingTime="2.610542858s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 01:05:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 01:05:08.93883581 +0000 UTC m=+443.590210058" watchObservedRunningTime="2026-09-11 01:05:08.939553111 +0000 UTC m=+443.590927355"
Sep 11 01:05:09.520284 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:05:09.520248    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-gqczs"]
Sep 11 01:05:09.523567 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:05:09.523551    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-gqczs"
Sep 11 01:05:09.525989 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:05:09.525964    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Sep 11 01:05:09.525989 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:05:09.525981    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-txtlm\""
Sep 11 01:05:09.526189 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:05:09.525972    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Sep 11 01:05:09.537486 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:05:09.537464    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-gqczs"]
Sep 11 01:05:09.625318 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:05:09.625291    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/73dee526-b179-4d7e-8ef0-375ef65b4cee-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-gqczs\" (UID: \"73dee526-b179-4d7e-8ef0-375ef65b4cee\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-gqczs"
Sep 11 01:05:09.625454 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:05:09.625328    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/73dee526-b179-4d7e-8ef0-375ef65b4cee-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-gqczs\" (UID: \"73dee526-b179-4d7e-8ef0-375ef65b4cee\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-gqczs"
Sep 11 01:05:09.625454 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:05:09.625400    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6nwdl\" (UniqueName: \"kubernetes.io/projected/73dee526-b179-4d7e-8ef0-375ef65b4cee-kube-api-access-6nwdl\") pod \"kuadrant-console-plugin-fbdd4f445-gqczs\" (UID: \"73dee526-b179-4d7e-8ef0-375ef65b4cee\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-gqczs"
Sep 11 01:05:09.726736 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:05:09.726708    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nwdl\" (UniqueName: \"kubernetes.io/projected/73dee526-b179-4d7e-8ef0-375ef65b4cee-kube-api-access-6nwdl\") pod \"kuadrant-console-plugin-fbdd4f445-gqczs\" (UID: \"73dee526-b179-4d7e-8ef0-375ef65b4cee\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-gqczs"
Sep 11 01:05:09.726886 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:05:09.726754    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/73dee526-b179-4d7e-8ef0-375ef65b4cee-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-gqczs\" (UID: \"73dee526-b179-4d7e-8ef0-375ef65b4cee\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-gqczs"
Sep 11 01:05:09.726886 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:05:09.726779    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/73dee526-b179-4d7e-8ef0-375ef65b4cee-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-gqczs\" (UID: \"73dee526-b179-4d7e-8ef0-375ef65b4cee\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-gqczs"
Sep 11 01:05:09.727396 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:05:09.727372    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/73dee526-b179-4d7e-8ef0-375ef65b4cee-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-gqczs\" (UID: \"73dee526-b179-4d7e-8ef0-375ef65b4cee\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-gqczs"
Sep 11 01:05:09.729175 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:05:09.729153    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/73dee526-b179-4d7e-8ef0-375ef65b4cee-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-gqczs\" (UID: \"73dee526-b179-4d7e-8ef0-375ef65b4cee\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-gqczs"
Sep 11 01:05:09.734792 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:05:09.734772    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nwdl\" (UniqueName: \"kubernetes.io/projected/73dee526-b179-4d7e-8ef0-375ef65b4cee-kube-api-access-6nwdl\") pod \"kuadrant-console-plugin-fbdd4f445-gqczs\" (UID: \"73dee526-b179-4d7e-8ef0-375ef65b4cee\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-gqczs"
Sep 11 01:05:09.844687 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:05:09.844665    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-gqczs"
Sep 11 01:05:10.113099 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:05:10.113026    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-gqczs"]
Sep 11 01:05:10.115283 ip-10-0-129-222 kubenswrapper[2569]: W0911 01:05:10.115255    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod73dee526_b179_4d7e_8ef0_375ef65b4cee.slice/crio-4747e218927d060e2394be96007006cdb0b53aa126a4f888a1f610588e275526 WatchSource:0}: Error finding container 4747e218927d060e2394be96007006cdb0b53aa126a4f888a1f610588e275526: Status 404 returned error can't find the container with id 4747e218927d060e2394be96007006cdb0b53aa126a4f888a1f610588e275526
Sep 11 01:05:10.932037 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:05:10.931998    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-gqczs" event={"ID":"73dee526-b179-4d7e-8ef0-375ef65b4cee","Type":"ContainerStarted","Data":"4747e218927d060e2394be96007006cdb0b53aa126a4f888a1f610588e275526"}
Sep 11 01:05:14.953649 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:05:14.953618    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-gqczs" event={"ID":"73dee526-b179-4d7e-8ef0-375ef65b4cee","Type":"ContainerStarted","Data":"6bad693c4ffd5b71984271c64aaae0c07e76ece27dced92a6e385479bbb9778c"}
Sep 11 01:05:14.969083 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:05:14.968991    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-gqczs" podStartSLOduration=1.3799588360000001 podStartE2EDuration="5.968976492s" totalImagesPullingTime="4.589017656s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 01:05:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 01:05:14.968179314 +0000 UTC m=+449.619553562" watchObservedRunningTime="2026-09-11 01:05:14.968976492 +0000 UTC m=+449.620350742"
Sep 11 01:05:19.926366 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:05:19.926334    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-wmkfk"
Sep 11 01:05:20.024233 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:05:20.024184    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-p5cvq"]
Sep 11 01:05:20.027949 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:05:20.027928    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-p5cvq"
Sep 11 01:05:20.030195 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:05:20.030170    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-sstc2\""
Sep 11 01:05:20.040022 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:05:20.039991    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-p5cvq"]
Sep 11 01:05:20.115028 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:05:20.115000    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/95cb4a17-bb29-4dfd-948a-2140f196c406-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5f685c496f-p5cvq\" (UID: \"95cb4a17-bb29-4dfd-948a-2140f196c406\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-p5cvq"
Sep 11 01:05:20.115164 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:05:20.115093    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wnj7h\" (UniqueName: \"kubernetes.io/projected/95cb4a17-bb29-4dfd-948a-2140f196c406-kube-api-access-wnj7h\") pod \"kuadrant-operator-controller-manager-5f685c496f-p5cvq\" (UID: \"95cb4a17-bb29-4dfd-948a-2140f196c406\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-p5cvq"
Sep 11 01:05:20.123305 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:05:20.123273    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-p5cvq"]
Sep 11 01:05:20.123526 ip-10-0-129-222 kubenswrapper[2569]: E0911 01:05:20.123504    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[extensions-socket-volume kube-api-access-wnj7h], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-p5cvq" podUID="95cb4a17-bb29-4dfd-948a-2140f196c406"
Sep 11 01:05:20.215797 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:05:20.215722    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wnj7h\" (UniqueName: \"kubernetes.io/projected/95cb4a17-bb29-4dfd-948a-2140f196c406-kube-api-access-wnj7h\") pod \"kuadrant-operator-controller-manager-5f685c496f-p5cvq\" (UID: \"95cb4a17-bb29-4dfd-948a-2140f196c406\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-p5cvq"
Sep 11 01:05:20.215797 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:05:20.215779    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/95cb4a17-bb29-4dfd-948a-2140f196c406-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5f685c496f-p5cvq\" (UID: \"95cb4a17-bb29-4dfd-948a-2140f196c406\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-p5cvq"
Sep 11 01:05:20.216124 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:05:20.216107    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/95cb4a17-bb29-4dfd-948a-2140f196c406-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5f685c496f-p5cvq\" (UID: \"95cb4a17-bb29-4dfd-948a-2140f196c406\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-p5cvq"
Sep 11 01:05:20.229017 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:05:20.228990    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wnj7h\" (UniqueName: \"kubernetes.io/projected/95cb4a17-bb29-4dfd-948a-2140f196c406-kube-api-access-wnj7h\") pod \"kuadrant-operator-controller-manager-5f685c496f-p5cvq\" (UID: \"95cb4a17-bb29-4dfd-948a-2140f196c406\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-p5cvq"
Sep 11 01:05:20.626219 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:05:20.626184    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-p5cvq"]
Sep 11 01:05:20.642687 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:05:20.642659    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-p5cvq"]
Sep 11 01:05:20.708985 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:05:20.708947    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-mrzxn"]
Sep 11 01:05:20.712502 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:05:20.712483    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-mrzxn"
Sep 11 01:05:20.736210 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:05:20.736184    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-mrzxn"]
Sep 11 01:05:20.820751 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:05:20.820714    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6fcb\" (UniqueName: \"kubernetes.io/projected/c6d102be-11cb-4e56-bfd7-4847abda44a6-kube-api-access-r6fcb\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-mrzxn\" (UID: \"c6d102be-11cb-4e56-bfd7-4847abda44a6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-mrzxn"
Sep 11 01:05:20.820933 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:05:20.820849    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c6d102be-11cb-4e56-bfd7-4847abda44a6-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-mrzxn\" (UID: \"c6d102be-11cb-4e56-bfd7-4847abda44a6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-mrzxn"
Sep 11 01:05:20.922244 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:05:20.922166    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c6d102be-11cb-4e56-bfd7-4847abda44a6-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-mrzxn\" (UID: \"c6d102be-11cb-4e56-bfd7-4847abda44a6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-mrzxn"
Sep 11 01:05:20.922244 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:05:20.922207    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6fcb\" (UniqueName: \"kubernetes.io/projected/c6d102be-11cb-4e56-bfd7-4847abda44a6-kube-api-access-r6fcb\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-mrzxn\" (UID: \"c6d102be-11cb-4e56-bfd7-4847abda44a6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-mrzxn"
Sep 11 01:05:20.922605 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:05:20.922582    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c6d102be-11cb-4e56-bfd7-4847abda44a6-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-mrzxn\" (UID: \"c6d102be-11cb-4e56-bfd7-4847abda44a6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-mrzxn"
Sep 11 01:05:20.930998 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:05:20.930969    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6fcb\" (UniqueName: \"kubernetes.io/projected/c6d102be-11cb-4e56-bfd7-4847abda44a6-kube-api-access-r6fcb\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-mrzxn\" (UID: \"c6d102be-11cb-4e56-bfd7-4847abda44a6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-mrzxn"
Sep 11 01:05:20.982249 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:05:20.982221    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-p5cvq"
Sep 11 01:05:20.984217 ip-10-0-129-222 kubenswrapper[2569]: E0911 01:05:20.984193    2569 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-5f685c496f-p5cvq\" is forbidden: User \"system:node:ip-10-0-129-222.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-222.ec2.internal' and this object" podUID="95cb4a17-bb29-4dfd-948a-2140f196c406" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-p5cvq"
Sep 11 01:05:20.989190 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:05:20.989174    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-p5cvq"
Sep 11 01:05:20.991152 ip-10-0-129-222 kubenswrapper[2569]: E0911 01:05:20.991131    2569 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-5f685c496f-p5cvq\" is forbidden: User \"system:node:ip-10-0-129-222.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-222.ec2.internal' and this object" podUID="95cb4a17-bb29-4dfd-948a-2140f196c406" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-p5cvq"
Sep 11 01:05:21.023348 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:05:21.023326    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-mrzxn"
Sep 11 01:05:21.124654 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:05:21.124620    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wnj7h\" (UniqueName: \"kubernetes.io/projected/95cb4a17-bb29-4dfd-948a-2140f196c406-kube-api-access-wnj7h\") pod \"95cb4a17-bb29-4dfd-948a-2140f196c406\" (UID: \"95cb4a17-bb29-4dfd-948a-2140f196c406\") "
Sep 11 01:05:21.124798 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:05:21.124692    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/95cb4a17-bb29-4dfd-948a-2140f196c406-extensions-socket-volume\") pod \"95cb4a17-bb29-4dfd-948a-2140f196c406\" (UID: \"95cb4a17-bb29-4dfd-948a-2140f196c406\") "
Sep 11 01:05:21.124983 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:05:21.124961    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95cb4a17-bb29-4dfd-948a-2140f196c406-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "95cb4a17-bb29-4dfd-948a-2140f196c406" (UID: "95cb4a17-bb29-4dfd-948a-2140f196c406"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 01:05:21.126628 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:05:21.126598    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95cb4a17-bb29-4dfd-948a-2140f196c406-kube-api-access-wnj7h" (OuterVolumeSpecName: "kube-api-access-wnj7h") pod "95cb4a17-bb29-4dfd-948a-2140f196c406" (UID: "95cb4a17-bb29-4dfd-948a-2140f196c406"). InnerVolumeSpecName "kube-api-access-wnj7h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 01:05:21.225609 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:05:21.225535    2569 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/95cb4a17-bb29-4dfd-948a-2140f196c406-extensions-socket-volume\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:05:21.225609 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:05:21.225561    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wnj7h\" (UniqueName: \"kubernetes.io/projected/95cb4a17-bb29-4dfd-948a-2140f196c406-kube-api-access-wnj7h\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:05:21.305509 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:05:21.305485    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-mrzxn"]
Sep 11 01:05:21.308482 ip-10-0-129-222 kubenswrapper[2569]: W0911 01:05:21.308454    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6d102be_11cb_4e56_bfd7_4847abda44a6.slice/crio-d323a45cdef670fd33efbf3a0e120edfdd17f35c9d03ee025b17243cf3894db4 WatchSource:0}: Error finding container d323a45cdef670fd33efbf3a0e120edfdd17f35c9d03ee025b17243cf3894db4: Status 404 returned error can't find the container with id d323a45cdef670fd33efbf3a0e120edfdd17f35c9d03ee025b17243cf3894db4
Sep 11 01:05:21.871044 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:05:21.871013    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="95cb4a17-bb29-4dfd-948a-2140f196c406" path="/var/lib/kubelet/pods/95cb4a17-bb29-4dfd-948a-2140f196c406/volumes"
Sep 11 01:05:21.988820 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:05:21.988790    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-mrzxn" event={"ID":"c6d102be-11cb-4e56-bfd7-4847abda44a6","Type":"ContainerStarted","Data":"d323a45cdef670fd33efbf3a0e120edfdd17f35c9d03ee025b17243cf3894db4"}
Sep 11 01:05:21.989200 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:05:21.988803    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-p5cvq"
Sep 11 01:05:21.993240 ip-10-0-129-222 kubenswrapper[2569]: E0911 01:05:21.993206    2569 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-5f685c496f-p5cvq\" is forbidden: User \"system:node:ip-10-0-129-222.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-222.ec2.internal' and this object" podUID="95cb4a17-bb29-4dfd-948a-2140f196c406" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-p5cvq"
Sep 11 01:05:26.008288 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:05:26.008253    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-mrzxn" event={"ID":"c6d102be-11cb-4e56-bfd7-4847abda44a6","Type":"ContainerStarted","Data":"d51db6e1235f1d3baa578e828be3c6f961534fe201fbec6fad5c20378548466e"}
Sep 11 01:05:26.008706 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:05:26.008335    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-mrzxn"
Sep 11 01:05:26.028517 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:05:26.028469    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-mrzxn" podStartSLOduration=2.070276062 podStartE2EDuration="6.028455972s" totalImagesPullingTime="3.95817991s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 01:05:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 01:05:26.027066014 +0000 UTC m=+460.678440265" watchObservedRunningTime="2026-09-11 01:05:26.028455972 +0000 UTC m=+460.679830220"
Sep 11 01:05:37.015439 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:05:37.015409    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-mrzxn"
Sep 11 01:05:49.696933 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:05:49.696893    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-mrzxn"]
Sep 11 01:05:49.697468 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:05:49.697178    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-mrzxn" podUID="c6d102be-11cb-4e56-bfd7-4847abda44a6" containerName="manager" containerID="cri-o://d51db6e1235f1d3baa578e828be3c6f961534fe201fbec6fad5c20378548466e" gracePeriod=10
Sep 11 01:05:49.946308 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:05:49.946284    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-mrzxn"
Sep 11 01:05:50.072775 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:05:50.072747    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c6d102be-11cb-4e56-bfd7-4847abda44a6-extensions-socket-volume\") pod \"c6d102be-11cb-4e56-bfd7-4847abda44a6\" (UID: \"c6d102be-11cb-4e56-bfd7-4847abda44a6\") "
Sep 11 01:05:50.072941 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:05:50.072789    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r6fcb\" (UniqueName: \"kubernetes.io/projected/c6d102be-11cb-4e56-bfd7-4847abda44a6-kube-api-access-r6fcb\") pod \"c6d102be-11cb-4e56-bfd7-4847abda44a6\" (UID: \"c6d102be-11cb-4e56-bfd7-4847abda44a6\") "
Sep 11 01:05:50.073134 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:05:50.073114    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6d102be-11cb-4e56-bfd7-4847abda44a6-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "c6d102be-11cb-4e56-bfd7-4847abda44a6" (UID: "c6d102be-11cb-4e56-bfd7-4847abda44a6"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 01:05:50.074910 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:05:50.074888    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6d102be-11cb-4e56-bfd7-4847abda44a6-kube-api-access-r6fcb" (OuterVolumeSpecName: "kube-api-access-r6fcb") pod "c6d102be-11cb-4e56-bfd7-4847abda44a6" (UID: "c6d102be-11cb-4e56-bfd7-4847abda44a6"). InnerVolumeSpecName "kube-api-access-r6fcb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 01:05:50.119193 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:05:50.119165    2569 generic.go:356] "Generic (PLEG): container finished" podID="c6d102be-11cb-4e56-bfd7-4847abda44a6" containerID="d51db6e1235f1d3baa578e828be3c6f961534fe201fbec6fad5c20378548466e" exitCode=0
Sep 11 01:05:50.119304 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:05:50.119203    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-mrzxn"
Sep 11 01:05:50.119304 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:05:50.119224    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-mrzxn" event={"ID":"c6d102be-11cb-4e56-bfd7-4847abda44a6","Type":"ContainerDied","Data":"d51db6e1235f1d3baa578e828be3c6f961534fe201fbec6fad5c20378548466e"}
Sep 11 01:05:50.119304 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:05:50.119262    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-mrzxn" event={"ID":"c6d102be-11cb-4e56-bfd7-4847abda44a6","Type":"ContainerDied","Data":"d323a45cdef670fd33efbf3a0e120edfdd17f35c9d03ee025b17243cf3894db4"}
Sep 11 01:05:50.119304 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:05:50.119282    2569 scope.go:117] "RemoveContainer" containerID="d51db6e1235f1d3baa578e828be3c6f961534fe201fbec6fad5c20378548466e"
Sep 11 01:05:50.127551 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:05:50.127531    2569 scope.go:117] "RemoveContainer" containerID="d51db6e1235f1d3baa578e828be3c6f961534fe201fbec6fad5c20378548466e"
Sep 11 01:05:50.127807 ip-10-0-129-222 kubenswrapper[2569]: E0911 01:05:50.127788    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d51db6e1235f1d3baa578e828be3c6f961534fe201fbec6fad5c20378548466e\": container with ID starting with d51db6e1235f1d3baa578e828be3c6f961534fe201fbec6fad5c20378548466e not found: ID does not exist" containerID="d51db6e1235f1d3baa578e828be3c6f961534fe201fbec6fad5c20378548466e"
Sep 11 01:05:50.127854 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:05:50.127816    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d51db6e1235f1d3baa578e828be3c6f961534fe201fbec6fad5c20378548466e"} err="failed to get container status \"d51db6e1235f1d3baa578e828be3c6f961534fe201fbec6fad5c20378548466e\": rpc error: code = NotFound desc = could not find container \"d51db6e1235f1d3baa578e828be3c6f961534fe201fbec6fad5c20378548466e\": container with ID starting with d51db6e1235f1d3baa578e828be3c6f961534fe201fbec6fad5c20378548466e not found: ID does not exist"
Sep 11 01:05:50.140745 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:05:50.140724    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-mrzxn"]
Sep 11 01:05:50.143972 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:05:50.143954    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-mrzxn"]
Sep 11 01:05:50.173709 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:05:50.173685    2569 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c6d102be-11cb-4e56-bfd7-4847abda44a6-extensions-socket-volume\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:05:50.173709 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:05:50.173707    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r6fcb\" (UniqueName: \"kubernetes.io/projected/c6d102be-11cb-4e56-bfd7-4847abda44a6-kube-api-access-r6fcb\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:05:51.870206 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:05:51.870174    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c6d102be-11cb-4e56-bfd7-4847abda44a6" path="/var/lib/kubelet/pods/c6d102be-11cb-4e56-bfd7-4847abda44a6/volumes"
Sep 11 01:05:53.421955 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:05:53.421917    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2t8jt"]
Sep 11 01:05:53.422357 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:05:53.422309    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6d102be-11cb-4e56-bfd7-4847abda44a6" containerName="manager"
Sep 11 01:05:53.422357 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:05:53.422321    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6d102be-11cb-4e56-bfd7-4847abda44a6" containerName="manager"
Sep 11 01:05:53.422437 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:05:53.422385    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="c6d102be-11cb-4e56-bfd7-4847abda44a6" containerName="manager"
Sep 11 01:05:53.426858 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:05:53.426836    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2t8jt"
Sep 11 01:05:53.429015 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:05:53.428993    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 11 01:05:53.429120 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:05:53.429080    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-lf64v\""
Sep 11 01:05:53.436321 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:05:53.436294    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2t8jt"]
Sep 11 01:05:53.498049 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:05:53.498021    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8f2fea7-8e9c-45dd-9291-6e261d2e69db-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2t8jt\" (UID: \"b8f2fea7-8e9c-45dd-9291-6e261d2e69db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2t8jt"
Sep 11 01:05:53.498185 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:05:53.498053    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8f2fea7-8e9c-45dd-9291-6e261d2e69db-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2t8jt\" (UID: \"b8f2fea7-8e9c-45dd-9291-6e261d2e69db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2t8jt"
Sep 11 01:05:53.498185 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:05:53.498089    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8f2fea7-8e9c-45dd-9291-6e261d2e69db-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2t8jt\" (UID: \"b8f2fea7-8e9c-45dd-9291-6e261d2e69db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2t8jt"
Sep 11 01:05:53.498185 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:05:53.498136    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8f2fea7-8e9c-45dd-9291-6e261d2e69db-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2t8jt\" (UID: \"b8f2fea7-8e9c-45dd-9291-6e261d2e69db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2t8jt"
Sep 11 01:05:53.498185 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:05:53.498162    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8f2fea7-8e9c-45dd-9291-6e261d2e69db-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2t8jt\" (UID: \"b8f2fea7-8e9c-45dd-9291-6e261d2e69db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2t8jt"
Sep 11 01:05:53.498326 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:05:53.498203    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8f2fea7-8e9c-45dd-9291-6e261d2e69db-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2t8jt\" (UID: \"b8f2fea7-8e9c-45dd-9291-6e261d2e69db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2t8jt"
Sep 11 01:05:53.498326 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:05:53.498222    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8f2fea7-8e9c-45dd-9291-6e261d2e69db-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2t8jt\" (UID: \"b8f2fea7-8e9c-45dd-9291-6e261d2e69db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2t8jt"
Sep 11 01:05:53.498326 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:05:53.498280    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8f2fea7-8e9c-45dd-9291-6e261d2e69db-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2t8jt\" (UID: \"b8f2fea7-8e9c-45dd-9291-6e261d2e69db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2t8jt"
Sep 11 01:05:53.498326 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:05:53.498310    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dkqqs\" (UniqueName: \"kubernetes.io/projected/b8f2fea7-8e9c-45dd-9291-6e261d2e69db-kube-api-access-dkqqs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2t8jt\" (UID: \"b8f2fea7-8e9c-45dd-9291-6e261d2e69db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2t8jt"
Sep 11 01:05:53.598651 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:05:53.598621    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8f2fea7-8e9c-45dd-9291-6e261d2e69db-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2t8jt\" (UID: \"b8f2fea7-8e9c-45dd-9291-6e261d2e69db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2t8jt"
Sep 11 01:05:53.598792 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:05:53.598656    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkqqs\" (UniqueName: \"kubernetes.io/projected/b8f2fea7-8e9c-45dd-9291-6e261d2e69db-kube-api-access-dkqqs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2t8jt\" (UID: \"b8f2fea7-8e9c-45dd-9291-6e261d2e69db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2t8jt"
Sep 11 01:05:53.598851 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:05:53.598835    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8f2fea7-8e9c-45dd-9291-6e261d2e69db-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2t8jt\" (UID: \"b8f2fea7-8e9c-45dd-9291-6e261d2e69db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2t8jt"
Sep 11 01:05:53.598909 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:05:53.598863    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8f2fea7-8e9c-45dd-9291-6e261d2e69db-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2t8jt\" (UID: \"b8f2fea7-8e9c-45dd-9291-6e261d2e69db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2t8jt"
Sep 11 01:05:53.598955 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:05:53.598923    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8f2fea7-8e9c-45dd-9291-6e261d2e69db-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2t8jt\" (UID: \"b8f2fea7-8e9c-45dd-9291-6e261d2e69db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2t8jt"
Sep 11 01:05:53.598955 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:05:53.598947    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8f2fea7-8e9c-45dd-9291-6e261d2e69db-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2t8jt\" (UID: \"b8f2fea7-8e9c-45dd-9291-6e261d2e69db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2t8jt"
Sep 11 01:05:53.599055 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:05:53.598977    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8f2fea7-8e9c-45dd-9291-6e261d2e69db-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2t8jt\" (UID: \"b8f2fea7-8e9c-45dd-9291-6e261d2e69db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2t8jt"
Sep 11 01:05:53.599055 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:05:53.599021    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8f2fea7-8e9c-45dd-9291-6e261d2e69db-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2t8jt\" (UID: \"b8f2fea7-8e9c-45dd-9291-6e261d2e69db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2t8jt"
Sep 11 01:05:53.599055 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:05:53.599047    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8f2fea7-8e9c-45dd-9291-6e261d2e69db-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2t8jt\" (UID: \"b8f2fea7-8e9c-45dd-9291-6e261d2e69db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2t8jt"
Sep 11 01:05:53.599202 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:05:53.599062    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8f2fea7-8e9c-45dd-9291-6e261d2e69db-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2t8jt\" (UID: \"b8f2fea7-8e9c-45dd-9291-6e261d2e69db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2t8jt"
Sep 11 01:05:53.599331 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:05:53.599225    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8f2fea7-8e9c-45dd-9291-6e261d2e69db-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2t8jt\" (UID: \"b8f2fea7-8e9c-45dd-9291-6e261d2e69db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2t8jt"
Sep 11 01:05:53.599416 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:05:53.599393    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8f2fea7-8e9c-45dd-9291-6e261d2e69db-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2t8jt\" (UID: \"b8f2fea7-8e9c-45dd-9291-6e261d2e69db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2t8jt"
Sep 11 01:05:53.599465 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:05:53.599406    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8f2fea7-8e9c-45dd-9291-6e261d2e69db-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2t8jt\" (UID: \"b8f2fea7-8e9c-45dd-9291-6e261d2e69db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2t8jt"
Sep 11 01:05:53.599614 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:05:53.599593    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8f2fea7-8e9c-45dd-9291-6e261d2e69db-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2t8jt\" (UID: \"b8f2fea7-8e9c-45dd-9291-6e261d2e69db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2t8jt"
Sep 11 01:05:53.601193 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:05:53.601167    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8f2fea7-8e9c-45dd-9291-6e261d2e69db-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2t8jt\" (UID: \"b8f2fea7-8e9c-45dd-9291-6e261d2e69db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2t8jt"
Sep 11 01:05:53.601447 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:05:53.601428    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8f2fea7-8e9c-45dd-9291-6e261d2e69db-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2t8jt\" (UID: \"b8f2fea7-8e9c-45dd-9291-6e261d2e69db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2t8jt"
Sep 11 01:05:53.606453 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:05:53.606428    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8f2fea7-8e9c-45dd-9291-6e261d2e69db-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2t8jt\" (UID: \"b8f2fea7-8e9c-45dd-9291-6e261d2e69db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2t8jt"
Sep 11 01:05:53.606638 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:05:53.606619    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkqqs\" (UniqueName: \"kubernetes.io/projected/b8f2fea7-8e9c-45dd-9291-6e261d2e69db-kube-api-access-dkqqs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2t8jt\" (UID: \"b8f2fea7-8e9c-45dd-9291-6e261d2e69db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2t8jt"
Sep 11 01:05:53.737743 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:05:53.737672    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2t8jt"
Sep 11 01:05:54.014365 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:05:54.014329    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2t8jt"]
Sep 11 01:05:54.019004 ip-10-0-129-222 kubenswrapper[2569]: W0911 01:05:54.018978    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8f2fea7_8e9c_45dd_9291_6e261d2e69db.slice/crio-555320b84609fb7b006440d9a06b7f953b2ee9a4a6e27511b8c709d6b752838d WatchSource:0}: Error finding container 555320b84609fb7b006440d9a06b7f953b2ee9a4a6e27511b8c709d6b752838d: Status 404 returned error can't find the container with id 555320b84609fb7b006440d9a06b7f953b2ee9a4a6e27511b8c709d6b752838d
Sep 11 01:05:54.139352 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:05:54.139312    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2t8jt" event={"ID":"b8f2fea7-8e9c-45dd-9291-6e261d2e69db","Type":"ContainerStarted","Data":"555320b84609fb7b006440d9a06b7f953b2ee9a4a6e27511b8c709d6b752838d"}
Sep 11 01:05:56.870237 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:05:56.870186    2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 01:05:56.870495 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:05:56.870270    2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 01:05:56.870495 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:05:56.870305    2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 01:05:57.157566 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:05:57.157484    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2t8jt" event={"ID":"b8f2fea7-8e9c-45dd-9291-6e261d2e69db","Type":"ContainerStarted","Data":"0e276f0c3fa3e504065a18c7e2e3e0c6aa4874ab469d09f70e7a692b774bcfbf"}
Sep 11 01:05:57.176773 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:05:57.176722    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2t8jt" podStartSLOduration=1.328159469 podStartE2EDuration="4.176705757s" totalImagesPullingTime="2.848546288s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 01:05:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 01:05:57.175988618 +0000 UTC m=+491.827362868" watchObservedRunningTime="2026-09-11 01:05:57.176705757 +0000 UTC m=+491.828080005"
Sep 11 01:05:57.737901 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:05:57.737843    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2t8jt"
Sep 11 01:05:57.742386 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:05:57.742360    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2t8jt"
Sep 11 01:05:58.167683 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:05:58.167646    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2t8jt"
Sep 11 01:05:58.168510 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:05:58.168487    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2t8jt"
Sep 11 01:05:58.589907 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:05:58.589864    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kmx5x"]
Sep 11 01:05:58.593428 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:05:58.593410    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-kmx5x"
Sep 11 01:05:58.596221 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:05:58.596199    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 11 01:05:58.604758 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:05:58.604738    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kmx5x"]
Sep 11 01:05:58.619742 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:05:58.619715    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kmx5x"]
Sep 11 01:05:58.634474 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:05:58.634450    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2r4z8\" (UniqueName: \"kubernetes.io/projected/257b2c19-d395-40eb-8e08-8ba9560fd070-kube-api-access-2r4z8\") pod \"limitador-limitador-84994f7ffc-kmx5x\" (UID: \"257b2c19-d395-40eb-8e08-8ba9560fd070\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-kmx5x"
Sep 11 01:05:58.634557 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:05:58.634487    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/257b2c19-d395-40eb-8e08-8ba9560fd070-config-file\") pod \"limitador-limitador-84994f7ffc-kmx5x\" (UID: \"257b2c19-d395-40eb-8e08-8ba9560fd070\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-kmx5x"
Sep 11 01:05:58.735134 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:05:58.735101    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2r4z8\" (UniqueName: \"kubernetes.io/projected/257b2c19-d395-40eb-8e08-8ba9560fd070-kube-api-access-2r4z8\") pod \"limitador-limitador-84994f7ffc-kmx5x\" (UID: \"257b2c19-d395-40eb-8e08-8ba9560fd070\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-kmx5x"
Sep 11 01:05:58.735283 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:05:58.735139    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/257b2c19-d395-40eb-8e08-8ba9560fd070-config-file\") pod \"limitador-limitador-84994f7ffc-kmx5x\" (UID: \"257b2c19-d395-40eb-8e08-8ba9560fd070\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-kmx5x"
Sep 11 01:05:58.735766 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:05:58.735747    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/257b2c19-d395-40eb-8e08-8ba9560fd070-config-file\") pod \"limitador-limitador-84994f7ffc-kmx5x\" (UID: \"257b2c19-d395-40eb-8e08-8ba9560fd070\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-kmx5x"
Sep 11 01:05:58.743185 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:05:58.743165    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2r4z8\" (UniqueName: \"kubernetes.io/projected/257b2c19-d395-40eb-8e08-8ba9560fd070-kube-api-access-2r4z8\") pod \"limitador-limitador-84994f7ffc-kmx5x\" (UID: \"257b2c19-d395-40eb-8e08-8ba9560fd070\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-kmx5x"
Sep 11 01:05:58.904030 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:05:58.903952    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-kmx5x"
Sep 11 01:05:59.009164 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:05:59.009132    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-kj6m4"]
Sep 11 01:05:59.013634 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:05:59.013618    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-kj6m4"
Sep 11 01:05:59.016348 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:05:59.016329    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-bjhvj\""
Sep 11 01:05:59.020385 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:05:59.020366    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-kj6m4"]
Sep 11 01:05:59.036692 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:05:59.036669    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7zrn\" (UniqueName: \"kubernetes.io/projected/9043f999-c9bf-4793-9159-c1153ee53518-kube-api-access-t7zrn\") pod \"authorino-7bcfd8f9cd-kj6m4\" (UID: \"9043f999-c9bf-4793-9159-c1153ee53518\") " pod="kuadrant-system/authorino-7bcfd8f9cd-kj6m4"
Sep 11 01:05:59.137564 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:05:59.137534    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7zrn\" (UniqueName: \"kubernetes.io/projected/9043f999-c9bf-4793-9159-c1153ee53518-kube-api-access-t7zrn\") pod \"authorino-7bcfd8f9cd-kj6m4\" (UID: \"9043f999-c9bf-4793-9159-c1153ee53518\") " pod="kuadrant-system/authorino-7bcfd8f9cd-kj6m4"
Sep 11 01:05:59.148006 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:05:59.147981    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7zrn\" (UniqueName: \"kubernetes.io/projected/9043f999-c9bf-4793-9159-c1153ee53518-kube-api-access-t7zrn\") pod \"authorino-7bcfd8f9cd-kj6m4\" (UID: \"9043f999-c9bf-4793-9159-c1153ee53518\") " pod="kuadrant-system/authorino-7bcfd8f9cd-kj6m4"
Sep 11 01:05:59.174156 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:05:59.174133    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kmx5x"]
Sep 11 01:05:59.176455 ip-10-0-129-222 kubenswrapper[2569]: W0911 01:05:59.176432    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod257b2c19_d395_40eb_8e08_8ba9560fd070.slice/crio-93bcefdee68647442b7596688b404a6cced66d3c6aebec69713d2e3f819a3a40 WatchSource:0}: Error finding container 93bcefdee68647442b7596688b404a6cced66d3c6aebec69713d2e3f819a3a40: Status 404 returned error can't find the container with id 93bcefdee68647442b7596688b404a6cced66d3c6aebec69713d2e3f819a3a40
Sep 11 01:05:59.324302 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:05:59.324274    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-kj6m4"
Sep 11 01:05:59.595217 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:05:59.595162    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-kj6m4"]
Sep 11 01:05:59.597902 ip-10-0-129-222 kubenswrapper[2569]: W0911 01:05:59.597859    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9043f999_c9bf_4793_9159_c1153ee53518.slice/crio-0d5598a51848b6495cabe138bfe166e54943eab698fd8b169b703aa9d08ce342 WatchSource:0}: Error finding container 0d5598a51848b6495cabe138bfe166e54943eab698fd8b169b703aa9d08ce342: Status 404 returned error can't find the container with id 0d5598a51848b6495cabe138bfe166e54943eab698fd8b169b703aa9d08ce342
Sep 11 01:06:00.175044 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:06:00.175004    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-kj6m4" event={"ID":"9043f999-c9bf-4793-9159-c1153ee53518","Type":"ContainerStarted","Data":"0d5598a51848b6495cabe138bfe166e54943eab698fd8b169b703aa9d08ce342"}
Sep 11 01:06:00.176496 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:06:00.176465    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-kmx5x" event={"ID":"257b2c19-d395-40eb-8e08-8ba9560fd070","Type":"ContainerStarted","Data":"93bcefdee68647442b7596688b404a6cced66d3c6aebec69713d2e3f819a3a40"}
Sep 11 01:06:01.185137 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:06:01.185101    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-kmx5x" event={"ID":"257b2c19-d395-40eb-8e08-8ba9560fd070","Type":"ContainerStarted","Data":"89d7cb8f80efd91494b938761eb1db409a9da300af8c64608fe9ccdfc4ec7ee5"}
Sep 11 01:06:01.185513 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:06:01.185228    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-kmx5x"
Sep 11 01:06:01.203393 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:06:01.203333    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-84994f7ffc-kmx5x" podStartSLOduration=2.063331217 podStartE2EDuration="3.203314502s" totalImagesPullingTime="1.139983285s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 01:05:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 01:06:01.203115601 +0000 UTC m=+495.854489853" watchObservedRunningTime="2026-09-11 01:06:01.203314502 +0000 UTC m=+495.854688749"
Sep 11 01:06:01.757136 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:06:01.757094    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-kj6m4"]
Sep 11 01:06:02.191326 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:06:02.191285    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-kj6m4" event={"ID":"9043f999-c9bf-4793-9159-c1153ee53518","Type":"ContainerStarted","Data":"7a05c8c63c4738a1c76f8a81f3679beadbebc66148e432ff6334b84101595b56"}
Sep 11 01:06:02.208231 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:06:02.208188    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7bcfd8f9cd-kj6m4" podStartSLOduration=2.193881739 podStartE2EDuration="4.208173339s" totalImagesPullingTime="2.0142916s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 01:05:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 01:06:02.206546801 +0000 UTC m=+496.857921053" watchObservedRunningTime="2026-09-11 01:06:02.208173339 +0000 UTC m=+496.859547587"
Sep 11 01:06:03.195858 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:06:03.195814    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7bcfd8f9cd-kj6m4" podUID="9043f999-c9bf-4793-9159-c1153ee53518" containerName="authorino" containerID="cri-o://7a05c8c63c4738a1c76f8a81f3679beadbebc66148e432ff6334b84101595b56" gracePeriod=30
Sep 11 01:06:03.442074 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:06:03.442048    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-kj6m4"
Sep 11 01:06:03.467985 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:06:03.467930    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t7zrn\" (UniqueName: \"kubernetes.io/projected/9043f999-c9bf-4793-9159-c1153ee53518-kube-api-access-t7zrn\") pod \"9043f999-c9bf-4793-9159-c1153ee53518\" (UID: \"9043f999-c9bf-4793-9159-c1153ee53518\") "
Sep 11 01:06:03.469809 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:06:03.469787    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9043f999-c9bf-4793-9159-c1153ee53518-kube-api-access-t7zrn" (OuterVolumeSpecName: "kube-api-access-t7zrn") pod "9043f999-c9bf-4793-9159-c1153ee53518" (UID: "9043f999-c9bf-4793-9159-c1153ee53518"). InnerVolumeSpecName "kube-api-access-t7zrn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 01:06:03.569148 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:06:03.569119    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t7zrn\" (UniqueName: \"kubernetes.io/projected/9043f999-c9bf-4793-9159-c1153ee53518-kube-api-access-t7zrn\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:06:04.202158 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:06:04.202075    2569 generic.go:356] "Generic (PLEG): container finished" podID="9043f999-c9bf-4793-9159-c1153ee53518" containerID="7a05c8c63c4738a1c76f8a81f3679beadbebc66148e432ff6334b84101595b56" exitCode=0
Sep 11 01:06:04.202158 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:06:04.202119    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-kj6m4"
Sep 11 01:06:04.202597 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:06:04.202166    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-kj6m4" event={"ID":"9043f999-c9bf-4793-9159-c1153ee53518","Type":"ContainerDied","Data":"7a05c8c63c4738a1c76f8a81f3679beadbebc66148e432ff6334b84101595b56"}
Sep 11 01:06:04.202597 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:06:04.202203    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-kj6m4" event={"ID":"9043f999-c9bf-4793-9159-c1153ee53518","Type":"ContainerDied","Data":"0d5598a51848b6495cabe138bfe166e54943eab698fd8b169b703aa9d08ce342"}
Sep 11 01:06:04.202597 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:06:04.202219    2569 scope.go:117] "RemoveContainer" containerID="7a05c8c63c4738a1c76f8a81f3679beadbebc66148e432ff6334b84101595b56"
Sep 11 01:06:04.210287 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:06:04.210271    2569 scope.go:117] "RemoveContainer" containerID="7a05c8c63c4738a1c76f8a81f3679beadbebc66148e432ff6334b84101595b56"
Sep 11 01:06:04.210547 ip-10-0-129-222 kubenswrapper[2569]: E0911 01:06:04.210528    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a05c8c63c4738a1c76f8a81f3679beadbebc66148e432ff6334b84101595b56\": container with ID starting with 7a05c8c63c4738a1c76f8a81f3679beadbebc66148e432ff6334b84101595b56 not found: ID does not exist" containerID="7a05c8c63c4738a1c76f8a81f3679beadbebc66148e432ff6334b84101595b56"
Sep 11 01:06:04.210608 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:06:04.210553    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a05c8c63c4738a1c76f8a81f3679beadbebc66148e432ff6334b84101595b56"} err="failed to get container status \"7a05c8c63c4738a1c76f8a81f3679beadbebc66148e432ff6334b84101595b56\": rpc error: code = NotFound desc = could not find container \"7a05c8c63c4738a1c76f8a81f3679beadbebc66148e432ff6334b84101595b56\": container with ID starting with 7a05c8c63c4738a1c76f8a81f3679beadbebc66148e432ff6334b84101595b56 not found: ID does not exist"
Sep 11 01:06:04.218015 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:06:04.217996    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-kj6m4"]
Sep 11 01:06:04.221482 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:06:04.221463    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-kj6m4"]
Sep 11 01:06:05.870645 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:06:05.870613    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9043f999-c9bf-4793-9159-c1153ee53518" path="/var/lib/kubelet/pods/9043f999-c9bf-4793-9159-c1153ee53518/volumes"
Sep 11 01:06:12.192436 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:06:12.192404    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-kmx5x"
Sep 11 01:06:27.405551 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:06:27.405518    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-557f876f69-9gj4m"]
Sep 11 01:06:27.405957 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:06:27.405943    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9043f999-c9bf-4793-9159-c1153ee53518" containerName="authorino"
Sep 11 01:06:27.406008 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:06:27.405959    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="9043f999-c9bf-4793-9159-c1153ee53518" containerName="authorino"
Sep 11 01:06:27.406042 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:06:27.406018    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="9043f999-c9bf-4793-9159-c1153ee53518" containerName="authorino"
Sep 11 01:06:27.415246 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:06:27.415219    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-557f876f69-9gj4m"]
Sep 11 01:06:27.415400 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:06:27.415299    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-9gj4m"
Sep 11 01:06:27.417492 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:06:27.417464    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-bjhvj\""
Sep 11 01:06:27.465250 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:06:27.465226    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shp9t\" (UniqueName: \"kubernetes.io/projected/80a947be-9ed5-4d99-9f08-213053b52248-kube-api-access-shp9t\") pod \"authorino-557f876f69-9gj4m\" (UID: \"80a947be-9ed5-4d99-9f08-213053b52248\") " pod="kuadrant-system/authorino-557f876f69-9gj4m"
Sep 11 01:06:27.474185 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:06:27.474157    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-9gj4m"]
Sep 11 01:06:27.474397 ip-10-0-129-222 kubenswrapper[2569]: E0911 01:06:27.474378    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-shp9t], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-557f876f69-9gj4m" podUID="80a947be-9ed5-4d99-9f08-213053b52248"
Sep 11 01:06:27.500406 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:06:27.500379    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-pjzfw"]
Sep 11 01:06:27.504004 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:06:27.503986    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-pjzfw"
Sep 11 01:06:27.506245 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:06:27.506220    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Sep 11 01:06:27.510725 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:06:27.510711    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-pjzfw"]
Sep 11 01:06:27.565776 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:06:27.565747    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmc9f\" (UniqueName: \"kubernetes.io/projected/16bfbead-3564-4cc9-9588-acaf626aa2e5-kube-api-access-fmc9f\") pod \"authorino-6d4dfc649f-pjzfw\" (UID: \"16bfbead-3564-4cc9-9588-acaf626aa2e5\") " pod="kuadrant-system/authorino-6d4dfc649f-pjzfw"
Sep 11 01:06:27.565909 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:06:27.565793    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/16bfbead-3564-4cc9-9588-acaf626aa2e5-tls-cert\") pod \"authorino-6d4dfc649f-pjzfw\" (UID: \"16bfbead-3564-4cc9-9588-acaf626aa2e5\") " pod="kuadrant-system/authorino-6d4dfc649f-pjzfw"
Sep 11 01:06:27.565909 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:06:27.565819    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shp9t\" (UniqueName: \"kubernetes.io/projected/80a947be-9ed5-4d99-9f08-213053b52248-kube-api-access-shp9t\") pod \"authorino-557f876f69-9gj4m\" (UID: \"80a947be-9ed5-4d99-9f08-213053b52248\") " pod="kuadrant-system/authorino-557f876f69-9gj4m"
Sep 11 01:06:27.573430 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:06:27.573408    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-shp9t\" (UniqueName: \"kubernetes.io/projected/80a947be-9ed5-4d99-9f08-213053b52248-kube-api-access-shp9t\") pod \"authorino-557f876f69-9gj4m\" (UID: \"80a947be-9ed5-4d99-9f08-213053b52248\") " pod="kuadrant-system/authorino-557f876f69-9gj4m"
Sep 11 01:06:27.646041 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:06:27.646014    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-pjzfw"]
Sep 11 01:06:27.646273 ip-10-0-129-222 kubenswrapper[2569]: E0911 01:06:27.646254    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-fmc9f tls-cert], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-6d4dfc649f-pjzfw" podUID="16bfbead-3564-4cc9-9588-acaf626aa2e5"
Sep 11 01:06:27.666353 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:06:27.666299    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/16bfbead-3564-4cc9-9588-acaf626aa2e5-tls-cert\") pod \"authorino-6d4dfc649f-pjzfw\" (UID: \"16bfbead-3564-4cc9-9588-acaf626aa2e5\") " pod="kuadrant-system/authorino-6d4dfc649f-pjzfw"
Sep 11 01:06:27.666443 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:06:27.666407    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmc9f\" (UniqueName: \"kubernetes.io/projected/16bfbead-3564-4cc9-9588-acaf626aa2e5-kube-api-access-fmc9f\") pod \"authorino-6d4dfc649f-pjzfw\" (UID: \"16bfbead-3564-4cc9-9588-acaf626aa2e5\") " pod="kuadrant-system/authorino-6d4dfc649f-pjzfw"
Sep 11 01:06:27.667831 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:06:27.667802    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6b968d659-fgnjd"]
Sep 11 01:06:27.668702 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:06:27.668674    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/16bfbead-3564-4cc9-9588-acaf626aa2e5-tls-cert\") pod \"authorino-6d4dfc649f-pjzfw\" (UID: \"16bfbead-3564-4cc9-9588-acaf626aa2e5\") " pod="kuadrant-system/authorino-6d4dfc649f-pjzfw"
Sep 11 01:06:27.671479 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:06:27.671463    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6b968d659-fgnjd"
Sep 11 01:06:27.676486 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:06:27.676464    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmc9f\" (UniqueName: \"kubernetes.io/projected/16bfbead-3564-4cc9-9588-acaf626aa2e5-kube-api-access-fmc9f\") pod \"authorino-6d4dfc649f-pjzfw\" (UID: \"16bfbead-3564-4cc9-9588-acaf626aa2e5\") " pod="kuadrant-system/authorino-6d4dfc649f-pjzfw"
Sep 11 01:06:27.677666 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:06:27.677647    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6b968d659-fgnjd"]
Sep 11 01:06:27.767394 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:06:27.767362    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ncnl\" (UniqueName: \"kubernetes.io/projected/54855bd3-d255-4bd1-af5f-7bf4c244119d-kube-api-access-5ncnl\") pod \"authorino-6b968d659-fgnjd\" (UID: \"54855bd3-d255-4bd1-af5f-7bf4c244119d\") " pod="kuadrant-system/authorino-6b968d659-fgnjd"
Sep 11 01:06:27.767521 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:06:27.767422    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/54855bd3-d255-4bd1-af5f-7bf4c244119d-tls-cert\") pod \"authorino-6b968d659-fgnjd\" (UID: \"54855bd3-d255-4bd1-af5f-7bf4c244119d\") " pod="kuadrant-system/authorino-6b968d659-fgnjd"
Sep 11 01:06:27.868284 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:06:27.868252    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ncnl\" (UniqueName: \"kubernetes.io/projected/54855bd3-d255-4bd1-af5f-7bf4c244119d-kube-api-access-5ncnl\") pod \"authorino-6b968d659-fgnjd\" (UID: \"54855bd3-d255-4bd1-af5f-7bf4c244119d\") " pod="kuadrant-system/authorino-6b968d659-fgnjd"
Sep 11 01:06:27.868410 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:06:27.868350    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/54855bd3-d255-4bd1-af5f-7bf4c244119d-tls-cert\") pod \"authorino-6b968d659-fgnjd\" (UID: \"54855bd3-d255-4bd1-af5f-7bf4c244119d\") " pod="kuadrant-system/authorino-6b968d659-fgnjd"
Sep 11 01:06:27.870934 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:06:27.870911    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/54855bd3-d255-4bd1-af5f-7bf4c244119d-tls-cert\") pod \"authorino-6b968d659-fgnjd\" (UID: \"54855bd3-d255-4bd1-af5f-7bf4c244119d\") " pod="kuadrant-system/authorino-6b968d659-fgnjd"
Sep 11 01:06:27.878690 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:06:27.878669    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ncnl\" (UniqueName: \"kubernetes.io/projected/54855bd3-d255-4bd1-af5f-7bf4c244119d-kube-api-access-5ncnl\") pod \"authorino-6b968d659-fgnjd\" (UID: \"54855bd3-d255-4bd1-af5f-7bf4c244119d\") " pod="kuadrant-system/authorino-6b968d659-fgnjd"
Sep 11 01:06:27.993800 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:06:27.993741    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6b968d659-fgnjd"
Sep 11 01:06:28.262779 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:06:28.262693    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6b968d659-fgnjd"]
Sep 11 01:06:28.265795 ip-10-0-129-222 kubenswrapper[2569]: W0911 01:06:28.265766    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod54855bd3_d255_4bd1_af5f_7bf4c244119d.slice/crio-1d11296778252dc112e3f015cc1c93c3c752b17171d4e47243bf3e57a0396763 WatchSource:0}: Error finding container 1d11296778252dc112e3f015cc1c93c3c752b17171d4e47243bf3e57a0396763: Status 404 returned error can't find the container with id 1d11296778252dc112e3f015cc1c93c3c752b17171d4e47243bf3e57a0396763
Sep 11 01:06:28.315153 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:06:28.315125    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6b968d659-fgnjd" event={"ID":"54855bd3-d255-4bd1-af5f-7bf4c244119d","Type":"ContainerStarted","Data":"1d11296778252dc112e3f015cc1c93c3c752b17171d4e47243bf3e57a0396763"}
Sep 11 01:06:28.315275 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:06:28.315157    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-pjzfw"
Sep 11 01:06:28.315275 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:06:28.315166    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-9gj4m"
Sep 11 01:06:28.323652 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:06:28.323637    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-pjzfw"
Sep 11 01:06:28.325833 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:06:28.325819    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-9gj4m"
Sep 11 01:06:28.372747 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:06:28.372722    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-shp9t\" (UniqueName: \"kubernetes.io/projected/80a947be-9ed5-4d99-9f08-213053b52248-kube-api-access-shp9t\") pod \"80a947be-9ed5-4d99-9f08-213053b52248\" (UID: \"80a947be-9ed5-4d99-9f08-213053b52248\") "
Sep 11 01:06:28.372747 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:06:28.372749    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/16bfbead-3564-4cc9-9588-acaf626aa2e5-tls-cert\") pod \"16bfbead-3564-4cc9-9588-acaf626aa2e5\" (UID: \"16bfbead-3564-4cc9-9588-acaf626aa2e5\") "
Sep 11 01:06:28.372931 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:06:28.372784    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fmc9f\" (UniqueName: \"kubernetes.io/projected/16bfbead-3564-4cc9-9588-acaf626aa2e5-kube-api-access-fmc9f\") pod \"16bfbead-3564-4cc9-9588-acaf626aa2e5\" (UID: \"16bfbead-3564-4cc9-9588-acaf626aa2e5\") "
Sep 11 01:06:28.374844 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:06:28.374815    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/16bfbead-3564-4cc9-9588-acaf626aa2e5-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "16bfbead-3564-4cc9-9588-acaf626aa2e5" (UID: "16bfbead-3564-4cc9-9588-acaf626aa2e5"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 01:06:28.374984 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:06:28.374857    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80a947be-9ed5-4d99-9f08-213053b52248-kube-api-access-shp9t" (OuterVolumeSpecName: "kube-api-access-shp9t") pod "80a947be-9ed5-4d99-9f08-213053b52248" (UID: "80a947be-9ed5-4d99-9f08-213053b52248"). InnerVolumeSpecName "kube-api-access-shp9t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 01:06:28.374984 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:06:28.374887    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16bfbead-3564-4cc9-9588-acaf626aa2e5-kube-api-access-fmc9f" (OuterVolumeSpecName: "kube-api-access-fmc9f") pod "16bfbead-3564-4cc9-9588-acaf626aa2e5" (UID: "16bfbead-3564-4cc9-9588-acaf626aa2e5"). InnerVolumeSpecName "kube-api-access-fmc9f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 01:06:28.473830 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:06:28.473806    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-shp9t\" (UniqueName: \"kubernetes.io/projected/80a947be-9ed5-4d99-9f08-213053b52248-kube-api-access-shp9t\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:06:28.473830 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:06:28.473827    2569 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/16bfbead-3564-4cc9-9588-acaf626aa2e5-tls-cert\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:06:28.474168 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:06:28.473836    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fmc9f\" (UniqueName: \"kubernetes.io/projected/16bfbead-3564-4cc9-9588-acaf626aa2e5-kube-api-access-fmc9f\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:06:29.322611 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:06:29.322577    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6b968d659-fgnjd" event={"ID":"54855bd3-d255-4bd1-af5f-7bf4c244119d","Type":"ContainerStarted","Data":"d4ccdaf11919fb1daa35a265e85f64ac75de70dc617dd1a49a2d2a74cc7e67c8"}
Sep 11 01:06:29.322611 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:06:29.322612    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-9gj4m"
Sep 11 01:06:29.322902 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:06:29.322644    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-pjzfw"
Sep 11 01:06:29.340643 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:06:29.340594    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-6b968d659-fgnjd" podStartSLOduration=1.958508881 podStartE2EDuration="2.340574929s" totalImagesPullingTime="382.066048ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 01:06:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 01:06:29.337924701 +0000 UTC m=+523.989298954" watchObservedRunningTime="2026-09-11 01:06:29.340574929 +0000 UTC m=+523.991949179"
Sep 11 01:06:29.378462 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:06:29.378428    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-9gj4m"]
Sep 11 01:06:29.386919 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:06:29.386893    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-557f876f69-9gj4m"]
Sep 11 01:06:29.411708 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:06:29.411676    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-pjzfw"]
Sep 11 01:06:29.413852 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:06:29.413831    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-pjzfw"]
Sep 11 01:06:29.870499 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:06:29.870471    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="16bfbead-3564-4cc9-9588-acaf626aa2e5" path="/var/lib/kubelet/pods/16bfbead-3564-4cc9-9588-acaf626aa2e5/volumes"
Sep 11 01:06:29.870855 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:06:29.870702    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="80a947be-9ed5-4d99-9f08-213053b52248" path="/var/lib/kubelet/pods/80a947be-9ed5-4d99-9f08-213053b52248/volumes"
Sep 11 01:07:10.146121 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:07:10.146086    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kmx5x"]
Sep 11 01:07:19.567348 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:07:19.567318    2569 generic.go:356] "Generic (PLEG): container finished" podID="b8f2fea7-8e9c-45dd-9291-6e261d2e69db" containerID="0e276f0c3fa3e504065a18c7e2e3e0c6aa4874ab469d09f70e7a692b774bcfbf" exitCode=0
Sep 11 01:07:19.567741 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:07:19.567388    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2t8jt" event={"ID":"b8f2fea7-8e9c-45dd-9291-6e261d2e69db","Type":"ContainerDied","Data":"0e276f0c3fa3e504065a18c7e2e3e0c6aa4874ab469d09f70e7a692b774bcfbf"}
Sep 11 01:07:19.567741 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:07:19.567732    2569 scope.go:117] "RemoveContainer" containerID="0e276f0c3fa3e504065a18c7e2e3e0c6aa4874ab469d09f70e7a692b774bcfbf"
Sep 11 01:07:19.568727 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:07:19.568698    2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 01:07:19.568795 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:07:19.568777    2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 01:07:19.568830 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:07:19.568807    2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 01:07:20.574447 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:07:20.574410    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2t8jt" event={"ID":"b8f2fea7-8e9c-45dd-9291-6e261d2e69db","Type":"ContainerStarted","Data":"fe0f2e8a655d82641ec78d7f39e76bf270acc5c28aa593b4568c5aec06edd660"}
Sep 11 01:07:20.738442 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:07:20.738400    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2t8jt"
Sep 11 01:07:20.738685 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:07:20.738653    2569 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-2t8jt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused" start-of-body=
Sep 11 01:07:20.738759 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:07:20.738716    2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2t8jt" podUID="b8f2fea7-8e9c-45dd-9291-6e261d2e69db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused"
Sep 11 01:07:21.738687 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:07:21.738645    2569 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-2t8jt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused" start-of-body=
Sep 11 01:07:21.739195 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:07:21.738723    2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2t8jt" podUID="b8f2fea7-8e9c-45dd-9291-6e261d2e69db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused"
Sep 11 01:07:22.738183 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:07:22.738137    2569 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-2t8jt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused" start-of-body=
Sep 11 01:07:22.738183 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:07:22.738206    2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2t8jt" podUID="b8f2fea7-8e9c-45dd-9291-6e261d2e69db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused"
Sep 11 01:07:23.738441 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:07:23.738400    2569 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-2t8jt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused" start-of-body=
Sep 11 01:07:23.738958 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:07:23.738475    2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2t8jt" podUID="b8f2fea7-8e9c-45dd-9291-6e261d2e69db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused"
Sep 11 01:07:24.739043 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:07:24.739000    2569 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-2t8jt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused" start-of-body=
Sep 11 01:07:24.739528 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:07:24.739070    2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2t8jt" podUID="b8f2fea7-8e9c-45dd-9291-6e261d2e69db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused"
Sep 11 01:07:25.500635 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:07:25.500594    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kmx5x"]
Sep 11 01:07:25.955777 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:07:25.955744    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2t8jt"
Sep 11 01:07:25.956243 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:07:25.956097    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2t8jt"
Sep 11 01:07:25.956949 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:07:25.956935    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2t8jt"
Sep 11 01:07:38.793866 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:07:38.793822    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kmx5x"]
Sep 11 01:07:44.592758 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:07:44.592724    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-ld7t4"]
Sep 11 01:07:44.596166 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:07:44.596148    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-ld7t4"
Sep 11 01:07:44.599416 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:07:44.599390    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-lg79n\""
Sep 11 01:07:44.599527 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:07:44.599436    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\""
Sep 11 01:07:44.599527 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:07:44.599444    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 11 01:07:44.599527 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:07:44.599479    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 11 01:07:44.605308 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:07:44.605288    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-ld7t4"]
Sep 11 01:07:44.739374 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:07:44.739326    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a7f9b621-8d87-49d1-a254-c7c8a7238f8c-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-ld7t4\" (UID: \"a7f9b621-8d87-49d1-a254-c7c8a7238f8c\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-ld7t4"
Sep 11 01:07:44.739533 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:07:44.739390    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a7f9b621-8d87-49d1-a254-c7c8a7238f8c-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-ld7t4\" (UID: \"a7f9b621-8d87-49d1-a254-c7c8a7238f8c\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-ld7t4"
Sep 11 01:07:44.739533 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:07:44.739422    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a7f9b621-8d87-49d1-a254-c7c8a7238f8c-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-ld7t4\" (UID: \"a7f9b621-8d87-49d1-a254-c7c8a7238f8c\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-ld7t4"
Sep 11 01:07:44.739533 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:07:44.739451    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a7f9b621-8d87-49d1-a254-c7c8a7238f8c-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-ld7t4\" (UID: \"a7f9b621-8d87-49d1-a254-c7c8a7238f8c\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-ld7t4"
Sep 11 01:07:44.739637 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:07:44.739534    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a7f9b621-8d87-49d1-a254-c7c8a7238f8c-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-ld7t4\" (UID: \"a7f9b621-8d87-49d1-a254-c7c8a7238f8c\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-ld7t4"
Sep 11 01:07:44.739637 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:07:44.739572    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j69kt\" (UniqueName: \"kubernetes.io/projected/a7f9b621-8d87-49d1-a254-c7c8a7238f8c-kube-api-access-j69kt\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-ld7t4\" (UID: \"a7f9b621-8d87-49d1-a254-c7c8a7238f8c\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-ld7t4"
Sep 11 01:07:44.840187 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:07:44.840154    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a7f9b621-8d87-49d1-a254-c7c8a7238f8c-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-ld7t4\" (UID: \"a7f9b621-8d87-49d1-a254-c7c8a7238f8c\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-ld7t4"
Sep 11 01:07:44.840187 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:07:44.840187    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a7f9b621-8d87-49d1-a254-c7c8a7238f8c-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-ld7t4\" (UID: \"a7f9b621-8d87-49d1-a254-c7c8a7238f8c\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-ld7t4"
Sep 11 01:07:44.840421 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:07:44.840216    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a7f9b621-8d87-49d1-a254-c7c8a7238f8c-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-ld7t4\" (UID: \"a7f9b621-8d87-49d1-a254-c7c8a7238f8c\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-ld7t4"
Sep 11 01:07:44.840421 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:07:44.840294    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a7f9b621-8d87-49d1-a254-c7c8a7238f8c-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-ld7t4\" (UID: \"a7f9b621-8d87-49d1-a254-c7c8a7238f8c\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-ld7t4"
Sep 11 01:07:44.840421 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:07:44.840347    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j69kt\" (UniqueName: \"kubernetes.io/projected/a7f9b621-8d87-49d1-a254-c7c8a7238f8c-kube-api-access-j69kt\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-ld7t4\" (UID: \"a7f9b621-8d87-49d1-a254-c7c8a7238f8c\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-ld7t4"
Sep 11 01:07:44.840421 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:07:44.840382    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a7f9b621-8d87-49d1-a254-c7c8a7238f8c-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-ld7t4\" (UID: \"a7f9b621-8d87-49d1-a254-c7c8a7238f8c\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-ld7t4"
Sep 11 01:07:44.840649 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:07:44.840625    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a7f9b621-8d87-49d1-a254-c7c8a7238f8c-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-ld7t4\" (UID: \"a7f9b621-8d87-49d1-a254-c7c8a7238f8c\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-ld7t4"
Sep 11 01:07:44.840699 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:07:44.840664    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a7f9b621-8d87-49d1-a254-c7c8a7238f8c-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-ld7t4\" (UID: \"a7f9b621-8d87-49d1-a254-c7c8a7238f8c\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-ld7t4"
Sep 11 01:07:44.840735 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:07:44.840707    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a7f9b621-8d87-49d1-a254-c7c8a7238f8c-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-ld7t4\" (UID: \"a7f9b621-8d87-49d1-a254-c7c8a7238f8c\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-ld7t4"
Sep 11 01:07:44.842657 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:07:44.842632    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a7f9b621-8d87-49d1-a254-c7c8a7238f8c-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-ld7t4\" (UID: \"a7f9b621-8d87-49d1-a254-c7c8a7238f8c\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-ld7t4"
Sep 11 01:07:44.842891 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:07:44.842829    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a7f9b621-8d87-49d1-a254-c7c8a7238f8c-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-ld7t4\" (UID: \"a7f9b621-8d87-49d1-a254-c7c8a7238f8c\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-ld7t4"
Sep 11 01:07:44.847369 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:07:44.847351    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j69kt\" (UniqueName: \"kubernetes.io/projected/a7f9b621-8d87-49d1-a254-c7c8a7238f8c-kube-api-access-j69kt\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-ld7t4\" (UID: \"a7f9b621-8d87-49d1-a254-c7c8a7238f8c\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-ld7t4"
Sep 11 01:07:44.906776 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:07:44.906748    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-ld7t4"
Sep 11 01:07:45.174166 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:07:45.174143    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-ld7t4"]
Sep 11 01:07:45.176788 ip-10-0-129-222 kubenswrapper[2569]: W0911 01:07:45.176753    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7f9b621_8d87_49d1_a254_c7c8a7238f8c.slice/crio-005d8ec91cb7cafb9c016ee88239ea30388a407616174b37fd13cd9f5a33c36b WatchSource:0}: Error finding container 005d8ec91cb7cafb9c016ee88239ea30388a407616174b37fd13cd9f5a33c36b: Status 404 returned error can't find the container with id 005d8ec91cb7cafb9c016ee88239ea30388a407616174b37fd13cd9f5a33c36b
Sep 11 01:07:45.497176 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:07:45.497092    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kmx5x"]
Sep 11 01:07:45.699051 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:07:45.699016    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-ld7t4" event={"ID":"a7f9b621-8d87-49d1-a254-c7c8a7238f8c","Type":"ContainerStarted","Data":"005d8ec91cb7cafb9c016ee88239ea30388a407616174b37fd13cd9f5a33c36b"}
Sep 11 01:07:46.706130 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:07:46.706090    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-ld7t4" event={"ID":"a7f9b621-8d87-49d1-a254-c7c8a7238f8c","Type":"ContainerStarted","Data":"2201ee5db5e28637201f38ab464f588deecf944afd321f27e896605d1928a6f7"}
Sep 11 01:07:46.723180 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:07:46.723130    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-ld7t4" podStartSLOduration=1.643478497 podStartE2EDuration="2.723115284s" totalImagesPullingTime="1.079636787s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 01:07:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 01:07:46.721381915 +0000 UTC m=+601.372756166" watchObservedRunningTime="2026-09-11 01:07:46.723115284 +0000 UTC m=+601.374489524"
Sep 11 01:07:54.907357 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:07:54.907319    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-ld7t4"
Sep 11 01:07:54.907806 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:07:54.907380    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-ld7t4"
Sep 11 01:07:54.920799 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:07:54.920769    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-ld7t4"
Sep 11 01:07:55.758540 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:07:55.758512    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-ld7t4"
Sep 11 01:07:58.106718 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:07:58.106680    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kmx5x"]
Sep 11 01:08:16.795889 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:08:16.795846    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kmx5x"]
Sep 11 01:08:35.505220 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:08:35.505186    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-pjbd2"]
Sep 11 01:08:35.508925 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:08:35.508908    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-pjbd2"
Sep 11 01:08:35.512018 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:08:35.511998    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Sep 11 01:08:35.518142 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:08:35.518121    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-pjbd2"]
Sep 11 01:08:35.665726 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:08:35.665693    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b0a0cbc-ae41-4060-8fc9-3b07ada1dd1f-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-pjbd2\" (UID: \"5b0a0cbc-ae41-4060-8fc9-3b07ada1dd1f\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-pjbd2"
Sep 11 01:08:35.665914 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:08:35.665768    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b0a0cbc-ae41-4060-8fc9-3b07ada1dd1f-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-pjbd2\" (UID: \"5b0a0cbc-ae41-4060-8fc9-3b07ada1dd1f\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-pjbd2"
Sep 11 01:08:35.665914 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:08:35.665791    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b0a0cbc-ae41-4060-8fc9-3b07ada1dd1f-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-pjbd2\" (UID: \"5b0a0cbc-ae41-4060-8fc9-3b07ada1dd1f\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-pjbd2"
Sep 11 01:08:35.665997 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:08:35.665915    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b0a0cbc-ae41-4060-8fc9-3b07ada1dd1f-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-pjbd2\" (UID: \"5b0a0cbc-ae41-4060-8fc9-3b07ada1dd1f\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-pjbd2"
Sep 11 01:08:35.665997 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:08:35.665951    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b0a0cbc-ae41-4060-8fc9-3b07ada1dd1f-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-pjbd2\" (UID: \"5b0a0cbc-ae41-4060-8fc9-3b07ada1dd1f\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-pjbd2"
Sep 11 01:08:35.665997 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:08:35.665971    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7dw2v\" (UniqueName: \"kubernetes.io/projected/5b0a0cbc-ae41-4060-8fc9-3b07ada1dd1f-kube-api-access-7dw2v\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-pjbd2\" (UID: \"5b0a0cbc-ae41-4060-8fc9-3b07ada1dd1f\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-pjbd2"
Sep 11 01:08:35.766768 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:08:35.766688    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b0a0cbc-ae41-4060-8fc9-3b07ada1dd1f-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-pjbd2\" (UID: \"5b0a0cbc-ae41-4060-8fc9-3b07ada1dd1f\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-pjbd2"
Sep 11 01:08:35.766768 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:08:35.766718    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b0a0cbc-ae41-4060-8fc9-3b07ada1dd1f-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-pjbd2\" (UID: \"5b0a0cbc-ae41-4060-8fc9-3b07ada1dd1f\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-pjbd2"
Sep 11 01:08:35.766768 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:08:35.766756    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b0a0cbc-ae41-4060-8fc9-3b07ada1dd1f-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-pjbd2\" (UID: \"5b0a0cbc-ae41-4060-8fc9-3b07ada1dd1f\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-pjbd2"
Sep 11 01:08:35.767023 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:08:35.766785    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b0a0cbc-ae41-4060-8fc9-3b07ada1dd1f-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-pjbd2\" (UID: \"5b0a0cbc-ae41-4060-8fc9-3b07ada1dd1f\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-pjbd2"
Sep 11 01:08:35.767023 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:08:35.766810    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7dw2v\" (UniqueName: \"kubernetes.io/projected/5b0a0cbc-ae41-4060-8fc9-3b07ada1dd1f-kube-api-access-7dw2v\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-pjbd2\" (UID: \"5b0a0cbc-ae41-4060-8fc9-3b07ada1dd1f\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-pjbd2"
Sep 11 01:08:35.767023 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:08:35.766986    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b0a0cbc-ae41-4060-8fc9-3b07ada1dd1f-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-pjbd2\" (UID: \"5b0a0cbc-ae41-4060-8fc9-3b07ada1dd1f\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-pjbd2"
Sep 11 01:08:35.767264 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:08:35.767245    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b0a0cbc-ae41-4060-8fc9-3b07ada1dd1f-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-pjbd2\" (UID: \"5b0a0cbc-ae41-4060-8fc9-3b07ada1dd1f\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-pjbd2"
Sep 11 01:08:35.767331 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:08:35.767256    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b0a0cbc-ae41-4060-8fc9-3b07ada1dd1f-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-pjbd2\" (UID: \"5b0a0cbc-ae41-4060-8fc9-3b07ada1dd1f\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-pjbd2"
Sep 11 01:08:35.767397 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:08:35.767376    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b0a0cbc-ae41-4060-8fc9-3b07ada1dd1f-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-pjbd2\" (UID: \"5b0a0cbc-ae41-4060-8fc9-3b07ada1dd1f\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-pjbd2"
Sep 11 01:08:35.769080 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:08:35.769054    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b0a0cbc-ae41-4060-8fc9-3b07ada1dd1f-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-pjbd2\" (UID: \"5b0a0cbc-ae41-4060-8fc9-3b07ada1dd1f\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-pjbd2"
Sep 11 01:08:35.769252 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:08:35.769236    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b0a0cbc-ae41-4060-8fc9-3b07ada1dd1f-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-pjbd2\" (UID: \"5b0a0cbc-ae41-4060-8fc9-3b07ada1dd1f\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-pjbd2"
Sep 11 01:08:35.773758 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:08:35.773730    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7dw2v\" (UniqueName: \"kubernetes.io/projected/5b0a0cbc-ae41-4060-8fc9-3b07ada1dd1f-kube-api-access-7dw2v\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-pjbd2\" (UID: \"5b0a0cbc-ae41-4060-8fc9-3b07ada1dd1f\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-pjbd2"
Sep 11 01:08:35.819260 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:08:35.819237    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-pjbd2"
Sep 11 01:08:36.087867 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:08:36.087837    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-pjbd2"]
Sep 11 01:08:36.090357 ip-10-0-129-222 kubenswrapper[2569]: W0911 01:08:36.090330    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b0a0cbc_ae41_4060_8fc9_3b07ada1dd1f.slice/crio-0ca83cbe4672fa74a51de893a9a413fc193b8b9356b71f344b99d7e2f82dcf06 WatchSource:0}: Error finding container 0ca83cbe4672fa74a51de893a9a413fc193b8b9356b71f344b99d7e2f82dcf06: Status 404 returned error can't find the container with id 0ca83cbe4672fa74a51de893a9a413fc193b8b9356b71f344b99d7e2f82dcf06
Sep 11 01:08:36.091656 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:08:36.091637    2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 01:08:36.948720 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:08:36.948679    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-pjbd2" event={"ID":"5b0a0cbc-ae41-4060-8fc9-3b07ada1dd1f","Type":"ContainerStarted","Data":"e983ec353b7ad59a3dcc57d9f39f474412009cf1b5cabe1235302376e99aa667"}
Sep 11 01:08:36.949158 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:08:36.948771    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-pjbd2" event={"ID":"5b0a0cbc-ae41-4060-8fc9-3b07ada1dd1f","Type":"ContainerStarted","Data":"0ca83cbe4672fa74a51de893a9a413fc193b8b9356b71f344b99d7e2f82dcf06"}
Sep 11 01:08:36.968664 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:08:36.968614    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-pjbd2" podStartSLOduration=1.754139667 podStartE2EDuration="1.968599181s" totalImagesPullingTime="214.459514ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 01:08:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 01:08:36.966724127 +0000 UTC m=+651.618098376" watchObservedRunningTime="2026-09-11 01:08:36.968599181 +0000 UTC m=+651.619973430"
Sep 11 01:08:37.507138 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:08:37.507094    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kmx5x"]
Sep 11 01:08:45.819808 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:08:45.819773    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-pjbd2"
Sep 11 01:08:45.820195 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:08:45.819864    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-pjbd2"
Sep 11 01:08:45.832282 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:08:45.832260    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-pjbd2"
Sep 11 01:08:46.007635 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:08:46.007608    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-pjbd2"
Sep 11 01:08:49.299390 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:08:49.299359    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-f8s9l"]
Sep 11 01:08:49.303023 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:08:49.303000    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-f8s9l"
Sep 11 01:08:49.305379 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:08:49.305356    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Sep 11 01:08:49.311534 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:08:49.311513    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-f8s9l"]
Sep 11 01:08:49.386686 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:08:49.386657    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b2a1717-67cc-41f4-b396-836fdf0e8036-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-f8s9l\" (UID: \"4b2a1717-67cc-41f4-b396-836fdf0e8036\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-f8s9l"
Sep 11 01:08:49.386807 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:08:49.386688    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b2a1717-67cc-41f4-b396-836fdf0e8036-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-f8s9l\" (UID: \"4b2a1717-67cc-41f4-b396-836fdf0e8036\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-f8s9l"
Sep 11 01:08:49.386807 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:08:49.386709    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7lwm\" (UniqueName: \"kubernetes.io/projected/4b2a1717-67cc-41f4-b396-836fdf0e8036-kube-api-access-b7lwm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-f8s9l\" (UID: \"4b2a1717-67cc-41f4-b396-836fdf0e8036\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-f8s9l"
Sep 11 01:08:49.386807 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:08:49.386740    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4b2a1717-67cc-41f4-b396-836fdf0e8036-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-f8s9l\" (UID: \"4b2a1717-67cc-41f4-b396-836fdf0e8036\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-f8s9l"
Sep 11 01:08:49.386807 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:08:49.386786    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b2a1717-67cc-41f4-b396-836fdf0e8036-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-f8s9l\" (UID: \"4b2a1717-67cc-41f4-b396-836fdf0e8036\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-f8s9l"
Sep 11 01:08:49.386807 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:08:49.386803    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b2a1717-67cc-41f4-b396-836fdf0e8036-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-f8s9l\" (UID: \"4b2a1717-67cc-41f4-b396-836fdf0e8036\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-f8s9l"
Sep 11 01:08:49.487620 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:08:49.487591    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b2a1717-67cc-41f4-b396-836fdf0e8036-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-f8s9l\" (UID: \"4b2a1717-67cc-41f4-b396-836fdf0e8036\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-f8s9l"
Sep 11 01:08:49.487749 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:08:49.487626    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b2a1717-67cc-41f4-b396-836fdf0e8036-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-f8s9l\" (UID: \"4b2a1717-67cc-41f4-b396-836fdf0e8036\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-f8s9l"
Sep 11 01:08:49.487749 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:08:49.487676    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b2a1717-67cc-41f4-b396-836fdf0e8036-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-f8s9l\" (UID: \"4b2a1717-67cc-41f4-b396-836fdf0e8036\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-f8s9l"
Sep 11 01:08:49.487749 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:08:49.487692    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b2a1717-67cc-41f4-b396-836fdf0e8036-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-f8s9l\" (UID: \"4b2a1717-67cc-41f4-b396-836fdf0e8036\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-f8s9l"
Sep 11 01:08:49.487749 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:08:49.487710    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7lwm\" (UniqueName: \"kubernetes.io/projected/4b2a1717-67cc-41f4-b396-836fdf0e8036-kube-api-access-b7lwm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-f8s9l\" (UID: \"4b2a1717-67cc-41f4-b396-836fdf0e8036\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-f8s9l"
Sep 11 01:08:49.487749 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:08:49.487732    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4b2a1717-67cc-41f4-b396-836fdf0e8036-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-f8s9l\" (UID: \"4b2a1717-67cc-41f4-b396-836fdf0e8036\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-f8s9l"
Sep 11 01:08:49.488171 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:08:49.488149    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4b2a1717-67cc-41f4-b396-836fdf0e8036-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-f8s9l\" (UID: \"4b2a1717-67cc-41f4-b396-836fdf0e8036\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-f8s9l"
Sep 11 01:08:49.488238 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:08:49.488172    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b2a1717-67cc-41f4-b396-836fdf0e8036-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-f8s9l\" (UID: \"4b2a1717-67cc-41f4-b396-836fdf0e8036\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-f8s9l"
Sep 11 01:08:49.488238 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:08:49.488224    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b2a1717-67cc-41f4-b396-836fdf0e8036-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-f8s9l\" (UID: \"4b2a1717-67cc-41f4-b396-836fdf0e8036\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-f8s9l"
Sep 11 01:08:49.489993 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:08:49.489969    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b2a1717-67cc-41f4-b396-836fdf0e8036-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-f8s9l\" (UID: \"4b2a1717-67cc-41f4-b396-836fdf0e8036\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-f8s9l"
Sep 11 01:08:49.490127 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:08:49.490109    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b2a1717-67cc-41f4-b396-836fdf0e8036-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-f8s9l\" (UID: \"4b2a1717-67cc-41f4-b396-836fdf0e8036\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-f8s9l"
Sep 11 01:08:49.496300 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:08:49.496278    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7lwm\" (UniqueName: \"kubernetes.io/projected/4b2a1717-67cc-41f4-b396-836fdf0e8036-kube-api-access-b7lwm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-f8s9l\" (UID: \"4b2a1717-67cc-41f4-b396-836fdf0e8036\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-f8s9l"
Sep 11 01:08:49.613544 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:08:49.613514    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-f8s9l"
Sep 11 01:08:49.893074 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:08:49.893048    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-f8s9l"]
Sep 11 01:08:49.893847 ip-10-0-129-222 kubenswrapper[2569]: W0911 01:08:49.893823    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4b2a1717_67cc_41f4_b396_836fdf0e8036.slice/crio-9ef97db81816f09764e5763205ebce9b20cc415b3b63451c9f29f3b0ece4a866 WatchSource:0}: Error finding container 9ef97db81816f09764e5763205ebce9b20cc415b3b63451c9f29f3b0ece4a866: Status 404 returned error can't find the container with id 9ef97db81816f09764e5763205ebce9b20cc415b3b63451c9f29f3b0ece4a866
Sep 11 01:08:50.014665 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:08:50.014633    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-f8s9l" event={"ID":"4b2a1717-67cc-41f4-b396-836fdf0e8036","Type":"ContainerStarted","Data":"9ef97db81816f09764e5763205ebce9b20cc415b3b63451c9f29f3b0ece4a866"}
Sep 11 01:08:51.020793 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:08:51.020752    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-f8s9l" event={"ID":"4b2a1717-67cc-41f4-b396-836fdf0e8036","Type":"ContainerStarted","Data":"a0fb495c0ed2d7ddee1ac3cdc0fb950f609a7ba9e77de356abb0f939b6b0846c"}
Sep 11 01:08:51.039227 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:08:51.039178    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-f8s9l" podStartSLOduration=1.852135643 podStartE2EDuration="2.039162522s" totalImagesPullingTime="187.026879ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 01:08:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 01:08:51.03823373 +0000 UTC m=+665.689607990" watchObservedRunningTime="2026-09-11 01:08:51.039162522 +0000 UTC m=+665.690536771"
Sep 11 01:08:51.704810 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:08:51.704777    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kmx5x"]
Sep 11 01:08:59.614579 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:08:59.614543    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-f8s9l"
Sep 11 01:08:59.615086 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:08:59.614592    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-f8s9l"
Sep 11 01:08:59.626572 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:08:59.626544    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-f8s9l"
Sep 11 01:09:00.075831 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:09:00.075805    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-f8s9l"
Sep 11 01:09:38.799068 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:09:38.798987    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6b968d659-fgnjd"]
Sep 11 01:09:38.799530 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:09:38.799173    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-6b968d659-fgnjd" podUID="54855bd3-d255-4bd1-af5f-7bf4c244119d" containerName="authorino" containerID="cri-o://d4ccdaf11919fb1daa35a265e85f64ac75de70dc617dd1a49a2d2a74cc7e67c8" gracePeriod=30
Sep 11 01:09:39.042519 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:09:39.042495    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6b968d659-fgnjd"
Sep 11 01:09:39.206427 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:09:39.206396    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5ncnl\" (UniqueName: \"kubernetes.io/projected/54855bd3-d255-4bd1-af5f-7bf4c244119d-kube-api-access-5ncnl\") pod \"54855bd3-d255-4bd1-af5f-7bf4c244119d\" (UID: \"54855bd3-d255-4bd1-af5f-7bf4c244119d\") "
Sep 11 01:09:39.206588 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:09:39.206444    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/54855bd3-d255-4bd1-af5f-7bf4c244119d-tls-cert\") pod \"54855bd3-d255-4bd1-af5f-7bf4c244119d\" (UID: \"54855bd3-d255-4bd1-af5f-7bf4c244119d\") "
Sep 11 01:09:39.208442 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:09:39.208413    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54855bd3-d255-4bd1-af5f-7bf4c244119d-kube-api-access-5ncnl" (OuterVolumeSpecName: "kube-api-access-5ncnl") pod "54855bd3-d255-4bd1-af5f-7bf4c244119d" (UID: "54855bd3-d255-4bd1-af5f-7bf4c244119d"). InnerVolumeSpecName "kube-api-access-5ncnl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 01:09:39.216550 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:09:39.216527    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54855bd3-d255-4bd1-af5f-7bf4c244119d-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "54855bd3-d255-4bd1-af5f-7bf4c244119d" (UID: "54855bd3-d255-4bd1-af5f-7bf4c244119d"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 01:09:39.262588 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:09:39.262563    2569 generic.go:356] "Generic (PLEG): container finished" podID="54855bd3-d255-4bd1-af5f-7bf4c244119d" containerID="d4ccdaf11919fb1daa35a265e85f64ac75de70dc617dd1a49a2d2a74cc7e67c8" exitCode=0
Sep 11 01:09:39.262687 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:09:39.262605    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6b968d659-fgnjd"
Sep 11 01:09:39.262687 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:09:39.262622    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6b968d659-fgnjd" event={"ID":"54855bd3-d255-4bd1-af5f-7bf4c244119d","Type":"ContainerDied","Data":"d4ccdaf11919fb1daa35a265e85f64ac75de70dc617dd1a49a2d2a74cc7e67c8"}
Sep 11 01:09:39.262687 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:09:39.262654    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6b968d659-fgnjd" event={"ID":"54855bd3-d255-4bd1-af5f-7bf4c244119d","Type":"ContainerDied","Data":"1d11296778252dc112e3f015cc1c93c3c752b17171d4e47243bf3e57a0396763"}
Sep 11 01:09:39.262687 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:09:39.262670    2569 scope.go:117] "RemoveContainer" containerID="d4ccdaf11919fb1daa35a265e85f64ac75de70dc617dd1a49a2d2a74cc7e67c8"
Sep 11 01:09:39.271792 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:09:39.271773    2569 scope.go:117] "RemoveContainer" containerID="d4ccdaf11919fb1daa35a265e85f64ac75de70dc617dd1a49a2d2a74cc7e67c8"
Sep 11 01:09:39.272120 ip-10-0-129-222 kubenswrapper[2569]: E0911 01:09:39.272101    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4ccdaf11919fb1daa35a265e85f64ac75de70dc617dd1a49a2d2a74cc7e67c8\": container with ID starting with d4ccdaf11919fb1daa35a265e85f64ac75de70dc617dd1a49a2d2a74cc7e67c8 not found: ID does not exist" containerID="d4ccdaf11919fb1daa35a265e85f64ac75de70dc617dd1a49a2d2a74cc7e67c8"
Sep 11 01:09:39.272184 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:09:39.272126    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4ccdaf11919fb1daa35a265e85f64ac75de70dc617dd1a49a2d2a74cc7e67c8"} err="failed to get container status \"d4ccdaf11919fb1daa35a265e85f64ac75de70dc617dd1a49a2d2a74cc7e67c8\": rpc error: code = NotFound desc = could not find container \"d4ccdaf11919fb1daa35a265e85f64ac75de70dc617dd1a49a2d2a74cc7e67c8\": container with ID starting with d4ccdaf11919fb1daa35a265e85f64ac75de70dc617dd1a49a2d2a74cc7e67c8 not found: ID does not exist"
Sep 11 01:09:39.298467 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:09:39.298437    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6b968d659-fgnjd"]
Sep 11 01:09:39.307324 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:09:39.307297    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5ncnl\" (UniqueName: \"kubernetes.io/projected/54855bd3-d255-4bd1-af5f-7bf4c244119d-kube-api-access-5ncnl\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:09:39.307324 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:09:39.307321    2569 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/54855bd3-d255-4bd1-af5f-7bf4c244119d-tls-cert\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:09:39.309135 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:09:39.309116    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6b968d659-fgnjd"]
Sep 11 01:09:39.870847 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:09:39.870814    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="54855bd3-d255-4bd1-af5f-7bf4c244119d" path="/var/lib/kubelet/pods/54855bd3-d255-4bd1-af5f-7bf4c244119d/volumes"
Sep 11 01:09:57.133848 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:09:57.133815    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-6ba087-openshift-default-85464b8f68-b5knz"]
Sep 11 01:09:57.134465 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:09:57.134444    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="54855bd3-d255-4bd1-af5f-7bf4c244119d" containerName="authorino"
Sep 11 01:09:57.134547 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:09:57.134468    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="54855bd3-d255-4bd1-af5f-7bf4c244119d" containerName="authorino"
Sep 11 01:09:57.134611 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:09:57.134593    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="54855bd3-d255-4bd1-af5f-7bf4c244119d" containerName="authorino"
Sep 11 01:09:57.138948 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:09:57.138929    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-6ba087-openshift-default-85464b8f68-b5knz"
Sep 11 01:09:57.141500 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:09:57.141480    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-6ba087-openshift-default-dockercfg-zkf5n\""
Sep 11 01:09:57.151893 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:09:57.151846    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-6ba087-openshift-default-85464b8f68-b5knz"]
Sep 11 01:09:57.262566 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:09:57.262536    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e147aad2-95bd-4135-82aa-7f015f6996a4-istio-podinfo\") pod \"e2e-worker-w0-6ba087-openshift-default-85464b8f68-b5knz\" (UID: \"e147aad2-95bd-4135-82aa-7f015f6996a4\") " pod="openshift-ingress/e2e-worker-w0-6ba087-openshift-default-85464b8f68-b5knz"
Sep 11 01:09:57.262746 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:09:57.262583    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e147aad2-95bd-4135-82aa-7f015f6996a4-istio-token\") pod \"e2e-worker-w0-6ba087-openshift-default-85464b8f68-b5knz\" (UID: \"e147aad2-95bd-4135-82aa-7f015f6996a4\") " pod="openshift-ingress/e2e-worker-w0-6ba087-openshift-default-85464b8f68-b5knz"
Sep 11 01:09:57.262746 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:09:57.262637    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e147aad2-95bd-4135-82aa-7f015f6996a4-workload-socket\") pod \"e2e-worker-w0-6ba087-openshift-default-85464b8f68-b5knz\" (UID: \"e147aad2-95bd-4135-82aa-7f015f6996a4\") " pod="openshift-ingress/e2e-worker-w0-6ba087-openshift-default-85464b8f68-b5knz"
Sep 11 01:09:57.262746 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:09:57.262669    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e147aad2-95bd-4135-82aa-7f015f6996a4-istio-envoy\") pod \"e2e-worker-w0-6ba087-openshift-default-85464b8f68-b5knz\" (UID: \"e147aad2-95bd-4135-82aa-7f015f6996a4\") " pod="openshift-ingress/e2e-worker-w0-6ba087-openshift-default-85464b8f68-b5knz"
Sep 11 01:09:57.262746 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:09:57.262702    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e147aad2-95bd-4135-82aa-7f015f6996a4-credential-socket\") pod \"e2e-worker-w0-6ba087-openshift-default-85464b8f68-b5knz\" (UID: \"e147aad2-95bd-4135-82aa-7f015f6996a4\") " pod="openshift-ingress/e2e-worker-w0-6ba087-openshift-default-85464b8f68-b5knz"
Sep 11 01:09:57.262982 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:09:57.262781    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e147aad2-95bd-4135-82aa-7f015f6996a4-istiod-ca-cert\") pod \"e2e-worker-w0-6ba087-openshift-default-85464b8f68-b5knz\" (UID: \"e147aad2-95bd-4135-82aa-7f015f6996a4\") " pod="openshift-ingress/e2e-worker-w0-6ba087-openshift-default-85464b8f68-b5knz"
Sep 11 01:09:57.262982 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:09:57.262825    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e147aad2-95bd-4135-82aa-7f015f6996a4-workload-certs\") pod \"e2e-worker-w0-6ba087-openshift-default-85464b8f68-b5knz\" (UID: \"e147aad2-95bd-4135-82aa-7f015f6996a4\") " pod="openshift-ingress/e2e-worker-w0-6ba087-openshift-default-85464b8f68-b5knz"
Sep 11 01:09:57.262982 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:09:57.262865    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e147aad2-95bd-4135-82aa-7f015f6996a4-istio-data\") pod \"e2e-worker-w0-6ba087-openshift-default-85464b8f68-b5knz\" (UID: \"e147aad2-95bd-4135-82aa-7f015f6996a4\") " pod="openshift-ingress/e2e-worker-w0-6ba087-openshift-default-85464b8f68-b5knz"
Sep 11 01:09:57.262982 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:09:57.262904    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqvjl\" (UniqueName: \"kubernetes.io/projected/e147aad2-95bd-4135-82aa-7f015f6996a4-kube-api-access-kqvjl\") pod \"e2e-worker-w0-6ba087-openshift-default-85464b8f68-b5knz\" (UID: \"e147aad2-95bd-4135-82aa-7f015f6996a4\") " pod="openshift-ingress/e2e-worker-w0-6ba087-openshift-default-85464b8f68-b5knz"
Sep 11 01:09:57.364150 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:09:57.364118    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e147aad2-95bd-4135-82aa-7f015f6996a4-workload-socket\") pod \"e2e-worker-w0-6ba087-openshift-default-85464b8f68-b5knz\" (UID: \"e147aad2-95bd-4135-82aa-7f015f6996a4\") " pod="openshift-ingress/e2e-worker-w0-6ba087-openshift-default-85464b8f68-b5knz"
Sep 11 01:09:57.364150 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:09:57.364157    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e147aad2-95bd-4135-82aa-7f015f6996a4-istio-envoy\") pod \"e2e-worker-w0-6ba087-openshift-default-85464b8f68-b5knz\" (UID: \"e147aad2-95bd-4135-82aa-7f015f6996a4\") " pod="openshift-ingress/e2e-worker-w0-6ba087-openshift-default-85464b8f68-b5knz"
Sep 11 01:09:57.364364 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:09:57.364183    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e147aad2-95bd-4135-82aa-7f015f6996a4-credential-socket\") pod \"e2e-worker-w0-6ba087-openshift-default-85464b8f68-b5knz\" (UID: \"e147aad2-95bd-4135-82aa-7f015f6996a4\") " pod="openshift-ingress/e2e-worker-w0-6ba087-openshift-default-85464b8f68-b5knz"
Sep 11 01:09:57.364364 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:09:57.364265    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e147aad2-95bd-4135-82aa-7f015f6996a4-istiod-ca-cert\") pod \"e2e-worker-w0-6ba087-openshift-default-85464b8f68-b5knz\" (UID: \"e147aad2-95bd-4135-82aa-7f015f6996a4\") " pod="openshift-ingress/e2e-worker-w0-6ba087-openshift-default-85464b8f68-b5knz"
Sep 11 01:09:57.364364 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:09:57.364303    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e147aad2-95bd-4135-82aa-7f015f6996a4-workload-certs\") pod \"e2e-worker-w0-6ba087-openshift-default-85464b8f68-b5knz\" (UID: \"e147aad2-95bd-4135-82aa-7f015f6996a4\") " pod="openshift-ingress/e2e-worker-w0-6ba087-openshift-default-85464b8f68-b5knz"
Sep 11 01:09:57.364364 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:09:57.364324    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e147aad2-95bd-4135-82aa-7f015f6996a4-istio-data\") pod \"e2e-worker-w0-6ba087-openshift-default-85464b8f68-b5knz\" (UID: \"e147aad2-95bd-4135-82aa-7f015f6996a4\") " pod="openshift-ingress/e2e-worker-w0-6ba087-openshift-default-85464b8f68-b5knz"
Sep 11 01:09:57.364364 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:09:57.364348    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqvjl\" (UniqueName: \"kubernetes.io/projected/e147aad2-95bd-4135-82aa-7f015f6996a4-kube-api-access-kqvjl\") pod \"e2e-worker-w0-6ba087-openshift-default-85464b8f68-b5knz\" (UID: \"e147aad2-95bd-4135-82aa-7f015f6996a4\") " pod="openshift-ingress/e2e-worker-w0-6ba087-openshift-default-85464b8f68-b5knz"
Sep 11 01:09:57.364684 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:09:57.364484    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e147aad2-95bd-4135-82aa-7f015f6996a4-istio-podinfo\") pod \"e2e-worker-w0-6ba087-openshift-default-85464b8f68-b5knz\" (UID: \"e147aad2-95bd-4135-82aa-7f015f6996a4\") " pod="openshift-ingress/e2e-worker-w0-6ba087-openshift-default-85464b8f68-b5knz"
Sep 11 01:09:57.364684 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:09:57.364562    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e147aad2-95bd-4135-82aa-7f015f6996a4-istio-token\") pod \"e2e-worker-w0-6ba087-openshift-default-85464b8f68-b5knz\" (UID: \"e147aad2-95bd-4135-82aa-7f015f6996a4\") " pod="openshift-ingress/e2e-worker-w0-6ba087-openshift-default-85464b8f68-b5knz"
Sep 11 01:09:57.364684 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:09:57.364575    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e147aad2-95bd-4135-82aa-7f015f6996a4-workload-socket\") pod \"e2e-worker-w0-6ba087-openshift-default-85464b8f68-b5knz\" (UID: \"e147aad2-95bd-4135-82aa-7f015f6996a4\") " pod="openshift-ingress/e2e-worker-w0-6ba087-openshift-default-85464b8f68-b5knz"
Sep 11 01:09:57.364845 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:09:57.364729    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e147aad2-95bd-4135-82aa-7f015f6996a4-istio-data\") pod \"e2e-worker-w0-6ba087-openshift-default-85464b8f68-b5knz\" (UID: \"e147aad2-95bd-4135-82aa-7f015f6996a4\") " pod="openshift-ingress/e2e-worker-w0-6ba087-openshift-default-85464b8f68-b5knz"
Sep 11 01:09:57.364845 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:09:57.364748    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e147aad2-95bd-4135-82aa-7f015f6996a4-workload-certs\") pod \"e2e-worker-w0-6ba087-openshift-default-85464b8f68-b5knz\" (UID: \"e147aad2-95bd-4135-82aa-7f015f6996a4\") " pod="openshift-ingress/e2e-worker-w0-6ba087-openshift-default-85464b8f68-b5knz"
Sep 11 01:09:57.364995 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:09:57.364847    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e147aad2-95bd-4135-82aa-7f015f6996a4-credential-socket\") pod \"e2e-worker-w0-6ba087-openshift-default-85464b8f68-b5knz\" (UID: \"e147aad2-95bd-4135-82aa-7f015f6996a4\") " pod="openshift-ingress/e2e-worker-w0-6ba087-openshift-default-85464b8f68-b5knz"
Sep 11 01:09:57.365078 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:09:57.365058    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e147aad2-95bd-4135-82aa-7f015f6996a4-istiod-ca-cert\") pod \"e2e-worker-w0-6ba087-openshift-default-85464b8f68-b5knz\" (UID: \"e147aad2-95bd-4135-82aa-7f015f6996a4\") " pod="openshift-ingress/e2e-worker-w0-6ba087-openshift-default-85464b8f68-b5knz"
Sep 11 01:09:57.366619 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:09:57.366598    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e147aad2-95bd-4135-82aa-7f015f6996a4-istio-envoy\") pod \"e2e-worker-w0-6ba087-openshift-default-85464b8f68-b5knz\" (UID: \"e147aad2-95bd-4135-82aa-7f015f6996a4\") " pod="openshift-ingress/e2e-worker-w0-6ba087-openshift-default-85464b8f68-b5knz"
Sep 11 01:09:57.367019 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:09:57.366999    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e147aad2-95bd-4135-82aa-7f015f6996a4-istio-podinfo\") pod \"e2e-worker-w0-6ba087-openshift-default-85464b8f68-b5knz\" (UID: \"e147aad2-95bd-4135-82aa-7f015f6996a4\") " pod="openshift-ingress/e2e-worker-w0-6ba087-openshift-default-85464b8f68-b5knz"
Sep 11 01:09:57.376445 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:09:57.376420    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e147aad2-95bd-4135-82aa-7f015f6996a4-istio-token\") pod \"e2e-worker-w0-6ba087-openshift-default-85464b8f68-b5knz\" (UID: \"e147aad2-95bd-4135-82aa-7f015f6996a4\") " pod="openshift-ingress/e2e-worker-w0-6ba087-openshift-default-85464b8f68-b5knz"
Sep 11 01:09:57.376572 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:09:57.376554    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqvjl\" (UniqueName: \"kubernetes.io/projected/e147aad2-95bd-4135-82aa-7f015f6996a4-kube-api-access-kqvjl\") pod \"e2e-worker-w0-6ba087-openshift-default-85464b8f68-b5knz\" (UID: \"e147aad2-95bd-4135-82aa-7f015f6996a4\") " pod="openshift-ingress/e2e-worker-w0-6ba087-openshift-default-85464b8f68-b5knz"
Sep 11 01:09:57.448863 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:09:57.448796    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-6ba087-openshift-default-85464b8f68-b5knz"
Sep 11 01:09:57.723494 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:09:57.723447    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-6ba087-openshift-default-85464b8f68-b5knz"]
Sep 11 01:09:57.726063 ip-10-0-129-222 kubenswrapper[2569]: W0911 01:09:57.726036    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode147aad2_95bd_4135_82aa_7f015f6996a4.slice/crio-af43b9ffa6f26f9491670781c27c3782857131dcfaff922f9e5fd009662a8f61 WatchSource:0}: Error finding container af43b9ffa6f26f9491670781c27c3782857131dcfaff922f9e5fd009662a8f61: Status 404 returned error can't find the container with id af43b9ffa6f26f9491670781c27c3782857131dcfaff922f9e5fd009662a8f61
Sep 11 01:09:57.728290 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:09:57.728258    2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 01:09:57.728349 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:09:57.728322    2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 01:09:57.728396 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:09:57.728354    2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 01:09:58.352822 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:09:58.352784    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-6ba087-openshift-default-85464b8f68-b5knz" event={"ID":"e147aad2-95bd-4135-82aa-7f015f6996a4","Type":"ContainerStarted","Data":"7dbec7d9e178cfcf55fd46d3858b4e3f1c87306bc6870f7adb05665f9802cd83"}
Sep 11 01:09:58.353262 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:09:58.352839    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-6ba087-openshift-default-85464b8f68-b5knz" event={"ID":"e147aad2-95bd-4135-82aa-7f015f6996a4","Type":"ContainerStarted","Data":"af43b9ffa6f26f9491670781c27c3782857131dcfaff922f9e5fd009662a8f61"}
Sep 11 01:09:58.381511 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:09:58.381450    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-6ba087-openshift-default-85464b8f68-b5knz" podStartSLOduration=1.381430345 podStartE2EDuration="1.381430345s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 01:09:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 01:09:58.380253138 +0000 UTC m=+733.031627398" watchObservedRunningTime="2026-09-11 01:09:58.381430345 +0000 UTC m=+733.032804595"
Sep 11 01:09:58.449124 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:09:58.449099    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-6ba087-openshift-default-85464b8f68-b5knz"
Sep 11 01:09:58.454188 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:09:58.454166    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-6ba087-openshift-default-85464b8f68-b5knz"
Sep 11 01:09:58.749070 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:09:58.748992    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7700-6723a0"]
Sep 11 01:09:58.752602 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:09:58.752579    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7700-6723a0"
Sep 11 01:09:58.754910 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:09:58.754889    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-d5ldp\""
Sep 11 01:09:58.762578 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:09:58.762553    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7700-6723a0"]
Sep 11 01:09:58.879199 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:09:58.879172    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjnfs\" (UniqueName: \"kubernetes.io/projected/b40c7038-d286-4735-83c2-c456d251d732-kube-api-access-xjnfs\") pod \"test-curl-7700-6723a0\" (UID: \"b40c7038-d286-4735-83c2-c456d251d732\") " pod="opendatahub/test-curl-7700-6723a0"
Sep 11 01:09:58.980153 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:09:58.980117    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjnfs\" (UniqueName: \"kubernetes.io/projected/b40c7038-d286-4735-83c2-c456d251d732-kube-api-access-xjnfs\") pod \"test-curl-7700-6723a0\" (UID: \"b40c7038-d286-4735-83c2-c456d251d732\") " pod="opendatahub/test-curl-7700-6723a0"
Sep 11 01:09:58.988454 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:09:58.988425    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjnfs\" (UniqueName: \"kubernetes.io/projected/b40c7038-d286-4735-83c2-c456d251d732-kube-api-access-xjnfs\") pod \"test-curl-7700-6723a0\" (UID: \"b40c7038-d286-4735-83c2-c456d251d732\") " pod="opendatahub/test-curl-7700-6723a0"
Sep 11 01:09:59.062629 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:09:59.062603    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7700-6723a0"
Sep 11 01:09:59.337739 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:09:59.337665    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7700-6723a0"]
Sep 11 01:09:59.341299 ip-10-0-129-222 kubenswrapper[2569]: W0911 01:09:59.341274    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb40c7038_d286_4735_83c2_c456d251d732.slice/crio-73af0e0f60d3b00d01923e9eef417165ae3b20d73c52e2e1614efa9806755d33 WatchSource:0}: Error finding container 73af0e0f60d3b00d01923e9eef417165ae3b20d73c52e2e1614efa9806755d33: Status 404 returned error can't find the container with id 73af0e0f60d3b00d01923e9eef417165ae3b20d73c52e2e1614efa9806755d33
Sep 11 01:09:59.358707 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:09:59.358682    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7700-6723a0" event={"ID":"b40c7038-d286-4735-83c2-c456d251d732","Type":"ContainerStarted","Data":"73af0e0f60d3b00d01923e9eef417165ae3b20d73c52e2e1614efa9806755d33"}
Sep 11 01:09:59.359048 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:09:59.358936    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-6ba087-openshift-default-85464b8f68-b5knz"
Sep 11 01:09:59.359745 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:09:59.359724    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-6ba087-openshift-default-85464b8f68-b5knz"
Sep 11 01:10:00.434931 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:00.434898    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kmx5x"]
Sep 11 01:10:01.372664 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:01.372630    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7700-6723a0" event={"ID":"b40c7038-d286-4735-83c2-c456d251d732","Type":"ContainerStarted","Data":"5b579580a40cf7fb4f5eb976a5591db77ed5a7dd0f50b84ab51ebafebe6c5cbf"}
Sep 11 01:10:01.389501 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:01.389444    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7700-6723a0" podStartSLOduration=1.462909494 podStartE2EDuration="3.389422744s" totalImagesPullingTime="1.92651325s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 01:09:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 01:10:01.386064306 +0000 UTC m=+736.037438556" watchObservedRunningTime="2026-09-11 01:10:01.389422744 +0000 UTC m=+736.040796995"
Sep 11 01:10:02.046240 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:02.046206    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7700-6723a0"]
Sep 11 01:10:02.055815 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:02.055788    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7700-6723a0"]
Sep 11 01:10:02.187779 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:02.187690    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7700-ca7d54"]
Sep 11 01:10:02.188200 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:02.188181    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b40c7038-d286-4735-83c2-c456d251d732" containerName="test-curl-7700-6723a0"
Sep 11 01:10:02.188302 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:02.188202    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="b40c7038-d286-4735-83c2-c456d251d732" containerName="test-curl-7700-6723a0"
Sep 11 01:10:02.188302 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:02.188296    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="b40c7038-d286-4735-83c2-c456d251d732" containerName="test-curl-7700-6723a0"
Sep 11 01:10:02.191513 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:02.191491    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7700-ca7d54"
Sep 11 01:10:02.199712 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:02.199683    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7700-ca7d54"]
Sep 11 01:10:02.309348 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:02.309312    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkbdt\" (UniqueName: \"kubernetes.io/projected/3862ebef-43f1-423a-b5d1-d110686731cd-kube-api-access-pkbdt\") pod \"test-curl-7700-ca7d54\" (UID: \"3862ebef-43f1-423a-b5d1-d110686731cd\") " pod="opendatahub/test-curl-7700-ca7d54"
Sep 11 01:10:02.377502 ip-10-0-129-222 kubenswrapper[2569]: E0911 01:10:02.377473    2569 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7700-6723a0" container="test-curl-7700-6723a0"
Sep 11 01:10:02.410789 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:02.410761    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkbdt\" (UniqueName: \"kubernetes.io/projected/3862ebef-43f1-423a-b5d1-d110686731cd-kube-api-access-pkbdt\") pod \"test-curl-7700-ca7d54\" (UID: \"3862ebef-43f1-423a-b5d1-d110686731cd\") " pod="opendatahub/test-curl-7700-ca7d54"
Sep 11 01:10:02.418525 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:02.418492    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkbdt\" (UniqueName: \"kubernetes.io/projected/3862ebef-43f1-423a-b5d1-d110686731cd-kube-api-access-pkbdt\") pod \"test-curl-7700-ca7d54\" (UID: \"3862ebef-43f1-423a-b5d1-d110686731cd\") " pod="opendatahub/test-curl-7700-ca7d54"
Sep 11 01:10:02.507303 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:02.507225    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7700-ca7d54"
Sep 11 01:10:02.777360 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:02.777278    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7700-ca7d54"]
Sep 11 01:10:02.780479 ip-10-0-129-222 kubenswrapper[2569]: W0911 01:10:02.780450    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3862ebef_43f1_423a_b5d1_d110686731cd.slice/crio-ac9ca9bdf73072b8736e3912912ca08cdd1e5446958b69bbb7f0723b3e7912e4 WatchSource:0}: Error finding container ac9ca9bdf73072b8736e3912912ca08cdd1e5446958b69bbb7f0723b3e7912e4: Status 404 returned error can't find the container with id ac9ca9bdf73072b8736e3912912ca08cdd1e5446958b69bbb7f0723b3e7912e4
Sep 11 01:10:03.384432 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:03.384399    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7700-ca7d54" event={"ID":"3862ebef-43f1-423a-b5d1-d110686731cd","Type":"ContainerStarted","Data":"c2ad6f5797420d497363fd7c5acc60fbba372a5cb64b3616d52c0f3091727b23"}
Sep 11 01:10:03.384833 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:03.384441    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7700-ca7d54" event={"ID":"3862ebef-43f1-423a-b5d1-d110686731cd","Type":"ContainerStarted","Data":"ac9ca9bdf73072b8736e3912912ca08cdd1e5446958b69bbb7f0723b3e7912e4"}
Sep 11 01:10:03.384833 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:03.384491    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7700-6723a0" podUID="b40c7038-d286-4735-83c2-c456d251d732" containerName="test-curl-7700-6723a0" containerID="cri-o://5b579580a40cf7fb4f5eb976a5591db77ed5a7dd0f50b84ab51ebafebe6c5cbf" gracePeriod=2
Sep 11 01:10:03.386927 ip-10-0-129-222 kubenswrapper[2569]: E0911 01:10:03.386906    2569 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7700-6723a0\" is forbidden: User \"system:node:ip-10-0-129-222.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-222.ec2.internal' and this object" podUID="b40c7038-d286-4735-83c2-c456d251d732" pod="opendatahub/test-curl-7700-6723a0"
Sep 11 01:10:03.388554 ip-10-0-129-222 kubenswrapper[2569]: E0911 01:10:03.388534    2569 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7700-6723a0\" is forbidden: User \"system:node:ip-10-0-129-222.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-222.ec2.internal' and this object" podUID="b40c7038-d286-4735-83c2-c456d251d732" pod="opendatahub/test-curl-7700-6723a0"
Sep 11 01:10:03.404098 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:03.404044    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7700-ca7d54" podStartSLOduration=1.404027135 podStartE2EDuration="1.404027135s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 01:10:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 01:10:03.403721439 +0000 UTC m=+738.055095690" watchObservedRunningTime="2026-09-11 01:10:03.404027135 +0000 UTC m=+738.055401384"
Sep 11 01:10:04.128219 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:04.128186    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7700-ca7d54"]
Sep 11 01:10:04.134560 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:04.134531    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7700-ca7d54"]
Sep 11 01:10:04.372705 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:04.372672    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-6ba087-bdcc845c4-cqzxs"]
Sep 11 01:10:04.373166 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:04.373145    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3862ebef-43f1-423a-b5d1-d110686731cd" containerName="test-curl-7700-ca7d54"
Sep 11 01:10:04.373166 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:04.373168    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="3862ebef-43f1-423a-b5d1-d110686731cd" containerName="test-curl-7700-ca7d54"
Sep 11 01:10:04.373322 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:04.373276    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="3862ebef-43f1-423a-b5d1-d110686731cd" containerName="test-curl-7700-ca7d54"
Sep 11 01:10:04.376569 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:04.376554    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-6ba087-bdcc845c4-cqzxs"
Sep 11 01:10:04.378924 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:04.378904    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-6ba087\""
Sep 11 01:10:04.379178 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:04.379156    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-6ba087-dockercfg-kxjhm\""
Sep 11 01:10:04.388166 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:04.388145    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-6ba087-bdcc845c4-cqzxs"]
Sep 11 01:10:04.390666 ip-10-0-129-222 kubenswrapper[2569]: E0911 01:10:04.390646    2569 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7700-ca7d54" container="test-curl-7700-ca7d54"
Sep 11 01:10:04.431523 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:04.431476    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7b5c45d-9578-4b0c-836b-e6d9b2d8dac2-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-6ba087-bdcc845c4-cqzxs\" (UID: \"a7b5c45d-9578-4b0c-836b-e6d9b2d8dac2\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-6ba087-bdcc845c4-cqzxs"
Sep 11 01:10:04.431726 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:04.431696    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmkl4\" (UniqueName: \"kubernetes.io/projected/a7b5c45d-9578-4b0c-836b-e6d9b2d8dac2-kube-api-access-gmkl4\") pod \"payload-pre-processing-e2e-worker-w0-6ba087-bdcc845c4-cqzxs\" (UID: \"a7b5c45d-9578-4b0c-836b-e6d9b2d8dac2\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-6ba087-bdcc845c4-cqzxs"
Sep 11 01:10:04.532996 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:04.532961    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7b5c45d-9578-4b0c-836b-e6d9b2d8dac2-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-6ba087-bdcc845c4-cqzxs\" (UID: \"a7b5c45d-9578-4b0c-836b-e6d9b2d8dac2\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-6ba087-bdcc845c4-cqzxs"
Sep 11 01:10:04.533147 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:04.533044    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmkl4\" (UniqueName: \"kubernetes.io/projected/a7b5c45d-9578-4b0c-836b-e6d9b2d8dac2-kube-api-access-gmkl4\") pod \"payload-pre-processing-e2e-worker-w0-6ba087-bdcc845c4-cqzxs\" (UID: \"a7b5c45d-9578-4b0c-836b-e6d9b2d8dac2\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-6ba087-bdcc845c4-cqzxs"
Sep 11 01:10:04.533636 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:04.533617    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7b5c45d-9578-4b0c-836b-e6d9b2d8dac2-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-6ba087-bdcc845c4-cqzxs\" (UID: \"a7b5c45d-9578-4b0c-836b-e6d9b2d8dac2\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-6ba087-bdcc845c4-cqzxs"
Sep 11 01:10:04.542595 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:04.542564    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmkl4\" (UniqueName: \"kubernetes.io/projected/a7b5c45d-9578-4b0c-836b-e6d9b2d8dac2-kube-api-access-gmkl4\") pod \"payload-pre-processing-e2e-worker-w0-6ba087-bdcc845c4-cqzxs\" (UID: \"a7b5c45d-9578-4b0c-836b-e6d9b2d8dac2\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-6ba087-bdcc845c4-cqzxs"
Sep 11 01:10:04.687852 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:04.687778    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-6ba087-bdcc845c4-cqzxs"
Sep 11 01:10:04.968734 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:04.968689    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-6ba087-bdcc845c4-cqzxs"]
Sep 11 01:10:04.971137 ip-10-0-129-222 kubenswrapper[2569]: W0911 01:10:04.971109    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7b5c45d_9578_4b0c_836b_e6d9b2d8dac2.slice/crio-367f743f198cb692ec1fb82d9b18b93285fc6645123ee0b660d18169aa454e2a WatchSource:0}: Error finding container 367f743f198cb692ec1fb82d9b18b93285fc6645123ee0b660d18169aa454e2a: Status 404 returned error can't find the container with id 367f743f198cb692ec1fb82d9b18b93285fc6645123ee0b660d18169aa454e2a
Sep 11 01:10:04.981602 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:04.981580    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-dc7dcd-openshift-default-5958f7974d-457sp"]
Sep 11 01:10:04.986820 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:04.986802    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-dc7dcd-openshift-default-5958f7974d-457sp"
Sep 11 01:10:04.989009 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:04.988989    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-dc7dcd-openshift-default-dockercfg-2dqhx\""
Sep 11 01:10:04.998410 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:04.998388    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-dc7dcd-openshift-default-5958f7974d-457sp"]
Sep 11 01:10:05.038581 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:05.038549    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ec0f171-1bff-4174-9905-fe855ce1ad42-istio-data\") pod \"e2e-autores-dc7dcd-openshift-default-5958f7974d-457sp\" (UID: \"8ec0f171-1bff-4174-9905-fe855ce1ad42\") " pod="openshift-ingress/e2e-autores-dc7dcd-openshift-default-5958f7974d-457sp"
Sep 11 01:10:05.038703 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:05.038587    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ec0f171-1bff-4174-9905-fe855ce1ad42-credential-socket\") pod \"e2e-autores-dc7dcd-openshift-default-5958f7974d-457sp\" (UID: \"8ec0f171-1bff-4174-9905-fe855ce1ad42\") " pod="openshift-ingress/e2e-autores-dc7dcd-openshift-default-5958f7974d-457sp"
Sep 11 01:10:05.038703 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:05.038617    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ec0f171-1bff-4174-9905-fe855ce1ad42-istio-podinfo\") pod \"e2e-autores-dc7dcd-openshift-default-5958f7974d-457sp\" (UID: \"8ec0f171-1bff-4174-9905-fe855ce1ad42\") " pod="openshift-ingress/e2e-autores-dc7dcd-openshift-default-5958f7974d-457sp"
Sep 11 01:10:05.038703 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:05.038685    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ec0f171-1bff-4174-9905-fe855ce1ad42-workload-socket\") pod \"e2e-autores-dc7dcd-openshift-default-5958f7974d-457sp\" (UID: \"8ec0f171-1bff-4174-9905-fe855ce1ad42\") " pod="openshift-ingress/e2e-autores-dc7dcd-openshift-default-5958f7974d-457sp"
Sep 11 01:10:05.038703 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:05.038702    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ec0f171-1bff-4174-9905-fe855ce1ad42-istiod-ca-cert\") pod \"e2e-autores-dc7dcd-openshift-default-5958f7974d-457sp\" (UID: \"8ec0f171-1bff-4174-9905-fe855ce1ad42\") " pod="openshift-ingress/e2e-autores-dc7dcd-openshift-default-5958f7974d-457sp"
Sep 11 01:10:05.038913 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:05.038731    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ec0f171-1bff-4174-9905-fe855ce1ad42-istio-envoy\") pod \"e2e-autores-dc7dcd-openshift-default-5958f7974d-457sp\" (UID: \"8ec0f171-1bff-4174-9905-fe855ce1ad42\") " pod="openshift-ingress/e2e-autores-dc7dcd-openshift-default-5958f7974d-457sp"
Sep 11 01:10:05.038913 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:05.038777    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ec0f171-1bff-4174-9905-fe855ce1ad42-workload-certs\") pod \"e2e-autores-dc7dcd-openshift-default-5958f7974d-457sp\" (UID: \"8ec0f171-1bff-4174-9905-fe855ce1ad42\") " pod="openshift-ingress/e2e-autores-dc7dcd-openshift-default-5958f7974d-457sp"
Sep 11 01:10:05.038913 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:05.038827    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ec0f171-1bff-4174-9905-fe855ce1ad42-istio-token\") pod \"e2e-autores-dc7dcd-openshift-default-5958f7974d-457sp\" (UID: \"8ec0f171-1bff-4174-9905-fe855ce1ad42\") " pod="openshift-ingress/e2e-autores-dc7dcd-openshift-default-5958f7974d-457sp"
Sep 11 01:10:05.039025 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:05.038916    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4pfh\" (UniqueName: \"kubernetes.io/projected/8ec0f171-1bff-4174-9905-fe855ce1ad42-kube-api-access-b4pfh\") pod \"e2e-autores-dc7dcd-openshift-default-5958f7974d-457sp\" (UID: \"8ec0f171-1bff-4174-9905-fe855ce1ad42\") " pod="openshift-ingress/e2e-autores-dc7dcd-openshift-default-5958f7974d-457sp"
Sep 11 01:10:05.139842 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:05.139807    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ec0f171-1bff-4174-9905-fe855ce1ad42-istio-data\") pod \"e2e-autores-dc7dcd-openshift-default-5958f7974d-457sp\" (UID: \"8ec0f171-1bff-4174-9905-fe855ce1ad42\") " pod="openshift-ingress/e2e-autores-dc7dcd-openshift-default-5958f7974d-457sp"
Sep 11 01:10:05.139842 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:05.139845    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ec0f171-1bff-4174-9905-fe855ce1ad42-credential-socket\") pod \"e2e-autores-dc7dcd-openshift-default-5958f7974d-457sp\" (UID: \"8ec0f171-1bff-4174-9905-fe855ce1ad42\") " pod="openshift-ingress/e2e-autores-dc7dcd-openshift-default-5958f7974d-457sp"
Sep 11 01:10:05.140078 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:05.139891    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ec0f171-1bff-4174-9905-fe855ce1ad42-istio-podinfo\") pod \"e2e-autores-dc7dcd-openshift-default-5958f7974d-457sp\" (UID: \"8ec0f171-1bff-4174-9905-fe855ce1ad42\") " pod="openshift-ingress/e2e-autores-dc7dcd-openshift-default-5958f7974d-457sp"
Sep 11 01:10:05.140078 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:05.139924    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ec0f171-1bff-4174-9905-fe855ce1ad42-workload-socket\") pod \"e2e-autores-dc7dcd-openshift-default-5958f7974d-457sp\" (UID: \"8ec0f171-1bff-4174-9905-fe855ce1ad42\") " pod="openshift-ingress/e2e-autores-dc7dcd-openshift-default-5958f7974d-457sp"
Sep 11 01:10:05.140078 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:05.139939    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ec0f171-1bff-4174-9905-fe855ce1ad42-istiod-ca-cert\") pod \"e2e-autores-dc7dcd-openshift-default-5958f7974d-457sp\" (UID: \"8ec0f171-1bff-4174-9905-fe855ce1ad42\") " pod="openshift-ingress/e2e-autores-dc7dcd-openshift-default-5958f7974d-457sp"
Sep 11 01:10:05.140078 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:05.139969    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ec0f171-1bff-4174-9905-fe855ce1ad42-istio-envoy\") pod \"e2e-autores-dc7dcd-openshift-default-5958f7974d-457sp\" (UID: \"8ec0f171-1bff-4174-9905-fe855ce1ad42\") " pod="openshift-ingress/e2e-autores-dc7dcd-openshift-default-5958f7974d-457sp"
Sep 11 01:10:05.140078 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:05.140020    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ec0f171-1bff-4174-9905-fe855ce1ad42-workload-certs\") pod \"e2e-autores-dc7dcd-openshift-default-5958f7974d-457sp\" (UID: \"8ec0f171-1bff-4174-9905-fe855ce1ad42\") " pod="openshift-ingress/e2e-autores-dc7dcd-openshift-default-5958f7974d-457sp"
Sep 11 01:10:05.140078 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:05.140058    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ec0f171-1bff-4174-9905-fe855ce1ad42-istio-token\") pod \"e2e-autores-dc7dcd-openshift-default-5958f7974d-457sp\" (UID: \"8ec0f171-1bff-4174-9905-fe855ce1ad42\") " pod="openshift-ingress/e2e-autores-dc7dcd-openshift-default-5958f7974d-457sp"
Sep 11 01:10:05.140395 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:05.140295    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ec0f171-1bff-4174-9905-fe855ce1ad42-credential-socket\") pod \"e2e-autores-dc7dcd-openshift-default-5958f7974d-457sp\" (UID: \"8ec0f171-1bff-4174-9905-fe855ce1ad42\") " pod="openshift-ingress/e2e-autores-dc7dcd-openshift-default-5958f7974d-457sp"
Sep 11 01:10:05.140395 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:05.140316    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ec0f171-1bff-4174-9905-fe855ce1ad42-workload-socket\") pod \"e2e-autores-dc7dcd-openshift-default-5958f7974d-457sp\" (UID: \"8ec0f171-1bff-4174-9905-fe855ce1ad42\") " pod="openshift-ingress/e2e-autores-dc7dcd-openshift-default-5958f7974d-457sp"
Sep 11 01:10:05.140395 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:05.140337    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ec0f171-1bff-4174-9905-fe855ce1ad42-istio-data\") pod \"e2e-autores-dc7dcd-openshift-default-5958f7974d-457sp\" (UID: \"8ec0f171-1bff-4174-9905-fe855ce1ad42\") " pod="openshift-ingress/e2e-autores-dc7dcd-openshift-default-5958f7974d-457sp"
Sep 11 01:10:05.140395 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:05.140367    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4pfh\" (UniqueName: \"kubernetes.io/projected/8ec0f171-1bff-4174-9905-fe855ce1ad42-kube-api-access-b4pfh\") pod \"e2e-autores-dc7dcd-openshift-default-5958f7974d-457sp\" (UID: \"8ec0f171-1bff-4174-9905-fe855ce1ad42\") " pod="openshift-ingress/e2e-autores-dc7dcd-openshift-default-5958f7974d-457sp"
Sep 11 01:10:05.140612 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:05.140560    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ec0f171-1bff-4174-9905-fe855ce1ad42-workload-certs\") pod \"e2e-autores-dc7dcd-openshift-default-5958f7974d-457sp\" (UID: \"8ec0f171-1bff-4174-9905-fe855ce1ad42\") " pod="openshift-ingress/e2e-autores-dc7dcd-openshift-default-5958f7974d-457sp"
Sep 11 01:10:05.140749 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:05.140732    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ec0f171-1bff-4174-9905-fe855ce1ad42-istiod-ca-cert\") pod \"e2e-autores-dc7dcd-openshift-default-5958f7974d-457sp\" (UID: \"8ec0f171-1bff-4174-9905-fe855ce1ad42\") " pod="openshift-ingress/e2e-autores-dc7dcd-openshift-default-5958f7974d-457sp"
Sep 11 01:10:05.142900 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:05.142860    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ec0f171-1bff-4174-9905-fe855ce1ad42-istio-envoy\") pod \"e2e-autores-dc7dcd-openshift-default-5958f7974d-457sp\" (UID: \"8ec0f171-1bff-4174-9905-fe855ce1ad42\") " pod="openshift-ingress/e2e-autores-dc7dcd-openshift-default-5958f7974d-457sp"
Sep 11 01:10:05.143096 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:05.143078    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ec0f171-1bff-4174-9905-fe855ce1ad42-istio-podinfo\") pod \"e2e-autores-dc7dcd-openshift-default-5958f7974d-457sp\" (UID: \"8ec0f171-1bff-4174-9905-fe855ce1ad42\") " pod="openshift-ingress/e2e-autores-dc7dcd-openshift-default-5958f7974d-457sp"
Sep 11 01:10:05.147710 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:05.147687    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ec0f171-1bff-4174-9905-fe855ce1ad42-istio-token\") pod \"e2e-autores-dc7dcd-openshift-default-5958f7974d-457sp\" (UID: \"8ec0f171-1bff-4174-9905-fe855ce1ad42\") " pod="openshift-ingress/e2e-autores-dc7dcd-openshift-default-5958f7974d-457sp"
Sep 11 01:10:05.148141 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:05.148120    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4pfh\" (UniqueName: \"kubernetes.io/projected/8ec0f171-1bff-4174-9905-fe855ce1ad42-kube-api-access-b4pfh\") pod \"e2e-autores-dc7dcd-openshift-default-5958f7974d-457sp\" (UID: \"8ec0f171-1bff-4174-9905-fe855ce1ad42\") " pod="openshift-ingress/e2e-autores-dc7dcd-openshift-default-5958f7974d-457sp"
Sep 11 01:10:05.298757 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:05.298677    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-dc7dcd-openshift-default-5958f7974d-457sp"
Sep 11 01:10:05.396163 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:05.396137    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-6ba087-bdcc845c4-cqzxs" event={"ID":"a7b5c45d-9578-4b0c-836b-e6d9b2d8dac2","Type":"ContainerStarted","Data":"367f743f198cb692ec1fb82d9b18b93285fc6645123ee0b660d18169aa454e2a"}
Sep 11 01:10:05.396485 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:05.396248    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7700-ca7d54" podUID="3862ebef-43f1-423a-b5d1-d110686731cd" containerName="test-curl-7700-ca7d54" containerID="cri-o://c2ad6f5797420d497363fd7c5acc60fbba372a5cb64b3616d52c0f3091727b23" gracePeriod=2
Sep 11 01:10:05.398415 ip-10-0-129-222 kubenswrapper[2569]: E0911 01:10:05.398384    2569 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7700-ca7d54\" is forbidden: User \"system:node:ip-10-0-129-222.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-222.ec2.internal' and this object" podUID="3862ebef-43f1-423a-b5d1-d110686731cd" pod="opendatahub/test-curl-7700-ca7d54"
Sep 11 01:10:05.570557 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:05.570535    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7700-6723a0"
Sep 11 01:10:05.573005 ip-10-0-129-222 kubenswrapper[2569]: E0911 01:10:05.572961    2569 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7700-ca7d54\" is forbidden: User \"system:node:ip-10-0-129-222.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-222.ec2.internal' and this object" podUID="3862ebef-43f1-423a-b5d1-d110686731cd" pod="opendatahub/test-curl-7700-ca7d54"
Sep 11 01:10:05.574591 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:05.574571    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-dc7dcd-openshift-default-5958f7974d-457sp"]
Sep 11 01:10:05.576656 ip-10-0-129-222 kubenswrapper[2569]: W0911 01:10:05.576624    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ec0f171_1bff_4174_9905_fe855ce1ad42.slice/crio-554b6711b5b0118399de69a8ffea15ea105d29185066540267766d9a13f9b48b WatchSource:0}: Error finding container 554b6711b5b0118399de69a8ffea15ea105d29185066540267766d9a13f9b48b: Status 404 returned error can't find the container with id 554b6711b5b0118399de69a8ffea15ea105d29185066540267766d9a13f9b48b
Sep 11 01:10:05.578791 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:05.578760    2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 01:10:05.578856 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:05.578820    2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 01:10:05.578856 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:05.578847    2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 01:10:05.644778 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:05.644748    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xjnfs\" (UniqueName: \"kubernetes.io/projected/b40c7038-d286-4735-83c2-c456d251d732-kube-api-access-xjnfs\") pod \"b40c7038-d286-4735-83c2-c456d251d732\" (UID: \"b40c7038-d286-4735-83c2-c456d251d732\") "
Sep 11 01:10:05.647303 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:05.647275    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b40c7038-d286-4735-83c2-c456d251d732-kube-api-access-xjnfs" (OuterVolumeSpecName: "kube-api-access-xjnfs") pod "b40c7038-d286-4735-83c2-c456d251d732" (UID: "b40c7038-d286-4735-83c2-c456d251d732"). InnerVolumeSpecName "kube-api-access-xjnfs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 01:10:05.746686 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:05.746649    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xjnfs\" (UniqueName: \"kubernetes.io/projected/b40c7038-d286-4735-83c2-c456d251d732-kube-api-access-xjnfs\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:10:05.870998 ip-10-0-129-222 kubenswrapper[2569]: E0911 01:10:05.870910    2569 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7700-ca7d54\" is forbidden: User \"system:node:ip-10-0-129-222.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-222.ec2.internal' and this object" podUID="3862ebef-43f1-423a-b5d1-d110686731cd" pod="opendatahub/test-curl-7700-ca7d54"
Sep 11 01:10:05.874172 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:05.874139    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b40c7038-d286-4735-83c2-c456d251d732" path="/var/lib/kubelet/pods/b40c7038-d286-4735-83c2-c456d251d732/volumes"
Sep 11 01:10:06.403686 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:06.403647    2569 generic.go:356] "Generic (PLEG): container finished" podID="b40c7038-d286-4735-83c2-c456d251d732" containerID="5b579580a40cf7fb4f5eb976a5591db77ed5a7dd0f50b84ab51ebafebe6c5cbf" exitCode=137
Sep 11 01:10:06.404164 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:06.403702    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7700-6723a0"
Sep 11 01:10:06.404164 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:06.403740    2569 scope.go:117] "RemoveContainer" containerID="5b579580a40cf7fb4f5eb976a5591db77ed5a7dd0f50b84ab51ebafebe6c5cbf"
Sep 11 01:10:06.406166 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:06.405917    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-dc7dcd-openshift-default-5958f7974d-457sp" event={"ID":"8ec0f171-1bff-4174-9905-fe855ce1ad42","Type":"ContainerStarted","Data":"46644ea2b24175c4fed14bd8c07f971173c7c4f2862c4187f966b7b691a88a21"}
Sep 11 01:10:06.406166 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:06.406016    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-dc7dcd-openshift-default-5958f7974d-457sp" event={"ID":"8ec0f171-1bff-4174-9905-fe855ce1ad42","Type":"ContainerStarted","Data":"554b6711b5b0118399de69a8ffea15ea105d29185066540267766d9a13f9b48b"}
Sep 11 01:10:06.413908 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:06.413885    2569 scope.go:117] "RemoveContainer" containerID="5b579580a40cf7fb4f5eb976a5591db77ed5a7dd0f50b84ab51ebafebe6c5cbf"
Sep 11 01:10:06.414203 ip-10-0-129-222 kubenswrapper[2569]: E0911 01:10:06.414179    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b579580a40cf7fb4f5eb976a5591db77ed5a7dd0f50b84ab51ebafebe6c5cbf\": container with ID starting with 5b579580a40cf7fb4f5eb976a5591db77ed5a7dd0f50b84ab51ebafebe6c5cbf not found: ID does not exist" containerID="5b579580a40cf7fb4f5eb976a5591db77ed5a7dd0f50b84ab51ebafebe6c5cbf"
Sep 11 01:10:06.414313 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:06.414209    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b579580a40cf7fb4f5eb976a5591db77ed5a7dd0f50b84ab51ebafebe6c5cbf"} err="failed to get container status \"5b579580a40cf7fb4f5eb976a5591db77ed5a7dd0f50b84ab51ebafebe6c5cbf\": rpc error: code = NotFound desc = could not find container \"5b579580a40cf7fb4f5eb976a5591db77ed5a7dd0f50b84ab51ebafebe6c5cbf\": container with ID starting with 5b579580a40cf7fb4f5eb976a5591db77ed5a7dd0f50b84ab51ebafebe6c5cbf not found: ID does not exist"
Sep 11 01:10:06.438165 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:06.438112    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-dc7dcd-openshift-default-5958f7974d-457sp" podStartSLOduration=2.438097901 podStartE2EDuration="2.438097901s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 01:10:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 01:10:06.435061785 +0000 UTC m=+741.086436057" watchObservedRunningTime="2026-09-11 01:10:06.438097901 +0000 UTC m=+741.089472149"
Sep 11 01:10:06.805498 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:06.805468    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-0bc9acd8-openshift-default-744d69d584-x6fhm"]
Sep 11 01:10:06.815241 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:06.815209    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-0bc9acd8-openshift-default-744d69d584-x6fhm"
Sep 11 01:10:06.817558 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:06.817531    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-0bc9acd8-openshift-default-dockercfg-glxtw\""
Sep 11 01:10:06.822220 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:06.822192    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-0bc9acd8-openshift-default-744d69d584-x6fhm"]
Sep 11 01:10:06.959103 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:06.959066    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03848fb4-bfa2-40ce-ae94-ab4ac9837b60-istio-envoy\") pod \"e2e-ait-0bc9acd8-openshift-default-744d69d584-x6fhm\" (UID: \"03848fb4-bfa2-40ce-ae94-ab4ac9837b60\") " pod="openshift-ingress/e2e-ait-0bc9acd8-openshift-default-744d69d584-x6fhm"
Sep 11 01:10:06.959285 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:06.959119    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03848fb4-bfa2-40ce-ae94-ab4ac9837b60-workload-certs\") pod \"e2e-ait-0bc9acd8-openshift-default-744d69d584-x6fhm\" (UID: \"03848fb4-bfa2-40ce-ae94-ab4ac9837b60\") " pod="openshift-ingress/e2e-ait-0bc9acd8-openshift-default-744d69d584-x6fhm"
Sep 11 01:10:06.959285 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:06.959195    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03848fb4-bfa2-40ce-ae94-ab4ac9837b60-istio-data\") pod \"e2e-ait-0bc9acd8-openshift-default-744d69d584-x6fhm\" (UID: \"03848fb4-bfa2-40ce-ae94-ab4ac9837b60\") " pod="openshift-ingress/e2e-ait-0bc9acd8-openshift-default-744d69d584-x6fhm"
Sep 11 01:10:06.959285 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:06.959234    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03848fb4-bfa2-40ce-ae94-ab4ac9837b60-workload-socket\") pod \"e2e-ait-0bc9acd8-openshift-default-744d69d584-x6fhm\" (UID: \"03848fb4-bfa2-40ce-ae94-ab4ac9837b60\") " pod="openshift-ingress/e2e-ait-0bc9acd8-openshift-default-744d69d584-x6fhm"
Sep 11 01:10:06.959285 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:06.959271    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03848fb4-bfa2-40ce-ae94-ab4ac9837b60-istio-token\") pod \"e2e-ait-0bc9acd8-openshift-default-744d69d584-x6fhm\" (UID: \"03848fb4-bfa2-40ce-ae94-ab4ac9837b60\") " pod="openshift-ingress/e2e-ait-0bc9acd8-openshift-default-744d69d584-x6fhm"
Sep 11 01:10:06.959485 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:06.959363    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03848fb4-bfa2-40ce-ae94-ab4ac9837b60-credential-socket\") pod \"e2e-ait-0bc9acd8-openshift-default-744d69d584-x6fhm\" (UID: \"03848fb4-bfa2-40ce-ae94-ab4ac9837b60\") " pod="openshift-ingress/e2e-ait-0bc9acd8-openshift-default-744d69d584-x6fhm"
Sep 11 01:10:06.959534 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:06.959490    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03848fb4-bfa2-40ce-ae94-ab4ac9837b60-istiod-ca-cert\") pod \"e2e-ait-0bc9acd8-openshift-default-744d69d584-x6fhm\" (UID: \"03848fb4-bfa2-40ce-ae94-ab4ac9837b60\") " pod="openshift-ingress/e2e-ait-0bc9acd8-openshift-default-744d69d584-x6fhm"
Sep 11 01:10:06.959534 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:06.959525    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hbnc\" (UniqueName: \"kubernetes.io/projected/03848fb4-bfa2-40ce-ae94-ab4ac9837b60-kube-api-access-2hbnc\") pod \"e2e-ait-0bc9acd8-openshift-default-744d69d584-x6fhm\" (UID: \"03848fb4-bfa2-40ce-ae94-ab4ac9837b60\") " pod="openshift-ingress/e2e-ait-0bc9acd8-openshift-default-744d69d584-x6fhm"
Sep 11 01:10:06.959644 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:06.959607    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03848fb4-bfa2-40ce-ae94-ab4ac9837b60-istio-podinfo\") pod \"e2e-ait-0bc9acd8-openshift-default-744d69d584-x6fhm\" (UID: \"03848fb4-bfa2-40ce-ae94-ab4ac9837b60\") " pod="openshift-ingress/e2e-ait-0bc9acd8-openshift-default-744d69d584-x6fhm"
Sep 11 01:10:07.061264 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:07.061183    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03848fb4-bfa2-40ce-ae94-ab4ac9837b60-istio-podinfo\") pod \"e2e-ait-0bc9acd8-openshift-default-744d69d584-x6fhm\" (UID: \"03848fb4-bfa2-40ce-ae94-ab4ac9837b60\") " pod="openshift-ingress/e2e-ait-0bc9acd8-openshift-default-744d69d584-x6fhm"
Sep 11 01:10:07.061264 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:07.061260    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03848fb4-bfa2-40ce-ae94-ab4ac9837b60-istio-envoy\") pod \"e2e-ait-0bc9acd8-openshift-default-744d69d584-x6fhm\" (UID: \"03848fb4-bfa2-40ce-ae94-ab4ac9837b60\") " pod="openshift-ingress/e2e-ait-0bc9acd8-openshift-default-744d69d584-x6fhm"
Sep 11 01:10:07.061497 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:07.061301    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03848fb4-bfa2-40ce-ae94-ab4ac9837b60-workload-certs\") pod \"e2e-ait-0bc9acd8-openshift-default-744d69d584-x6fhm\" (UID: \"03848fb4-bfa2-40ce-ae94-ab4ac9837b60\") " pod="openshift-ingress/e2e-ait-0bc9acd8-openshift-default-744d69d584-x6fhm"
Sep 11 01:10:07.061497 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:07.061331    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03848fb4-bfa2-40ce-ae94-ab4ac9837b60-istio-data\") pod \"e2e-ait-0bc9acd8-openshift-default-744d69d584-x6fhm\" (UID: \"03848fb4-bfa2-40ce-ae94-ab4ac9837b60\") " pod="openshift-ingress/e2e-ait-0bc9acd8-openshift-default-744d69d584-x6fhm"
Sep 11 01:10:07.061497 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:07.061361    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03848fb4-bfa2-40ce-ae94-ab4ac9837b60-workload-socket\") pod \"e2e-ait-0bc9acd8-openshift-default-744d69d584-x6fhm\" (UID: \"03848fb4-bfa2-40ce-ae94-ab4ac9837b60\") " pod="openshift-ingress/e2e-ait-0bc9acd8-openshift-default-744d69d584-x6fhm"
Sep 11 01:10:07.061497 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:07.061409    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03848fb4-bfa2-40ce-ae94-ab4ac9837b60-istio-token\") pod \"e2e-ait-0bc9acd8-openshift-default-744d69d584-x6fhm\" (UID: \"03848fb4-bfa2-40ce-ae94-ab4ac9837b60\") " pod="openshift-ingress/e2e-ait-0bc9acd8-openshift-default-744d69d584-x6fhm"
Sep 11 01:10:07.061497 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:07.061478    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03848fb4-bfa2-40ce-ae94-ab4ac9837b60-credential-socket\") pod \"e2e-ait-0bc9acd8-openshift-default-744d69d584-x6fhm\" (UID: \"03848fb4-bfa2-40ce-ae94-ab4ac9837b60\") " pod="openshift-ingress/e2e-ait-0bc9acd8-openshift-default-744d69d584-x6fhm"
Sep 11 01:10:07.061759 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:07.061640    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03848fb4-bfa2-40ce-ae94-ab4ac9837b60-istiod-ca-cert\") pod \"e2e-ait-0bc9acd8-openshift-default-744d69d584-x6fhm\" (UID: \"03848fb4-bfa2-40ce-ae94-ab4ac9837b60\") " pod="openshift-ingress/e2e-ait-0bc9acd8-openshift-default-744d69d584-x6fhm"
Sep 11 01:10:07.061759 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:07.061672    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hbnc\" (UniqueName: \"kubernetes.io/projected/03848fb4-bfa2-40ce-ae94-ab4ac9837b60-kube-api-access-2hbnc\") pod \"e2e-ait-0bc9acd8-openshift-default-744d69d584-x6fhm\" (UID: \"03848fb4-bfa2-40ce-ae94-ab4ac9837b60\") " pod="openshift-ingress/e2e-ait-0bc9acd8-openshift-default-744d69d584-x6fhm"
Sep 11 01:10:07.061892 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:07.061758    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03848fb4-bfa2-40ce-ae94-ab4ac9837b60-istio-data\") pod \"e2e-ait-0bc9acd8-openshift-default-744d69d584-x6fhm\" (UID: \"03848fb4-bfa2-40ce-ae94-ab4ac9837b60\") " pod="openshift-ingress/e2e-ait-0bc9acd8-openshift-default-744d69d584-x6fhm"
Sep 11 01:10:07.061892 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:07.061796    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03848fb4-bfa2-40ce-ae94-ab4ac9837b60-workload-socket\") pod \"e2e-ait-0bc9acd8-openshift-default-744d69d584-x6fhm\" (UID: \"03848fb4-bfa2-40ce-ae94-ab4ac9837b60\") " pod="openshift-ingress/e2e-ait-0bc9acd8-openshift-default-744d69d584-x6fhm"
Sep 11 01:10:07.062104 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:07.062077    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03848fb4-bfa2-40ce-ae94-ab4ac9837b60-credential-socket\") pod \"e2e-ait-0bc9acd8-openshift-default-744d69d584-x6fhm\" (UID: \"03848fb4-bfa2-40ce-ae94-ab4ac9837b60\") " pod="openshift-ingress/e2e-ait-0bc9acd8-openshift-default-744d69d584-x6fhm"
Sep 11 01:10:07.062261 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:07.062114    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03848fb4-bfa2-40ce-ae94-ab4ac9837b60-workload-certs\") pod \"e2e-ait-0bc9acd8-openshift-default-744d69d584-x6fhm\" (UID: \"03848fb4-bfa2-40ce-ae94-ab4ac9837b60\") " pod="openshift-ingress/e2e-ait-0bc9acd8-openshift-default-744d69d584-x6fhm"
Sep 11 01:10:07.062422 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:07.062396    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03848fb4-bfa2-40ce-ae94-ab4ac9837b60-istiod-ca-cert\") pod \"e2e-ait-0bc9acd8-openshift-default-744d69d584-x6fhm\" (UID: \"03848fb4-bfa2-40ce-ae94-ab4ac9837b60\") " pod="openshift-ingress/e2e-ait-0bc9acd8-openshift-default-744d69d584-x6fhm"
Sep 11 01:10:07.063927 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:07.063907    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03848fb4-bfa2-40ce-ae94-ab4ac9837b60-istio-envoy\") pod \"e2e-ait-0bc9acd8-openshift-default-744d69d584-x6fhm\" (UID: \"03848fb4-bfa2-40ce-ae94-ab4ac9837b60\") " pod="openshift-ingress/e2e-ait-0bc9acd8-openshift-default-744d69d584-x6fhm"
Sep 11 01:10:07.064116 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:07.064097    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03848fb4-bfa2-40ce-ae94-ab4ac9837b60-istio-podinfo\") pod \"e2e-ait-0bc9acd8-openshift-default-744d69d584-x6fhm\" (UID: \"03848fb4-bfa2-40ce-ae94-ab4ac9837b60\") " pod="openshift-ingress/e2e-ait-0bc9acd8-openshift-default-744d69d584-x6fhm"
Sep 11 01:10:07.069544 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:07.069518    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03848fb4-bfa2-40ce-ae94-ab4ac9837b60-istio-token\") pod \"e2e-ait-0bc9acd8-openshift-default-744d69d584-x6fhm\" (UID: \"03848fb4-bfa2-40ce-ae94-ab4ac9837b60\") " pod="openshift-ingress/e2e-ait-0bc9acd8-openshift-default-744d69d584-x6fhm"
Sep 11 01:10:07.069805 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:07.069787    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hbnc\" (UniqueName: \"kubernetes.io/projected/03848fb4-bfa2-40ce-ae94-ab4ac9837b60-kube-api-access-2hbnc\") pod \"e2e-ait-0bc9acd8-openshift-default-744d69d584-x6fhm\" (UID: \"03848fb4-bfa2-40ce-ae94-ab4ac9837b60\") " pod="openshift-ingress/e2e-ait-0bc9acd8-openshift-default-744d69d584-x6fhm"
Sep 11 01:10:07.130235 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:07.130198    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-0bc9acd8-openshift-default-744d69d584-x6fhm"
Sep 11 01:10:07.299293 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:07.299262    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-dc7dcd-openshift-default-5958f7974d-457sp"
Sep 11 01:10:07.305519 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:07.305488    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-dc7dcd-openshift-default-5958f7974d-457sp"
Sep 11 01:10:07.413888 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:07.413782    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-dc7dcd-openshift-default-5958f7974d-457sp"
Sep 11 01:10:07.414902 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:07.414863    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-dc7dcd-openshift-default-5958f7974d-457sp"
Sep 11 01:10:07.682953 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:07.682929    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7700-ca7d54"
Sep 11 01:10:07.770880 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:07.770840    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pkbdt\" (UniqueName: \"kubernetes.io/projected/3862ebef-43f1-423a-b5d1-d110686731cd-kube-api-access-pkbdt\") pod \"3862ebef-43f1-423a-b5d1-d110686731cd\" (UID: \"3862ebef-43f1-423a-b5d1-d110686731cd\") "
Sep 11 01:10:07.772840 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:07.772812    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3862ebef-43f1-423a-b5d1-d110686731cd-kube-api-access-pkbdt" (OuterVolumeSpecName: "kube-api-access-pkbdt") pod "3862ebef-43f1-423a-b5d1-d110686731cd" (UID: "3862ebef-43f1-423a-b5d1-d110686731cd"). InnerVolumeSpecName "kube-api-access-pkbdt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 01:10:07.818467 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:07.818438    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-0bc9acd8-openshift-default-744d69d584-x6fhm"]
Sep 11 01:10:07.820274 ip-10-0-129-222 kubenswrapper[2569]: W0911 01:10:07.820248    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03848fb4_bfa2_40ce_ae94_ab4ac9837b60.slice/crio-9ceb0bd7e17734a7e8386d6dd9f58da3133dd1ad60e951cc5cdcd2f4cd570af3 WatchSource:0}: Error finding container 9ceb0bd7e17734a7e8386d6dd9f58da3133dd1ad60e951cc5cdcd2f4cd570af3: Status 404 returned error can't find the container with id 9ceb0bd7e17734a7e8386d6dd9f58da3133dd1ad60e951cc5cdcd2f4cd570af3
Sep 11 01:10:07.822593 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:07.822563    2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 01:10:07.822669 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:07.822621    2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 01:10:07.822669 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:07.822651    2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 01:10:07.871900 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:07.871830    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pkbdt\" (UniqueName: \"kubernetes.io/projected/3862ebef-43f1-423a-b5d1-d110686731cd-kube-api-access-pkbdt\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:10:07.872624 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:07.872580    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3862ebef-43f1-423a-b5d1-d110686731cd" path="/var/lib/kubelet/pods/3862ebef-43f1-423a-b5d1-d110686731cd/volumes"
Sep 11 01:10:08.421990 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:08.421908    2569 generic.go:356] "Generic (PLEG): container finished" podID="3862ebef-43f1-423a-b5d1-d110686731cd" containerID="c2ad6f5797420d497363fd7c5acc60fbba372a5cb64b3616d52c0f3091727b23" exitCode=137
Sep 11 01:10:08.421990 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:08.421947    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7700-ca7d54"
Sep 11 01:10:08.422459 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:08.422006    2569 scope.go:117] "RemoveContainer" containerID="c2ad6f5797420d497363fd7c5acc60fbba372a5cb64b3616d52c0f3091727b23"
Sep 11 01:10:08.423919 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:08.423830    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-6ba087-bdcc845c4-cqzxs" event={"ID":"a7b5c45d-9578-4b0c-836b-e6d9b2d8dac2","Type":"ContainerStarted","Data":"69b9e8ae3d1252ce4d2afccd74286028dca0d451ddc16c6944f47e24c4c40f19"}
Sep 11 01:10:08.424058 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:08.423985    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-6ba087-bdcc845c4-cqzxs"
Sep 11 01:10:08.425906 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:08.425856    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0bc9acd8-openshift-default-744d69d584-x6fhm" event={"ID":"03848fb4-bfa2-40ce-ae94-ab4ac9837b60","Type":"ContainerStarted","Data":"813382bc441f6727902aa5f80216c47bcbbb716711293edc312e69b191e50d63"}
Sep 11 01:10:08.426003 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:08.425927    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0bc9acd8-openshift-default-744d69d584-x6fhm" event={"ID":"03848fb4-bfa2-40ce-ae94-ab4ac9837b60","Type":"ContainerStarted","Data":"9ceb0bd7e17734a7e8386d6dd9f58da3133dd1ad60e951cc5cdcd2f4cd570af3"}
Sep 11 01:10:08.430673 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:08.430654    2569 scope.go:117] "RemoveContainer" containerID="c2ad6f5797420d497363fd7c5acc60fbba372a5cb64b3616d52c0f3091727b23"
Sep 11 01:10:08.430924 ip-10-0-129-222 kubenswrapper[2569]: E0911 01:10:08.430907    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2ad6f5797420d497363fd7c5acc60fbba372a5cb64b3616d52c0f3091727b23\": container with ID starting with c2ad6f5797420d497363fd7c5acc60fbba372a5cb64b3616d52c0f3091727b23 not found: ID does not exist" containerID="c2ad6f5797420d497363fd7c5acc60fbba372a5cb64b3616d52c0f3091727b23"
Sep 11 01:10:08.430994 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:08.430929    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2ad6f5797420d497363fd7c5acc60fbba372a5cb64b3616d52c0f3091727b23"} err="failed to get container status \"c2ad6f5797420d497363fd7c5acc60fbba372a5cb64b3616d52c0f3091727b23\": rpc error: code = NotFound desc = could not find container \"c2ad6f5797420d497363fd7c5acc60fbba372a5cb64b3616d52c0f3091727b23\": container with ID starting with c2ad6f5797420d497363fd7c5acc60fbba372a5cb64b3616d52c0f3091727b23 not found: ID does not exist"
Sep 11 01:10:08.441938 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:08.441889    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-6ba087-bdcc845c4-cqzxs" podStartSLOduration=1.810073941 podStartE2EDuration="4.441859675s" totalImagesPullingTime="2.631785734s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 01:10:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 01:10:08.440310704 +0000 UTC m=+743.091684955" watchObservedRunningTime="2026-09-11 01:10:08.441859675 +0000 UTC m=+743.093233923"
Sep 11 01:10:08.471505 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:08.471466    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-0bc9acd8-openshift-default-744d69d584-x6fhm" podStartSLOduration=2.471452923 podStartE2EDuration="2.471452923s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 01:10:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 01:10:08.468355896 +0000 UTC m=+743.119730147" watchObservedRunningTime="2026-09-11 01:10:08.471452923 +0000 UTC m=+743.122827216"
Sep 11 01:10:09.131034 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:09.130997    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-0bc9acd8-openshift-default-744d69d584-x6fhm"
Sep 11 01:10:09.135941 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:09.135910    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-0bc9acd8-openshift-default-744d69d584-x6fhm"
Sep 11 01:10:09.440921 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:09.440825    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-0bc9acd8-openshift-default-744d69d584-x6fhm"
Sep 11 01:10:09.441556 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:09.441535    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-0bc9acd8-openshift-default-744d69d584-x6fhm"
Sep 11 01:10:10.571510 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:10.571476    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-dc7dcd-846dffff5d-zffpq"]
Sep 11 01:10:10.576641 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:10.576620    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-dc7dcd-846dffff5d-zffpq"
Sep 11 01:10:10.579151 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:10.579125    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-dc7dcd\""
Sep 11 01:10:10.579279 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:10.579235    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-dc7dcd-dockercfg-8jnck\""
Sep 11 01:10:10.588867 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:10.588847    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-dc7dcd-846dffff5d-zffpq"]
Sep 11 01:10:10.702400 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:10.702363    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/076eaa74-0639-4df0-9392-3c6a3ec60a9f-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-dc7dcd-846dffff5d-zffpq\" (UID: \"076eaa74-0639-4df0-9392-3c6a3ec60a9f\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-dc7dcd-846dffff5d-zffpq"
Sep 11 01:10:10.702400 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:10.702400    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9r75\" (UniqueName: \"kubernetes.io/projected/076eaa74-0639-4df0-9392-3c6a3ec60a9f-kube-api-access-h9r75\") pod \"payload-pre-processing-e2e-autores-dc7dcd-846dffff5d-zffpq\" (UID: \"076eaa74-0639-4df0-9392-3c6a3ec60a9f\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-dc7dcd-846dffff5d-zffpq"
Sep 11 01:10:10.803850 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:10.803814    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/076eaa74-0639-4df0-9392-3c6a3ec60a9f-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-dc7dcd-846dffff5d-zffpq\" (UID: \"076eaa74-0639-4df0-9392-3c6a3ec60a9f\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-dc7dcd-846dffff5d-zffpq"
Sep 11 01:10:10.803850 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:10.803853    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9r75\" (UniqueName: \"kubernetes.io/projected/076eaa74-0639-4df0-9392-3c6a3ec60a9f-kube-api-access-h9r75\") pod \"payload-pre-processing-e2e-autores-dc7dcd-846dffff5d-zffpq\" (UID: \"076eaa74-0639-4df0-9392-3c6a3ec60a9f\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-dc7dcd-846dffff5d-zffpq"
Sep 11 01:10:10.804501 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:10.804482    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/076eaa74-0639-4df0-9392-3c6a3ec60a9f-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-dc7dcd-846dffff5d-zffpq\" (UID: \"076eaa74-0639-4df0-9392-3c6a3ec60a9f\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-dc7dcd-846dffff5d-zffpq"
Sep 11 01:10:10.811702 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:10.811678    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9r75\" (UniqueName: \"kubernetes.io/projected/076eaa74-0639-4df0-9392-3c6a3ec60a9f-kube-api-access-h9r75\") pod \"payload-pre-processing-e2e-autores-dc7dcd-846dffff5d-zffpq\" (UID: \"076eaa74-0639-4df0-9392-3c6a3ec60a9f\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-dc7dcd-846dffff5d-zffpq"
Sep 11 01:10:10.887439 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:10.887357    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-dc7dcd-846dffff5d-zffpq"
Sep 11 01:10:11.180813 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:11.180785    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-dc7dcd-846dffff5d-zffpq"]
Sep 11 01:10:11.182377 ip-10-0-129-222 kubenswrapper[2569]: W0911 01:10:11.182347    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod076eaa74_0639_4df0_9392_3c6a3ec60a9f.slice/crio-36cca2a74965bcd82c8a348515b750630239aefd530cf37bef65d3823dd9830d WatchSource:0}: Error finding container 36cca2a74965bcd82c8a348515b750630239aefd530cf37bef65d3823dd9830d: Status 404 returned error can't find the container with id 36cca2a74965bcd82c8a348515b750630239aefd530cf37bef65d3823dd9830d
Sep 11 01:10:11.488146 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:11.488054    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-dc7dcd-846dffff5d-zffpq" event={"ID":"076eaa74-0639-4df0-9392-3c6a3ec60a9f","Type":"ContainerStarted","Data":"c8685432b33fe8b2762613b30fe9210a676db5556041e97fdc7affb0a99a254f"}
Sep 11 01:10:11.488146 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:11.488112    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-dc7dcd-846dffff5d-zffpq" event={"ID":"076eaa74-0639-4df0-9392-3c6a3ec60a9f","Type":"ContainerStarted","Data":"36cca2a74965bcd82c8a348515b750630239aefd530cf37bef65d3823dd9830d"}
Sep 11 01:10:11.505481 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:11.505419    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-autores-dc7dcd-846dffff5d-zffpq" podStartSLOduration=1.505398366 podStartE2EDuration="1.505398366s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 01:10:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 01:10:11.504285234 +0000 UTC m=+746.155659487" watchObservedRunningTime="2026-09-11 01:10:11.505398366 +0000 UTC m=+746.156772617"
Sep 11 01:10:12.492925 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:12.492894    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-dc7dcd-846dffff5d-zffpq"
Sep 11 01:10:16.798240 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:16.798203    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kmx5x"]
Sep 11 01:10:19.443122 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:19.443091    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-6ba087-bdcc845c4-cqzxs"
Sep 11 01:10:19.531591 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:19.531512    2569 generic.go:356] "Generic (PLEG): container finished" podID="e147aad2-95bd-4135-82aa-7f015f6996a4" containerID="7dbec7d9e178cfcf55fd46d3858b4e3f1c87306bc6870f7adb05665f9802cd83" exitCode=0
Sep 11 01:10:19.531734 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:19.531581    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-6ba087-openshift-default-85464b8f68-b5knz" event={"ID":"e147aad2-95bd-4135-82aa-7f015f6996a4","Type":"ContainerDied","Data":"7dbec7d9e178cfcf55fd46d3858b4e3f1c87306bc6870f7adb05665f9802cd83"}
Sep 11 01:10:19.532010 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:19.531994    2569 scope.go:117] "RemoveContainer" containerID="7dbec7d9e178cfcf55fd46d3858b4e3f1c87306bc6870f7adb05665f9802cd83"
Sep 11 01:10:19.533305 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:19.533274    2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 01:10:19.533389 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:19.533350    2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 01:10:19.533389 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:19.533386    2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 01:10:20.541173 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:20.541134    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-6ba087-openshift-default-85464b8f68-b5knz" event={"ID":"e147aad2-95bd-4135-82aa-7f015f6996a4","Type":"ContainerStarted","Data":"25f4d1bac28bf3430c1ea1d494cee8a6f0f57677871637a577bf47c2eabf5066"}
Sep 11 01:10:21.449597 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:21.449560    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-6ba087-openshift-default-85464b8f68-b5knz"
Sep 11 01:10:21.449981 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:21.449956    2569 patch_prober.go:28] interesting pod/e2e-worker-w0-6ba087-openshift-default-85464b8f68-b5knz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body=
Sep 11 01:10:21.450183 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:21.450157    2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6ba087-openshift-default-85464b8f68-b5knz" podUID="e147aad2-95bd-4135-82aa-7f015f6996a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused"
Sep 11 01:10:22.449290 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:22.449250    2569 patch_prober.go:28] interesting pod/e2e-worker-w0-6ba087-openshift-default-85464b8f68-b5knz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body=
Sep 11 01:10:22.449890 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:22.449838    2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6ba087-openshift-default-85464b8f68-b5knz" podUID="e147aad2-95bd-4135-82aa-7f015f6996a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused"
Sep 11 01:10:23.449603 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:23.449561    2569 patch_prober.go:28] interesting pod/e2e-worker-w0-6ba087-openshift-default-85464b8f68-b5knz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body=
Sep 11 01:10:23.450223 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:23.450190    2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6ba087-openshift-default-85464b8f68-b5knz" podUID="e147aad2-95bd-4135-82aa-7f015f6996a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused"
Sep 11 01:10:23.500119 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:23.500088    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-dc7dcd-846dffff5d-zffpq"
Sep 11 01:10:24.450252 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:24.450214    2569 patch_prober.go:28] interesting pod/e2e-worker-w0-6ba087-openshift-default-85464b8f68-b5knz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body=
Sep 11 01:10:24.451210 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:24.450794    2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6ba087-openshift-default-85464b8f68-b5knz" podUID="e147aad2-95bd-4135-82aa-7f015f6996a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused"
Sep 11 01:10:25.450228 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:25.449707    2569 patch_prober.go:28] interesting pod/e2e-worker-w0-6ba087-openshift-default-85464b8f68-b5knz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body=
Sep 11 01:10:25.450228 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:25.449780    2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6ba087-openshift-default-85464b8f68-b5knz" podUID="e147aad2-95bd-4135-82aa-7f015f6996a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused"
Sep 11 01:10:25.747481 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:25.747396    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-891e67-openshift-default-564fcdf9fb-xw8kt"]
Sep 11 01:10:25.824098 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:25.824049    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-891e67-openshift-default-564fcdf9fb-xw8kt"]
Sep 11 01:10:25.824274 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:25.824196    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-891e67-openshift-default-564fcdf9fb-xw8kt"
Sep 11 01:10:25.828453 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:25.828430    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-891e67-openshift-default-dockercfg-p2q58\""
Sep 11 01:10:25.888817 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:25.888782    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b6bb107-1129-43c9-aa58-25cd7c70bfbf-workload-socket\") pod \"e2e-inf-b-891e67-openshift-default-564fcdf9fb-xw8kt\" (UID: \"2b6bb107-1129-43c9-aa58-25cd7c70bfbf\") " pod="openshift-ingress/e2e-inf-b-891e67-openshift-default-564fcdf9fb-xw8kt"
Sep 11 01:10:25.888817 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:25.888833    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b6bb107-1129-43c9-aa58-25cd7c70bfbf-istio-podinfo\") pod \"e2e-inf-b-891e67-openshift-default-564fcdf9fb-xw8kt\" (UID: \"2b6bb107-1129-43c9-aa58-25cd7c70bfbf\") " pod="openshift-ingress/e2e-inf-b-891e67-openshift-default-564fcdf9fb-xw8kt"
Sep 11 01:10:25.889104 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:25.888856    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b6bb107-1129-43c9-aa58-25cd7c70bfbf-istio-data\") pod \"e2e-inf-b-891e67-openshift-default-564fcdf9fb-xw8kt\" (UID: \"2b6bb107-1129-43c9-aa58-25cd7c70bfbf\") " pod="openshift-ingress/e2e-inf-b-891e67-openshift-default-564fcdf9fb-xw8kt"
Sep 11 01:10:25.889510 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:25.889078    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b6bb107-1129-43c9-aa58-25cd7c70bfbf-istio-envoy\") pod \"e2e-inf-b-891e67-openshift-default-564fcdf9fb-xw8kt\" (UID: \"2b6bb107-1129-43c9-aa58-25cd7c70bfbf\") " pod="openshift-ingress/e2e-inf-b-891e67-openshift-default-564fcdf9fb-xw8kt"
Sep 11 01:10:25.889510 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:25.889250    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xtssr\" (UniqueName: \"kubernetes.io/projected/2b6bb107-1129-43c9-aa58-25cd7c70bfbf-kube-api-access-xtssr\") pod \"e2e-inf-b-891e67-openshift-default-564fcdf9fb-xw8kt\" (UID: \"2b6bb107-1129-43c9-aa58-25cd7c70bfbf\") " pod="openshift-ingress/e2e-inf-b-891e67-openshift-default-564fcdf9fb-xw8kt"
Sep 11 01:10:25.889510 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:25.889335    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b6bb107-1129-43c9-aa58-25cd7c70bfbf-istiod-ca-cert\") pod \"e2e-inf-b-891e67-openshift-default-564fcdf9fb-xw8kt\" (UID: \"2b6bb107-1129-43c9-aa58-25cd7c70bfbf\") " pod="openshift-ingress/e2e-inf-b-891e67-openshift-default-564fcdf9fb-xw8kt"
Sep 11 01:10:25.889510 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:25.889390    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b6bb107-1129-43c9-aa58-25cd7c70bfbf-istio-token\") pod \"e2e-inf-b-891e67-openshift-default-564fcdf9fb-xw8kt\" (UID: \"2b6bb107-1129-43c9-aa58-25cd7c70bfbf\") " pod="openshift-ingress/e2e-inf-b-891e67-openshift-default-564fcdf9fb-xw8kt"
Sep 11 01:10:25.889510 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:25.889421    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b6bb107-1129-43c9-aa58-25cd7c70bfbf-credential-socket\") pod \"e2e-inf-b-891e67-openshift-default-564fcdf9fb-xw8kt\" (UID: \"2b6bb107-1129-43c9-aa58-25cd7c70bfbf\") " pod="openshift-ingress/e2e-inf-b-891e67-openshift-default-564fcdf9fb-xw8kt"
Sep 11 01:10:25.889510 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:25.889457    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b6bb107-1129-43c9-aa58-25cd7c70bfbf-workload-certs\") pod \"e2e-inf-b-891e67-openshift-default-564fcdf9fb-xw8kt\" (UID: \"2b6bb107-1129-43c9-aa58-25cd7c70bfbf\") " pod="openshift-ingress/e2e-inf-b-891e67-openshift-default-564fcdf9fb-xw8kt"
Sep 11 01:10:25.990471 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:25.990429    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b6bb107-1129-43c9-aa58-25cd7c70bfbf-istiod-ca-cert\") pod \"e2e-inf-b-891e67-openshift-default-564fcdf9fb-xw8kt\" (UID: \"2b6bb107-1129-43c9-aa58-25cd7c70bfbf\") " pod="openshift-ingress/e2e-inf-b-891e67-openshift-default-564fcdf9fb-xw8kt"
Sep 11 01:10:25.990678 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:25.990500    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b6bb107-1129-43c9-aa58-25cd7c70bfbf-istio-token\") pod \"e2e-inf-b-891e67-openshift-default-564fcdf9fb-xw8kt\" (UID: \"2b6bb107-1129-43c9-aa58-25cd7c70bfbf\") " pod="openshift-ingress/e2e-inf-b-891e67-openshift-default-564fcdf9fb-xw8kt"
Sep 11 01:10:25.990678 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:25.990531    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b6bb107-1129-43c9-aa58-25cd7c70bfbf-credential-socket\") pod \"e2e-inf-b-891e67-openshift-default-564fcdf9fb-xw8kt\" (UID: \"2b6bb107-1129-43c9-aa58-25cd7c70bfbf\") " pod="openshift-ingress/e2e-inf-b-891e67-openshift-default-564fcdf9fb-xw8kt"
Sep 11 01:10:25.990678 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:25.990566    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b6bb107-1129-43c9-aa58-25cd7c70bfbf-workload-certs\") pod \"e2e-inf-b-891e67-openshift-default-564fcdf9fb-xw8kt\" (UID: \"2b6bb107-1129-43c9-aa58-25cd7c70bfbf\") " pod="openshift-ingress/e2e-inf-b-891e67-openshift-default-564fcdf9fb-xw8kt"
Sep 11 01:10:25.990678 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:25.990626    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b6bb107-1129-43c9-aa58-25cd7c70bfbf-workload-socket\") pod \"e2e-inf-b-891e67-openshift-default-564fcdf9fb-xw8kt\" (UID: \"2b6bb107-1129-43c9-aa58-25cd7c70bfbf\") " pod="openshift-ingress/e2e-inf-b-891e67-openshift-default-564fcdf9fb-xw8kt"
Sep 11 01:10:25.990678 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:25.990655    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b6bb107-1129-43c9-aa58-25cd7c70bfbf-istio-podinfo\") pod \"e2e-inf-b-891e67-openshift-default-564fcdf9fb-xw8kt\" (UID: \"2b6bb107-1129-43c9-aa58-25cd7c70bfbf\") " pod="openshift-ingress/e2e-inf-b-891e67-openshift-default-564fcdf9fb-xw8kt"
Sep 11 01:10:25.991022 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:25.990680    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b6bb107-1129-43c9-aa58-25cd7c70bfbf-istio-data\") pod \"e2e-inf-b-891e67-openshift-default-564fcdf9fb-xw8kt\" (UID: \"2b6bb107-1129-43c9-aa58-25cd7c70bfbf\") " pod="openshift-ingress/e2e-inf-b-891e67-openshift-default-564fcdf9fb-xw8kt"
Sep 11 01:10:25.991022 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:25.990771    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b6bb107-1129-43c9-aa58-25cd7c70bfbf-istio-envoy\") pod \"e2e-inf-b-891e67-openshift-default-564fcdf9fb-xw8kt\" (UID: \"2b6bb107-1129-43c9-aa58-25cd7c70bfbf\") " pod="openshift-ingress/e2e-inf-b-891e67-openshift-default-564fcdf9fb-xw8kt"
Sep 11 01:10:25.991022 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:25.990805    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtssr\" (UniqueName: \"kubernetes.io/projected/2b6bb107-1129-43c9-aa58-25cd7c70bfbf-kube-api-access-xtssr\") pod \"e2e-inf-b-891e67-openshift-default-564fcdf9fb-xw8kt\" (UID: \"2b6bb107-1129-43c9-aa58-25cd7c70bfbf\") " pod="openshift-ingress/e2e-inf-b-891e67-openshift-default-564fcdf9fb-xw8kt"
Sep 11 01:10:25.991339 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:25.991276    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b6bb107-1129-43c9-aa58-25cd7c70bfbf-workload-socket\") pod \"e2e-inf-b-891e67-openshift-default-564fcdf9fb-xw8kt\" (UID: \"2b6bb107-1129-43c9-aa58-25cd7c70bfbf\") " pod="openshift-ingress/e2e-inf-b-891e67-openshift-default-564fcdf9fb-xw8kt"
Sep 11 01:10:25.991339 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:25.991311    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b6bb107-1129-43c9-aa58-25cd7c70bfbf-istiod-ca-cert\") pod \"e2e-inf-b-891e67-openshift-default-564fcdf9fb-xw8kt\" (UID: \"2b6bb107-1129-43c9-aa58-25cd7c70bfbf\") " pod="openshift-ingress/e2e-inf-b-891e67-openshift-default-564fcdf9fb-xw8kt"
Sep 11 01:10:25.991558 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:25.991528    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b6bb107-1129-43c9-aa58-25cd7c70bfbf-credential-socket\") pod \"e2e-inf-b-891e67-openshift-default-564fcdf9fb-xw8kt\" (UID: \"2b6bb107-1129-43c9-aa58-25cd7c70bfbf\") " pod="openshift-ingress/e2e-inf-b-891e67-openshift-default-564fcdf9fb-xw8kt"
Sep 11 01:10:25.991723 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:25.991703    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b6bb107-1129-43c9-aa58-25cd7c70bfbf-workload-certs\") pod \"e2e-inf-b-891e67-openshift-default-564fcdf9fb-xw8kt\" (UID: \"2b6bb107-1129-43c9-aa58-25cd7c70bfbf\") " pod="openshift-ingress/e2e-inf-b-891e67-openshift-default-564fcdf9fb-xw8kt"
Sep 11 01:10:25.991798 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:25.991779    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b6bb107-1129-43c9-aa58-25cd7c70bfbf-istio-data\") pod \"e2e-inf-b-891e67-openshift-default-564fcdf9fb-xw8kt\" (UID: \"2b6bb107-1129-43c9-aa58-25cd7c70bfbf\") " pod="openshift-ingress/e2e-inf-b-891e67-openshift-default-564fcdf9fb-xw8kt"
Sep 11 01:10:25.994090 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:25.994049    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b6bb107-1129-43c9-aa58-25cd7c70bfbf-istio-envoy\") pod \"e2e-inf-b-891e67-openshift-default-564fcdf9fb-xw8kt\" (UID: \"2b6bb107-1129-43c9-aa58-25cd7c70bfbf\") " pod="openshift-ingress/e2e-inf-b-891e67-openshift-default-564fcdf9fb-xw8kt"
Sep 11 01:10:25.994443 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:25.994421    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b6bb107-1129-43c9-aa58-25cd7c70bfbf-istio-podinfo\") pod \"e2e-inf-b-891e67-openshift-default-564fcdf9fb-xw8kt\" (UID: \"2b6bb107-1129-43c9-aa58-25cd7c70bfbf\") " pod="openshift-ingress/e2e-inf-b-891e67-openshift-default-564fcdf9fb-xw8kt"
Sep 11 01:10:26.003914 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:26.003788    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b6bb107-1129-43c9-aa58-25cd7c70bfbf-istio-token\") pod \"e2e-inf-b-891e67-openshift-default-564fcdf9fb-xw8kt\" (UID: \"2b6bb107-1129-43c9-aa58-25cd7c70bfbf\") " pod="openshift-ingress/e2e-inf-b-891e67-openshift-default-564fcdf9fb-xw8kt"
Sep 11 01:10:26.003914 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:26.003837    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtssr\" (UniqueName: \"kubernetes.io/projected/2b6bb107-1129-43c9-aa58-25cd7c70bfbf-kube-api-access-xtssr\") pod \"e2e-inf-b-891e67-openshift-default-564fcdf9fb-xw8kt\" (UID: \"2b6bb107-1129-43c9-aa58-25cd7c70bfbf\") " pod="openshift-ingress/e2e-inf-b-891e67-openshift-default-564fcdf9fb-xw8kt"
Sep 11 01:10:26.140838 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:26.140794    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-891e67-openshift-default-564fcdf9fb-xw8kt"
Sep 11 01:10:26.430930 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:26.430901    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-891e67-openshift-default-564fcdf9fb-xw8kt"]
Sep 11 01:10:26.433054 ip-10-0-129-222 kubenswrapper[2569]: W0911 01:10:26.433026    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b6bb107_1129_43c9_aa58_25cd7c70bfbf.slice/crio-5a44b163ff36da0150500bee5427dda3ef62d30b43f7f3822a847cd0b7371941 WatchSource:0}: Error finding container 5a44b163ff36da0150500bee5427dda3ef62d30b43f7f3822a847cd0b7371941: Status 404 returned error can't find the container with id 5a44b163ff36da0150500bee5427dda3ef62d30b43f7f3822a847cd0b7371941
Sep 11 01:10:26.436176 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:26.436138    2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 01:10:26.436286 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:26.436216    2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 01:10:26.436286 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:26.436263    2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 01:10:26.454351 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:26.454328    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-6ba087-openshift-default-85464b8f68-b5knz"
Sep 11 01:10:26.454982 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:26.454922    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-6ba087-openshift-default-85464b8f68-b5knz"
Sep 11 01:10:26.455858 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:26.455837    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-6ba087-openshift-default-85464b8f68-b5knz"
Sep 11 01:10:26.596844 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:26.596780    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-891e67-openshift-default-564fcdf9fb-xw8kt" event={"ID":"2b6bb107-1129-43c9-aa58-25cd7c70bfbf","Type":"ContainerStarted","Data":"8a84288b9a80622c7d651d301fd4c8a9eb292140b04df4f1b33521cb7801710f"}
Sep 11 01:10:26.597039 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:26.596858    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-891e67-openshift-default-564fcdf9fb-xw8kt" event={"ID":"2b6bb107-1129-43c9-aa58-25cd7c70bfbf","Type":"ContainerStarted","Data":"5a44b163ff36da0150500bee5427dda3ef62d30b43f7f3822a847cd0b7371941"}
Sep 11 01:10:26.620468 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:26.620411    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-891e67-openshift-default-564fcdf9fb-xw8kt" podStartSLOduration=1.620393424 podStartE2EDuration="1.620393424s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 01:10:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 01:10:26.618192462 +0000 UTC m=+761.269566711" watchObservedRunningTime="2026-09-11 01:10:26.620393424 +0000 UTC m=+761.271767674"
Sep 11 01:10:27.141541 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:27.141494    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-891e67-openshift-default-564fcdf9fb-xw8kt"
Sep 11 01:10:27.148615 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:27.148589    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-891e67-openshift-default-564fcdf9fb-xw8kt"
Sep 11 01:10:27.603409 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:27.603376    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-891e67-openshift-default-564fcdf9fb-xw8kt"
Sep 11 01:10:27.604461 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:27.604433    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-891e67-openshift-default-564fcdf9fb-xw8kt"
Sep 11 01:10:28.612333 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:28.612299    2569 generic.go:356] "Generic (PLEG): container finished" podID="8ec0f171-1bff-4174-9905-fe855ce1ad42" containerID="46644ea2b24175c4fed14bd8c07f971173c7c4f2862c4187f966b7b691a88a21" exitCode=0
Sep 11 01:10:28.612841 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:28.612374    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-dc7dcd-openshift-default-5958f7974d-457sp" event={"ID":"8ec0f171-1bff-4174-9905-fe855ce1ad42","Type":"ContainerDied","Data":"46644ea2b24175c4fed14bd8c07f971173c7c4f2862c4187f966b7b691a88a21"}
Sep 11 01:10:28.612841 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:28.612822    2569 scope.go:117] "RemoveContainer" containerID="46644ea2b24175c4fed14bd8c07f971173c7c4f2862c4187f966b7b691a88a21"
Sep 11 01:10:28.614043 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:28.614023    2569 generic.go:356] "Generic (PLEG): container finished" podID="03848fb4-bfa2-40ce-ae94-ab4ac9837b60" containerID="813382bc441f6727902aa5f80216c47bcbbb716711293edc312e69b191e50d63" exitCode=0
Sep 11 01:10:28.614141 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:28.614062    2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 01:10:28.614141 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:28.614091    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0bc9acd8-openshift-default-744d69d584-x6fhm" event={"ID":"03848fb4-bfa2-40ce-ae94-ab4ac9837b60","Type":"ContainerDied","Data":"813382bc441f6727902aa5f80216c47bcbbb716711293edc312e69b191e50d63"}
Sep 11 01:10:28.614141 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:28.614125    2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 01:10:28.614273 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:28.614156    2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 01:10:28.614591 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:28.614573    2569 scope.go:117] "RemoveContainer" containerID="813382bc441f6727902aa5f80216c47bcbbb716711293edc312e69b191e50d63"
Sep 11 01:10:28.615689 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:28.615666    2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 01:10:28.615739 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:28.615722    2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 01:10:28.615777 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:28.615756    2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 01:10:29.621910 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:29.621855    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0bc9acd8-openshift-default-744d69d584-x6fhm" event={"ID":"03848fb4-bfa2-40ce-ae94-ab4ac9837b60","Type":"ContainerStarted","Data":"66177f3ee7684b7f932ebbbe00661e9ac96a1609e4cc1a6b307376918273cdba"}
Sep 11 01:10:29.623858 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:29.623836    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-dc7dcd-openshift-default-5958f7974d-457sp" event={"ID":"8ec0f171-1bff-4174-9905-fe855ce1ad42","Type":"ContainerStarted","Data":"288ad8bcd36951b193884c4c5d9e687666efe6edd615bf7fa1a2d2b907184732"}
Sep 11 01:10:30.130539 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:30.130500    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-0bc9acd8-openshift-default-744d69d584-x6fhm"
Sep 11 01:10:30.130745 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:30.130707    2569 patch_prober.go:28] interesting pod/e2e-ait-0bc9acd8-openshift-default-744d69d584-x6fhm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body=
Sep 11 01:10:30.130822 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:30.130765    2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-0bc9acd8-openshift-default-744d69d584-x6fhm" podUID="03848fb4-bfa2-40ce-ae94-ab4ac9837b60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused"
Sep 11 01:10:30.299900 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:30.299839    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-dc7dcd-openshift-default-5958f7974d-457sp"
Sep 11 01:10:30.300099 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:30.300010    2569 patch_prober.go:28] interesting pod/e2e-autores-dc7dcd-openshift-default-5958f7974d-457sp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body=
Sep 11 01:10:30.300099 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:30.300055    2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-dc7dcd-openshift-default-5958f7974d-457sp" podUID="8ec0f171-1bff-4174-9905-fe855ce1ad42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused"
Sep 11 01:10:30.489880 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:30.489774    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-891e67-fc5bccbf4-4w5pk"]
Sep 11 01:10:30.492705 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:30.492689    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-891e67-fc5bccbf4-4w5pk"
Sep 11 01:10:30.494974 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:30.494952    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-891e67\""
Sep 11 01:10:30.503775 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:30.503754    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-891e67-fc5bccbf4-4w5pk"]
Sep 11 01:10:30.516012 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:30.515984    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-891e67-55d8c8cf55-j7jmn"]
Sep 11 01:10:30.518512 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:30.518497    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-891e67-55d8c8cf55-j7jmn"
Sep 11 01:10:30.521390 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:30.521367    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-891e67-dockercfg-9p58t\""
Sep 11 01:10:30.532421 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:30.532397    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-891e67-55d8c8cf55-j7jmn"]
Sep 11 01:10:30.571894 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:30.556421    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70a044e1-e833-4cb8-a58d-2e41dc53ec40-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-891e67-fc5bccbf4-4w5pk\" (UID: \"70a044e1-e833-4cb8-a58d-2e41dc53ec40\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-891e67-fc5bccbf4-4w5pk"
Sep 11 01:10:30.571894 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:30.556489    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28rcb\" (UniqueName: \"kubernetes.io/projected/70a044e1-e833-4cb8-a58d-2e41dc53ec40-kube-api-access-28rcb\") pod \"payload-pre-processing-e2e-inf-b-891e67-fc5bccbf4-4w5pk\" (UID: \"70a044e1-e833-4cb8-a58d-2e41dc53ec40\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-891e67-fc5bccbf4-4w5pk"
Sep 11 01:10:30.571894 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:30.556596    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5f59\" (UniqueName: \"kubernetes.io/projected/67a94c83-7633-4bec-8ad9-37f452ad8f8f-kube-api-access-d5f59\") pod \"payload-processing-e2e-inf-b-891e67-55d8c8cf55-j7jmn\" (UID: \"67a94c83-7633-4bec-8ad9-37f452ad8f8f\") " pod="openshift-ingress/payload-processing-e2e-inf-b-891e67-55d8c8cf55-j7jmn"
Sep 11 01:10:30.571894 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:30.556642    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67a94c83-7633-4bec-8ad9-37f452ad8f8f-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-891e67-55d8c8cf55-j7jmn\" (UID: \"67a94c83-7633-4bec-8ad9-37f452ad8f8f\") " pod="openshift-ingress/payload-processing-e2e-inf-b-891e67-55d8c8cf55-j7jmn"
Sep 11 01:10:30.657896 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:30.657771    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5f59\" (UniqueName: \"kubernetes.io/projected/67a94c83-7633-4bec-8ad9-37f452ad8f8f-kube-api-access-d5f59\") pod \"payload-processing-e2e-inf-b-891e67-55d8c8cf55-j7jmn\" (UID: \"67a94c83-7633-4bec-8ad9-37f452ad8f8f\") " pod="openshift-ingress/payload-processing-e2e-inf-b-891e67-55d8c8cf55-j7jmn"
Sep 11 01:10:30.657896 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:30.657850    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67a94c83-7633-4bec-8ad9-37f452ad8f8f-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-891e67-55d8c8cf55-j7jmn\" (UID: \"67a94c83-7633-4bec-8ad9-37f452ad8f8f\") " pod="openshift-ingress/payload-processing-e2e-inf-b-891e67-55d8c8cf55-j7jmn"
Sep 11 01:10:30.658441 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:30.657940    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70a044e1-e833-4cb8-a58d-2e41dc53ec40-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-891e67-fc5bccbf4-4w5pk\" (UID: \"70a044e1-e833-4cb8-a58d-2e41dc53ec40\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-891e67-fc5bccbf4-4w5pk"
Sep 11 01:10:30.658441 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:30.658005    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28rcb\" (UniqueName: \"kubernetes.io/projected/70a044e1-e833-4cb8-a58d-2e41dc53ec40-kube-api-access-28rcb\") pod \"payload-pre-processing-e2e-inf-b-891e67-fc5bccbf4-4w5pk\" (UID: \"70a044e1-e833-4cb8-a58d-2e41dc53ec40\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-891e67-fc5bccbf4-4w5pk"
Sep 11 01:10:30.661893 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:30.659081    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67a94c83-7633-4bec-8ad9-37f452ad8f8f-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-891e67-55d8c8cf55-j7jmn\" (UID: \"67a94c83-7633-4bec-8ad9-37f452ad8f8f\") " pod="openshift-ingress/payload-processing-e2e-inf-b-891e67-55d8c8cf55-j7jmn"
Sep 11 01:10:30.661893 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:30.659502    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70a044e1-e833-4cb8-a58d-2e41dc53ec40-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-891e67-fc5bccbf4-4w5pk\" (UID: \"70a044e1-e833-4cb8-a58d-2e41dc53ec40\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-891e67-fc5bccbf4-4w5pk"
Sep 11 01:10:30.672892 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:30.669822    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5f59\" (UniqueName: \"kubernetes.io/projected/67a94c83-7633-4bec-8ad9-37f452ad8f8f-kube-api-access-d5f59\") pod \"payload-processing-e2e-inf-b-891e67-55d8c8cf55-j7jmn\" (UID: \"67a94c83-7633-4bec-8ad9-37f452ad8f8f\") " pod="openshift-ingress/payload-processing-e2e-inf-b-891e67-55d8c8cf55-j7jmn"
Sep 11 01:10:30.689890 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:30.688957    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-28rcb\" (UniqueName: \"kubernetes.io/projected/70a044e1-e833-4cb8-a58d-2e41dc53ec40-kube-api-access-28rcb\") pod \"payload-pre-processing-e2e-inf-b-891e67-fc5bccbf4-4w5pk\" (UID: \"70a044e1-e833-4cb8-a58d-2e41dc53ec40\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-891e67-fc5bccbf4-4w5pk"
Sep 11 01:10:30.804570 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:30.804545    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-891e67-fc5bccbf4-4w5pk"
Sep 11 01:10:30.830451 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:30.830424    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-891e67-55d8c8cf55-j7jmn"
Sep 11 01:10:31.131653 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:31.131491    2569 patch_prober.go:28] interesting pod/e2e-ait-0bc9acd8-openshift-default-744d69d584-x6fhm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body=
Sep 11 01:10:31.131653 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:31.131560    2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-0bc9acd8-openshift-default-744d69d584-x6fhm" podUID="03848fb4-bfa2-40ce-ae94-ab4ac9837b60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused"
Sep 11 01:10:31.152896 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:31.148974    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-891e67-fc5bccbf4-4w5pk"]
Sep 11 01:10:31.224351 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:31.224324    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-891e67-55d8c8cf55-j7jmn"]
Sep 11 01:10:31.230031 ip-10-0-129-222 kubenswrapper[2569]: W0911 01:10:31.229989    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67a94c83_7633_4bec_8ad9_37f452ad8f8f.slice/crio-24ef9e102909e3bf0ff9fcc2654c2a8a13fc6534ca483de3ccccf203d945457e WatchSource:0}: Error finding container 24ef9e102909e3bf0ff9fcc2654c2a8a13fc6534ca483de3ccccf203d945457e: Status 404 returned error can't find the container with id 24ef9e102909e3bf0ff9fcc2654c2a8a13fc6534ca483de3ccccf203d945457e
Sep 11 01:10:31.299622 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:31.299599    2569 patch_prober.go:28] interesting pod/e2e-autores-dc7dcd-openshift-default-5958f7974d-457sp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body=
Sep 11 01:10:31.299735 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:31.299638    2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-dc7dcd-openshift-default-5958f7974d-457sp" podUID="8ec0f171-1bff-4174-9905-fe855ce1ad42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused"
Sep 11 01:10:31.638239 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:31.638198    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-891e67-fc5bccbf4-4w5pk" event={"ID":"70a044e1-e833-4cb8-a58d-2e41dc53ec40","Type":"ContainerStarted","Data":"c1a4e6d05f2374204cf0932b9dbb4c9ab0cb960f5d04d75c69806964415b9fcc"}
Sep 11 01:10:31.638460 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:31.638275    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-891e67-fc5bccbf4-4w5pk" event={"ID":"70a044e1-e833-4cb8-a58d-2e41dc53ec40","Type":"ContainerStarted","Data":"1519da419f76e363216ca942aba5394ab3d69dbb8fa346145a79cb286b89b25e"}
Sep 11 01:10:31.639144 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:31.639118    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-891e67-fc5bccbf4-4w5pk"
Sep 11 01:10:31.641440 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:31.641411    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-891e67-55d8c8cf55-j7jmn" event={"ID":"67a94c83-7633-4bec-8ad9-37f452ad8f8f","Type":"ContainerStarted","Data":"e214384c4fb05c9d4e21f0087cc5cfdf5f465b72d8c51dcd53d65e70746d1f7f"}
Sep 11 01:10:31.641555 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:31.641480    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-891e67-55d8c8cf55-j7jmn" event={"ID":"67a94c83-7633-4bec-8ad9-37f452ad8f8f","Type":"ContainerStarted","Data":"24ef9e102909e3bf0ff9fcc2654c2a8a13fc6534ca483de3ccccf203d945457e"}
Sep 11 01:10:31.642174 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:31.642151    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-891e67-55d8c8cf55-j7jmn"
Sep 11 01:10:31.658039 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:31.657976    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-891e67-fc5bccbf4-4w5pk" podStartSLOduration=1.6579565889999999 podStartE2EDuration="1.657956589s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 01:10:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 01:10:31.65552558 +0000 UTC m=+766.306899841" watchObservedRunningTime="2026-09-11 01:10:31.657956589 +0000 UTC m=+766.309330843"
Sep 11 01:10:31.710625 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:31.710564    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-891e67-55d8c8cf55-j7jmn" podStartSLOduration=1.7105466470000001 podStartE2EDuration="1.710546647s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 01:10:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 01:10:31.680070505 +0000 UTC m=+766.331444756" watchObservedRunningTime="2026-09-11 01:10:31.710546647 +0000 UTC m=+766.361920942"
Sep 11 01:10:31.711893 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:31.711848    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kmx5x"]
Sep 11 01:10:32.133890 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:32.133819    2569 patch_prober.go:28] interesting pod/e2e-ait-0bc9acd8-openshift-default-744d69d584-x6fhm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body=
Sep 11 01:10:32.134060 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:32.133938    2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-0bc9acd8-openshift-default-744d69d584-x6fhm" podUID="03848fb4-bfa2-40ce-ae94-ab4ac9837b60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused"
Sep 11 01:10:32.299985 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:32.299940    2569 patch_prober.go:28] interesting pod/e2e-autores-dc7dcd-openshift-default-5958f7974d-457sp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body=
Sep 11 01:10:32.300563 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:32.300481    2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-dc7dcd-openshift-default-5958f7974d-457sp" podUID="8ec0f171-1bff-4174-9905-fe855ce1ad42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused"
Sep 11 01:10:33.130765 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:33.130730    2569 patch_prober.go:28] interesting pod/e2e-ait-0bc9acd8-openshift-default-744d69d584-x6fhm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body=
Sep 11 01:10:33.131227 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:33.130802    2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-0bc9acd8-openshift-default-744d69d584-x6fhm" podUID="03848fb4-bfa2-40ce-ae94-ab4ac9837b60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused"
Sep 11 01:10:33.299362 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:33.299322    2569 patch_prober.go:28] interesting pod/e2e-autores-dc7dcd-openshift-default-5958f7974d-457sp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body=
Sep 11 01:10:33.299529 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:33.299392    2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-dc7dcd-openshift-default-5958f7974d-457sp" podUID="8ec0f171-1bff-4174-9905-fe855ce1ad42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused"
Sep 11 01:10:34.131367 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:34.131328    2569 patch_prober.go:28] interesting pod/e2e-ait-0bc9acd8-openshift-default-744d69d584-x6fhm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body=
Sep 11 01:10:34.131751 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:34.131386    2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-0bc9acd8-openshift-default-744d69d584-x6fhm" podUID="03848fb4-bfa2-40ce-ae94-ab4ac9837b60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused"
Sep 11 01:10:34.299825 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:34.299792    2569 patch_prober.go:28] interesting pod/e2e-autores-dc7dcd-openshift-default-5958f7974d-457sp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body=
Sep 11 01:10:34.300048 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:34.300004    2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-dc7dcd-openshift-default-5958f7974d-457sp" podUID="8ec0f171-1bff-4174-9905-fe855ce1ad42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused"
Sep 11 01:10:35.303722 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:35.303689    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-dc7dcd-openshift-default-5958f7974d-457sp"
Sep 11 01:10:35.304313 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:35.304286    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-dc7dcd-openshift-default-5958f7974d-457sp"
Sep 11 01:10:35.305092 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:35.305074    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-dc7dcd-openshift-default-5958f7974d-457sp"
Sep 11 01:10:35.379597 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:35.379550    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-0bc9acd8-openshift-default-744d69d584-x6fhm"
Sep 11 01:10:35.380082 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:35.380053    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-0bc9acd8-openshift-default-744d69d584-x6fhm"
Sep 11 01:10:35.381259 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:35.381235    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-0bc9acd8-openshift-default-744d69d584-x6fhm"
Sep 11 01:10:40.561779 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:40.561739    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0bc9acd8-2gz7w"]
Sep 11 01:10:40.565306 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:40.565280    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0bc9acd8-2gz7w"
Sep 11 01:10:40.567608 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:40.567585    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 01:10:40.567739 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:40.567625    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pppkt\""
Sep 11 01:10:40.575603 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:40.575570    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0bc9acd8-2gz7w"]
Sep 11 01:10:40.683894 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:40.683834    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/098e6596-9a52-48e6-91ce-69762f83ef78-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-0bc9acd8-2gz7w\" (UID: \"098e6596-9a52-48e6-91ce-69762f83ef78\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0bc9acd8-2gz7w"
Sep 11 01:10:40.784684 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:40.784632    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/098e6596-9a52-48e6-91ce-69762f83ef78-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-0bc9acd8-2gz7w\" (UID: \"098e6596-9a52-48e6-91ce-69762f83ef78\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0bc9acd8-2gz7w"
Sep 11 01:10:40.785326 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:40.785306    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/098e6596-9a52-48e6-91ce-69762f83ef78-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-0bc9acd8-2gz7w\" (UID: \"098e6596-9a52-48e6-91ce-69762f83ef78\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0bc9acd8-2gz7w"
Sep 11 01:10:40.876847 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:40.876759    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0bc9acd8-2gz7w"
Sep 11 01:10:41.172193 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:41.172169    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0bc9acd8-2gz7w"]
Sep 11 01:10:41.174051 ip-10-0-129-222 kubenswrapper[2569]: W0911 01:10:41.174007    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod098e6596_9a52_48e6_91ce_69762f83ef78.slice/crio-766beda780f363359290d7c83039d639c00891fe7ac5379e05b382079ff10f48 WatchSource:0}: Error finding container 766beda780f363359290d7c83039d639c00891fe7ac5379e05b382079ff10f48: Status 404 returned error can't find the container with id 766beda780f363359290d7c83039d639c00891fe7ac5379e05b382079ff10f48
Sep 11 01:10:41.707754 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:41.707720    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0bc9acd8-2gz7w" event={"ID":"098e6596-9a52-48e6-91ce-69762f83ef78","Type":"ContainerStarted","Data":"766beda780f363359290d7c83039d639c00891fe7ac5379e05b382079ff10f48"}
Sep 11 01:10:43.501386 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:43.501354    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kmx5x"]
Sep 11 01:10:43.664839 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:43.664810    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-891e67-55d8c8cf55-j7jmn"
Sep 11 01:10:43.665127 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:43.664908    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-891e67-fc5bccbf4-4w5pk"
Sep 11 01:10:44.726868 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:44.726783    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0bc9acd8-2gz7w" event={"ID":"098e6596-9a52-48e6-91ce-69762f83ef78","Type":"ContainerStarted","Data":"a10f093132f3d6707bd15636c1bfcea4dfd863d26e754f8ca415924dfc1b368c"}
Sep 11 01:10:44.744365 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:44.744320    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0bc9acd8-2gz7w" podStartSLOduration=1.928283864 podStartE2EDuration="4.744303326s" totalImagesPullingTime="2.816019462s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 01:10:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 01:10:44.7415535 +0000 UTC m=+779.392927750" watchObservedRunningTime="2026-09-11 01:10:44.744303326 +0000 UTC m=+779.395677578"
Sep 11 01:10:45.733582 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:45.733550    2569 generic.go:356] "Generic (PLEG): container finished" podID="098e6596-9a52-48e6-91ce-69762f83ef78" containerID="a10f093132f3d6707bd15636c1bfcea4dfd863d26e754f8ca415924dfc1b368c" exitCode=0
Sep 11 01:10:45.734015 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:45.733634    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0bc9acd8-2gz7w" event={"ID":"098e6596-9a52-48e6-91ce-69762f83ef78","Type":"ContainerDied","Data":"a10f093132f3d6707bd15636c1bfcea4dfd863d26e754f8ca415924dfc1b368c"}
Sep 11 01:10:46.883654 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:46.883627    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0bc9acd8-2gz7w"
Sep 11 01:10:47.052166 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:47.052140    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/098e6596-9a52-48e6-91ce-69762f83ef78-maas-api-service-ca\") pod \"098e6596-9a52-48e6-91ce-69762f83ef78\" (UID: \"098e6596-9a52-48e6-91ce-69762f83ef78\") "
Sep 11 01:10:47.052526 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:47.052496    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/098e6596-9a52-48e6-91ce-69762f83ef78-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "098e6596-9a52-48e6-91ce-69762f83ef78" (UID: "098e6596-9a52-48e6-91ce-69762f83ef78"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 01:10:47.052724 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:47.052705    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/098e6596-9a52-48e6-91ce-69762f83ef78-maas-api-service-ca\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:10:47.745232 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:47.745158    2569 generic.go:356] "Generic (PLEG): container finished" podID="2b6bb107-1129-43c9-aa58-25cd7c70bfbf" containerID="8a84288b9a80622c7d651d301fd4c8a9eb292140b04df4f1b33521cb7801710f" exitCode=0
Sep 11 01:10:47.745414 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:47.745230    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-891e67-openshift-default-564fcdf9fb-xw8kt" event={"ID":"2b6bb107-1129-43c9-aa58-25cd7c70bfbf","Type":"ContainerDied","Data":"8a84288b9a80622c7d651d301fd4c8a9eb292140b04df4f1b33521cb7801710f"}
Sep 11 01:10:47.745733 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:47.745716    2569 scope.go:117] "RemoveContainer" containerID="8a84288b9a80622c7d651d301fd4c8a9eb292140b04df4f1b33521cb7801710f"
Sep 11 01:10:47.746679 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:47.746659    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0bc9acd8-2gz7w"
Sep 11 01:10:47.746753 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:47.746680    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0bc9acd8-2gz7w" event={"ID":"098e6596-9a52-48e6-91ce-69762f83ef78","Type":"ContainerDied","Data":"766beda780f363359290d7c83039d639c00891fe7ac5379e05b382079ff10f48"}
Sep 11 01:10:47.746753 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:47.746719    2569 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="766beda780f363359290d7c83039d639c00891fe7ac5379e05b382079ff10f48"
Sep 11 01:10:47.747113 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:47.747077    2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 01:10:47.747185 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:47.747168    2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 01:10:47.747232 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:47.747216    2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 01:10:48.757910 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:48.757855    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-891e67-openshift-default-564fcdf9fb-xw8kt" event={"ID":"2b6bb107-1129-43c9-aa58-25cd7c70bfbf","Type":"ContainerStarted","Data":"d9cf58c9d35182fe0447734cffe15a3d7d83220501297c6b56bb6f36589f5666"}
Sep 11 01:10:49.141995 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:49.141538    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-891e67-openshift-default-564fcdf9fb-xw8kt"
Sep 11 01:10:49.141995 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:49.141665    2569 patch_prober.go:28] interesting pod/e2e-inf-b-891e67-openshift-default-564fcdf9fb-xw8kt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body=
Sep 11 01:10:49.141995 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:49.141709    2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-891e67-openshift-default-564fcdf9fb-xw8kt" podUID="2b6bb107-1129-43c9-aa58-25cd7c70bfbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused"
Sep 11 01:10:50.141664 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:50.141623    2569 patch_prober.go:28] interesting pod/e2e-inf-b-891e67-openshift-default-564fcdf9fb-xw8kt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body=
Sep 11 01:10:50.142281 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:50.142239    2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-891e67-openshift-default-564fcdf9fb-xw8kt" podUID="2b6bb107-1129-43c9-aa58-25cd7c70bfbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused"
Sep 11 01:10:51.141770 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:51.141722    2569 patch_prober.go:28] interesting pod/e2e-inf-b-891e67-openshift-default-564fcdf9fb-xw8kt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body=
Sep 11 01:10:51.142292 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:51.141794    2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-891e67-openshift-default-564fcdf9fb-xw8kt" podUID="2b6bb107-1129-43c9-aa58-25cd7c70bfbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused"
Sep 11 01:10:52.141391 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:52.141355    2569 patch_prober.go:28] interesting pod/e2e-inf-b-891e67-openshift-default-564fcdf9fb-xw8kt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body=
Sep 11 01:10:52.141671 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:52.141644    2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-891e67-openshift-default-564fcdf9fb-xw8kt" podUID="2b6bb107-1129-43c9-aa58-25cd7c70bfbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused"
Sep 11 01:10:52.859970 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:52.859924    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0bc9acd8-2gz7w"]
Sep 11 01:10:52.861841 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:52.861814    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0bc9acd8-2gz7w"]
Sep 11 01:10:52.999956 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:52.999907    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-0bc9acd8-openshift-default-744d69d584-x6fhm"]
Sep 11 01:10:53.000224 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:53.000195    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-0bc9acd8-openshift-default-744d69d584-x6fhm" podUID="03848fb4-bfa2-40ce-ae94-ab4ac9837b60" containerName="istio-proxy" containerID="cri-o://66177f3ee7684b7f932ebbbe00661e9ac96a1609e4cc1a6b307376918273cdba" gracePeriod=30
Sep 11 01:10:53.141854 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:53.141752    2569 patch_prober.go:28] interesting pod/e2e-inf-b-891e67-openshift-default-564fcdf9fb-xw8kt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body=
Sep 11 01:10:53.141854 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:53.141822    2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-891e67-openshift-default-564fcdf9fb-xw8kt" podUID="2b6bb107-1129-43c9-aa58-25cd7c70bfbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused"
Sep 11 01:10:53.871157 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:53.871127    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="098e6596-9a52-48e6-91ce-69762f83ef78" path="/var/lib/kubelet/pods/098e6596-9a52-48e6-91ce-69762f83ef78/volumes"
Sep 11 01:10:54.183865 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:54.183774    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-891e67-openshift-default-564fcdf9fb-xw8kt"
Sep 11 01:10:54.185363 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:54.185323    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-891e67-openshift-default-564fcdf9fb-xw8kt"
Sep 11 01:10:54.192543 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:54.192514    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-891e67-openshift-default-564fcdf9fb-xw8kt"
Sep 11 01:10:57.900033 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:57.900002    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-1d3c7cc9-fd48988c8-65qvp"]
Sep 11 01:10:57.900477 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:57.900463    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="098e6596-9a52-48e6-91ce-69762f83ef78" containerName="revoke-keys"
Sep 11 01:10:57.900529 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:57.900481    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="098e6596-9a52-48e6-91ce-69762f83ef78" containerName="revoke-keys"
Sep 11 01:10:57.900577 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:57.900564    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="098e6596-9a52-48e6-91ce-69762f83ef78" containerName="revoke-keys"
Sep 11 01:10:57.908380 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:57.908359    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-1d3c7cc9-fd48988c8-65qvp"
Sep 11 01:10:57.910450 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:57.910428    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-1d3c7cc9-dockercfg-xtnj8\""
Sep 11 01:10:57.910574 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:57.910437    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-1d3c7cc9\""
Sep 11 01:10:57.914766 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:57.914743    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-1d3c7cc9-fd48988c8-65qvp"]
Sep 11 01:10:57.961534 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:57.961496    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17e94e18-1d22-4ed1-94e1-f1020470a635-plugins-config-volume\") pod \"payload-processing-e2e-migrate-1d3c7cc9-fd48988c8-65qvp\" (UID: \"17e94e18-1d22-4ed1-94e1-f1020470a635\") " pod="openshift-ingress/payload-processing-e2e-migrate-1d3c7cc9-fd48988c8-65qvp"
Sep 11 01:10:57.961859 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:57.961829    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85l49\" (UniqueName: \"kubernetes.io/projected/17e94e18-1d22-4ed1-94e1-f1020470a635-kube-api-access-85l49\") pod \"payload-processing-e2e-migrate-1d3c7cc9-fd48988c8-65qvp\" (UID: \"17e94e18-1d22-4ed1-94e1-f1020470a635\") " pod="openshift-ingress/payload-processing-e2e-migrate-1d3c7cc9-fd48988c8-65qvp"
Sep 11 01:10:58.062982 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:58.062948    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17e94e18-1d22-4ed1-94e1-f1020470a635-plugins-config-volume\") pod \"payload-processing-e2e-migrate-1d3c7cc9-fd48988c8-65qvp\" (UID: \"17e94e18-1d22-4ed1-94e1-f1020470a635\") " pod="openshift-ingress/payload-processing-e2e-migrate-1d3c7cc9-fd48988c8-65qvp"
Sep 11 01:10:58.063110 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:58.063020    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85l49\" (UniqueName: \"kubernetes.io/projected/17e94e18-1d22-4ed1-94e1-f1020470a635-kube-api-access-85l49\") pod \"payload-processing-e2e-migrate-1d3c7cc9-fd48988c8-65qvp\" (UID: \"17e94e18-1d22-4ed1-94e1-f1020470a635\") " pod="openshift-ingress/payload-processing-e2e-migrate-1d3c7cc9-fd48988c8-65qvp"
Sep 11 01:10:58.063643 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:58.063622    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17e94e18-1d22-4ed1-94e1-f1020470a635-plugins-config-volume\") pod \"payload-processing-e2e-migrate-1d3c7cc9-fd48988c8-65qvp\" (UID: \"17e94e18-1d22-4ed1-94e1-f1020470a635\") " pod="openshift-ingress/payload-processing-e2e-migrate-1d3c7cc9-fd48988c8-65qvp"
Sep 11 01:10:58.071489 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:58.071462    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-85l49\" (UniqueName: \"kubernetes.io/projected/17e94e18-1d22-4ed1-94e1-f1020470a635-kube-api-access-85l49\") pod \"payload-processing-e2e-migrate-1d3c7cc9-fd48988c8-65qvp\" (UID: \"17e94e18-1d22-4ed1-94e1-f1020470a635\") " pod="openshift-ingress/payload-processing-e2e-migrate-1d3c7cc9-fd48988c8-65qvp"
Sep 11 01:10:58.221072 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:58.221042    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-1d3c7cc9-fd48988c8-65qvp"
Sep 11 01:10:58.265114 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:58.264738    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-0bc9acd8-openshift-default-744d69d584-x6fhm"
Sep 11 01:10:58.366432 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:58.366397    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03848fb4-bfa2-40ce-ae94-ab4ac9837b60-workload-socket\") pod \"03848fb4-bfa2-40ce-ae94-ab4ac9837b60\" (UID: \"03848fb4-bfa2-40ce-ae94-ab4ac9837b60\") "
Sep 11 01:10:58.366432 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:58.366446    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03848fb4-bfa2-40ce-ae94-ab4ac9837b60-istio-token\") pod \"03848fb4-bfa2-40ce-ae94-ab4ac9837b60\" (UID: \"03848fb4-bfa2-40ce-ae94-ab4ac9837b60\") "
Sep 11 01:10:58.366641 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:58.366479    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03848fb4-bfa2-40ce-ae94-ab4ac9837b60-istio-envoy\") pod \"03848fb4-bfa2-40ce-ae94-ab4ac9837b60\" (UID: \"03848fb4-bfa2-40ce-ae94-ab4ac9837b60\") "
Sep 11 01:10:58.366641 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:58.366515    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2hbnc\" (UniqueName: \"kubernetes.io/projected/03848fb4-bfa2-40ce-ae94-ab4ac9837b60-kube-api-access-2hbnc\") pod \"03848fb4-bfa2-40ce-ae94-ab4ac9837b60\" (UID: \"03848fb4-bfa2-40ce-ae94-ab4ac9837b60\") "
Sep 11 01:10:58.366641 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:58.366576    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03848fb4-bfa2-40ce-ae94-ab4ac9837b60-istio-podinfo\") pod \"03848fb4-bfa2-40ce-ae94-ab4ac9837b60\" (UID: \"03848fb4-bfa2-40ce-ae94-ab4ac9837b60\") "
Sep 11 01:10:58.366641 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:58.366603    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03848fb4-bfa2-40ce-ae94-ab4ac9837b60-credential-socket\") pod \"03848fb4-bfa2-40ce-ae94-ab4ac9837b60\" (UID: \"03848fb4-bfa2-40ce-ae94-ab4ac9837b60\") "
Sep 11 01:10:58.366831 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:58.366693    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03848fb4-bfa2-40ce-ae94-ab4ac9837b60-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "03848fb4-bfa2-40ce-ae94-ab4ac9837b60" (UID: "03848fb4-bfa2-40ce-ae94-ab4ac9837b60"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 01:10:58.366831 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:58.366755    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03848fb4-bfa2-40ce-ae94-ab4ac9837b60-istio-data\") pod \"03848fb4-bfa2-40ce-ae94-ab4ac9837b60\" (UID: \"03848fb4-bfa2-40ce-ae94-ab4ac9837b60\") "
Sep 11 01:10:58.366831 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:58.366815    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03848fb4-bfa2-40ce-ae94-ab4ac9837b60-workload-certs\") pod \"03848fb4-bfa2-40ce-ae94-ab4ac9837b60\" (UID: \"03848fb4-bfa2-40ce-ae94-ab4ac9837b60\") "
Sep 11 01:10:58.367026 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:58.366866    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03848fb4-bfa2-40ce-ae94-ab4ac9837b60-istiod-ca-cert\") pod \"03848fb4-bfa2-40ce-ae94-ab4ac9837b60\" (UID: \"03848fb4-bfa2-40ce-ae94-ab4ac9837b60\") "
Sep 11 01:10:58.367026 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:58.366949    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03848fb4-bfa2-40ce-ae94-ab4ac9837b60-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "03848fb4-bfa2-40ce-ae94-ab4ac9837b60" (UID: "03848fb4-bfa2-40ce-ae94-ab4ac9837b60"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 01:10:58.367147 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:58.367029    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03848fb4-bfa2-40ce-ae94-ab4ac9837b60-istio-data" (OuterVolumeSpecName: "istio-data") pod "03848fb4-bfa2-40ce-ae94-ab4ac9837b60" (UID: "03848fb4-bfa2-40ce-ae94-ab4ac9837b60"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 01:10:58.367147 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:58.367126    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03848fb4-bfa2-40ce-ae94-ab4ac9837b60-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "03848fb4-bfa2-40ce-ae94-ab4ac9837b60" (UID: "03848fb4-bfa2-40ce-ae94-ab4ac9837b60"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 01:10:58.367329 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:58.367309    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03848fb4-bfa2-40ce-ae94-ab4ac9837b60-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "03848fb4-bfa2-40ce-ae94-ab4ac9837b60" (UID: "03848fb4-bfa2-40ce-ae94-ab4ac9837b60"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 01:10:58.367405 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:58.367373    2569 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03848fb4-bfa2-40ce-ae94-ab4ac9837b60-istio-data\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:10:58.367405 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:58.367387    2569 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03848fb4-bfa2-40ce-ae94-ab4ac9837b60-workload-certs\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:10:58.367405 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:58.367395    2569 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03848fb4-bfa2-40ce-ae94-ab4ac9837b60-workload-socket\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:10:58.367405 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:58.367404    2569 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03848fb4-bfa2-40ce-ae94-ab4ac9837b60-credential-socket\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:10:58.368940 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:58.368907    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03848fb4-bfa2-40ce-ae94-ab4ac9837b60-kube-api-access-2hbnc" (OuterVolumeSpecName: "kube-api-access-2hbnc") pod "03848fb4-bfa2-40ce-ae94-ab4ac9837b60" (UID: "03848fb4-bfa2-40ce-ae94-ab4ac9837b60"). InnerVolumeSpecName "kube-api-access-2hbnc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 01:10:58.369041 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:58.368984    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03848fb4-bfa2-40ce-ae94-ab4ac9837b60-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "03848fb4-bfa2-40ce-ae94-ab4ac9837b60" (UID: "03848fb4-bfa2-40ce-ae94-ab4ac9837b60"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 01:10:58.369161 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:58.369125    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03848fb4-bfa2-40ce-ae94-ab4ac9837b60-istio-token" (OuterVolumeSpecName: "istio-token") pod "03848fb4-bfa2-40ce-ae94-ab4ac9837b60" (UID: "03848fb4-bfa2-40ce-ae94-ab4ac9837b60"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 01:10:58.369239 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:58.369189    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/03848fb4-bfa2-40ce-ae94-ab4ac9837b60-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "03848fb4-bfa2-40ce-ae94-ab4ac9837b60" (UID: "03848fb4-bfa2-40ce-ae94-ab4ac9837b60"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 01:10:58.468042 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:58.467978    2569 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03848fb4-bfa2-40ce-ae94-ab4ac9837b60-istio-token\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:10:58.468042 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:58.468003    2569 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03848fb4-bfa2-40ce-ae94-ab4ac9837b60-istio-envoy\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:10:58.468042 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:58.468013    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2hbnc\" (UniqueName: \"kubernetes.io/projected/03848fb4-bfa2-40ce-ae94-ab4ac9837b60-kube-api-access-2hbnc\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:10:58.468042 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:58.468023    2569 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03848fb4-bfa2-40ce-ae94-ab4ac9837b60-istio-podinfo\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:10:58.468042 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:58.468032    2569 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03848fb4-bfa2-40ce-ae94-ab4ac9837b60-istiod-ca-cert\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:10:58.497515 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:58.497494    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-1d3c7cc9-fd48988c8-65qvp"]
Sep 11 01:10:58.499640 ip-10-0-129-222 kubenswrapper[2569]: W0911 01:10:58.499615    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17e94e18_1d22_4ed1_94e1_f1020470a635.slice/crio-770a0b93d97b4e33ca8087995bf7f7d9fee3fb65fa78117329368a54570670c3 WatchSource:0}: Error finding container 770a0b93d97b4e33ca8087995bf7f7d9fee3fb65fa78117329368a54570670c3: Status 404 returned error can't find the container with id 770a0b93d97b4e33ca8087995bf7f7d9fee3fb65fa78117329368a54570670c3
Sep 11 01:10:58.816452 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:58.816419    2569 generic.go:356] "Generic (PLEG): container finished" podID="03848fb4-bfa2-40ce-ae94-ab4ac9837b60" containerID="66177f3ee7684b7f932ebbbe00661e9ac96a1609e4cc1a6b307376918273cdba" exitCode=0
Sep 11 01:10:58.816636 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:58.816475    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-0bc9acd8-openshift-default-744d69d584-x6fhm"
Sep 11 01:10:58.816636 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:58.816494    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0bc9acd8-openshift-default-744d69d584-x6fhm" event={"ID":"03848fb4-bfa2-40ce-ae94-ab4ac9837b60","Type":"ContainerDied","Data":"66177f3ee7684b7f932ebbbe00661e9ac96a1609e4cc1a6b307376918273cdba"}
Sep 11 01:10:58.816636 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:58.816547    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0bc9acd8-openshift-default-744d69d584-x6fhm" event={"ID":"03848fb4-bfa2-40ce-ae94-ab4ac9837b60","Type":"ContainerDied","Data":"9ceb0bd7e17734a7e8386d6dd9f58da3133dd1ad60e951cc5cdcd2f4cd570af3"}
Sep 11 01:10:58.816636 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:58.816573    2569 scope.go:117] "RemoveContainer" containerID="66177f3ee7684b7f932ebbbe00661e9ac96a1609e4cc1a6b307376918273cdba"
Sep 11 01:10:58.818335 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:58.818307    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-1d3c7cc9-fd48988c8-65qvp" event={"ID":"17e94e18-1d22-4ed1-94e1-f1020470a635","Type":"ContainerStarted","Data":"8e7a6cb37dff6d06622056ef5d675871024a0f50ce1c37bcb37dce990239d984"}
Sep 11 01:10:58.818462 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:58.818359    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-1d3c7cc9-fd48988c8-65qvp" event={"ID":"17e94e18-1d22-4ed1-94e1-f1020470a635","Type":"ContainerStarted","Data":"770a0b93d97b4e33ca8087995bf7f7d9fee3fb65fa78117329368a54570670c3"}
Sep 11 01:10:58.818462 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:58.818414    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-1d3c7cc9-fd48988c8-65qvp"
Sep 11 01:10:58.827644 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:58.827625    2569 scope.go:117] "RemoveContainer" containerID="813382bc441f6727902aa5f80216c47bcbbb716711293edc312e69b191e50d63"
Sep 11 01:10:58.836580 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:58.836558    2569 scope.go:117] "RemoveContainer" containerID="66177f3ee7684b7f932ebbbe00661e9ac96a1609e4cc1a6b307376918273cdba"
Sep 11 01:10:58.836837 ip-10-0-129-222 kubenswrapper[2569]: E0911 01:10:58.836819    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66177f3ee7684b7f932ebbbe00661e9ac96a1609e4cc1a6b307376918273cdba\": container with ID starting with 66177f3ee7684b7f932ebbbe00661e9ac96a1609e4cc1a6b307376918273cdba not found: ID does not exist" containerID="66177f3ee7684b7f932ebbbe00661e9ac96a1609e4cc1a6b307376918273cdba"
Sep 11 01:10:58.836959 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:58.836848    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66177f3ee7684b7f932ebbbe00661e9ac96a1609e4cc1a6b307376918273cdba"} err="failed to get container status \"66177f3ee7684b7f932ebbbe00661e9ac96a1609e4cc1a6b307376918273cdba\": rpc error: code = NotFound desc = could not find container \"66177f3ee7684b7f932ebbbe00661e9ac96a1609e4cc1a6b307376918273cdba\": container with ID starting with 66177f3ee7684b7f932ebbbe00661e9ac96a1609e4cc1a6b307376918273cdba not found: ID does not exist"
Sep 11 01:10:58.836959 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:58.836899    2569 scope.go:117] "RemoveContainer" containerID="813382bc441f6727902aa5f80216c47bcbbb716711293edc312e69b191e50d63"
Sep 11 01:10:58.837160 ip-10-0-129-222 kubenswrapper[2569]: E0911 01:10:58.837136    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"813382bc441f6727902aa5f80216c47bcbbb716711293edc312e69b191e50d63\": container with ID starting with 813382bc441f6727902aa5f80216c47bcbbb716711293edc312e69b191e50d63 not found: ID does not exist" containerID="813382bc441f6727902aa5f80216c47bcbbb716711293edc312e69b191e50d63"
Sep 11 01:10:58.837203 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:58.837169    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"813382bc441f6727902aa5f80216c47bcbbb716711293edc312e69b191e50d63"} err="failed to get container status \"813382bc441f6727902aa5f80216c47bcbbb716711293edc312e69b191e50d63\": rpc error: code = NotFound desc = could not find container \"813382bc441f6727902aa5f80216c47bcbbb716711293edc312e69b191e50d63\": container with ID starting with 813382bc441f6727902aa5f80216c47bcbbb716711293edc312e69b191e50d63 not found: ID does not exist"
Sep 11 01:10:58.837242 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:58.837182    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-1d3c7cc9-fd48988c8-65qvp" podStartSLOduration=1.8371683349999999 podStartE2EDuration="1.837168335s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 01:10:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 01:10:58.835699022 +0000 UTC m=+793.487073292" watchObservedRunningTime="2026-09-11 01:10:58.837168335 +0000 UTC m=+793.488542584"
Sep 11 01:10:58.850469 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:58.850442    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-0bc9acd8-openshift-default-744d69d584-x6fhm"]
Sep 11 01:10:58.857727 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:58.857702    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-0bc9acd8-openshift-default-744d69d584-x6fhm"]
Sep 11 01:10:59.310883 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:59.310836    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kmx5x"]
Sep 11 01:10:59.872230 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:10:59.872197    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="03848fb4-bfa2-40ce-ae94-ab4ac9837b60" path="/var/lib/kubelet/pods/03848fb4-bfa2-40ce-ae94-ab4ac9837b60/volumes"
Sep 11 01:11:09.828125 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:11:09.828040    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-1d3c7cc9-fd48988c8-65qvp"
Sep 11 01:11:15.594660 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:11:15.594624    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kmx5x"]
Sep 11 01:11:21.977910 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:11:21.977863    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1d3c7cc9-4rs8z"]
Sep 11 01:11:21.978339 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:11:21.978325    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="03848fb4-bfa2-40ce-ae94-ab4ac9837b60" containerName="istio-proxy"
Sep 11 01:11:21.978393 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:11:21.978340    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="03848fb4-bfa2-40ce-ae94-ab4ac9837b60" containerName="istio-proxy"
Sep 11 01:11:21.978393 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:11:21.978357    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="03848fb4-bfa2-40ce-ae94-ab4ac9837b60" containerName="istio-proxy"
Sep 11 01:11:21.978393 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:11:21.978362    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="03848fb4-bfa2-40ce-ae94-ab4ac9837b60" containerName="istio-proxy"
Sep 11 01:11:21.978498 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:11:21.978428    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="03848fb4-bfa2-40ce-ae94-ab4ac9837b60" containerName="istio-proxy"
Sep 11 01:11:21.978498 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:11:21.978438    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="03848fb4-bfa2-40ce-ae94-ab4ac9837b60" containerName="istio-proxy"
Sep 11 01:11:21.981715 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:11:21.981698    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1d3c7cc9-4rs8z"
Sep 11 01:11:21.983885 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:11:21.983854    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pppkt\""
Sep 11 01:11:21.983981 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:11:21.983911    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 01:11:21.988785 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:11:21.988751    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1d3c7cc9-4rs8z"]
Sep 11 01:11:22.083058 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:11:22.083029    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/35bb2587-d49a-41b5-aa73-554e3b9f2073-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-1d3c7cc9-4rs8z\" (UID: \"35bb2587-d49a-41b5-aa73-554e3b9f2073\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1d3c7cc9-4rs8z"
Sep 11 01:11:22.184311 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:11:22.184279    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/35bb2587-d49a-41b5-aa73-554e3b9f2073-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-1d3c7cc9-4rs8z\" (UID: \"35bb2587-d49a-41b5-aa73-554e3b9f2073\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1d3c7cc9-4rs8z"
Sep 11 01:11:22.184921 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:11:22.184903    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/35bb2587-d49a-41b5-aa73-554e3b9f2073-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-1d3c7cc9-4rs8z\" (UID: \"35bb2587-d49a-41b5-aa73-554e3b9f2073\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1d3c7cc9-4rs8z"
Sep 11 01:11:22.294505 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:11:22.294441    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1d3c7cc9-4rs8z"
Sep 11 01:11:22.564392 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:11:22.564370    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1d3c7cc9-4rs8z"]
Sep 11 01:11:22.566984 ip-10-0-129-222 kubenswrapper[2569]: W0911 01:11:22.566961    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35bb2587_d49a_41b5_aa73_554e3b9f2073.slice/crio-3d6d4ce6b3401b21ac0248f4cabdeb0ad86a9e08839f60b533d67d7c7f107d7b WatchSource:0}: Error finding container 3d6d4ce6b3401b21ac0248f4cabdeb0ad86a9e08839f60b533d67d7c7f107d7b: Status 404 returned error can't find the container with id 3d6d4ce6b3401b21ac0248f4cabdeb0ad86a9e08839f60b533d67d7c7f107d7b
Sep 11 01:11:22.947471 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:11:22.947440    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1d3c7cc9-4rs8z" event={"ID":"35bb2587-d49a-41b5-aa73-554e3b9f2073","Type":"ContainerStarted","Data":"f10b765d3c713facd1626eb056fe21a57fdd621053f68b5a10b72293b351b9fa"}
Sep 11 01:11:22.947613 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:11:22.947491    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1d3c7cc9-4rs8z" event={"ID":"35bb2587-d49a-41b5-aa73-554e3b9f2073","Type":"ContainerStarted","Data":"3d6d4ce6b3401b21ac0248f4cabdeb0ad86a9e08839f60b533d67d7c7f107d7b"}
Sep 11 01:11:22.968425 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:11:22.968378    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1d3c7cc9-4rs8z" podStartSLOduration=1.9683652839999999 podStartE2EDuration="1.968365284s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 01:11:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 01:11:22.965971768 +0000 UTC m=+817.617346019" watchObservedRunningTime="2026-09-11 01:11:22.968365284 +0000 UTC m=+817.619739533"
Sep 11 01:11:23.953450 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:11:23.953415    2569 generic.go:356] "Generic (PLEG): container finished" podID="35bb2587-d49a-41b5-aa73-554e3b9f2073" containerID="f10b765d3c713facd1626eb056fe21a57fdd621053f68b5a10b72293b351b9fa" exitCode=0
Sep 11 01:11:23.953845 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:11:23.953488    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1d3c7cc9-4rs8z" event={"ID":"35bb2587-d49a-41b5-aa73-554e3b9f2073","Type":"ContainerDied","Data":"f10b765d3c713facd1626eb056fe21a57fdd621053f68b5a10b72293b351b9fa"}
Sep 11 01:11:25.110500 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:11:25.110476    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1d3c7cc9-4rs8z"
Sep 11 01:11:25.211640 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:11:25.211571    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/35bb2587-d49a-41b5-aa73-554e3b9f2073-maas-api-service-ca\") pod \"35bb2587-d49a-41b5-aa73-554e3b9f2073\" (UID: \"35bb2587-d49a-41b5-aa73-554e3b9f2073\") "
Sep 11 01:11:25.211930 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:11:25.211909    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35bb2587-d49a-41b5-aa73-554e3b9f2073-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "35bb2587-d49a-41b5-aa73-554e3b9f2073" (UID: "35bb2587-d49a-41b5-aa73-554e3b9f2073"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 01:11:25.312600 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:11:25.312573    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/35bb2587-d49a-41b5-aa73-554e3b9f2073-maas-api-service-ca\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:11:25.965338 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:11:25.965308    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1d3c7cc9-4rs8z" event={"ID":"35bb2587-d49a-41b5-aa73-554e3b9f2073","Type":"ContainerDied","Data":"3d6d4ce6b3401b21ac0248f4cabdeb0ad86a9e08839f60b533d67d7c7f107d7b"}
Sep 11 01:11:25.965338 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:11:25.965332    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1d3c7cc9-4rs8z"
Sep 11 01:11:25.965519 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:11:25.965354    2569 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3d6d4ce6b3401b21ac0248f4cabdeb0ad86a9e08839f60b533d67d7c7f107d7b"
Sep 11 01:11:27.694936 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:11:27.694898    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kmx5x"]
Sep 11 01:11:27.784902 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:11:27.784844    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-1d3c7cc9-fd48988c8-65qvp"]
Sep 11 01:11:27.785098 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:11:27.785066    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-1d3c7cc9-fd48988c8-65qvp" podUID="17e94e18-1d22-4ed1-94e1-f1020470a635" containerName="payload-processing" containerID="cri-o://8e7a6cb37dff6d06622056ef5d675871024a0f50ce1c37bcb37dce990239d984" gracePeriod=30
Sep 11 01:11:27.807393 ip-10-0-129-222 kubenswrapper[2569]: E0911 01:11:27.807366    2569 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17e94e18_1d22_4ed1_94e1_f1020470a635.slice/crio-8e7a6cb37dff6d06622056ef5d675871024a0f50ce1c37bcb37dce990239d984.scope\": RecentStats: unable to find data in memory cache]"
Sep 11 01:11:27.963023 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:11:27.962992    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1d3c7cc9-4rs8z"]
Sep 11 01:11:27.968435 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:11:27.968407    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1d3c7cc9-4rs8z"]
Sep 11 01:11:27.979837 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:11:27.979812    2569 generic.go:356] "Generic (PLEG): container finished" podID="17e94e18-1d22-4ed1-94e1-f1020470a635" containerID="8e7a6cb37dff6d06622056ef5d675871024a0f50ce1c37bcb37dce990239d984" exitCode=0
Sep 11 01:11:27.979968 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:11:27.979910    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-1d3c7cc9-fd48988c8-65qvp" event={"ID":"17e94e18-1d22-4ed1-94e1-f1020470a635","Type":"ContainerDied","Data":"8e7a6cb37dff6d06622056ef5d675871024a0f50ce1c37bcb37dce990239d984"}
Sep 11 01:11:28.047770 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:11:28.047735    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-1d3c7cc9-fd48988c8-65qvp"
Sep 11 01:11:28.137133 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:11:28.137101    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-85l49\" (UniqueName: \"kubernetes.io/projected/17e94e18-1d22-4ed1-94e1-f1020470a635-kube-api-access-85l49\") pod \"17e94e18-1d22-4ed1-94e1-f1020470a635\" (UID: \"17e94e18-1d22-4ed1-94e1-f1020470a635\") "
Sep 11 01:11:28.137317 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:11:28.137156    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17e94e18-1d22-4ed1-94e1-f1020470a635-plugins-config-volume\") pod \"17e94e18-1d22-4ed1-94e1-f1020470a635\" (UID: \"17e94e18-1d22-4ed1-94e1-f1020470a635\") "
Sep 11 01:11:28.137532 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:11:28.137505    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17e94e18-1d22-4ed1-94e1-f1020470a635-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "17e94e18-1d22-4ed1-94e1-f1020470a635" (UID: "17e94e18-1d22-4ed1-94e1-f1020470a635"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 01:11:28.139211 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:11:28.139180    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17e94e18-1d22-4ed1-94e1-f1020470a635-kube-api-access-85l49" (OuterVolumeSpecName: "kube-api-access-85l49") pod "17e94e18-1d22-4ed1-94e1-f1020470a635" (UID: "17e94e18-1d22-4ed1-94e1-f1020470a635"). InnerVolumeSpecName "kube-api-access-85l49". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 01:11:28.238817 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:11:28.238733    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-85l49\" (UniqueName: \"kubernetes.io/projected/17e94e18-1d22-4ed1-94e1-f1020470a635-kube-api-access-85l49\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:11:28.238817 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:11:28.238764    2569 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17e94e18-1d22-4ed1-94e1-f1020470a635-plugins-config-volume\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:11:28.866826 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:11:28.866797    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-aba078c2-openshift-default-6c4db56c6d-xr2wt"]
Sep 11 01:11:28.867246 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:11:28.867229    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17e94e18-1d22-4ed1-94e1-f1020470a635" containerName="payload-processing"
Sep 11 01:11:28.867334 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:11:28.867250    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="17e94e18-1d22-4ed1-94e1-f1020470a635" containerName="payload-processing"
Sep 11 01:11:28.867334 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:11:28.867265    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="35bb2587-d49a-41b5-aa73-554e3b9f2073" containerName="revoke-keys"
Sep 11 01:11:28.867334 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:11:28.867274    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="35bb2587-d49a-41b5-aa73-554e3b9f2073" containerName="revoke-keys"
Sep 11 01:11:28.867491 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:11:28.867393    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="35bb2587-d49a-41b5-aa73-554e3b9f2073" containerName="revoke-keys"
Sep 11 01:11:28.867491 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:11:28.867406    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="17e94e18-1d22-4ed1-94e1-f1020470a635" containerName="payload-processing"
Sep 11 01:11:28.870832 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:11:28.870816    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-aba078c2-openshift-default-6c4db56c6d-xr2wt"
Sep 11 01:11:28.873008 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:11:28.872988    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-aba078c2-openshift-default-dockercfg-jm54n\""
Sep 11 01:11:28.883232 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:11:28.883209    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-aba078c2-openshift-default-6c4db56c6d-xr2wt"]
Sep 11 01:11:28.987348 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:11:28.987307    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-1d3c7cc9-fd48988c8-65qvp" event={"ID":"17e94e18-1d22-4ed1-94e1-f1020470a635","Type":"ContainerDied","Data":"770a0b93d97b4e33ca8087995bf7f7d9fee3fb65fa78117329368a54570670c3"}
Sep 11 01:11:28.987672 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:11:28.987646    2569 scope.go:117] "RemoveContainer" containerID="8e7a6cb37dff6d06622056ef5d675871024a0f50ce1c37bcb37dce990239d984"
Sep 11 01:11:28.987825 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:11:28.987358    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-1d3c7cc9-fd48988c8-65qvp"
Sep 11 01:11:29.015250 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:11:29.015220    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-1d3c7cc9-fd48988c8-65qvp"]
Sep 11 01:11:29.021449 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:11:29.021412    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-1d3c7cc9-fd48988c8-65qvp"]
Sep 11 01:11:29.047097 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:11:29.047064    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjwqp\" (UniqueName: \"kubernetes.io/projected/f3b8c998-7ffc-4a54-b50a-a35e66789f58-kube-api-access-fjwqp\") pod \"e2e-ait-aba078c2-openshift-default-6c4db56c6d-xr2wt\" (UID: \"f3b8c998-7ffc-4a54-b50a-a35e66789f58\") " pod="openshift-ingress/e2e-ait-aba078c2-openshift-default-6c4db56c6d-xr2wt"
Sep 11 01:11:29.047277 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:11:29.047207    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3b8c998-7ffc-4a54-b50a-a35e66789f58-istio-data\") pod \"e2e-ait-aba078c2-openshift-default-6c4db56c6d-xr2wt\" (UID: \"f3b8c998-7ffc-4a54-b50a-a35e66789f58\") " pod="openshift-ingress/e2e-ait-aba078c2-openshift-default-6c4db56c6d-xr2wt"
Sep 11 01:11:29.047277 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:11:29.047258    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3b8c998-7ffc-4a54-b50a-a35e66789f58-workload-socket\") pod \"e2e-ait-aba078c2-openshift-default-6c4db56c6d-xr2wt\" (UID: \"f3b8c998-7ffc-4a54-b50a-a35e66789f58\") " pod="openshift-ingress/e2e-ait-aba078c2-openshift-default-6c4db56c6d-xr2wt"
Sep 11 01:11:29.047409 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:11:29.047289    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3b8c998-7ffc-4a54-b50a-a35e66789f58-istio-token\") pod \"e2e-ait-aba078c2-openshift-default-6c4db56c6d-xr2wt\" (UID: \"f3b8c998-7ffc-4a54-b50a-a35e66789f58\") " pod="openshift-ingress/e2e-ait-aba078c2-openshift-default-6c4db56c6d-xr2wt"
Sep 11 01:11:29.047409 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:11:29.047360    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3b8c998-7ffc-4a54-b50a-a35e66789f58-workload-certs\") pod \"e2e-ait-aba078c2-openshift-default-6c4db56c6d-xr2wt\" (UID: \"f3b8c998-7ffc-4a54-b50a-a35e66789f58\") " pod="openshift-ingress/e2e-ait-aba078c2-openshift-default-6c4db56c6d-xr2wt"
Sep 11 01:11:29.047498 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:11:29.047477    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3b8c998-7ffc-4a54-b50a-a35e66789f58-credential-socket\") pod \"e2e-ait-aba078c2-openshift-default-6c4db56c6d-xr2wt\" (UID: \"f3b8c998-7ffc-4a54-b50a-a35e66789f58\") " pod="openshift-ingress/e2e-ait-aba078c2-openshift-default-6c4db56c6d-xr2wt"
Sep 11 01:11:29.047558 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:11:29.047515    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3b8c998-7ffc-4a54-b50a-a35e66789f58-istiod-ca-cert\") pod \"e2e-ait-aba078c2-openshift-default-6c4db56c6d-xr2wt\" (UID: \"f3b8c998-7ffc-4a54-b50a-a35e66789f58\") " pod="openshift-ingress/e2e-ait-aba078c2-openshift-default-6c4db56c6d-xr2wt"
Sep 11 01:11:29.047612 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:11:29.047551    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3b8c998-7ffc-4a54-b50a-a35e66789f58-istio-envoy\") pod \"e2e-ait-aba078c2-openshift-default-6c4db56c6d-xr2wt\" (UID: \"f3b8c998-7ffc-4a54-b50a-a35e66789f58\") " pod="openshift-ingress/e2e-ait-aba078c2-openshift-default-6c4db56c6d-xr2wt"
Sep 11 01:11:29.047669 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:11:29.047615    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3b8c998-7ffc-4a54-b50a-a35e66789f58-istio-podinfo\") pod \"e2e-ait-aba078c2-openshift-default-6c4db56c6d-xr2wt\" (UID: \"f3b8c998-7ffc-4a54-b50a-a35e66789f58\") " pod="openshift-ingress/e2e-ait-aba078c2-openshift-default-6c4db56c6d-xr2wt"
Sep 11 01:11:29.149093 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:11:29.148981    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjwqp\" (UniqueName: \"kubernetes.io/projected/f3b8c998-7ffc-4a54-b50a-a35e66789f58-kube-api-access-fjwqp\") pod \"e2e-ait-aba078c2-openshift-default-6c4db56c6d-xr2wt\" (UID: \"f3b8c998-7ffc-4a54-b50a-a35e66789f58\") " pod="openshift-ingress/e2e-ait-aba078c2-openshift-default-6c4db56c6d-xr2wt"
Sep 11 01:11:29.149093 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:11:29.149053    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3b8c998-7ffc-4a54-b50a-a35e66789f58-istio-data\") pod \"e2e-ait-aba078c2-openshift-default-6c4db56c6d-xr2wt\" (UID: \"f3b8c998-7ffc-4a54-b50a-a35e66789f58\") " pod="openshift-ingress/e2e-ait-aba078c2-openshift-default-6c4db56c6d-xr2wt"
Sep 11 01:11:29.149093 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:11:29.149089    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3b8c998-7ffc-4a54-b50a-a35e66789f58-workload-socket\") pod \"e2e-ait-aba078c2-openshift-default-6c4db56c6d-xr2wt\" (UID: \"f3b8c998-7ffc-4a54-b50a-a35e66789f58\") " pod="openshift-ingress/e2e-ait-aba078c2-openshift-default-6c4db56c6d-xr2wt"
Sep 11 01:11:29.149397 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:11:29.149114    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3b8c998-7ffc-4a54-b50a-a35e66789f58-istio-token\") pod \"e2e-ait-aba078c2-openshift-default-6c4db56c6d-xr2wt\" (UID: \"f3b8c998-7ffc-4a54-b50a-a35e66789f58\") " pod="openshift-ingress/e2e-ait-aba078c2-openshift-default-6c4db56c6d-xr2wt"
Sep 11 01:11:29.149397 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:11:29.149150    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3b8c998-7ffc-4a54-b50a-a35e66789f58-workload-certs\") pod \"e2e-ait-aba078c2-openshift-default-6c4db56c6d-xr2wt\" (UID: \"f3b8c998-7ffc-4a54-b50a-a35e66789f58\") " pod="openshift-ingress/e2e-ait-aba078c2-openshift-default-6c4db56c6d-xr2wt"
Sep 11 01:11:29.149397 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:11:29.149216    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3b8c998-7ffc-4a54-b50a-a35e66789f58-credential-socket\") pod \"e2e-ait-aba078c2-openshift-default-6c4db56c6d-xr2wt\" (UID: \"f3b8c998-7ffc-4a54-b50a-a35e66789f58\") " pod="openshift-ingress/e2e-ait-aba078c2-openshift-default-6c4db56c6d-xr2wt"
Sep 11 01:11:29.149397 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:11:29.149239    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3b8c998-7ffc-4a54-b50a-a35e66789f58-istiod-ca-cert\") pod \"e2e-ait-aba078c2-openshift-default-6c4db56c6d-xr2wt\" (UID: \"f3b8c998-7ffc-4a54-b50a-a35e66789f58\") " pod="openshift-ingress/e2e-ait-aba078c2-openshift-default-6c4db56c6d-xr2wt"
Sep 11 01:11:29.149397 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:11:29.149266    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3b8c998-7ffc-4a54-b50a-a35e66789f58-istio-envoy\") pod \"e2e-ait-aba078c2-openshift-default-6c4db56c6d-xr2wt\" (UID: \"f3b8c998-7ffc-4a54-b50a-a35e66789f58\") " pod="openshift-ingress/e2e-ait-aba078c2-openshift-default-6c4db56c6d-xr2wt"
Sep 11 01:11:29.149397 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:11:29.149287    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3b8c998-7ffc-4a54-b50a-a35e66789f58-istio-podinfo\") pod \"e2e-ait-aba078c2-openshift-default-6c4db56c6d-xr2wt\" (UID: \"f3b8c998-7ffc-4a54-b50a-a35e66789f58\") " pod="openshift-ingress/e2e-ait-aba078c2-openshift-default-6c4db56c6d-xr2wt"
Sep 11 01:11:29.149724 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:11:29.149512    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3b8c998-7ffc-4a54-b50a-a35e66789f58-istio-data\") pod \"e2e-ait-aba078c2-openshift-default-6c4db56c6d-xr2wt\" (UID: \"f3b8c998-7ffc-4a54-b50a-a35e66789f58\") " pod="openshift-ingress/e2e-ait-aba078c2-openshift-default-6c4db56c6d-xr2wt"
Sep 11 01:11:29.149724 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:11:29.149540    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3b8c998-7ffc-4a54-b50a-a35e66789f58-workload-socket\") pod \"e2e-ait-aba078c2-openshift-default-6c4db56c6d-xr2wt\" (UID: \"f3b8c998-7ffc-4a54-b50a-a35e66789f58\") " pod="openshift-ingress/e2e-ait-aba078c2-openshift-default-6c4db56c6d-xr2wt"
Sep 11 01:11:29.149724 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:11:29.149712    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3b8c998-7ffc-4a54-b50a-a35e66789f58-credential-socket\") pod \"e2e-ait-aba078c2-openshift-default-6c4db56c6d-xr2wt\" (UID: \"f3b8c998-7ffc-4a54-b50a-a35e66789f58\") " pod="openshift-ingress/e2e-ait-aba078c2-openshift-default-6c4db56c6d-xr2wt"
Sep 11 01:11:29.149933 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:11:29.149782    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3b8c998-7ffc-4a54-b50a-a35e66789f58-workload-certs\") pod \"e2e-ait-aba078c2-openshift-default-6c4db56c6d-xr2wt\" (UID: \"f3b8c998-7ffc-4a54-b50a-a35e66789f58\") " pod="openshift-ingress/e2e-ait-aba078c2-openshift-default-6c4db56c6d-xr2wt"
Sep 11 01:11:29.150445 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:11:29.150416    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3b8c998-7ffc-4a54-b50a-a35e66789f58-istiod-ca-cert\") pod \"e2e-ait-aba078c2-openshift-default-6c4db56c6d-xr2wt\" (UID: \"f3b8c998-7ffc-4a54-b50a-a35e66789f58\") " pod="openshift-ingress/e2e-ait-aba078c2-openshift-default-6c4db56c6d-xr2wt"
Sep 11 01:11:29.152118 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:11:29.152094    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3b8c998-7ffc-4a54-b50a-a35e66789f58-istio-podinfo\") pod \"e2e-ait-aba078c2-openshift-default-6c4db56c6d-xr2wt\" (UID: \"f3b8c998-7ffc-4a54-b50a-a35e66789f58\") " pod="openshift-ingress/e2e-ait-aba078c2-openshift-default-6c4db56c6d-xr2wt"
Sep 11 01:11:29.152381 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:11:29.152361    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3b8c998-7ffc-4a54-b50a-a35e66789f58-istio-envoy\") pod \"e2e-ait-aba078c2-openshift-default-6c4db56c6d-xr2wt\" (UID: \"f3b8c998-7ffc-4a54-b50a-a35e66789f58\") " pod="openshift-ingress/e2e-ait-aba078c2-openshift-default-6c4db56c6d-xr2wt"
Sep 11 01:11:29.166656 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:11:29.166627    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjwqp\" (UniqueName: \"kubernetes.io/projected/f3b8c998-7ffc-4a54-b50a-a35e66789f58-kube-api-access-fjwqp\") pod \"e2e-ait-aba078c2-openshift-default-6c4db56c6d-xr2wt\" (UID: \"f3b8c998-7ffc-4a54-b50a-a35e66789f58\") " pod="openshift-ingress/e2e-ait-aba078c2-openshift-default-6c4db56c6d-xr2wt"
Sep 11 01:11:29.167848 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:11:29.167828    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3b8c998-7ffc-4a54-b50a-a35e66789f58-istio-token\") pod \"e2e-ait-aba078c2-openshift-default-6c4db56c6d-xr2wt\" (UID: \"f3b8c998-7ffc-4a54-b50a-a35e66789f58\") " pod="openshift-ingress/e2e-ait-aba078c2-openshift-default-6c4db56c6d-xr2wt"
Sep 11 01:11:29.182745 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:11:29.182720    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-aba078c2-openshift-default-6c4db56c6d-xr2wt"
Sep 11 01:11:29.463840 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:11:29.463814    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-aba078c2-openshift-default-6c4db56c6d-xr2wt"]
Sep 11 01:11:29.465566 ip-10-0-129-222 kubenswrapper[2569]: W0911 01:11:29.465540    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3b8c998_7ffc_4a54_b50a_a35e66789f58.slice/crio-430ba3082fb8d5dfd2f8b8a78f3e8e6606a3f9b3b92c6813711b6722764f1ccd WatchSource:0}: Error finding container 430ba3082fb8d5dfd2f8b8a78f3e8e6606a3f9b3b92c6813711b6722764f1ccd: Status 404 returned error can't find the container with id 430ba3082fb8d5dfd2f8b8a78f3e8e6606a3f9b3b92c6813711b6722764f1ccd
Sep 11 01:11:29.467799 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:11:29.467767    2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 01:11:29.467857 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:11:29.467834    2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 01:11:29.467917 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:11:29.467861    2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 01:11:29.873158 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:11:29.873117    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="17e94e18-1d22-4ed1-94e1-f1020470a635" path="/var/lib/kubelet/pods/17e94e18-1d22-4ed1-94e1-f1020470a635/volumes"
Sep 11 01:11:29.873614 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:11:29.873504    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="35bb2587-d49a-41b5-aa73-554e3b9f2073" path="/var/lib/kubelet/pods/35bb2587-d49a-41b5-aa73-554e3b9f2073/volumes"
Sep 11 01:11:29.994939 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:11:29.994901    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-aba078c2-openshift-default-6c4db56c6d-xr2wt" event={"ID":"f3b8c998-7ffc-4a54-b50a-a35e66789f58","Type":"ContainerStarted","Data":"4552812d0703b5de8752a3ae8b71125619d759e4200f690029a8f8630c222ac2"}
Sep 11 01:11:29.995130 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:11:29.994987    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-aba078c2-openshift-default-6c4db56c6d-xr2wt" event={"ID":"f3b8c998-7ffc-4a54-b50a-a35e66789f58","Type":"ContainerStarted","Data":"430ba3082fb8d5dfd2f8b8a78f3e8e6606a3f9b3b92c6813711b6722764f1ccd"}
Sep 11 01:11:30.015428 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:11:30.015374    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-aba078c2-openshift-default-6c4db56c6d-xr2wt" podStartSLOduration=2.015360085 podStartE2EDuration="2.015360085s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 01:11:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 01:11:30.013239065 +0000 UTC m=+824.664613317" watchObservedRunningTime="2026-09-11 01:11:30.015360085 +0000 UTC m=+824.666734333"
Sep 11 01:11:30.183330 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:11:30.183250    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-aba078c2-openshift-default-6c4db56c6d-xr2wt"
Sep 11 01:11:30.188048 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:11:30.188025    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-aba078c2-openshift-default-6c4db56c6d-xr2wt"
Sep 11 01:11:31.001320 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:11:31.001288    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-aba078c2-openshift-default-6c4db56c6d-xr2wt"
Sep 11 01:11:31.002260 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:11:31.002240    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-aba078c2-openshift-default-6c4db56c6d-xr2wt"
Sep 11 01:11:33.105814 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:11:33.105779    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-aba078c2-84c9648747-khpz7"]
Sep 11 01:11:33.110932 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:11:33.110914    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-aba078c2-84c9648747-khpz7"
Sep 11 01:11:33.113432 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:11:33.113405    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-aba078c2\""
Sep 11 01:11:33.113569 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:11:33.113405    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-aba078c2-dockercfg-2z6z6\""
Sep 11 01:11:33.120520 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:11:33.120496    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-aba078c2-84c9648747-khpz7"]
Sep 11 01:11:33.287370 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:11:33.287335    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e703c866-c5bc-46c1-8331-bd1780c66732-plugins-config-volume\") pod \"payload-processing-e2e-ait-aba078c2-84c9648747-khpz7\" (UID: \"e703c866-c5bc-46c1-8331-bd1780c66732\") " pod="openshift-ingress/payload-processing-e2e-ait-aba078c2-84c9648747-khpz7"
Sep 11 01:11:33.287581 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:11:33.287532    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qnj5\" (UniqueName: \"kubernetes.io/projected/e703c866-c5bc-46c1-8331-bd1780c66732-kube-api-access-2qnj5\") pod \"payload-processing-e2e-ait-aba078c2-84c9648747-khpz7\" (UID: \"e703c866-c5bc-46c1-8331-bd1780c66732\") " pod="openshift-ingress/payload-processing-e2e-ait-aba078c2-84c9648747-khpz7"
Sep 11 01:11:33.388337 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:11:33.388259    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qnj5\" (UniqueName: \"kubernetes.io/projected/e703c866-c5bc-46c1-8331-bd1780c66732-kube-api-access-2qnj5\") pod \"payload-processing-e2e-ait-aba078c2-84c9648747-khpz7\" (UID: \"e703c866-c5bc-46c1-8331-bd1780c66732\") " pod="openshift-ingress/payload-processing-e2e-ait-aba078c2-84c9648747-khpz7"
Sep 11 01:11:33.388337 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:11:33.388322    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e703c866-c5bc-46c1-8331-bd1780c66732-plugins-config-volume\") pod \"payload-processing-e2e-ait-aba078c2-84c9648747-khpz7\" (UID: \"e703c866-c5bc-46c1-8331-bd1780c66732\") " pod="openshift-ingress/payload-processing-e2e-ait-aba078c2-84c9648747-khpz7"
Sep 11 01:11:33.389086 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:11:33.389062    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e703c866-c5bc-46c1-8331-bd1780c66732-plugins-config-volume\") pod \"payload-processing-e2e-ait-aba078c2-84c9648747-khpz7\" (UID: \"e703c866-c5bc-46c1-8331-bd1780c66732\") " pod="openshift-ingress/payload-processing-e2e-ait-aba078c2-84c9648747-khpz7"
Sep 11 01:11:33.402009 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:11:33.401979    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qnj5\" (UniqueName: \"kubernetes.io/projected/e703c866-c5bc-46c1-8331-bd1780c66732-kube-api-access-2qnj5\") pod \"payload-processing-e2e-ait-aba078c2-84c9648747-khpz7\" (UID: \"e703c866-c5bc-46c1-8331-bd1780c66732\") " pod="openshift-ingress/payload-processing-e2e-ait-aba078c2-84c9648747-khpz7"
Sep 11 01:11:33.423834 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:11:33.423807    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-aba078c2-84c9648747-khpz7"
Sep 11 01:11:33.693825 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:11:33.693802    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-aba078c2-84c9648747-khpz7"]
Sep 11 01:11:33.695373 ip-10-0-129-222 kubenswrapper[2569]: W0911 01:11:33.695350    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode703c866_c5bc_46c1_8331_bd1780c66732.slice/crio-3a2bf341bd1ee1f84716d238ece780a60f84c6721b442b3730a09a03ef1cdb57 WatchSource:0}: Error finding container 3a2bf341bd1ee1f84716d238ece780a60f84c6721b442b3730a09a03ef1cdb57: Status 404 returned error can't find the container with id 3a2bf341bd1ee1f84716d238ece780a60f84c6721b442b3730a09a03ef1cdb57
Sep 11 01:11:34.020441 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:11:34.020350    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-aba078c2-84c9648747-khpz7" event={"ID":"e703c866-c5bc-46c1-8331-bd1780c66732","Type":"ContainerStarted","Data":"5c60c9a0c7824c477d3e342606e1668e13a381c73e43aac4c03e0761ed9d2771"}
Sep 11 01:11:34.020441 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:11:34.020401    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-aba078c2-84c9648747-khpz7" event={"ID":"e703c866-c5bc-46c1-8331-bd1780c66732","Type":"ContainerStarted","Data":"3a2bf341bd1ee1f84716d238ece780a60f84c6721b442b3730a09a03ef1cdb57"}
Sep 11 01:11:34.020631 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:11:34.020459    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-aba078c2-84c9648747-khpz7"
Sep 11 01:11:34.038926 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:11:34.038857    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-aba078c2-84c9648747-khpz7" podStartSLOduration=1.038837971 podStartE2EDuration="1.038837971s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 01:11:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 01:11:34.035266963 +0000 UTC m=+828.686641215" watchObservedRunningTime="2026-09-11 01:11:34.038837971 +0000 UTC m=+828.690212222"
Sep 11 01:11:45.032799 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:11:45.032766    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-aba078c2-84c9648747-khpz7"
Sep 11 01:11:51.130655 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:11:51.130618    2569 generic.go:356] "Generic (PLEG): container finished" podID="f3b8c998-7ffc-4a54-b50a-a35e66789f58" containerID="4552812d0703b5de8752a3ae8b71125619d759e4200f690029a8f8630c222ac2" exitCode=0
Sep 11 01:11:51.131133 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:11:51.130690    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-aba078c2-openshift-default-6c4db56c6d-xr2wt" event={"ID":"f3b8c998-7ffc-4a54-b50a-a35e66789f58","Type":"ContainerDied","Data":"4552812d0703b5de8752a3ae8b71125619d759e4200f690029a8f8630c222ac2"}
Sep 11 01:11:51.131199 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:11:51.131181    2569 scope.go:117] "RemoveContainer" containerID="4552812d0703b5de8752a3ae8b71125619d759e4200f690029a8f8630c222ac2"
Sep 11 01:11:51.132348 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:11:51.132314    2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 01:11:51.132464 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:11:51.132419    2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 01:11:51.132527 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:11:51.132470    2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 01:11:51.692658 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:11:51.692624    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kmx5x"]
Sep 11 01:11:52.141688 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:11:52.141643    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-aba078c2-openshift-default-6c4db56c6d-xr2wt" event={"ID":"f3b8c998-7ffc-4a54-b50a-a35e66789f58","Type":"ContainerStarted","Data":"33efcb481cfe9c59fd637a18fd9999f6e15b4116a22d3f03dc0b83af21891809"}
Sep 11 01:11:52.183292 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:11:52.183259    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-aba078c2-openshift-default-6c4db56c6d-xr2wt"
Sep 11 01:11:52.183470 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:11:52.183444    2569 patch_prober.go:28] interesting pod/e2e-ait-aba078c2-openshift-default-6c4db56c6d-xr2wt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body=
Sep 11 01:11:52.183514 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:11:52.183500    2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-aba078c2-openshift-default-6c4db56c6d-xr2wt" podUID="f3b8c998-7ffc-4a54-b50a-a35e66789f58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused"
Sep 11 01:11:53.183840 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:11:53.183805    2569 patch_prober.go:28] interesting pod/e2e-ait-aba078c2-openshift-default-6c4db56c6d-xr2wt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body=
Sep 11 01:11:53.184256 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:11:53.183857    2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-aba078c2-openshift-default-6c4db56c6d-xr2wt" podUID="f3b8c998-7ffc-4a54-b50a-a35e66789f58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused"
Sep 11 01:11:54.184175 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:11:54.184099    2569 patch_prober.go:28] interesting pod/e2e-ait-aba078c2-openshift-default-6c4db56c6d-xr2wt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body=
Sep 11 01:11:54.184175 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:11:54.184162    2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-aba078c2-openshift-default-6c4db56c6d-xr2wt" podUID="f3b8c998-7ffc-4a54-b50a-a35e66789f58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused"
Sep 11 01:11:55.183444 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:11:55.183413    2569 patch_prober.go:28] interesting pod/e2e-ait-aba078c2-openshift-default-6c4db56c6d-xr2wt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body=
Sep 11 01:11:55.183618 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:11:55.183468    2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-aba078c2-openshift-default-6c4db56c6d-xr2wt" podUID="f3b8c998-7ffc-4a54-b50a-a35e66789f58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused"
Sep 11 01:11:56.183213 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:11:56.183179    2569 patch_prober.go:28] interesting pod/e2e-ait-aba078c2-openshift-default-6c4db56c6d-xr2wt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body=
Sep 11 01:11:56.183597 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:11:56.183238    2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-aba078c2-openshift-default-6c4db56c6d-xr2wt" podUID="f3b8c998-7ffc-4a54-b50a-a35e66789f58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused"
Sep 11 01:11:57.183533 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:11:57.183498    2569 patch_prober.go:28] interesting pod/e2e-ait-aba078c2-openshift-default-6c4db56c6d-xr2wt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body=
Sep 11 01:11:57.183922 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:11:57.183551    2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-aba078c2-openshift-default-6c4db56c6d-xr2wt" podUID="f3b8c998-7ffc-4a54-b50a-a35e66789f58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused"
Sep 11 01:11:58.187642 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:11:58.187615    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-aba078c2-openshift-default-6c4db56c6d-xr2wt"
Sep 11 01:11:58.188115 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:11:58.188012    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-aba078c2-openshift-default-6c4db56c6d-xr2wt"
Sep 11 01:11:58.188657 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:11:58.188639    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-aba078c2-openshift-default-6c4db56c6d-xr2wt"
Sep 11 01:12:12.314008 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:12.313976    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kmx5x"]
Sep 11 01:12:13.244371 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:13.244339    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-aba078c2-84c9648747-khpz7"]
Sep 11 01:12:13.244589 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:13.244550    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-aba078c2-84c9648747-khpz7" podUID="e703c866-c5bc-46c1-8331-bd1780c66732" containerName="payload-processing" containerID="cri-o://5c60c9a0c7824c477d3e342606e1668e13a381c73e43aac4c03e0761ed9d2771" gracePeriod=30
Sep 11 01:12:13.505068 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:13.505011    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-aba078c2-84c9648747-khpz7"
Sep 11 01:12:13.592578 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:13.592540    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e703c866-c5bc-46c1-8331-bd1780c66732-plugins-config-volume\") pod \"e703c866-c5bc-46c1-8331-bd1780c66732\" (UID: \"e703c866-c5bc-46c1-8331-bd1780c66732\") "
Sep 11 01:12:13.592781 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:13.592627    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2qnj5\" (UniqueName: \"kubernetes.io/projected/e703c866-c5bc-46c1-8331-bd1780c66732-kube-api-access-2qnj5\") pod \"e703c866-c5bc-46c1-8331-bd1780c66732\" (UID: \"e703c866-c5bc-46c1-8331-bd1780c66732\") "
Sep 11 01:12:13.593027 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:13.592992    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e703c866-c5bc-46c1-8331-bd1780c66732-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e703c866-c5bc-46c1-8331-bd1780c66732" (UID: "e703c866-c5bc-46c1-8331-bd1780c66732"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 01:12:13.596033 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:13.596004    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e703c866-c5bc-46c1-8331-bd1780c66732-kube-api-access-2qnj5" (OuterVolumeSpecName: "kube-api-access-2qnj5") pod "e703c866-c5bc-46c1-8331-bd1780c66732" (UID: "e703c866-c5bc-46c1-8331-bd1780c66732"). InnerVolumeSpecName "kube-api-access-2qnj5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 01:12:13.694252 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:13.694220    2569 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e703c866-c5bc-46c1-8331-bd1780c66732-plugins-config-volume\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:12:13.694252 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:13.694246    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2qnj5\" (UniqueName: \"kubernetes.io/projected/e703c866-c5bc-46c1-8331-bd1780c66732-kube-api-access-2qnj5\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:12:14.272250 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:14.272161    2569 generic.go:356] "Generic (PLEG): container finished" podID="e703c866-c5bc-46c1-8331-bd1780c66732" containerID="5c60c9a0c7824c477d3e342606e1668e13a381c73e43aac4c03e0761ed9d2771" exitCode=0
Sep 11 01:12:14.272250 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:14.272207    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-aba078c2-84c9648747-khpz7"
Sep 11 01:12:14.272447 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:14.272245    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-aba078c2-84c9648747-khpz7" event={"ID":"e703c866-c5bc-46c1-8331-bd1780c66732","Type":"ContainerDied","Data":"5c60c9a0c7824c477d3e342606e1668e13a381c73e43aac4c03e0761ed9d2771"}
Sep 11 01:12:14.272447 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:14.272296    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-aba078c2-84c9648747-khpz7" event={"ID":"e703c866-c5bc-46c1-8331-bd1780c66732","Type":"ContainerDied","Data":"3a2bf341bd1ee1f84716d238ece780a60f84c6721b442b3730a09a03ef1cdb57"}
Sep 11 01:12:14.272447 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:14.272318    2569 scope.go:117] "RemoveContainer" containerID="5c60c9a0c7824c477d3e342606e1668e13a381c73e43aac4c03e0761ed9d2771"
Sep 11 01:12:14.286367 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:14.286344    2569 scope.go:117] "RemoveContainer" containerID="5c60c9a0c7824c477d3e342606e1668e13a381c73e43aac4c03e0761ed9d2771"
Sep 11 01:12:14.286660 ip-10-0-129-222 kubenswrapper[2569]: E0911 01:12:14.286642    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c60c9a0c7824c477d3e342606e1668e13a381c73e43aac4c03e0761ed9d2771\": container with ID starting with 5c60c9a0c7824c477d3e342606e1668e13a381c73e43aac4c03e0761ed9d2771 not found: ID does not exist" containerID="5c60c9a0c7824c477d3e342606e1668e13a381c73e43aac4c03e0761ed9d2771"
Sep 11 01:12:14.286714 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:14.286667    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c60c9a0c7824c477d3e342606e1668e13a381c73e43aac4c03e0761ed9d2771"} err="failed to get container status \"5c60c9a0c7824c477d3e342606e1668e13a381c73e43aac4c03e0761ed9d2771\": rpc error: code = NotFound desc = could not find container \"5c60c9a0c7824c477d3e342606e1668e13a381c73e43aac4c03e0761ed9d2771\": container with ID starting with 5c60c9a0c7824c477d3e342606e1668e13a381c73e43aac4c03e0761ed9d2771 not found: ID does not exist"
Sep 11 01:12:14.292984 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:14.292961    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-aba078c2-84c9648747-khpz7"]
Sep 11 01:12:14.299697 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:14.299677    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-aba078c2-84c9648747-khpz7"]
Sep 11 01:12:14.566825 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:14.566795    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-aba078c2-openshift-default-6c4db56c6d-xr2wt"]
Sep 11 01:12:14.567284 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:14.567053    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-aba078c2-openshift-default-6c4db56c6d-xr2wt" podUID="f3b8c998-7ffc-4a54-b50a-a35e66789f58" containerName="istio-proxy" containerID="cri-o://33efcb481cfe9c59fd637a18fd9999f6e15b4116a22d3f03dc0b83af21891809" gracePeriod=30
Sep 11 01:12:15.319537 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:15.319506    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-39ea7c6f-openshift-default-566bd9c7d4-j2v5j"]
Sep 11 01:12:15.320006 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:15.319992    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e703c866-c5bc-46c1-8331-bd1780c66732" containerName="payload-processing"
Sep 11 01:12:15.320069 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:15.320008    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="e703c866-c5bc-46c1-8331-bd1780c66732" containerName="payload-processing"
Sep 11 01:12:15.320108 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:15.320075    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="e703c866-c5bc-46c1-8331-bd1780c66732" containerName="payload-processing"
Sep 11 01:12:15.324468 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:15.324448    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-39ea7c6f-openshift-default-566bd9c7d4-j2v5j"
Sep 11 01:12:15.326938 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:15.326912    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-39ea7c6f-openshift-default-dockercfg-v9vhk\""
Sep 11 01:12:15.333980 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:15.333957    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-39ea7c6f-openshift-default-566bd9c7d4-j2v5j"]
Sep 11 01:12:15.412216 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:15.412179    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69502d0e-5d63-45aa-8724-678b467e51b4-istio-token\") pod \"e2e-derive-39ea7c6f-openshift-default-566bd9c7d4-j2v5j\" (UID: \"69502d0e-5d63-45aa-8724-678b467e51b4\") " pod="openshift-ingress/e2e-derive-39ea7c6f-openshift-default-566bd9c7d4-j2v5j"
Sep 11 01:12:15.412426 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:15.412248    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69502d0e-5d63-45aa-8724-678b467e51b4-workload-socket\") pod \"e2e-derive-39ea7c6f-openshift-default-566bd9c7d4-j2v5j\" (UID: \"69502d0e-5d63-45aa-8724-678b467e51b4\") " pod="openshift-ingress/e2e-derive-39ea7c6f-openshift-default-566bd9c7d4-j2v5j"
Sep 11 01:12:15.412426 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:15.412279    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69502d0e-5d63-45aa-8724-678b467e51b4-istiod-ca-cert\") pod \"e2e-derive-39ea7c6f-openshift-default-566bd9c7d4-j2v5j\" (UID: \"69502d0e-5d63-45aa-8724-678b467e51b4\") " pod="openshift-ingress/e2e-derive-39ea7c6f-openshift-default-566bd9c7d4-j2v5j"
Sep 11 01:12:15.412426 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:15.412352    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69502d0e-5d63-45aa-8724-678b467e51b4-credential-socket\") pod \"e2e-derive-39ea7c6f-openshift-default-566bd9c7d4-j2v5j\" (UID: \"69502d0e-5d63-45aa-8724-678b467e51b4\") " pod="openshift-ingress/e2e-derive-39ea7c6f-openshift-default-566bd9c7d4-j2v5j"
Sep 11 01:12:15.412426 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:15.412401    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69502d0e-5d63-45aa-8724-678b467e51b4-workload-certs\") pod \"e2e-derive-39ea7c6f-openshift-default-566bd9c7d4-j2v5j\" (UID: \"69502d0e-5d63-45aa-8724-678b467e51b4\") " pod="openshift-ingress/e2e-derive-39ea7c6f-openshift-default-566bd9c7d4-j2v5j"
Sep 11 01:12:15.412693 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:15.412460    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69502d0e-5d63-45aa-8724-678b467e51b4-istio-podinfo\") pod \"e2e-derive-39ea7c6f-openshift-default-566bd9c7d4-j2v5j\" (UID: \"69502d0e-5d63-45aa-8724-678b467e51b4\") " pod="openshift-ingress/e2e-derive-39ea7c6f-openshift-default-566bd9c7d4-j2v5j"
Sep 11 01:12:15.412693 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:15.412499    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69502d0e-5d63-45aa-8724-678b467e51b4-istio-envoy\") pod \"e2e-derive-39ea7c6f-openshift-default-566bd9c7d4-j2v5j\" (UID: \"69502d0e-5d63-45aa-8724-678b467e51b4\") " pod="openshift-ingress/e2e-derive-39ea7c6f-openshift-default-566bd9c7d4-j2v5j"
Sep 11 01:12:15.412693 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:15.412549    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69502d0e-5d63-45aa-8724-678b467e51b4-istio-data\") pod \"e2e-derive-39ea7c6f-openshift-default-566bd9c7d4-j2v5j\" (UID: \"69502d0e-5d63-45aa-8724-678b467e51b4\") " pod="openshift-ingress/e2e-derive-39ea7c6f-openshift-default-566bd9c7d4-j2v5j"
Sep 11 01:12:15.412693 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:15.412598    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vs95\" (UniqueName: \"kubernetes.io/projected/69502d0e-5d63-45aa-8724-678b467e51b4-kube-api-access-6vs95\") pod \"e2e-derive-39ea7c6f-openshift-default-566bd9c7d4-j2v5j\" (UID: \"69502d0e-5d63-45aa-8724-678b467e51b4\") " pod="openshift-ingress/e2e-derive-39ea7c6f-openshift-default-566bd9c7d4-j2v5j"
Sep 11 01:12:15.513393 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:15.513346    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69502d0e-5d63-45aa-8724-678b467e51b4-workload-socket\") pod \"e2e-derive-39ea7c6f-openshift-default-566bd9c7d4-j2v5j\" (UID: \"69502d0e-5d63-45aa-8724-678b467e51b4\") " pod="openshift-ingress/e2e-derive-39ea7c6f-openshift-default-566bd9c7d4-j2v5j"
Sep 11 01:12:15.513393 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:15.513400    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69502d0e-5d63-45aa-8724-678b467e51b4-istiod-ca-cert\") pod \"e2e-derive-39ea7c6f-openshift-default-566bd9c7d4-j2v5j\" (UID: \"69502d0e-5d63-45aa-8724-678b467e51b4\") " pod="openshift-ingress/e2e-derive-39ea7c6f-openshift-default-566bd9c7d4-j2v5j"
Sep 11 01:12:15.513641 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:15.513461    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69502d0e-5d63-45aa-8724-678b467e51b4-credential-socket\") pod \"e2e-derive-39ea7c6f-openshift-default-566bd9c7d4-j2v5j\" (UID: \"69502d0e-5d63-45aa-8724-678b467e51b4\") " pod="openshift-ingress/e2e-derive-39ea7c6f-openshift-default-566bd9c7d4-j2v5j"
Sep 11 01:12:15.513641 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:15.513500    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69502d0e-5d63-45aa-8724-678b467e51b4-workload-certs\") pod \"e2e-derive-39ea7c6f-openshift-default-566bd9c7d4-j2v5j\" (UID: \"69502d0e-5d63-45aa-8724-678b467e51b4\") " pod="openshift-ingress/e2e-derive-39ea7c6f-openshift-default-566bd9c7d4-j2v5j"
Sep 11 01:12:15.513641 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:15.513528    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69502d0e-5d63-45aa-8724-678b467e51b4-istio-podinfo\") pod \"e2e-derive-39ea7c6f-openshift-default-566bd9c7d4-j2v5j\" (UID: \"69502d0e-5d63-45aa-8724-678b467e51b4\") " pod="openshift-ingress/e2e-derive-39ea7c6f-openshift-default-566bd9c7d4-j2v5j"
Sep 11 01:12:15.513641 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:15.513554    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69502d0e-5d63-45aa-8724-678b467e51b4-istio-envoy\") pod \"e2e-derive-39ea7c6f-openshift-default-566bd9c7d4-j2v5j\" (UID: \"69502d0e-5d63-45aa-8724-678b467e51b4\") " pod="openshift-ingress/e2e-derive-39ea7c6f-openshift-default-566bd9c7d4-j2v5j"
Sep 11 01:12:15.513641 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:15.513603    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69502d0e-5d63-45aa-8724-678b467e51b4-istio-data\") pod \"e2e-derive-39ea7c6f-openshift-default-566bd9c7d4-j2v5j\" (UID: \"69502d0e-5d63-45aa-8724-678b467e51b4\") " pod="openshift-ingress/e2e-derive-39ea7c6f-openshift-default-566bd9c7d4-j2v5j"
Sep 11 01:12:15.513962 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:15.513647    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vs95\" (UniqueName: \"kubernetes.io/projected/69502d0e-5d63-45aa-8724-678b467e51b4-kube-api-access-6vs95\") pod \"e2e-derive-39ea7c6f-openshift-default-566bd9c7d4-j2v5j\" (UID: \"69502d0e-5d63-45aa-8724-678b467e51b4\") " pod="openshift-ingress/e2e-derive-39ea7c6f-openshift-default-566bd9c7d4-j2v5j"
Sep 11 01:12:15.513962 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:15.513719    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69502d0e-5d63-45aa-8724-678b467e51b4-istio-token\") pod \"e2e-derive-39ea7c6f-openshift-default-566bd9c7d4-j2v5j\" (UID: \"69502d0e-5d63-45aa-8724-678b467e51b4\") " pod="openshift-ingress/e2e-derive-39ea7c6f-openshift-default-566bd9c7d4-j2v5j"
Sep 11 01:12:15.513962 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:15.513735    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69502d0e-5d63-45aa-8724-678b467e51b4-workload-socket\") pod \"e2e-derive-39ea7c6f-openshift-default-566bd9c7d4-j2v5j\" (UID: \"69502d0e-5d63-45aa-8724-678b467e51b4\") " pod="openshift-ingress/e2e-derive-39ea7c6f-openshift-default-566bd9c7d4-j2v5j"
Sep 11 01:12:15.513962 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:15.513840    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69502d0e-5d63-45aa-8724-678b467e51b4-credential-socket\") pod \"e2e-derive-39ea7c6f-openshift-default-566bd9c7d4-j2v5j\" (UID: \"69502d0e-5d63-45aa-8724-678b467e51b4\") " pod="openshift-ingress/e2e-derive-39ea7c6f-openshift-default-566bd9c7d4-j2v5j"
Sep 11 01:12:15.513962 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:15.513931    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69502d0e-5d63-45aa-8724-678b467e51b4-workload-certs\") pod \"e2e-derive-39ea7c6f-openshift-default-566bd9c7d4-j2v5j\" (UID: \"69502d0e-5d63-45aa-8724-678b467e51b4\") " pod="openshift-ingress/e2e-derive-39ea7c6f-openshift-default-566bd9c7d4-j2v5j"
Sep 11 01:12:15.514202 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:15.514123    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69502d0e-5d63-45aa-8724-678b467e51b4-istio-data\") pod \"e2e-derive-39ea7c6f-openshift-default-566bd9c7d4-j2v5j\" (UID: \"69502d0e-5d63-45aa-8724-678b467e51b4\") " pod="openshift-ingress/e2e-derive-39ea7c6f-openshift-default-566bd9c7d4-j2v5j"
Sep 11 01:12:15.514250 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:15.514236    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69502d0e-5d63-45aa-8724-678b467e51b4-istiod-ca-cert\") pod \"e2e-derive-39ea7c6f-openshift-default-566bd9c7d4-j2v5j\" (UID: \"69502d0e-5d63-45aa-8724-678b467e51b4\") " pod="openshift-ingress/e2e-derive-39ea7c6f-openshift-default-566bd9c7d4-j2v5j"
Sep 11 01:12:15.515930 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:15.515903    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69502d0e-5d63-45aa-8724-678b467e51b4-istio-envoy\") pod \"e2e-derive-39ea7c6f-openshift-default-566bd9c7d4-j2v5j\" (UID: \"69502d0e-5d63-45aa-8724-678b467e51b4\") " pod="openshift-ingress/e2e-derive-39ea7c6f-openshift-default-566bd9c7d4-j2v5j"
Sep 11 01:12:15.516151 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:15.516134    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69502d0e-5d63-45aa-8724-678b467e51b4-istio-podinfo\") pod \"e2e-derive-39ea7c6f-openshift-default-566bd9c7d4-j2v5j\" (UID: \"69502d0e-5d63-45aa-8724-678b467e51b4\") " pod="openshift-ingress/e2e-derive-39ea7c6f-openshift-default-566bd9c7d4-j2v5j"
Sep 11 01:12:15.521141 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:15.521117    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69502d0e-5d63-45aa-8724-678b467e51b4-istio-token\") pod \"e2e-derive-39ea7c6f-openshift-default-566bd9c7d4-j2v5j\" (UID: \"69502d0e-5d63-45aa-8724-678b467e51b4\") " pod="openshift-ingress/e2e-derive-39ea7c6f-openshift-default-566bd9c7d4-j2v5j"
Sep 11 01:12:15.521228 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:15.521187    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vs95\" (UniqueName: \"kubernetes.io/projected/69502d0e-5d63-45aa-8724-678b467e51b4-kube-api-access-6vs95\") pod \"e2e-derive-39ea7c6f-openshift-default-566bd9c7d4-j2v5j\" (UID: \"69502d0e-5d63-45aa-8724-678b467e51b4\") " pod="openshift-ingress/e2e-derive-39ea7c6f-openshift-default-566bd9c7d4-j2v5j"
Sep 11 01:12:15.635681 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:15.635611    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-39ea7c6f-openshift-default-566bd9c7d4-j2v5j"
Sep 11 01:12:15.871853 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:15.871819    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e703c866-c5bc-46c1-8331-bd1780c66732" path="/var/lib/kubelet/pods/e703c866-c5bc-46c1-8331-bd1780c66732/volumes"
Sep 11 01:12:15.913405 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:15.913380    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-39ea7c6f-openshift-default-566bd9c7d4-j2v5j"]
Sep 11 01:12:15.915177 ip-10-0-129-222 kubenswrapper[2569]: W0911 01:12:15.915151    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69502d0e_5d63_45aa_8724_678b467e51b4.slice/crio-18cd488bd0cd01d1e9e4ce80d50686d4faec6bb2e9acfdba1f821ece6d653415 WatchSource:0}: Error finding container 18cd488bd0cd01d1e9e4ce80d50686d4faec6bb2e9acfdba1f821ece6d653415: Status 404 returned error can't find the container with id 18cd488bd0cd01d1e9e4ce80d50686d4faec6bb2e9acfdba1f821ece6d653415
Sep 11 01:12:15.917429 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:15.917394    2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 01:12:15.917529 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:15.917458    2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 01:12:15.917529 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:15.917487    2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 01:12:16.285232 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:16.285132    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-39ea7c6f-openshift-default-566bd9c7d4-j2v5j" event={"ID":"69502d0e-5d63-45aa-8724-678b467e51b4","Type":"ContainerStarted","Data":"76211d9197ad0889fabe0d7e3090b011bd3df6a5f7ac19cff8407aedd9acb20b"}
Sep 11 01:12:16.285232 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:16.285197    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-39ea7c6f-openshift-default-566bd9c7d4-j2v5j" event={"ID":"69502d0e-5d63-45aa-8724-678b467e51b4","Type":"ContainerStarted","Data":"18cd488bd0cd01d1e9e4ce80d50686d4faec6bb2e9acfdba1f821ece6d653415"}
Sep 11 01:12:16.304571 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:16.304522    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-39ea7c6f-openshift-default-566bd9c7d4-j2v5j" podStartSLOduration=1.304503743 podStartE2EDuration="1.304503743s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 01:12:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 01:12:16.304080184 +0000 UTC m=+870.955454435" watchObservedRunningTime="2026-09-11 01:12:16.304503743 +0000 UTC m=+870.955877993"
Sep 11 01:12:16.635942 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:16.635898    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-39ea7c6f-openshift-default-566bd9c7d4-j2v5j"
Sep 11 01:12:16.641014 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:16.640992    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-39ea7c6f-openshift-default-566bd9c7d4-j2v5j"
Sep 11 01:12:17.068438 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:17.068402    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-dc7dcd-846dffff5d-zffpq"]
Sep 11 01:12:17.068817 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:17.068699    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-autores-dc7dcd-846dffff5d-zffpq" podUID="076eaa74-0639-4df0-9392-3c6a3ec60a9f" containerName="payload-pre-processing" containerID="cri-o://c8685432b33fe8b2762613b30fe9210a676db5556041e97fdc7affb0a99a254f" gracePeriod=30
Sep 11 01:12:17.293838 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:17.293807    2569 generic.go:356] "Generic (PLEG): container finished" podID="076eaa74-0639-4df0-9392-3c6a3ec60a9f" containerID="c8685432b33fe8b2762613b30fe9210a676db5556041e97fdc7affb0a99a254f" exitCode=0
Sep 11 01:12:17.294019 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:17.293902    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-dc7dcd-846dffff5d-zffpq" event={"ID":"076eaa74-0639-4df0-9392-3c6a3ec60a9f","Type":"ContainerDied","Data":"c8685432b33fe8b2762613b30fe9210a676db5556041e97fdc7affb0a99a254f"}
Sep 11 01:12:17.294311 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:17.294292    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-39ea7c6f-openshift-default-566bd9c7d4-j2v5j"
Sep 11 01:12:17.295248 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:17.295223    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-39ea7c6f-openshift-default-566bd9c7d4-j2v5j"
Sep 11 01:12:17.338418 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:17.338392    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-dc7dcd-846dffff5d-zffpq"
Sep 11 01:12:17.432329 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:17.432288    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/076eaa74-0639-4df0-9392-3c6a3ec60a9f-plugins-config-volume\") pod \"076eaa74-0639-4df0-9392-3c6a3ec60a9f\" (UID: \"076eaa74-0639-4df0-9392-3c6a3ec60a9f\") "
Sep 11 01:12:17.432329 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:17.432332    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h9r75\" (UniqueName: \"kubernetes.io/projected/076eaa74-0639-4df0-9392-3c6a3ec60a9f-kube-api-access-h9r75\") pod \"076eaa74-0639-4df0-9392-3c6a3ec60a9f\" (UID: \"076eaa74-0639-4df0-9392-3c6a3ec60a9f\") "
Sep 11 01:12:17.432669 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:17.432643    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/076eaa74-0639-4df0-9392-3c6a3ec60a9f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "076eaa74-0639-4df0-9392-3c6a3ec60a9f" (UID: "076eaa74-0639-4df0-9392-3c6a3ec60a9f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 01:12:17.434480 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:17.434453    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/076eaa74-0639-4df0-9392-3c6a3ec60a9f-kube-api-access-h9r75" (OuterVolumeSpecName: "kube-api-access-h9r75") pod "076eaa74-0639-4df0-9392-3c6a3ec60a9f" (UID: "076eaa74-0639-4df0-9392-3c6a3ec60a9f"). InnerVolumeSpecName "kube-api-access-h9r75". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 01:12:17.534076 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:17.534043    2569 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/076eaa74-0639-4df0-9392-3c6a3ec60a9f-plugins-config-volume\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:12:17.534076 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:17.534072    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h9r75\" (UniqueName: \"kubernetes.io/projected/076eaa74-0639-4df0-9392-3c6a3ec60a9f-kube-api-access-h9r75\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:12:18.300520 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:18.300434    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-dc7dcd-846dffff5d-zffpq"
Sep 11 01:12:18.300520 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:18.300472    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-dc7dcd-846dffff5d-zffpq" event={"ID":"076eaa74-0639-4df0-9392-3c6a3ec60a9f","Type":"ContainerDied","Data":"36cca2a74965bcd82c8a348515b750630239aefd530cf37bef65d3823dd9830d"}
Sep 11 01:12:18.300984 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:18.300534    2569 scope.go:117] "RemoveContainer" containerID="c8685432b33fe8b2762613b30fe9210a676db5556041e97fdc7affb0a99a254f"
Sep 11 01:12:18.321320 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:18.321284    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-dc7dcd-846dffff5d-zffpq"]
Sep 11 01:12:18.325859 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:18.325837    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-dc7dcd-846dffff5d-zffpq"]
Sep 11 01:12:19.822173 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:19.822148    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-aba078c2-openshift-default-6c4db56c6d-xr2wt"
Sep 11 01:12:19.856592 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:19.856523    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3b8c998-7ffc-4a54-b50a-a35e66789f58-credential-socket\") pod \"f3b8c998-7ffc-4a54-b50a-a35e66789f58\" (UID: \"f3b8c998-7ffc-4a54-b50a-a35e66789f58\") "
Sep 11 01:12:19.856719 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:19.856624    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3b8c998-7ffc-4a54-b50a-a35e66789f58-istio-podinfo\") pod \"f3b8c998-7ffc-4a54-b50a-a35e66789f58\" (UID: \"f3b8c998-7ffc-4a54-b50a-a35e66789f58\") "
Sep 11 01:12:19.856719 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:19.856658    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3b8c998-7ffc-4a54-b50a-a35e66789f58-istio-token\") pod \"f3b8c998-7ffc-4a54-b50a-a35e66789f58\" (UID: \"f3b8c998-7ffc-4a54-b50a-a35e66789f58\") "
Sep 11 01:12:19.856719 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:19.856686    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3b8c998-7ffc-4a54-b50a-a35e66789f58-workload-certs\") pod \"f3b8c998-7ffc-4a54-b50a-a35e66789f58\" (UID: \"f3b8c998-7ffc-4a54-b50a-a35e66789f58\") "
Sep 11 01:12:19.856904 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:19.856734    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3b8c998-7ffc-4a54-b50a-a35e66789f58-workload-socket\") pod \"f3b8c998-7ffc-4a54-b50a-a35e66789f58\" (UID: \"f3b8c998-7ffc-4a54-b50a-a35e66789f58\") "
Sep 11 01:12:19.856904 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:19.856763    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fjwqp\" (UniqueName: \"kubernetes.io/projected/f3b8c998-7ffc-4a54-b50a-a35e66789f58-kube-api-access-fjwqp\") pod \"f3b8c998-7ffc-4a54-b50a-a35e66789f58\" (UID: \"f3b8c998-7ffc-4a54-b50a-a35e66789f58\") "
Sep 11 01:12:19.856904 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:19.856791    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3b8c998-7ffc-4a54-b50a-a35e66789f58-istio-data\") pod \"f3b8c998-7ffc-4a54-b50a-a35e66789f58\" (UID: \"f3b8c998-7ffc-4a54-b50a-a35e66789f58\") "
Sep 11 01:12:19.856904 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:19.856810    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3b8c998-7ffc-4a54-b50a-a35e66789f58-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "f3b8c998-7ffc-4a54-b50a-a35e66789f58" (UID: "f3b8c998-7ffc-4a54-b50a-a35e66789f58"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 01:12:19.856904 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:19.856829    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3b8c998-7ffc-4a54-b50a-a35e66789f58-istio-envoy\") pod \"f3b8c998-7ffc-4a54-b50a-a35e66789f58\" (UID: \"f3b8c998-7ffc-4a54-b50a-a35e66789f58\") "
Sep 11 01:12:19.856904 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:19.856866    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3b8c998-7ffc-4a54-b50a-a35e66789f58-istiod-ca-cert\") pod \"f3b8c998-7ffc-4a54-b50a-a35e66789f58\" (UID: \"f3b8c998-7ffc-4a54-b50a-a35e66789f58\") "
Sep 11 01:12:19.857240 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:19.857167    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3b8c998-7ffc-4a54-b50a-a35e66789f58-istio-data" (OuterVolumeSpecName: "istio-data") pod "f3b8c998-7ffc-4a54-b50a-a35e66789f58" (UID: "f3b8c998-7ffc-4a54-b50a-a35e66789f58"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 01:12:19.857297 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:19.857239    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3b8c998-7ffc-4a54-b50a-a35e66789f58-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "f3b8c998-7ffc-4a54-b50a-a35e66789f58" (UID: "f3b8c998-7ffc-4a54-b50a-a35e66789f58"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 01:12:19.857353 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:19.857315    2569 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3b8c998-7ffc-4a54-b50a-a35e66789f58-credential-socket\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:12:19.857353 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:19.857335    2569 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3b8c998-7ffc-4a54-b50a-a35e66789f58-istio-data\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:12:19.857463 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:19.857396    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3b8c998-7ffc-4a54-b50a-a35e66789f58-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "f3b8c998-7ffc-4a54-b50a-a35e66789f58" (UID: "f3b8c998-7ffc-4a54-b50a-a35e66789f58"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 01:12:19.857520 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:19.857478    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3b8c998-7ffc-4a54-b50a-a35e66789f58-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "f3b8c998-7ffc-4a54-b50a-a35e66789f58" (UID: "f3b8c998-7ffc-4a54-b50a-a35e66789f58"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 01:12:19.859933 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:19.859865    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3b8c998-7ffc-4a54-b50a-a35e66789f58-istio-token" (OuterVolumeSpecName: "istio-token") pod "f3b8c998-7ffc-4a54-b50a-a35e66789f58" (UID: "f3b8c998-7ffc-4a54-b50a-a35e66789f58"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 01:12:19.860093 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:19.860073    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f3b8c998-7ffc-4a54-b50a-a35e66789f58-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "f3b8c998-7ffc-4a54-b50a-a35e66789f58" (UID: "f3b8c998-7ffc-4a54-b50a-a35e66789f58"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 01:12:19.860325 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:19.860305    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3b8c998-7ffc-4a54-b50a-a35e66789f58-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "f3b8c998-7ffc-4a54-b50a-a35e66789f58" (UID: "f3b8c998-7ffc-4a54-b50a-a35e66789f58"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 01:12:19.860385 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:19.860330    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3b8c998-7ffc-4a54-b50a-a35e66789f58-kube-api-access-fjwqp" (OuterVolumeSpecName: "kube-api-access-fjwqp") pod "f3b8c998-7ffc-4a54-b50a-a35e66789f58" (UID: "f3b8c998-7ffc-4a54-b50a-a35e66789f58"). InnerVolumeSpecName "kube-api-access-fjwqp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 01:12:19.871963 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:19.871929    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="076eaa74-0639-4df0-9392-3c6a3ec60a9f" path="/var/lib/kubelet/pods/076eaa74-0639-4df0-9392-3c6a3ec60a9f/volumes"
Sep 11 01:12:19.958648 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:19.958624    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fjwqp\" (UniqueName: \"kubernetes.io/projected/f3b8c998-7ffc-4a54-b50a-a35e66789f58-kube-api-access-fjwqp\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:12:19.958648 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:19.958647    2569 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3b8c998-7ffc-4a54-b50a-a35e66789f58-istio-envoy\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:12:19.958790 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:19.958657    2569 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3b8c998-7ffc-4a54-b50a-a35e66789f58-istiod-ca-cert\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:12:19.958790 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:19.958665    2569 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3b8c998-7ffc-4a54-b50a-a35e66789f58-istio-podinfo\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:12:19.958790 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:19.958673    2569 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3b8c998-7ffc-4a54-b50a-a35e66789f58-istio-token\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:12:19.958790 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:19.958683    2569 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3b8c998-7ffc-4a54-b50a-a35e66789f58-workload-certs\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:12:19.958790 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:19.958690    2569 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3b8c998-7ffc-4a54-b50a-a35e66789f58-workload-socket\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:12:20.314422 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:20.314386    2569 generic.go:356] "Generic (PLEG): container finished" podID="f3b8c998-7ffc-4a54-b50a-a35e66789f58" containerID="33efcb481cfe9c59fd637a18fd9999f6e15b4116a22d3f03dc0b83af21891809" exitCode=0
Sep 11 01:12:20.314580 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:20.314429    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-aba078c2-openshift-default-6c4db56c6d-xr2wt" event={"ID":"f3b8c998-7ffc-4a54-b50a-a35e66789f58","Type":"ContainerDied","Data":"33efcb481cfe9c59fd637a18fd9999f6e15b4116a22d3f03dc0b83af21891809"}
Sep 11 01:12:20.314580 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:20.314435    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-aba078c2-openshift-default-6c4db56c6d-xr2wt"
Sep 11 01:12:20.314580 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:20.314469    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-aba078c2-openshift-default-6c4db56c6d-xr2wt" event={"ID":"f3b8c998-7ffc-4a54-b50a-a35e66789f58","Type":"ContainerDied","Data":"430ba3082fb8d5dfd2f8b8a78f3e8e6606a3f9b3b92c6813711b6722764f1ccd"}
Sep 11 01:12:20.314580 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:20.314493    2569 scope.go:117] "RemoveContainer" containerID="33efcb481cfe9c59fd637a18fd9999f6e15b4116a22d3f03dc0b83af21891809"
Sep 11 01:12:20.325074 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:20.325052    2569 scope.go:117] "RemoveContainer" containerID="4552812d0703b5de8752a3ae8b71125619d759e4200f690029a8f8630c222ac2"
Sep 11 01:12:20.333444 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:20.333426    2569 scope.go:117] "RemoveContainer" containerID="33efcb481cfe9c59fd637a18fd9999f6e15b4116a22d3f03dc0b83af21891809"
Sep 11 01:12:20.333717 ip-10-0-129-222 kubenswrapper[2569]: E0911 01:12:20.333698    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33efcb481cfe9c59fd637a18fd9999f6e15b4116a22d3f03dc0b83af21891809\": container with ID starting with 33efcb481cfe9c59fd637a18fd9999f6e15b4116a22d3f03dc0b83af21891809 not found: ID does not exist" containerID="33efcb481cfe9c59fd637a18fd9999f6e15b4116a22d3f03dc0b83af21891809"
Sep 11 01:12:20.333791 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:20.333729    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33efcb481cfe9c59fd637a18fd9999f6e15b4116a22d3f03dc0b83af21891809"} err="failed to get container status \"33efcb481cfe9c59fd637a18fd9999f6e15b4116a22d3f03dc0b83af21891809\": rpc error: code = NotFound desc = could not find container \"33efcb481cfe9c59fd637a18fd9999f6e15b4116a22d3f03dc0b83af21891809\": container with ID starting with 33efcb481cfe9c59fd637a18fd9999f6e15b4116a22d3f03dc0b83af21891809 not found: ID does not exist"
Sep 11 01:12:20.333791 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:20.333754    2569 scope.go:117] "RemoveContainer" containerID="4552812d0703b5de8752a3ae8b71125619d759e4200f690029a8f8630c222ac2"
Sep 11 01:12:20.334069 ip-10-0-129-222 kubenswrapper[2569]: E0911 01:12:20.334049    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4552812d0703b5de8752a3ae8b71125619d759e4200f690029a8f8630c222ac2\": container with ID starting with 4552812d0703b5de8752a3ae8b71125619d759e4200f690029a8f8630c222ac2 not found: ID does not exist" containerID="4552812d0703b5de8752a3ae8b71125619d759e4200f690029a8f8630c222ac2"
Sep 11 01:12:20.334112 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:20.334074    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4552812d0703b5de8752a3ae8b71125619d759e4200f690029a8f8630c222ac2"} err="failed to get container status \"4552812d0703b5de8752a3ae8b71125619d759e4200f690029a8f8630c222ac2\": rpc error: code = NotFound desc = could not find container \"4552812d0703b5de8752a3ae8b71125619d759e4200f690029a8f8630c222ac2\": container with ID starting with 4552812d0703b5de8752a3ae8b71125619d759e4200f690029a8f8630c222ac2 not found: ID does not exist"
Sep 11 01:12:20.363315 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:20.363285    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-aba078c2-openshift-default-6c4db56c6d-xr2wt"]
Sep 11 01:12:20.368590 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:20.368566    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-aba078c2-openshift-default-6c4db56c6d-xr2wt"]
Sep 11 01:12:21.871173 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:21.871141    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f3b8c998-7ffc-4a54-b50a-a35e66789f58" path="/var/lib/kubelet/pods/f3b8c998-7ffc-4a54-b50a-a35e66789f58/volumes"
Sep 11 01:12:29.484253 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:29.484222    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-dc7dcd-openshift-default-5958f7974d-457sp"]
Sep 11 01:12:29.484751 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:29.484482    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-dc7dcd-openshift-default-5958f7974d-457sp" podUID="8ec0f171-1bff-4174-9905-fe855ce1ad42" containerName="istio-proxy" containerID="cri-o://288ad8bcd36951b193884c4c5d9e687666efe6edd615bf7fa1a2d2b907184732" gracePeriod=30
Sep 11 01:12:30.218238 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:30.218201    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-explicit-8d451c-openshift-default-7b5868fcdd-g9rzj"]
Sep 11 01:12:30.218666 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:30.218649    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f3b8c998-7ffc-4a54-b50a-a35e66789f58" containerName="istio-proxy"
Sep 11 01:12:30.218666 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:30.218665    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="f3b8c998-7ffc-4a54-b50a-a35e66789f58" containerName="istio-proxy"
Sep 11 01:12:30.218817 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:30.218685    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="076eaa74-0639-4df0-9392-3c6a3ec60a9f" containerName="payload-pre-processing"
Sep 11 01:12:30.218817 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:30.218691    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="076eaa74-0639-4df0-9392-3c6a3ec60a9f" containerName="payload-pre-processing"
Sep 11 01:12:30.218817 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:30.218700    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f3b8c998-7ffc-4a54-b50a-a35e66789f58" containerName="istio-proxy"
Sep 11 01:12:30.218817 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:30.218706    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="f3b8c998-7ffc-4a54-b50a-a35e66789f58" containerName="istio-proxy"
Sep 11 01:12:30.218817 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:30.218768    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="076eaa74-0639-4df0-9392-3c6a3ec60a9f" containerName="payload-pre-processing"
Sep 11 01:12:30.218817 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:30.218777    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="f3b8c998-7ffc-4a54-b50a-a35e66789f58" containerName="istio-proxy"
Sep 11 01:12:30.223494 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:30.223471    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-8d451c-openshift-default-7b5868fcdd-g9rzj"
Sep 11 01:12:30.225694 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:30.225672    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-explicit-8d451c-openshift-default-dockercfg-px5gd\""
Sep 11 01:12:30.243945 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:30.235448    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-8d451c-openshift-default-7b5868fcdd-g9rzj"]
Sep 11 01:12:30.262492 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:30.262456    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6c2f5c6-7f99-4519-acf1-7af6c423bc0f-workload-certs\") pod \"e2e-explicit-8d451c-openshift-default-7b5868fcdd-g9rzj\" (UID: \"f6c2f5c6-7f99-4519-acf1-7af6c423bc0f\") " pod="openshift-ingress/e2e-explicit-8d451c-openshift-default-7b5868fcdd-g9rzj"
Sep 11 01:12:30.262748 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:30.262728    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6c2f5c6-7f99-4519-acf1-7af6c423bc0f-istio-data\") pod \"e2e-explicit-8d451c-openshift-default-7b5868fcdd-g9rzj\" (UID: \"f6c2f5c6-7f99-4519-acf1-7af6c423bc0f\") " pod="openshift-ingress/e2e-explicit-8d451c-openshift-default-7b5868fcdd-g9rzj"
Sep 11 01:12:30.262911 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:30.262894    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6c2f5c6-7f99-4519-acf1-7af6c423bc0f-istiod-ca-cert\") pod \"e2e-explicit-8d451c-openshift-default-7b5868fcdd-g9rzj\" (UID: \"f6c2f5c6-7f99-4519-acf1-7af6c423bc0f\") " pod="openshift-ingress/e2e-explicit-8d451c-openshift-default-7b5868fcdd-g9rzj"
Sep 11 01:12:30.263057 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:30.263042    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6c2f5c6-7f99-4519-acf1-7af6c423bc0f-workload-socket\") pod \"e2e-explicit-8d451c-openshift-default-7b5868fcdd-g9rzj\" (UID: \"f6c2f5c6-7f99-4519-acf1-7af6c423bc0f\") " pod="openshift-ingress/e2e-explicit-8d451c-openshift-default-7b5868fcdd-g9rzj"
Sep 11 01:12:30.263186 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:30.263171    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6c2f5c6-7f99-4519-acf1-7af6c423bc0f-istio-token\") pod \"e2e-explicit-8d451c-openshift-default-7b5868fcdd-g9rzj\" (UID: \"f6c2f5c6-7f99-4519-acf1-7af6c423bc0f\") " pod="openshift-ingress/e2e-explicit-8d451c-openshift-default-7b5868fcdd-g9rzj"
Sep 11 01:12:30.263335 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:30.263321    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6c2f5c6-7f99-4519-acf1-7af6c423bc0f-credential-socket\") pod \"e2e-explicit-8d451c-openshift-default-7b5868fcdd-g9rzj\" (UID: \"f6c2f5c6-7f99-4519-acf1-7af6c423bc0f\") " pod="openshift-ingress/e2e-explicit-8d451c-openshift-default-7b5868fcdd-g9rzj"
Sep 11 01:12:30.263470 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:30.263454    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6c2f5c6-7f99-4519-acf1-7af6c423bc0f-istio-envoy\") pod \"e2e-explicit-8d451c-openshift-default-7b5868fcdd-g9rzj\" (UID: \"f6c2f5c6-7f99-4519-acf1-7af6c423bc0f\") " pod="openshift-ingress/e2e-explicit-8d451c-openshift-default-7b5868fcdd-g9rzj"
Sep 11 01:12:30.263653 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:30.263637    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6c2f5c6-7f99-4519-acf1-7af6c423bc0f-istio-podinfo\") pod \"e2e-explicit-8d451c-openshift-default-7b5868fcdd-g9rzj\" (UID: \"f6c2f5c6-7f99-4519-acf1-7af6c423bc0f\") " pod="openshift-ingress/e2e-explicit-8d451c-openshift-default-7b5868fcdd-g9rzj"
Sep 11 01:12:30.263815 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:30.263799    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24wfx\" (UniqueName: \"kubernetes.io/projected/f6c2f5c6-7f99-4519-acf1-7af6c423bc0f-kube-api-access-24wfx\") pod \"e2e-explicit-8d451c-openshift-default-7b5868fcdd-g9rzj\" (UID: \"f6c2f5c6-7f99-4519-acf1-7af6c423bc0f\") " pod="openshift-ingress/e2e-explicit-8d451c-openshift-default-7b5868fcdd-g9rzj"
Sep 11 01:12:30.365960 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:30.365486    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6c2f5c6-7f99-4519-acf1-7af6c423bc0f-istio-podinfo\") pod \"e2e-explicit-8d451c-openshift-default-7b5868fcdd-g9rzj\" (UID: \"f6c2f5c6-7f99-4519-acf1-7af6c423bc0f\") " pod="openshift-ingress/e2e-explicit-8d451c-openshift-default-7b5868fcdd-g9rzj"
Sep 11 01:12:30.365960 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:30.365563    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24wfx\" (UniqueName: \"kubernetes.io/projected/f6c2f5c6-7f99-4519-acf1-7af6c423bc0f-kube-api-access-24wfx\") pod \"e2e-explicit-8d451c-openshift-default-7b5868fcdd-g9rzj\" (UID: \"f6c2f5c6-7f99-4519-acf1-7af6c423bc0f\") " pod="openshift-ingress/e2e-explicit-8d451c-openshift-default-7b5868fcdd-g9rzj"
Sep 11 01:12:30.365960 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:30.365633    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6c2f5c6-7f99-4519-acf1-7af6c423bc0f-workload-certs\") pod \"e2e-explicit-8d451c-openshift-default-7b5868fcdd-g9rzj\" (UID: \"f6c2f5c6-7f99-4519-acf1-7af6c423bc0f\") " pod="openshift-ingress/e2e-explicit-8d451c-openshift-default-7b5868fcdd-g9rzj"
Sep 11 01:12:30.365960 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:30.365675    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6c2f5c6-7f99-4519-acf1-7af6c423bc0f-istio-data\") pod \"e2e-explicit-8d451c-openshift-default-7b5868fcdd-g9rzj\" (UID: \"f6c2f5c6-7f99-4519-acf1-7af6c423bc0f\") " pod="openshift-ingress/e2e-explicit-8d451c-openshift-default-7b5868fcdd-g9rzj"
Sep 11 01:12:30.365960 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:30.365706    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6c2f5c6-7f99-4519-acf1-7af6c423bc0f-istiod-ca-cert\") pod \"e2e-explicit-8d451c-openshift-default-7b5868fcdd-g9rzj\" (UID: \"f6c2f5c6-7f99-4519-acf1-7af6c423bc0f\") " pod="openshift-ingress/e2e-explicit-8d451c-openshift-default-7b5868fcdd-g9rzj"
Sep 11 01:12:30.365960 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:30.365750    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6c2f5c6-7f99-4519-acf1-7af6c423bc0f-workload-socket\") pod \"e2e-explicit-8d451c-openshift-default-7b5868fcdd-g9rzj\" (UID: \"f6c2f5c6-7f99-4519-acf1-7af6c423bc0f\") " pod="openshift-ingress/e2e-explicit-8d451c-openshift-default-7b5868fcdd-g9rzj"
Sep 11 01:12:30.365960 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:30.365775    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6c2f5c6-7f99-4519-acf1-7af6c423bc0f-istio-token\") pod \"e2e-explicit-8d451c-openshift-default-7b5868fcdd-g9rzj\" (UID: \"f6c2f5c6-7f99-4519-acf1-7af6c423bc0f\") " pod="openshift-ingress/e2e-explicit-8d451c-openshift-default-7b5868fcdd-g9rzj"
Sep 11 01:12:30.365960 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:30.365811    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6c2f5c6-7f99-4519-acf1-7af6c423bc0f-credential-socket\") pod \"e2e-explicit-8d451c-openshift-default-7b5868fcdd-g9rzj\" (UID: \"f6c2f5c6-7f99-4519-acf1-7af6c423bc0f\") " pod="openshift-ingress/e2e-explicit-8d451c-openshift-default-7b5868fcdd-g9rzj"
Sep 11 01:12:30.365960 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:30.365837    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6c2f5c6-7f99-4519-acf1-7af6c423bc0f-istio-envoy\") pod \"e2e-explicit-8d451c-openshift-default-7b5868fcdd-g9rzj\" (UID: \"f6c2f5c6-7f99-4519-acf1-7af6c423bc0f\") " pod="openshift-ingress/e2e-explicit-8d451c-openshift-default-7b5868fcdd-g9rzj"
Sep 11 01:12:30.367056 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:30.367026    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6c2f5c6-7f99-4519-acf1-7af6c423bc0f-istio-data\") pod \"e2e-explicit-8d451c-openshift-default-7b5868fcdd-g9rzj\" (UID: \"f6c2f5c6-7f99-4519-acf1-7af6c423bc0f\") " pod="openshift-ingress/e2e-explicit-8d451c-openshift-default-7b5868fcdd-g9rzj"
Sep 11 01:12:30.367289 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:30.367260    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6c2f5c6-7f99-4519-acf1-7af6c423bc0f-workload-socket\") pod \"e2e-explicit-8d451c-openshift-default-7b5868fcdd-g9rzj\" (UID: \"f6c2f5c6-7f99-4519-acf1-7af6c423bc0f\") " pod="openshift-ingress/e2e-explicit-8d451c-openshift-default-7b5868fcdd-g9rzj"
Sep 11 01:12:30.367468 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:30.367290    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6c2f5c6-7f99-4519-acf1-7af6c423bc0f-credential-socket\") pod \"e2e-explicit-8d451c-openshift-default-7b5868fcdd-g9rzj\" (UID: \"f6c2f5c6-7f99-4519-acf1-7af6c423bc0f\") " pod="openshift-ingress/e2e-explicit-8d451c-openshift-default-7b5868fcdd-g9rzj"
Sep 11 01:12:30.367617 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:30.367596    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6c2f5c6-7f99-4519-acf1-7af6c423bc0f-workload-certs\") pod \"e2e-explicit-8d451c-openshift-default-7b5868fcdd-g9rzj\" (UID: \"f6c2f5c6-7f99-4519-acf1-7af6c423bc0f\") " pod="openshift-ingress/e2e-explicit-8d451c-openshift-default-7b5868fcdd-g9rzj"
Sep 11 01:12:30.367688 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:30.367626    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6c2f5c6-7f99-4519-acf1-7af6c423bc0f-istiod-ca-cert\") pod \"e2e-explicit-8d451c-openshift-default-7b5868fcdd-g9rzj\" (UID: \"f6c2f5c6-7f99-4519-acf1-7af6c423bc0f\") " pod="openshift-ingress/e2e-explicit-8d451c-openshift-default-7b5868fcdd-g9rzj"
Sep 11 01:12:30.368683 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:30.368651    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6c2f5c6-7f99-4519-acf1-7af6c423bc0f-istio-envoy\") pod \"e2e-explicit-8d451c-openshift-default-7b5868fcdd-g9rzj\" (UID: \"f6c2f5c6-7f99-4519-acf1-7af6c423bc0f\") " pod="openshift-ingress/e2e-explicit-8d451c-openshift-default-7b5868fcdd-g9rzj"
Sep 11 01:12:30.369347 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:30.369326    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6c2f5c6-7f99-4519-acf1-7af6c423bc0f-istio-podinfo\") pod \"e2e-explicit-8d451c-openshift-default-7b5868fcdd-g9rzj\" (UID: \"f6c2f5c6-7f99-4519-acf1-7af6c423bc0f\") " pod="openshift-ingress/e2e-explicit-8d451c-openshift-default-7b5868fcdd-g9rzj"
Sep 11 01:12:30.375918 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:30.375893    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-24wfx\" (UniqueName: \"kubernetes.io/projected/f6c2f5c6-7f99-4519-acf1-7af6c423bc0f-kube-api-access-24wfx\") pod \"e2e-explicit-8d451c-openshift-default-7b5868fcdd-g9rzj\" (UID: \"f6c2f5c6-7f99-4519-acf1-7af6c423bc0f\") " pod="openshift-ingress/e2e-explicit-8d451c-openshift-default-7b5868fcdd-g9rzj"
Sep 11 01:12:30.376709 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:30.376688    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6c2f5c6-7f99-4519-acf1-7af6c423bc0f-istio-token\") pod \"e2e-explicit-8d451c-openshift-default-7b5868fcdd-g9rzj\" (UID: \"f6c2f5c6-7f99-4519-acf1-7af6c423bc0f\") " pod="openshift-ingress/e2e-explicit-8d451c-openshift-default-7b5868fcdd-g9rzj"
Sep 11 01:12:30.533797 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:30.533706    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-8d451c-openshift-default-7b5868fcdd-g9rzj"
Sep 11 01:12:30.827749 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:30.827726    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-8d451c-openshift-default-7b5868fcdd-g9rzj"]
Sep 11 01:12:30.829613 ip-10-0-129-222 kubenswrapper[2569]: W0911 01:12:30.829587    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6c2f5c6_7f99_4519_acf1_7af6c423bc0f.slice/crio-8ddc4e2ebfb667d4d21fb6df54fc539ac819aef5142a14942b9c1a636b0f8758 WatchSource:0}: Error finding container 8ddc4e2ebfb667d4d21fb6df54fc539ac819aef5142a14942b9c1a636b0f8758: Status 404 returned error can't find the container with id 8ddc4e2ebfb667d4d21fb6df54fc539ac819aef5142a14942b9c1a636b0f8758
Sep 11 01:12:30.832003 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:30.831966    2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 01:12:30.832109 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:30.832031    2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 01:12:30.832109 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:30.832061    2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 01:12:31.383958 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:31.383911    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-8d451c-openshift-default-7b5868fcdd-g9rzj" event={"ID":"f6c2f5c6-7f99-4519-acf1-7af6c423bc0f","Type":"ContainerStarted","Data":"17c42f6abbe8776859d57d14880e986f7c0b9a8b71ee157f8440c0109b535f1a"}
Sep 11 01:12:31.384146 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:31.383979    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-8d451c-openshift-default-7b5868fcdd-g9rzj" event={"ID":"f6c2f5c6-7f99-4519-acf1-7af6c423bc0f","Type":"ContainerStarted","Data":"8ddc4e2ebfb667d4d21fb6df54fc539ac819aef5142a14942b9c1a636b0f8758"}
Sep 11 01:12:31.406393 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:31.406343    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-explicit-8d451c-openshift-default-7b5868fcdd-g9rzj" podStartSLOduration=1.406328533 podStartE2EDuration="1.406328533s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 01:12:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 01:12:31.405110631 +0000 UTC m=+886.056484894" watchObservedRunningTime="2026-09-11 01:12:31.406328533 +0000 UTC m=+886.057702842"
Sep 11 01:12:31.534116 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:31.534082    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-8d451c-openshift-default-7b5868fcdd-g9rzj"
Sep 11 01:12:31.549148 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:31.549120    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-8d451c-openshift-default-7b5868fcdd-g9rzj"
Sep 11 01:12:32.391107 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:32.391068    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-8d451c-openshift-default-7b5868fcdd-g9rzj"
Sep 11 01:12:32.392207 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:32.392176    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-8d451c-openshift-default-7b5868fcdd-g9rzj"
Sep 11 01:12:33.336043 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:33.336000    2569 patch_prober.go:28] interesting pod/e2e-derive-39ea7c6f-openshift-default-566bd9c7d4-j2v5j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 11 01:12:33.336404 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:33.336067    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-39ea7c6f-openshift-default-566bd9c7d4-j2v5j" podUID="69502d0e-5d63-45aa-8724-678b467e51b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 11 01:12:34.740371 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:34.740340    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-dc7dcd-openshift-default-5958f7974d-457sp"
Sep 11 01:12:34.809164 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:34.809135    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ec0f171-1bff-4174-9905-fe855ce1ad42-istiod-ca-cert\") pod \"8ec0f171-1bff-4174-9905-fe855ce1ad42\" (UID: \"8ec0f171-1bff-4174-9905-fe855ce1ad42\") "
Sep 11 01:12:34.809334 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:34.809208    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ec0f171-1bff-4174-9905-fe855ce1ad42-workload-socket\") pod \"8ec0f171-1bff-4174-9905-fe855ce1ad42\" (UID: \"8ec0f171-1bff-4174-9905-fe855ce1ad42\") "
Sep 11 01:12:34.809334 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:34.809236    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ec0f171-1bff-4174-9905-fe855ce1ad42-istio-data\") pod \"8ec0f171-1bff-4174-9905-fe855ce1ad42\" (UID: \"8ec0f171-1bff-4174-9905-fe855ce1ad42\") "
Sep 11 01:12:34.809334 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:34.809260    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b4pfh\" (UniqueName: \"kubernetes.io/projected/8ec0f171-1bff-4174-9905-fe855ce1ad42-kube-api-access-b4pfh\") pod \"8ec0f171-1bff-4174-9905-fe855ce1ad42\" (UID: \"8ec0f171-1bff-4174-9905-fe855ce1ad42\") "
Sep 11 01:12:34.809334 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:34.809291    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ec0f171-1bff-4174-9905-fe855ce1ad42-credential-socket\") pod \"8ec0f171-1bff-4174-9905-fe855ce1ad42\" (UID: \"8ec0f171-1bff-4174-9905-fe855ce1ad42\") "
Sep 11 01:12:34.809334 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:34.809313    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ec0f171-1bff-4174-9905-fe855ce1ad42-workload-certs\") pod \"8ec0f171-1bff-4174-9905-fe855ce1ad42\" (UID: \"8ec0f171-1bff-4174-9905-fe855ce1ad42\") "
Sep 11 01:12:34.809591 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:34.809424    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ec0f171-1bff-4174-9905-fe855ce1ad42-istio-envoy\") pod \"8ec0f171-1bff-4174-9905-fe855ce1ad42\" (UID: \"8ec0f171-1bff-4174-9905-fe855ce1ad42\") "
Sep 11 01:12:34.809591 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:34.809473    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ec0f171-1bff-4174-9905-fe855ce1ad42-istio-podinfo\") pod \"8ec0f171-1bff-4174-9905-fe855ce1ad42\" (UID: \"8ec0f171-1bff-4174-9905-fe855ce1ad42\") "
Sep 11 01:12:34.809591 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:34.809481    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ec0f171-1bff-4174-9905-fe855ce1ad42-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "8ec0f171-1bff-4174-9905-fe855ce1ad42" (UID: "8ec0f171-1bff-4174-9905-fe855ce1ad42"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 01:12:34.809591 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:34.809516    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ec0f171-1bff-4174-9905-fe855ce1ad42-istio-token\") pod \"8ec0f171-1bff-4174-9905-fe855ce1ad42\" (UID: \"8ec0f171-1bff-4174-9905-fe855ce1ad42\") "
Sep 11 01:12:34.809591 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:34.809540    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ec0f171-1bff-4174-9905-fe855ce1ad42-istio-data" (OuterVolumeSpecName: "istio-data") pod "8ec0f171-1bff-4174-9905-fe855ce1ad42" (UID: "8ec0f171-1bff-4174-9905-fe855ce1ad42"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 01:12:34.809591 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:34.809558    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ec0f171-1bff-4174-9905-fe855ce1ad42-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "8ec0f171-1bff-4174-9905-fe855ce1ad42" (UID: "8ec0f171-1bff-4174-9905-fe855ce1ad42"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 01:12:34.809918 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:34.809620    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ec0f171-1bff-4174-9905-fe855ce1ad42-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "8ec0f171-1bff-4174-9905-fe855ce1ad42" (UID: "8ec0f171-1bff-4174-9905-fe855ce1ad42"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 01:12:34.809918 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:34.809647    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ec0f171-1bff-4174-9905-fe855ce1ad42-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "8ec0f171-1bff-4174-9905-fe855ce1ad42" (UID: "8ec0f171-1bff-4174-9905-fe855ce1ad42"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 01:12:34.810036 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:34.809969    2569 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ec0f171-1bff-4174-9905-fe855ce1ad42-istiod-ca-cert\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:12:34.810036 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:34.810007    2569 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ec0f171-1bff-4174-9905-fe855ce1ad42-workload-socket\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:12:34.810036 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:34.810022    2569 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ec0f171-1bff-4174-9905-fe855ce1ad42-istio-data\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:12:34.810172 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:34.810036    2569 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ec0f171-1bff-4174-9905-fe855ce1ad42-credential-socket\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:12:34.810172 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:34.810049    2569 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ec0f171-1bff-4174-9905-fe855ce1ad42-workload-certs\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:12:34.811636 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:34.811610    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ec0f171-1bff-4174-9905-fe855ce1ad42-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "8ec0f171-1bff-4174-9905-fe855ce1ad42" (UID: "8ec0f171-1bff-4174-9905-fe855ce1ad42"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 01:12:34.811729 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:34.811620    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ec0f171-1bff-4174-9905-fe855ce1ad42-istio-token" (OuterVolumeSpecName: "istio-token") pod "8ec0f171-1bff-4174-9905-fe855ce1ad42" (UID: "8ec0f171-1bff-4174-9905-fe855ce1ad42"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 01:12:34.811925 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:34.811905    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8ec0f171-1bff-4174-9905-fe855ce1ad42-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "8ec0f171-1bff-4174-9905-fe855ce1ad42" (UID: "8ec0f171-1bff-4174-9905-fe855ce1ad42"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 01:12:34.811980 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:34.811958    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ec0f171-1bff-4174-9905-fe855ce1ad42-kube-api-access-b4pfh" (OuterVolumeSpecName: "kube-api-access-b4pfh") pod "8ec0f171-1bff-4174-9905-fe855ce1ad42" (UID: "8ec0f171-1bff-4174-9905-fe855ce1ad42"). InnerVolumeSpecName "kube-api-access-b4pfh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 01:12:34.910739 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:34.910712    2569 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ec0f171-1bff-4174-9905-fe855ce1ad42-istio-envoy\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:12:34.912097 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:34.910742    2569 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ec0f171-1bff-4174-9905-fe855ce1ad42-istio-podinfo\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:12:34.912097 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:34.910756    2569 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ec0f171-1bff-4174-9905-fe855ce1ad42-istio-token\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:12:34.912097 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:34.910769    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b4pfh\" (UniqueName: \"kubernetes.io/projected/8ec0f171-1bff-4174-9905-fe855ce1ad42-kube-api-access-b4pfh\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:12:35.078656 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:35.078586    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-8d451c-7d7b4b9f74-xw47v"]
Sep 11 01:12:35.079088 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:35.079073    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ec0f171-1bff-4174-9905-fe855ce1ad42" containerName="istio-proxy"
Sep 11 01:12:35.079143 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:35.079091    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ec0f171-1bff-4174-9905-fe855ce1ad42" containerName="istio-proxy"
Sep 11 01:12:35.079143 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:35.079115    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ec0f171-1bff-4174-9905-fe855ce1ad42" containerName="istio-proxy"
Sep 11 01:12:35.079143 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:35.079122    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ec0f171-1bff-4174-9905-fe855ce1ad42" containerName="istio-proxy"
Sep 11 01:12:35.079246 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:35.079196    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ec0f171-1bff-4174-9905-fe855ce1ad42" containerName="istio-proxy"
Sep 11 01:12:35.079246 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:35.079208    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="f3b8c998-7ffc-4a54-b50a-a35e66789f58" containerName="istio-proxy"
Sep 11 01:12:35.081328 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:35.081313    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-8d451c-7d7b4b9f74-xw47v"
Sep 11 01:12:35.083940 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:35.083920    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-8d451c-dockercfg-nlhpd\""
Sep 11 01:12:35.084058 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:35.083928    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-8d451c\""
Sep 11 01:12:35.092578 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:35.092556    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-8d451c-7d7b4b9f74-xw47v"]
Sep 11 01:12:35.213780 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:35.213698    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6qg5\" (UniqueName: \"kubernetes.io/projected/f9868c6b-59cb-4e62-9979-b3a36b4fa2e2-kube-api-access-c6qg5\") pod \"payload-pre-processing-e2e-explicit-8d451c-7d7b4b9f74-xw47v\" (UID: \"f9868c6b-59cb-4e62-9979-b3a36b4fa2e2\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-8d451c-7d7b4b9f74-xw47v"
Sep 11 01:12:35.213780 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:35.213737    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9868c6b-59cb-4e62-9979-b3a36b4fa2e2-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-8d451c-7d7b4b9f74-xw47v\" (UID: \"f9868c6b-59cb-4e62-9979-b3a36b4fa2e2\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-8d451c-7d7b4b9f74-xw47v"
Sep 11 01:12:35.315390 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:35.315360    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6qg5\" (UniqueName: \"kubernetes.io/projected/f9868c6b-59cb-4e62-9979-b3a36b4fa2e2-kube-api-access-c6qg5\") pod \"payload-pre-processing-e2e-explicit-8d451c-7d7b4b9f74-xw47v\" (UID: \"f9868c6b-59cb-4e62-9979-b3a36b4fa2e2\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-8d451c-7d7b4b9f74-xw47v"
Sep 11 01:12:35.315548 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:35.315404    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9868c6b-59cb-4e62-9979-b3a36b4fa2e2-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-8d451c-7d7b4b9f74-xw47v\" (UID: \"f9868c6b-59cb-4e62-9979-b3a36b4fa2e2\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-8d451c-7d7b4b9f74-xw47v"
Sep 11 01:12:35.316197 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:35.316177    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9868c6b-59cb-4e62-9979-b3a36b4fa2e2-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-8d451c-7d7b4b9f74-xw47v\" (UID: \"f9868c6b-59cb-4e62-9979-b3a36b4fa2e2\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-8d451c-7d7b4b9f74-xw47v"
Sep 11 01:12:35.326867 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:35.326837    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6qg5\" (UniqueName: \"kubernetes.io/projected/f9868c6b-59cb-4e62-9979-b3a36b4fa2e2-kube-api-access-c6qg5\") pod \"payload-pre-processing-e2e-explicit-8d451c-7d7b4b9f74-xw47v\" (UID: \"f9868c6b-59cb-4e62-9979-b3a36b4fa2e2\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-8d451c-7d7b4b9f74-xw47v"
Sep 11 01:12:35.392686 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:35.392616    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-8d451c-7d7b4b9f74-xw47v"
Sep 11 01:12:35.411114 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:35.411091    2569 generic.go:356] "Generic (PLEG): container finished" podID="8ec0f171-1bff-4174-9905-fe855ce1ad42" containerID="288ad8bcd36951b193884c4c5d9e687666efe6edd615bf7fa1a2d2b907184732" exitCode=0
Sep 11 01:12:35.411249 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:35.411151    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-dc7dcd-openshift-default-5958f7974d-457sp"
Sep 11 01:12:35.411249 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:35.411169    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-dc7dcd-openshift-default-5958f7974d-457sp" event={"ID":"8ec0f171-1bff-4174-9905-fe855ce1ad42","Type":"ContainerDied","Data":"288ad8bcd36951b193884c4c5d9e687666efe6edd615bf7fa1a2d2b907184732"}
Sep 11 01:12:35.411249 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:35.411217    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-dc7dcd-openshift-default-5958f7974d-457sp" event={"ID":"8ec0f171-1bff-4174-9905-fe855ce1ad42","Type":"ContainerDied","Data":"554b6711b5b0118399de69a8ffea15ea105d29185066540267766d9a13f9b48b"}
Sep 11 01:12:35.411249 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:35.411232    2569 scope.go:117] "RemoveContainer" containerID="288ad8bcd36951b193884c4c5d9e687666efe6edd615bf7fa1a2d2b907184732"
Sep 11 01:12:35.413174 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:35.413150    2569 generic.go:356] "Generic (PLEG): container finished" podID="69502d0e-5d63-45aa-8724-678b467e51b4" containerID="76211d9197ad0889fabe0d7e3090b011bd3df6a5f7ac19cff8407aedd9acb20b" exitCode=0
Sep 11 01:12:35.413301 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:35.413226    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-39ea7c6f-openshift-default-566bd9c7d4-j2v5j" event={"ID":"69502d0e-5d63-45aa-8724-678b467e51b4","Type":"ContainerDied","Data":"76211d9197ad0889fabe0d7e3090b011bd3df6a5f7ac19cff8407aedd9acb20b"}
Sep 11 01:12:35.413632 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:35.413611    2569 scope.go:117] "RemoveContainer" containerID="76211d9197ad0889fabe0d7e3090b011bd3df6a5f7ac19cff8407aedd9acb20b"
Sep 11 01:12:35.414989 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:35.414956    2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 01:12:35.415093 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:35.415040    2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 01:12:35.415153 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:35.415089    2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 01:12:35.421668 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:35.421636    2569 scope.go:117] "RemoveContainer" containerID="46644ea2b24175c4fed14bd8c07f971173c7c4f2862c4187f966b7b691a88a21"
Sep 11 01:12:35.430554 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:35.430535    2569 scope.go:117] "RemoveContainer" containerID="288ad8bcd36951b193884c4c5d9e687666efe6edd615bf7fa1a2d2b907184732"
Sep 11 01:12:35.430853 ip-10-0-129-222 kubenswrapper[2569]: E0911 01:12:35.430827    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"288ad8bcd36951b193884c4c5d9e687666efe6edd615bf7fa1a2d2b907184732\": container with ID starting with 288ad8bcd36951b193884c4c5d9e687666efe6edd615bf7fa1a2d2b907184732 not found: ID does not exist" containerID="288ad8bcd36951b193884c4c5d9e687666efe6edd615bf7fa1a2d2b907184732"
Sep 11 01:12:35.430963 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:35.430864    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"288ad8bcd36951b193884c4c5d9e687666efe6edd615bf7fa1a2d2b907184732"} err="failed to get container status \"288ad8bcd36951b193884c4c5d9e687666efe6edd615bf7fa1a2d2b907184732\": rpc error: code = NotFound desc = could not find container \"288ad8bcd36951b193884c4c5d9e687666efe6edd615bf7fa1a2d2b907184732\": container with ID starting with 288ad8bcd36951b193884c4c5d9e687666efe6edd615bf7fa1a2d2b907184732 not found: ID does not exist"
Sep 11 01:12:35.430963 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:35.430929    2569 scope.go:117] "RemoveContainer" containerID="46644ea2b24175c4fed14bd8c07f971173c7c4f2862c4187f966b7b691a88a21"
Sep 11 01:12:35.432098 ip-10-0-129-222 kubenswrapper[2569]: E0911 01:12:35.431309    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46644ea2b24175c4fed14bd8c07f971173c7c4f2862c4187f966b7b691a88a21\": container with ID starting with 46644ea2b24175c4fed14bd8c07f971173c7c4f2862c4187f966b7b691a88a21 not found: ID does not exist" containerID="46644ea2b24175c4fed14bd8c07f971173c7c4f2862c4187f966b7b691a88a21"
Sep 11 01:12:35.432098 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:35.431347    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46644ea2b24175c4fed14bd8c07f971173c7c4f2862c4187f966b7b691a88a21"} err="failed to get container status \"46644ea2b24175c4fed14bd8c07f971173c7c4f2862c4187f966b7b691a88a21\": rpc error: code = NotFound desc = could not find container \"46644ea2b24175c4fed14bd8c07f971173c7c4f2862c4187f966b7b691a88a21\": container with ID starting with 46644ea2b24175c4fed14bd8c07f971173c7c4f2862c4187f966b7b691a88a21 not found: ID does not exist"
Sep 11 01:12:35.462205 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:35.461992    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-dc7dcd-openshift-default-5958f7974d-457sp"]
Sep 11 01:12:35.466330 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:35.466307    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-dc7dcd-openshift-default-5958f7974d-457sp"]
Sep 11 01:12:35.670459 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:35.670375    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-8d451c-7d7b4b9f74-xw47v"]
Sep 11 01:12:35.673622 ip-10-0-129-222 kubenswrapper[2569]: W0911 01:12:35.673589    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9868c6b_59cb_4e62_9979_b3a36b4fa2e2.slice/crio-34b7453f92c5a29ee2de8fe97f067ccd84621349aa446e4e717170e8c223b92d WatchSource:0}: Error finding container 34b7453f92c5a29ee2de8fe97f067ccd84621349aa446e4e717170e8c223b92d: Status 404 returned error can't find the container with id 34b7453f92c5a29ee2de8fe97f067ccd84621349aa446e4e717170e8c223b92d
Sep 11 01:12:35.871854 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:35.871824    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8ec0f171-1bff-4174-9905-fe855ce1ad42" path="/var/lib/kubelet/pods/8ec0f171-1bff-4174-9905-fe855ce1ad42/volumes"
Sep 11 01:12:36.210354 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:36.210324    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kmx5x"]
Sep 11 01:12:36.421129 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:36.421094    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-39ea7c6f-openshift-default-566bd9c7d4-j2v5j" event={"ID":"69502d0e-5d63-45aa-8724-678b467e51b4","Type":"ContainerStarted","Data":"c67e13264c70eb5e6146ff39d6650cd96aa0e44d8d9c8241ec0b696720f4502b"}
Sep 11 01:12:36.422726 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:36.422696    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-8d451c-7d7b4b9f74-xw47v" event={"ID":"f9868c6b-59cb-4e62-9979-b3a36b4fa2e2","Type":"ContainerStarted","Data":"9984423a1c56e1420832ef81af056f596a5f08aced94667feae807195604db4e"}
Sep 11 01:12:36.422860 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:36.422737    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-8d451c-7d7b4b9f74-xw47v" event={"ID":"f9868c6b-59cb-4e62-9979-b3a36b4fa2e2","Type":"ContainerStarted","Data":"34b7453f92c5a29ee2de8fe97f067ccd84621349aa446e4e717170e8c223b92d"}
Sep 11 01:12:36.422860 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:36.422794    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-8d451c-7d7b4b9f74-xw47v"
Sep 11 01:12:36.461811 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:36.461670    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-explicit-8d451c-7d7b4b9f74-xw47v" podStartSLOduration=1.461646917 podStartE2EDuration="1.461646917s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 01:12:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 01:12:36.458132459 +0000 UTC m=+891.109506710" watchObservedRunningTime="2026-09-11 01:12:36.461646917 +0000 UTC m=+891.113021158"
Sep 11 01:12:36.636364 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:36.636320    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-39ea7c6f-openshift-default-566bd9c7d4-j2v5j"
Sep 11 01:12:36.636631 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:36.636604    2569 patch_prober.go:28] interesting pod/e2e-derive-39ea7c6f-openshift-default-566bd9c7d4-j2v5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body=
Sep 11 01:12:36.636735 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:36.636666    2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-39ea7c6f-openshift-default-566bd9c7d4-j2v5j" podUID="69502d0e-5d63-45aa-8724-678b467e51b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused"
Sep 11 01:12:37.636420 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:37.636377    2569 patch_prober.go:28] interesting pod/e2e-derive-39ea7c6f-openshift-default-566bd9c7d4-j2v5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body=
Sep 11 01:12:37.637033 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:37.636999    2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-39ea7c6f-openshift-default-566bd9c7d4-j2v5j" podUID="69502d0e-5d63-45aa-8724-678b467e51b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused"
Sep 11 01:12:38.637029 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:38.636984    2569 patch_prober.go:28] interesting pod/e2e-derive-39ea7c6f-openshift-default-566bd9c7d4-j2v5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body=
Sep 11 01:12:38.637496 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:38.637057    2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-39ea7c6f-openshift-default-566bd9c7d4-j2v5j" podUID="69502d0e-5d63-45aa-8724-678b467e51b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused"
Sep 11 01:12:39.636776 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:39.636734    2569 patch_prober.go:28] interesting pod/e2e-derive-39ea7c6f-openshift-default-566bd9c7d4-j2v5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body=
Sep 11 01:12:39.636972 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:39.636804    2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-39ea7c6f-openshift-default-566bd9c7d4-j2v5j" podUID="69502d0e-5d63-45aa-8724-678b467e51b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused"
Sep 11 01:12:40.636405 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:40.636365    2569 patch_prober.go:28] interesting pod/e2e-derive-39ea7c6f-openshift-default-566bd9c7d4-j2v5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body=
Sep 11 01:12:40.637000 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:40.636966    2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-39ea7c6f-openshift-default-566bd9c7d4-j2v5j" podUID="69502d0e-5d63-45aa-8724-678b467e51b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused"
Sep 11 01:12:41.636392 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:41.636352    2569 patch_prober.go:28] interesting pod/e2e-derive-39ea7c6f-openshift-default-566bd9c7d4-j2v5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body=
Sep 11 01:12:41.636567 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:41.636420    2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-39ea7c6f-openshift-default-566bd9c7d4-j2v5j" podUID="69502d0e-5d63-45aa-8724-678b467e51b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused"
Sep 11 01:12:42.524260 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:42.524219    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-39ea7c6f-t85x5"]
Sep 11 01:12:42.524913 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:42.524867    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ec0f171-1bff-4174-9905-fe855ce1ad42" containerName="istio-proxy"
Sep 11 01:12:42.526881 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:42.526856    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-39ea7c6f-t85x5"
Sep 11 01:12:42.529189 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:42.529172    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 01:12:42.529252 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:42.529220    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pppkt\""
Sep 11 01:12:42.534667 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:42.534603    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-39ea7c6f-t85x5"]
Sep 11 01:12:42.598720 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:42.598690    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/113d244f-3929-4f2f-a84b-260a11c19683-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-39ea7c6f-t85x5\" (UID: \"113d244f-3929-4f2f-a84b-260a11c19683\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-39ea7c6f-t85x5"
Sep 11 01:12:42.640473 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:42.640450    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-39ea7c6f-openshift-default-566bd9c7d4-j2v5j"
Sep 11 01:12:42.640807 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:42.640709    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-39ea7c6f-openshift-default-566bd9c7d4-j2v5j"
Sep 11 01:12:42.641462 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:42.641445    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-39ea7c6f-openshift-default-566bd9c7d4-j2v5j"
Sep 11 01:12:42.699813 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:42.699775    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/113d244f-3929-4f2f-a84b-260a11c19683-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-39ea7c6f-t85x5\" (UID: \"113d244f-3929-4f2f-a84b-260a11c19683\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-39ea7c6f-t85x5"
Sep 11 01:12:42.700432 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:42.700411    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/113d244f-3929-4f2f-a84b-260a11c19683-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-39ea7c6f-t85x5\" (UID: \"113d244f-3929-4f2f-a84b-260a11c19683\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-39ea7c6f-t85x5"
Sep 11 01:12:42.838098 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:42.838058    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-39ea7c6f-t85x5"
Sep 11 01:12:43.112090 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:43.112065    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-39ea7c6f-t85x5"]
Sep 11 01:12:43.114398 ip-10-0-129-222 kubenswrapper[2569]: W0911 01:12:43.114371    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod113d244f_3929_4f2f_a84b_260a11c19683.slice/crio-6cfd1be70342b714e3dbf2279a1f61437ebee8a988ab1232bb172669ee2d4573 WatchSource:0}: Error finding container 6cfd1be70342b714e3dbf2279a1f61437ebee8a988ab1232bb172669ee2d4573: Status 404 returned error can't find the container with id 6cfd1be70342b714e3dbf2279a1f61437ebee8a988ab1232bb172669ee2d4573
Sep 11 01:12:43.473496 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:43.473464    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-39ea7c6f-t85x5" event={"ID":"113d244f-3929-4f2f-a84b-260a11c19683","Type":"ContainerStarted","Data":"c005a9d9db59f4d32357a4db73232df4aed5442391d7b3668253aae3a64d6712"}
Sep 11 01:12:43.473614 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:43.473525    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-39ea7c6f-t85x5" event={"ID":"113d244f-3929-4f2f-a84b-260a11c19683","Type":"ContainerStarted","Data":"6cfd1be70342b714e3dbf2279a1f61437ebee8a988ab1232bb172669ee2d4573"}
Sep 11 01:12:43.488457 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:43.488403    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-39ea7c6f-t85x5" podStartSLOduration=1.488389998 podStartE2EDuration="1.488389998s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 01:12:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 01:12:43.485890375 +0000 UTC m=+898.137264618" watchObservedRunningTime="2026-09-11 01:12:43.488389998 +0000 UTC m=+898.139764248"
Sep 11 01:12:44.480668 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:44.480633    2569 generic.go:356] "Generic (PLEG): container finished" podID="113d244f-3929-4f2f-a84b-260a11c19683" containerID="c005a9d9db59f4d32357a4db73232df4aed5442391d7b3668253aae3a64d6712" exitCode=0
Sep 11 01:12:44.481064 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:44.480717    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-39ea7c6f-t85x5" event={"ID":"113d244f-3929-4f2f-a84b-260a11c19683","Type":"ContainerDied","Data":"c005a9d9db59f4d32357a4db73232df4aed5442391d7b3668253aae3a64d6712"}
Sep 11 01:12:45.620810 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:45.620788    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-39ea7c6f-t85x5"
Sep 11 01:12:45.727620 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:45.727590    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/113d244f-3929-4f2f-a84b-260a11c19683-maas-api-service-ca\") pod \"113d244f-3929-4f2f-a84b-260a11c19683\" (UID: \"113d244f-3929-4f2f-a84b-260a11c19683\") "
Sep 11 01:12:45.727945 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:45.727924    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/113d244f-3929-4f2f-a84b-260a11c19683-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "113d244f-3929-4f2f-a84b-260a11c19683" (UID: "113d244f-3929-4f2f-a84b-260a11c19683"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 01:12:45.829333 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:45.829303    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/113d244f-3929-4f2f-a84b-260a11c19683-maas-api-service-ca\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:12:46.492641 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:46.492602    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-39ea7c6f-t85x5" event={"ID":"113d244f-3929-4f2f-a84b-260a11c19683","Type":"ContainerDied","Data":"6cfd1be70342b714e3dbf2279a1f61437ebee8a988ab1232bb172669ee2d4573"}
Sep 11 01:12:46.492641 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:46.492643    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-39ea7c6f-t85x5"
Sep 11 01:12:46.492847 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:46.492688    2569 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6cfd1be70342b714e3dbf2279a1f61437ebee8a988ab1232bb172669ee2d4573"
Sep 11 01:12:47.431053 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:47.431023    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-8d451c-7d7b4b9f74-xw47v"
Sep 11 01:12:49.753109 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:49.753068    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-39ea7c6f-t85x5"]
Sep 11 01:12:49.759215 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:49.759180    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-39ea7c6f-t85x5"]
Sep 11 01:12:49.871767 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:49.871733    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="113d244f-3929-4f2f-a84b-260a11c19683" path="/var/lib/kubelet/pods/113d244f-3929-4f2f-a84b-260a11c19683/volumes"
Sep 11 01:12:49.912982 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:49.912952    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-39ea7c6f-openshift-default-566bd9c7d4-j2v5j"]
Sep 11 01:12:49.913191 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:49.913168    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-39ea7c6f-openshift-default-566bd9c7d4-j2v5j" podUID="69502d0e-5d63-45aa-8724-678b467e51b4" containerName="istio-proxy" containerID="cri-o://c67e13264c70eb5e6146ff39d6650cd96aa0e44d8d9c8241ec0b696720f4502b" gracePeriod=30
Sep 11 01:12:55.169277 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:55.169250    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-39ea7c6f-openshift-default-566bd9c7d4-j2v5j"
Sep 11 01:12:55.322189 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:55.322162    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69502d0e-5d63-45aa-8724-678b467e51b4-istiod-ca-cert\") pod \"69502d0e-5d63-45aa-8724-678b467e51b4\" (UID: \"69502d0e-5d63-45aa-8724-678b467e51b4\") "
Sep 11 01:12:55.322379 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:55.322205    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69502d0e-5d63-45aa-8724-678b467e51b4-credential-socket\") pod \"69502d0e-5d63-45aa-8724-678b467e51b4\" (UID: \"69502d0e-5d63-45aa-8724-678b467e51b4\") "
Sep 11 01:12:55.322379 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:55.322234    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69502d0e-5d63-45aa-8724-678b467e51b4-workload-socket\") pod \"69502d0e-5d63-45aa-8724-678b467e51b4\" (UID: \"69502d0e-5d63-45aa-8724-678b467e51b4\") "
Sep 11 01:12:55.322508 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:55.322434    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69502d0e-5d63-45aa-8724-678b467e51b4-istio-envoy\") pod \"69502d0e-5d63-45aa-8724-678b467e51b4\" (UID: \"69502d0e-5d63-45aa-8724-678b467e51b4\") "
Sep 11 01:12:55.322508 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:55.322451    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69502d0e-5d63-45aa-8724-678b467e51b4-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "69502d0e-5d63-45aa-8724-678b467e51b4" (UID: "69502d0e-5d63-45aa-8724-678b467e51b4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 01:12:55.322508 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:55.322480    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6vs95\" (UniqueName: \"kubernetes.io/projected/69502d0e-5d63-45aa-8724-678b467e51b4-kube-api-access-6vs95\") pod \"69502d0e-5d63-45aa-8724-678b467e51b4\" (UID: \"69502d0e-5d63-45aa-8724-678b467e51b4\") "
Sep 11 01:12:55.322508 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:55.322485    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69502d0e-5d63-45aa-8724-678b467e51b4-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "69502d0e-5d63-45aa-8724-678b467e51b4" (UID: "69502d0e-5d63-45aa-8724-678b467e51b4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 01:12:55.322717 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:55.322565    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69502d0e-5d63-45aa-8724-678b467e51b4-workload-certs\") pod \"69502d0e-5d63-45aa-8724-678b467e51b4\" (UID: \"69502d0e-5d63-45aa-8724-678b467e51b4\") "
Sep 11 01:12:55.322717 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:55.322605    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69502d0e-5d63-45aa-8724-678b467e51b4-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "69502d0e-5d63-45aa-8724-678b467e51b4" (UID: "69502d0e-5d63-45aa-8724-678b467e51b4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 01:12:55.322717 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:55.322644    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69502d0e-5d63-45aa-8724-678b467e51b4-istio-data\") pod \"69502d0e-5d63-45aa-8724-678b467e51b4\" (UID: \"69502d0e-5d63-45aa-8724-678b467e51b4\") "
Sep 11 01:12:55.322717 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:55.322678    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69502d0e-5d63-45aa-8724-678b467e51b4-istio-token\") pod \"69502d0e-5d63-45aa-8724-678b467e51b4\" (UID: \"69502d0e-5d63-45aa-8724-678b467e51b4\") "
Sep 11 01:12:55.322956 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:55.322720    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69502d0e-5d63-45aa-8724-678b467e51b4-istio-podinfo\") pod \"69502d0e-5d63-45aa-8724-678b467e51b4\" (UID: \"69502d0e-5d63-45aa-8724-678b467e51b4\") "
Sep 11 01:12:55.322956 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:55.322752    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69502d0e-5d63-45aa-8724-678b467e51b4-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "69502d0e-5d63-45aa-8724-678b467e51b4" (UID: "69502d0e-5d63-45aa-8724-678b467e51b4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 01:12:55.322956 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:55.322814    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69502d0e-5d63-45aa-8724-678b467e51b4-istio-data" (OuterVolumeSpecName: "istio-data") pod "69502d0e-5d63-45aa-8724-678b467e51b4" (UID: "69502d0e-5d63-45aa-8724-678b467e51b4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 01:12:55.323148 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:55.323127    2569 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69502d0e-5d63-45aa-8724-678b467e51b4-istio-data\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:12:55.323209 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:55.323153    2569 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69502d0e-5d63-45aa-8724-678b467e51b4-istiod-ca-cert\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:12:55.323209 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:55.323166    2569 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69502d0e-5d63-45aa-8724-678b467e51b4-credential-socket\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:12:55.323209 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:55.323179    2569 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69502d0e-5d63-45aa-8724-678b467e51b4-workload-socket\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:12:55.323209 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:55.323194    2569 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69502d0e-5d63-45aa-8724-678b467e51b4-workload-certs\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:12:55.325165 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:55.325131    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69502d0e-5d63-45aa-8724-678b467e51b4-istio-token" (OuterVolumeSpecName: "istio-token") pod "69502d0e-5d63-45aa-8724-678b467e51b4" (UID: "69502d0e-5d63-45aa-8724-678b467e51b4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 01:12:55.325274 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:55.325212    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69502d0e-5d63-45aa-8724-678b467e51b4-kube-api-access-6vs95" (OuterVolumeSpecName: "kube-api-access-6vs95") pod "69502d0e-5d63-45aa-8724-678b467e51b4" (UID: "69502d0e-5d63-45aa-8724-678b467e51b4"). InnerVolumeSpecName "kube-api-access-6vs95". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 01:12:55.325274 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:55.325234    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69502d0e-5d63-45aa-8724-678b467e51b4-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "69502d0e-5d63-45aa-8724-678b467e51b4" (UID: "69502d0e-5d63-45aa-8724-678b467e51b4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 01:12:55.325274 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:55.325247    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/69502d0e-5d63-45aa-8724-678b467e51b4-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "69502d0e-5d63-45aa-8724-678b467e51b4" (UID: "69502d0e-5d63-45aa-8724-678b467e51b4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 01:12:55.424205 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:55.424174    2569 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69502d0e-5d63-45aa-8724-678b467e51b4-istio-envoy\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:12:55.424205 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:55.424201    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6vs95\" (UniqueName: \"kubernetes.io/projected/69502d0e-5d63-45aa-8724-678b467e51b4-kube-api-access-6vs95\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:12:55.424205 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:55.424212    2569 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69502d0e-5d63-45aa-8724-678b467e51b4-istio-token\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:12:55.424407 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:55.424220    2569 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69502d0e-5d63-45aa-8724-678b467e51b4-istio-podinfo\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:12:55.548750 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:55.548719    2569 generic.go:356] "Generic (PLEG): container finished" podID="69502d0e-5d63-45aa-8724-678b467e51b4" containerID="c67e13264c70eb5e6146ff39d6650cd96aa0e44d8d9c8241ec0b696720f4502b" exitCode=0
Sep 11 01:12:55.548934 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:55.548787    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-39ea7c6f-openshift-default-566bd9c7d4-j2v5j" event={"ID":"69502d0e-5d63-45aa-8724-678b467e51b4","Type":"ContainerDied","Data":"c67e13264c70eb5e6146ff39d6650cd96aa0e44d8d9c8241ec0b696720f4502b"}
Sep 11 01:12:55.548934 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:55.548848    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-39ea7c6f-openshift-default-566bd9c7d4-j2v5j"
Sep 11 01:12:55.548934 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:55.548864    2569 scope.go:117] "RemoveContainer" containerID="c67e13264c70eb5e6146ff39d6650cd96aa0e44d8d9c8241ec0b696720f4502b"
Sep 11 01:12:55.549078 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:55.548851    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-39ea7c6f-openshift-default-566bd9c7d4-j2v5j" event={"ID":"69502d0e-5d63-45aa-8724-678b467e51b4","Type":"ContainerDied","Data":"18cd488bd0cd01d1e9e4ce80d50686d4faec6bb2e9acfdba1f821ece6d653415"}
Sep 11 01:12:55.559232 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:55.559213    2569 scope.go:117] "RemoveContainer" containerID="76211d9197ad0889fabe0d7e3090b011bd3df6a5f7ac19cff8407aedd9acb20b"
Sep 11 01:12:55.568060 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:55.568039    2569 scope.go:117] "RemoveContainer" containerID="c67e13264c70eb5e6146ff39d6650cd96aa0e44d8d9c8241ec0b696720f4502b"
Sep 11 01:12:55.568472 ip-10-0-129-222 kubenswrapper[2569]: E0911 01:12:55.568364    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c67e13264c70eb5e6146ff39d6650cd96aa0e44d8d9c8241ec0b696720f4502b\": container with ID starting with c67e13264c70eb5e6146ff39d6650cd96aa0e44d8d9c8241ec0b696720f4502b not found: ID does not exist" containerID="c67e13264c70eb5e6146ff39d6650cd96aa0e44d8d9c8241ec0b696720f4502b"
Sep 11 01:12:55.568472 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:55.568402    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c67e13264c70eb5e6146ff39d6650cd96aa0e44d8d9c8241ec0b696720f4502b"} err="failed to get container status \"c67e13264c70eb5e6146ff39d6650cd96aa0e44d8d9c8241ec0b696720f4502b\": rpc error: code = NotFound desc = could not find container \"c67e13264c70eb5e6146ff39d6650cd96aa0e44d8d9c8241ec0b696720f4502b\": container with ID starting with c67e13264c70eb5e6146ff39d6650cd96aa0e44d8d9c8241ec0b696720f4502b not found: ID does not exist"
Sep 11 01:12:55.568472 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:55.568436    2569 scope.go:117] "RemoveContainer" containerID="76211d9197ad0889fabe0d7e3090b011bd3df6a5f7ac19cff8407aedd9acb20b"
Sep 11 01:12:55.568890 ip-10-0-129-222 kubenswrapper[2569]: E0911 01:12:55.568849    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76211d9197ad0889fabe0d7e3090b011bd3df6a5f7ac19cff8407aedd9acb20b\": container with ID starting with 76211d9197ad0889fabe0d7e3090b011bd3df6a5f7ac19cff8407aedd9acb20b not found: ID does not exist" containerID="76211d9197ad0889fabe0d7e3090b011bd3df6a5f7ac19cff8407aedd9acb20b"
Sep 11 01:12:55.568989 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:55.568910    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76211d9197ad0889fabe0d7e3090b011bd3df6a5f7ac19cff8407aedd9acb20b"} err="failed to get container status \"76211d9197ad0889fabe0d7e3090b011bd3df6a5f7ac19cff8407aedd9acb20b\": rpc error: code = NotFound desc = could not find container \"76211d9197ad0889fabe0d7e3090b011bd3df6a5f7ac19cff8407aedd9acb20b\": container with ID starting with 76211d9197ad0889fabe0d7e3090b011bd3df6a5f7ac19cff8407aedd9acb20b not found: ID does not exist"
Sep 11 01:12:55.571402 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:55.571379    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-39ea7c6f-openshift-default-566bd9c7d4-j2v5j"]
Sep 11 01:12:55.574369 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:55.574315    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-39ea7c6f-openshift-default-566bd9c7d4-j2v5j"]
Sep 11 01:12:55.872608 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:55.872534    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="69502d0e-5d63-45aa-8724-678b467e51b4" path="/var/lib/kubelet/pods/69502d0e-5d63-45aa-8724-678b467e51b4/volumes"
Sep 11 01:12:56.595909 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:56.595862    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kmx5x"]
Sep 11 01:12:57.562518 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:57.562484    2569 generic.go:356] "Generic (PLEG): container finished" podID="f6c2f5c6-7f99-4519-acf1-7af6c423bc0f" containerID="17c42f6abbe8776859d57d14880e986f7c0b9a8b71ee157f8440c0109b535f1a" exitCode=0
Sep 11 01:12:57.562719 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:57.562552    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-8d451c-openshift-default-7b5868fcdd-g9rzj" event={"ID":"f6c2f5c6-7f99-4519-acf1-7af6c423bc0f","Type":"ContainerDied","Data":"17c42f6abbe8776859d57d14880e986f7c0b9a8b71ee157f8440c0109b535f1a"}
Sep 11 01:12:57.562991 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:57.562974    2569 scope.go:117] "RemoveContainer" containerID="17c42f6abbe8776859d57d14880e986f7c0b9a8b71ee157f8440c0109b535f1a"
Sep 11 01:12:57.564164 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:57.564135    2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 01:12:57.564219 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:57.564205    2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 01:12:57.564254 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:57.564236    2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 01:12:58.570462 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:58.570427    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-8d451c-openshift-default-7b5868fcdd-g9rzj" event={"ID":"f6c2f5c6-7f99-4519-acf1-7af6c423bc0f","Type":"ContainerStarted","Data":"7b5de0a04a22a15504b11f4f57e4dd7aca0dab092588ff88591adc7a51571373"}
Sep 11 01:12:59.534094 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:59.534059    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-8d451c-openshift-default-7b5868fcdd-g9rzj"
Sep 11 01:12:59.535385 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:59.535065    2569 patch_prober.go:28] interesting pod/e2e-explicit-8d451c-openshift-default-7b5868fcdd-g9rzj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body=
Sep 11 01:12:59.535385 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:12:59.535125    2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-8d451c-openshift-default-7b5868fcdd-g9rzj" podUID="f6c2f5c6-7f99-4519-acf1-7af6c423bc0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused"
Sep 11 01:13:00.534647 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:13:00.534610    2569 patch_prober.go:28] interesting pod/e2e-explicit-8d451c-openshift-default-7b5868fcdd-g9rzj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body=
Sep 11 01:13:00.535118 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:13:00.534680    2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-8d451c-openshift-default-7b5868fcdd-g9rzj" podUID="f6c2f5c6-7f99-4519-acf1-7af6c423bc0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused"
Sep 11 01:13:01.535257 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:13:01.535213    2569 patch_prober.go:28] interesting pod/e2e-explicit-8d451c-openshift-default-7b5868fcdd-g9rzj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body=
Sep 11 01:13:01.535741 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:13:01.535283    2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-8d451c-openshift-default-7b5868fcdd-g9rzj" podUID="f6c2f5c6-7f99-4519-acf1-7af6c423bc0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused"
Sep 11 01:13:02.391826 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:13:02.391783    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-8d451c-openshift-default-7b5868fcdd-g9rzj"
Sep 11 01:13:02.534530 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:13:02.534494    2569 patch_prober.go:28] interesting pod/e2e-explicit-8d451c-openshift-default-7b5868fcdd-g9rzj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body=
Sep 11 01:13:02.534699 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:13:02.534556    2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-8d451c-openshift-default-7b5868fcdd-g9rzj" podUID="f6c2f5c6-7f99-4519-acf1-7af6c423bc0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused"
Sep 11 01:13:03.534855 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:13:03.534819    2569 patch_prober.go:28] interesting pod/e2e-explicit-8d451c-openshift-default-7b5868fcdd-g9rzj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body=
Sep 11 01:13:03.535253 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:13:03.534906    2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-8d451c-openshift-default-7b5868fcdd-g9rzj" podUID="f6c2f5c6-7f99-4519-acf1-7af6c423bc0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused"
Sep 11 01:13:04.538254 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:13:04.538227    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-8d451c-openshift-default-7b5868fcdd-g9rzj"
Sep 11 01:13:04.539237 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:13:04.539216    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-8d451c-openshift-default-7b5868fcdd-g9rzj"
Sep 11 01:13:21.892991 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:13:21.892956    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kmx5x"]
Sep 11 01:13:38.911368 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:13:38.911336    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kmx5x"]
Sep 11 01:13:57.005707 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:13:57.005667    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kmx5x"]
Sep 11 01:14:21.398488 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:14:21.398447    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kmx5x"]
Sep 11 01:14:25.857142 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:14:25.857111    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-7b8e72c7-openshift-default-69b8bddc8d-9t2bk"]
Sep 11 01:14:25.857581 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:14:25.857567    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="113d244f-3929-4f2f-a84b-260a11c19683" containerName="revoke-keys"
Sep 11 01:14:25.857632 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:14:25.857582    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="113d244f-3929-4f2f-a84b-260a11c19683" containerName="revoke-keys"
Sep 11 01:14:25.857632 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:14:25.857596    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="69502d0e-5d63-45aa-8724-678b467e51b4" containerName="istio-proxy"
Sep 11 01:14:25.857632 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:14:25.857605    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="69502d0e-5d63-45aa-8724-678b467e51b4" containerName="istio-proxy"
Sep 11 01:14:25.857632 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:14:25.857622    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="69502d0e-5d63-45aa-8724-678b467e51b4" containerName="istio-proxy"
Sep 11 01:14:25.857632 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:14:25.857628    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="69502d0e-5d63-45aa-8724-678b467e51b4" containerName="istio-proxy"
Sep 11 01:14:25.857792 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:14:25.857705    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="69502d0e-5d63-45aa-8724-678b467e51b4" containerName="istio-proxy"
Sep 11 01:14:25.857792 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:14:25.857712    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="113d244f-3929-4f2f-a84b-260a11c19683" containerName="revoke-keys"
Sep 11 01:14:25.861081 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:14:25.861061    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7b8e72c7-openshift-default-69b8bddc8d-9t2bk"
Sep 11 01:14:25.864267 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:14:25.864244    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-7b8e72c7-openshift-default-dockercfg-w9d7w\""
Sep 11 01:14:25.872648 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:14:25.872573    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-7b8e72c7-openshift-default-69b8bddc8d-9t2bk"]
Sep 11 01:14:25.960340 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:14:25.960306    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0387913-3dec-4201-8294-9fef9626afab-credential-socket\") pod \"e2e-mt-7b8e72c7-openshift-default-69b8bddc8d-9t2bk\" (UID: \"e0387913-3dec-4201-8294-9fef9626afab\") " pod="openshift-ingress/e2e-mt-7b8e72c7-openshift-default-69b8bddc8d-9t2bk"
Sep 11 01:14:25.960525 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:14:25.960362    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2f2n7\" (UniqueName: \"kubernetes.io/projected/e0387913-3dec-4201-8294-9fef9626afab-kube-api-access-2f2n7\") pod \"e2e-mt-7b8e72c7-openshift-default-69b8bddc8d-9t2bk\" (UID: \"e0387913-3dec-4201-8294-9fef9626afab\") " pod="openshift-ingress/e2e-mt-7b8e72c7-openshift-default-69b8bddc8d-9t2bk"
Sep 11 01:14:25.960525 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:14:25.960430    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0387913-3dec-4201-8294-9fef9626afab-workload-socket\") pod \"e2e-mt-7b8e72c7-openshift-default-69b8bddc8d-9t2bk\" (UID: \"e0387913-3dec-4201-8294-9fef9626afab\") " pod="openshift-ingress/e2e-mt-7b8e72c7-openshift-default-69b8bddc8d-9t2bk"
Sep 11 01:14:25.960525 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:14:25.960484    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0387913-3dec-4201-8294-9fef9626afab-istiod-ca-cert\") pod \"e2e-mt-7b8e72c7-openshift-default-69b8bddc8d-9t2bk\" (UID: \"e0387913-3dec-4201-8294-9fef9626afab\") " pod="openshift-ingress/e2e-mt-7b8e72c7-openshift-default-69b8bddc8d-9t2bk"
Sep 11 01:14:25.960668 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:14:25.960546    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0387913-3dec-4201-8294-9fef9626afab-workload-certs\") pod \"e2e-mt-7b8e72c7-openshift-default-69b8bddc8d-9t2bk\" (UID: \"e0387913-3dec-4201-8294-9fef9626afab\") " pod="openshift-ingress/e2e-mt-7b8e72c7-openshift-default-69b8bddc8d-9t2bk"
Sep 11 01:14:25.960668 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:14:25.960573    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0387913-3dec-4201-8294-9fef9626afab-istio-token\") pod \"e2e-mt-7b8e72c7-openshift-default-69b8bddc8d-9t2bk\" (UID: \"e0387913-3dec-4201-8294-9fef9626afab\") " pod="openshift-ingress/e2e-mt-7b8e72c7-openshift-default-69b8bddc8d-9t2bk"
Sep 11 01:14:25.960668 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:14:25.960637    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0387913-3dec-4201-8294-9fef9626afab-istio-envoy\") pod \"e2e-mt-7b8e72c7-openshift-default-69b8bddc8d-9t2bk\" (UID: \"e0387913-3dec-4201-8294-9fef9626afab\") " pod="openshift-ingress/e2e-mt-7b8e72c7-openshift-default-69b8bddc8d-9t2bk"
Sep 11 01:14:25.960787 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:14:25.960722    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0387913-3dec-4201-8294-9fef9626afab-istio-data\") pod \"e2e-mt-7b8e72c7-openshift-default-69b8bddc8d-9t2bk\" (UID: \"e0387913-3dec-4201-8294-9fef9626afab\") " pod="openshift-ingress/e2e-mt-7b8e72c7-openshift-default-69b8bddc8d-9t2bk"
Sep 11 01:14:25.960787 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:14:25.960757    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0387913-3dec-4201-8294-9fef9626afab-istio-podinfo\") pod \"e2e-mt-7b8e72c7-openshift-default-69b8bddc8d-9t2bk\" (UID: \"e0387913-3dec-4201-8294-9fef9626afab\") " pod="openshift-ingress/e2e-mt-7b8e72c7-openshift-default-69b8bddc8d-9t2bk"
Sep 11 01:14:26.061757 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:14:26.061716    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0387913-3dec-4201-8294-9fef9626afab-istio-data\") pod \"e2e-mt-7b8e72c7-openshift-default-69b8bddc8d-9t2bk\" (UID: \"e0387913-3dec-4201-8294-9fef9626afab\") " pod="openshift-ingress/e2e-mt-7b8e72c7-openshift-default-69b8bddc8d-9t2bk"
Sep 11 01:14:26.061976 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:14:26.061764    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0387913-3dec-4201-8294-9fef9626afab-istio-podinfo\") pod \"e2e-mt-7b8e72c7-openshift-default-69b8bddc8d-9t2bk\" (UID: \"e0387913-3dec-4201-8294-9fef9626afab\") " pod="openshift-ingress/e2e-mt-7b8e72c7-openshift-default-69b8bddc8d-9t2bk"
Sep 11 01:14:26.061976 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:14:26.061831    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0387913-3dec-4201-8294-9fef9626afab-credential-socket\") pod \"e2e-mt-7b8e72c7-openshift-default-69b8bddc8d-9t2bk\" (UID: \"e0387913-3dec-4201-8294-9fef9626afab\") " pod="openshift-ingress/e2e-mt-7b8e72c7-openshift-default-69b8bddc8d-9t2bk"
Sep 11 01:14:26.061976 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:14:26.061957    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2f2n7\" (UniqueName: \"kubernetes.io/projected/e0387913-3dec-4201-8294-9fef9626afab-kube-api-access-2f2n7\") pod \"e2e-mt-7b8e72c7-openshift-default-69b8bddc8d-9t2bk\" (UID: \"e0387913-3dec-4201-8294-9fef9626afab\") " pod="openshift-ingress/e2e-mt-7b8e72c7-openshift-default-69b8bddc8d-9t2bk"
Sep 11 01:14:26.062167 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:14:26.062010    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0387913-3dec-4201-8294-9fef9626afab-workload-socket\") pod \"e2e-mt-7b8e72c7-openshift-default-69b8bddc8d-9t2bk\" (UID: \"e0387913-3dec-4201-8294-9fef9626afab\") " pod="openshift-ingress/e2e-mt-7b8e72c7-openshift-default-69b8bddc8d-9t2bk"
Sep 11 01:14:26.062167 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:14:26.062056    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0387913-3dec-4201-8294-9fef9626afab-istiod-ca-cert\") pod \"e2e-mt-7b8e72c7-openshift-default-69b8bddc8d-9t2bk\" (UID: \"e0387913-3dec-4201-8294-9fef9626afab\") " pod="openshift-ingress/e2e-mt-7b8e72c7-openshift-default-69b8bddc8d-9t2bk"
Sep 11 01:14:26.062167 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:14:26.062113    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0387913-3dec-4201-8294-9fef9626afab-workload-certs\") pod \"e2e-mt-7b8e72c7-openshift-default-69b8bddc8d-9t2bk\" (UID: \"e0387913-3dec-4201-8294-9fef9626afab\") " pod="openshift-ingress/e2e-mt-7b8e72c7-openshift-default-69b8bddc8d-9t2bk"
Sep 11 01:14:26.062167 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:14:26.062144    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0387913-3dec-4201-8294-9fef9626afab-istio-token\") pod \"e2e-mt-7b8e72c7-openshift-default-69b8bddc8d-9t2bk\" (UID: \"e0387913-3dec-4201-8294-9fef9626afab\") " pod="openshift-ingress/e2e-mt-7b8e72c7-openshift-default-69b8bddc8d-9t2bk"
Sep 11 01:14:26.062398 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:14:26.062160    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0387913-3dec-4201-8294-9fef9626afab-istio-data\") pod \"e2e-mt-7b8e72c7-openshift-default-69b8bddc8d-9t2bk\" (UID: \"e0387913-3dec-4201-8294-9fef9626afab\") " pod="openshift-ingress/e2e-mt-7b8e72c7-openshift-default-69b8bddc8d-9t2bk"
Sep 11 01:14:26.062398 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:14:26.062203    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0387913-3dec-4201-8294-9fef9626afab-istio-envoy\") pod \"e2e-mt-7b8e72c7-openshift-default-69b8bddc8d-9t2bk\" (UID: \"e0387913-3dec-4201-8294-9fef9626afab\") " pod="openshift-ingress/e2e-mt-7b8e72c7-openshift-default-69b8bddc8d-9t2bk"
Sep 11 01:14:26.062398 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:14:26.062207    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0387913-3dec-4201-8294-9fef9626afab-credential-socket\") pod \"e2e-mt-7b8e72c7-openshift-default-69b8bddc8d-9t2bk\" (UID: \"e0387913-3dec-4201-8294-9fef9626afab\") " pod="openshift-ingress/e2e-mt-7b8e72c7-openshift-default-69b8bddc8d-9t2bk"
Sep 11 01:14:26.062398 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:14:26.062350    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0387913-3dec-4201-8294-9fef9626afab-workload-socket\") pod \"e2e-mt-7b8e72c7-openshift-default-69b8bddc8d-9t2bk\" (UID: \"e0387913-3dec-4201-8294-9fef9626afab\") " pod="openshift-ingress/e2e-mt-7b8e72c7-openshift-default-69b8bddc8d-9t2bk"
Sep 11 01:14:26.062543 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:14:26.062432    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0387913-3dec-4201-8294-9fef9626afab-workload-certs\") pod \"e2e-mt-7b8e72c7-openshift-default-69b8bddc8d-9t2bk\" (UID: \"e0387913-3dec-4201-8294-9fef9626afab\") " pod="openshift-ingress/e2e-mt-7b8e72c7-openshift-default-69b8bddc8d-9t2bk"
Sep 11 01:14:26.062857 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:14:26.062839    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0387913-3dec-4201-8294-9fef9626afab-istiod-ca-cert\") pod \"e2e-mt-7b8e72c7-openshift-default-69b8bddc8d-9t2bk\" (UID: \"e0387913-3dec-4201-8294-9fef9626afab\") " pod="openshift-ingress/e2e-mt-7b8e72c7-openshift-default-69b8bddc8d-9t2bk"
Sep 11 01:14:26.064289 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:14:26.064272    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0387913-3dec-4201-8294-9fef9626afab-istio-envoy\") pod \"e2e-mt-7b8e72c7-openshift-default-69b8bddc8d-9t2bk\" (UID: \"e0387913-3dec-4201-8294-9fef9626afab\") " pod="openshift-ingress/e2e-mt-7b8e72c7-openshift-default-69b8bddc8d-9t2bk"
Sep 11 01:14:26.064392 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:14:26.064377    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0387913-3dec-4201-8294-9fef9626afab-istio-podinfo\") pod \"e2e-mt-7b8e72c7-openshift-default-69b8bddc8d-9t2bk\" (UID: \"e0387913-3dec-4201-8294-9fef9626afab\") " pod="openshift-ingress/e2e-mt-7b8e72c7-openshift-default-69b8bddc8d-9t2bk"
Sep 11 01:14:26.069735 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:14:26.069706    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0387913-3dec-4201-8294-9fef9626afab-istio-token\") pod \"e2e-mt-7b8e72c7-openshift-default-69b8bddc8d-9t2bk\" (UID: \"e0387913-3dec-4201-8294-9fef9626afab\") " pod="openshift-ingress/e2e-mt-7b8e72c7-openshift-default-69b8bddc8d-9t2bk"
Sep 11 01:14:26.069977 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:14:26.069959    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2f2n7\" (UniqueName: \"kubernetes.io/projected/e0387913-3dec-4201-8294-9fef9626afab-kube-api-access-2f2n7\") pod \"e2e-mt-7b8e72c7-openshift-default-69b8bddc8d-9t2bk\" (UID: \"e0387913-3dec-4201-8294-9fef9626afab\") " pod="openshift-ingress/e2e-mt-7b8e72c7-openshift-default-69b8bddc8d-9t2bk"
Sep 11 01:14:26.172916 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:14:26.172836    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7b8e72c7-openshift-default-69b8bddc8d-9t2bk"
Sep 11 01:14:26.229010 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:14:26.228637    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-8d451c-b8cjp"]
Sep 11 01:14:26.229374 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:14:26.229352    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="69502d0e-5d63-45aa-8724-678b467e51b4" containerName="istio-proxy"
Sep 11 01:14:26.233392 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:14:26.233327    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-8d451c-b8cjp"
Sep 11 01:14:26.236330 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:14:26.236298    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pppkt\""
Sep 11 01:14:26.236606 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:14:26.236584    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 01:14:26.241586 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:14:26.241526    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-8d451c-b8cjp"]
Sep 11 01:14:26.364949 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:14:26.364918    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/437556d2-b3e8-40c4-8cff-6571c34c2083-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-8d451c-b8cjp\" (UID: \"437556d2-b3e8-40c4-8cff-6571c34c2083\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-8d451c-b8cjp"
Sep 11 01:14:26.454006 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:14:26.453983    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-7b8e72c7-openshift-default-69b8bddc8d-9t2bk"]
Sep 11 01:14:26.455561 ip-10-0-129-222 kubenswrapper[2569]: W0911 01:14:26.455531    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0387913_3dec_4201_8294_9fef9626afab.slice/crio-e68d2d6301052d7e6240daa0df9a2f4c5c31d2ccafa8d58c814ade07fe8d0855 WatchSource:0}: Error finding container e68d2d6301052d7e6240daa0df9a2f4c5c31d2ccafa8d58c814ade07fe8d0855: Status 404 returned error can't find the container with id e68d2d6301052d7e6240daa0df9a2f4c5c31d2ccafa8d58c814ade07fe8d0855
Sep 11 01:14:26.458325 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:14:26.458276    2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 01:14:26.458419 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:14:26.458376    2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 01:14:26.458468 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:14:26.458447    2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 01:14:26.466199 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:14:26.466178    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/437556d2-b3e8-40c4-8cff-6571c34c2083-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-8d451c-b8cjp\" (UID: \"437556d2-b3e8-40c4-8cff-6571c34c2083\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-8d451c-b8cjp"
Sep 11 01:14:26.466755 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:14:26.466727    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/437556d2-b3e8-40c4-8cff-6571c34c2083-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-8d451c-b8cjp\" (UID: \"437556d2-b3e8-40c4-8cff-6571c34c2083\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-8d451c-b8cjp"
Sep 11 01:14:26.551293 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:14:26.551258    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-8d451c-b8cjp"
Sep 11 01:14:26.831571 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:14:26.831545    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-8d451c-b8cjp"]
Sep 11 01:14:26.833759 ip-10-0-129-222 kubenswrapper[2569]: W0911 01:14:26.833730    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod437556d2_b3e8_40c4_8cff_6571c34c2083.slice/crio-8387bc01f5ccf03b217aee873315c0bee1afe76e8360b44c82259f14fad1927c WatchSource:0}: Error finding container 8387bc01f5ccf03b217aee873315c0bee1afe76e8360b44c82259f14fad1927c: Status 404 returned error can't find the container with id 8387bc01f5ccf03b217aee873315c0bee1afe76e8360b44c82259f14fad1927c
Sep 11 01:14:27.061797 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:14:27.061764    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7b8e72c7-openshift-default-69b8bddc8d-9t2bk" event={"ID":"e0387913-3dec-4201-8294-9fef9626afab","Type":"ContainerStarted","Data":"3bce9347574786343feaffbca610f346303b227c049b424d09cdbb521d51a9f2"}
Sep 11 01:14:27.062235 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:14:27.061856    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7b8e72c7-openshift-default-69b8bddc8d-9t2bk" event={"ID":"e0387913-3dec-4201-8294-9fef9626afab","Type":"ContainerStarted","Data":"e68d2d6301052d7e6240daa0df9a2f4c5c31d2ccafa8d58c814ade07fe8d0855"}
Sep 11 01:14:27.063056 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:14:27.063032    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-8d451c-b8cjp" event={"ID":"437556d2-b3e8-40c4-8cff-6571c34c2083","Type":"ContainerStarted","Data":"8387bc01f5ccf03b217aee873315c0bee1afe76e8360b44c82259f14fad1927c"}
Sep 11 01:14:27.083851 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:14:27.083772    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-7b8e72c7-openshift-default-69b8bddc8d-9t2bk" podStartSLOduration=2.08375749 podStartE2EDuration="2.08375749s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 01:14:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 01:14:27.081759141 +0000 UTC m=+1001.733133404" watchObservedRunningTime="2026-09-11 01:14:27.08375749 +0000 UTC m=+1001.735131783"
Sep 11 01:14:27.173503 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:14:27.173477    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-7b8e72c7-openshift-default-69b8bddc8d-9t2bk"
Sep 11 01:14:27.178515 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:14:27.178496    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-7b8e72c7-openshift-default-69b8bddc8d-9t2bk"
Sep 11 01:14:28.069938 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:14:28.069903    2569 generic.go:356] "Generic (PLEG): container finished" podID="437556d2-b3e8-40c4-8cff-6571c34c2083" containerID="b822c221c1c92f53c142996033cba8f57ecf90ef6f60a2c02c83208dff3eb11e" exitCode=0
Sep 11 01:14:28.070326 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:14:28.070038    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-8d451c-b8cjp" event={"ID":"437556d2-b3e8-40c4-8cff-6571c34c2083","Type":"ContainerDied","Data":"b822c221c1c92f53c142996033cba8f57ecf90ef6f60a2c02c83208dff3eb11e"}
Sep 11 01:14:28.070416 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:14:28.070399    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-7b8e72c7-openshift-default-69b8bddc8d-9t2bk"
Sep 11 01:14:28.071284 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:14:28.071268    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-7b8e72c7-openshift-default-69b8bddc8d-9t2bk"
Sep 11 01:14:29.210943 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:14:29.210919    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-8d451c-b8cjp"
Sep 11 01:14:29.293755 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:14:29.293730    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/437556d2-b3e8-40c4-8cff-6571c34c2083-maas-api-service-ca\") pod \"437556d2-b3e8-40c4-8cff-6571c34c2083\" (UID: \"437556d2-b3e8-40c4-8cff-6571c34c2083\") "
Sep 11 01:14:29.294122 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:14:29.294097    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/437556d2-b3e8-40c4-8cff-6571c34c2083-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "437556d2-b3e8-40c4-8cff-6571c34c2083" (UID: "437556d2-b3e8-40c4-8cff-6571c34c2083"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 01:14:29.394610 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:14:29.394527    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/437556d2-b3e8-40c4-8cff-6571c34c2083-maas-api-service-ca\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:14:30.082762 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:14:30.082730    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-8d451c-b8cjp"
Sep 11 01:14:30.082960 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:14:30.082730    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-8d451c-b8cjp" event={"ID":"437556d2-b3e8-40c4-8cff-6571c34c2083","Type":"ContainerDied","Data":"8387bc01f5ccf03b217aee873315c0bee1afe76e8360b44c82259f14fad1927c"}
Sep 11 01:14:30.082960 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:14:30.082856    2569 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8387bc01f5ccf03b217aee873315c0bee1afe76e8360b44c82259f14fad1927c"
Sep 11 01:14:33.477275 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:14:33.477238    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-7b8e72c7-7df96bbb58-cjwl7"]
Sep 11 01:14:33.477698 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:14:33.477685    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="437556d2-b3e8-40c4-8cff-6571c34c2083" containerName="revoke-keys"
Sep 11 01:14:33.477747 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:14:33.477700    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="437556d2-b3e8-40c4-8cff-6571c34c2083" containerName="revoke-keys"
Sep 11 01:14:33.477782 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:14:33.477772    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="437556d2-b3e8-40c4-8cff-6571c34c2083" containerName="revoke-keys"
Sep 11 01:14:33.481054 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:14:33.481036    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-7b8e72c7-7df96bbb58-cjwl7"
Sep 11 01:14:33.484273 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:14:33.484249    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-7b8e72c7\""
Sep 11 01:14:33.484423 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:14:33.484407    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-7b8e72c7-dockercfg-8tdwh\""
Sep 11 01:14:33.514300 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:14:33.514268    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-7b8e72c7-7df96bbb58-cjwl7"]
Sep 11 01:14:33.527688 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:14:33.527665    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-7b8e72c7-66f454f4bf-k6l45"]
Sep 11 01:14:33.531297 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:14:33.531281    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-7b8e72c7-66f454f4bf-k6l45"
Sep 11 01:14:33.537367 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:14:33.537339    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39491142-6059-4835-9c24-fef3f72af78c-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-7b8e72c7-7df96bbb58-cjwl7\" (UID: \"39491142-6059-4835-9c24-fef3f72af78c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-7b8e72c7-7df96bbb58-cjwl7"
Sep 11 01:14:33.537517 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:14:33.537456    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2fsd\" (UniqueName: \"kubernetes.io/projected/39491142-6059-4835-9c24-fef3f72af78c-kube-api-access-m2fsd\") pod \"payload-pre-processing-e2e-mt-7b8e72c7-7df96bbb58-cjwl7\" (UID: \"39491142-6059-4835-9c24-fef3f72af78c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-7b8e72c7-7df96bbb58-cjwl7"
Sep 11 01:14:33.553109 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:14:33.553083    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-7b8e72c7-66f454f4bf-k6l45"]
Sep 11 01:14:33.638523 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:14:33.638481    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39491142-6059-4835-9c24-fef3f72af78c-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-7b8e72c7-7df96bbb58-cjwl7\" (UID: \"39491142-6059-4835-9c24-fef3f72af78c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-7b8e72c7-7df96bbb58-cjwl7"
Sep 11 01:14:33.638742 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:14:33.638545    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-647ng\" (UniqueName: \"kubernetes.io/projected/a926ff1c-074b-4592-9089-4b9278c44414-kube-api-access-647ng\") pod \"payload-processing-e2e-mt-7b8e72c7-66f454f4bf-k6l45\" (UID: \"a926ff1c-074b-4592-9089-4b9278c44414\") " pod="openshift-ingress/payload-processing-e2e-mt-7b8e72c7-66f454f4bf-k6l45"
Sep 11 01:14:33.638742 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:14:33.638580    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a926ff1c-074b-4592-9089-4b9278c44414-plugins-config-volume\") pod \"payload-processing-e2e-mt-7b8e72c7-66f454f4bf-k6l45\" (UID: \"a926ff1c-074b-4592-9089-4b9278c44414\") " pod="openshift-ingress/payload-processing-e2e-mt-7b8e72c7-66f454f4bf-k6l45"
Sep 11 01:14:33.638742 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:14:33.638611    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2fsd\" (UniqueName: \"kubernetes.io/projected/39491142-6059-4835-9c24-fef3f72af78c-kube-api-access-m2fsd\") pod \"payload-pre-processing-e2e-mt-7b8e72c7-7df96bbb58-cjwl7\" (UID: \"39491142-6059-4835-9c24-fef3f72af78c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-7b8e72c7-7df96bbb58-cjwl7"
Sep 11 01:14:33.639325 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:14:33.639298    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39491142-6059-4835-9c24-fef3f72af78c-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-7b8e72c7-7df96bbb58-cjwl7\" (UID: \"39491142-6059-4835-9c24-fef3f72af78c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-7b8e72c7-7df96bbb58-cjwl7"
Sep 11 01:14:33.652767 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:14:33.652737    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2fsd\" (UniqueName: \"kubernetes.io/projected/39491142-6059-4835-9c24-fef3f72af78c-kube-api-access-m2fsd\") pod \"payload-pre-processing-e2e-mt-7b8e72c7-7df96bbb58-cjwl7\" (UID: \"39491142-6059-4835-9c24-fef3f72af78c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-7b8e72c7-7df96bbb58-cjwl7"
Sep 11 01:14:33.739718 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:14:33.739644    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-647ng\" (UniqueName: \"kubernetes.io/projected/a926ff1c-074b-4592-9089-4b9278c44414-kube-api-access-647ng\") pod \"payload-processing-e2e-mt-7b8e72c7-66f454f4bf-k6l45\" (UID: \"a926ff1c-074b-4592-9089-4b9278c44414\") " pod="openshift-ingress/payload-processing-e2e-mt-7b8e72c7-66f454f4bf-k6l45"
Sep 11 01:14:33.739718 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:14:33.739684    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a926ff1c-074b-4592-9089-4b9278c44414-plugins-config-volume\") pod \"payload-processing-e2e-mt-7b8e72c7-66f454f4bf-k6l45\" (UID: \"a926ff1c-074b-4592-9089-4b9278c44414\") " pod="openshift-ingress/payload-processing-e2e-mt-7b8e72c7-66f454f4bf-k6l45"
Sep 11 01:14:33.740359 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:14:33.740340    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a926ff1c-074b-4592-9089-4b9278c44414-plugins-config-volume\") pod \"payload-processing-e2e-mt-7b8e72c7-66f454f4bf-k6l45\" (UID: \"a926ff1c-074b-4592-9089-4b9278c44414\") " pod="openshift-ingress/payload-processing-e2e-mt-7b8e72c7-66f454f4bf-k6l45"
Sep 11 01:14:33.747559 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:14:33.747535    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-647ng\" (UniqueName: \"kubernetes.io/projected/a926ff1c-074b-4592-9089-4b9278c44414-kube-api-access-647ng\") pod \"payload-processing-e2e-mt-7b8e72c7-66f454f4bf-k6l45\" (UID: \"a926ff1c-074b-4592-9089-4b9278c44414\") " pod="openshift-ingress/payload-processing-e2e-mt-7b8e72c7-66f454f4bf-k6l45"
Sep 11 01:14:33.791695 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:14:33.791670    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-7b8e72c7-7df96bbb58-cjwl7"
Sep 11 01:14:33.841998 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:14:33.841967    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-7b8e72c7-66f454f4bf-k6l45"
Sep 11 01:14:34.065024 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:14:34.065000    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-7b8e72c7-7df96bbb58-cjwl7"]
Sep 11 01:14:34.066913 ip-10-0-129-222 kubenswrapper[2569]: W0911 01:14:34.066883    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39491142_6059_4835_9c24_fef3f72af78c.slice/crio-40c514fc2dfcc976b8b1a9262101c040cb42176710c820c9520c1a60c42075b0 WatchSource:0}: Error finding container 40c514fc2dfcc976b8b1a9262101c040cb42176710c820c9520c1a60c42075b0: Status 404 returned error can't find the container with id 40c514fc2dfcc976b8b1a9262101c040cb42176710c820c9520c1a60c42075b0
Sep 11 01:14:34.107535 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:14:34.107499    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-7b8e72c7-7df96bbb58-cjwl7" event={"ID":"39491142-6059-4835-9c24-fef3f72af78c","Type":"ContainerStarted","Data":"40c514fc2dfcc976b8b1a9262101c040cb42176710c820c9520c1a60c42075b0"}
Sep 11 01:14:34.126426 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:14:34.126386    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-7b8e72c7-66f454f4bf-k6l45"]
Sep 11 01:14:34.128555 ip-10-0-129-222 kubenswrapper[2569]: W0911 01:14:34.128528    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda926ff1c_074b_4592_9089_4b9278c44414.slice/crio-d864dc84d6d4ef05a36505f80a2b7b5adf75fd0abe47bc177954f66ed98bef57 WatchSource:0}: Error finding container d864dc84d6d4ef05a36505f80a2b7b5adf75fd0abe47bc177954f66ed98bef57: Status 404 returned error can't find the container with id d864dc84d6d4ef05a36505f80a2b7b5adf75fd0abe47bc177954f66ed98bef57
Sep 11 01:14:35.115693 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:14:35.115657    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-7b8e72c7-66f454f4bf-k6l45" event={"ID":"a926ff1c-074b-4592-9089-4b9278c44414","Type":"ContainerStarted","Data":"f07ab2d05aa332a84ef9a86682eed74471beac6f39f2255c664b86f8e7ab519b"}
Sep 11 01:14:35.116165 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:14:35.115714    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-7b8e72c7-66f454f4bf-k6l45" event={"ID":"a926ff1c-074b-4592-9089-4b9278c44414","Type":"ContainerStarted","Data":"d864dc84d6d4ef05a36505f80a2b7b5adf75fd0abe47bc177954f66ed98bef57"}
Sep 11 01:14:35.116165 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:14:35.115745    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-7b8e72c7-66f454f4bf-k6l45"
Sep 11 01:14:35.117416 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:14:35.117389    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-7b8e72c7-7df96bbb58-cjwl7" event={"ID":"39491142-6059-4835-9c24-fef3f72af78c","Type":"ContainerStarted","Data":"9274786b2222c6c07b810aa5e103afbe60e71d14df462e51aa5302d17fe2580b"}
Sep 11 01:14:35.117535 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:14:35.117516    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-7b8e72c7-7df96bbb58-cjwl7"
Sep 11 01:14:35.134520 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:14:35.134481    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-7b8e72c7-66f454f4bf-k6l45" podStartSLOduration=2.134466542 podStartE2EDuration="2.134466542s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 01:14:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 01:14:35.131535255 +0000 UTC m=+1009.782909505" watchObservedRunningTime="2026-09-11 01:14:35.134466542 +0000 UTC m=+1009.785840847"
Sep 11 01:14:35.147985 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:14:35.147947    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-7b8e72c7-7df96bbb58-cjwl7" podStartSLOduration=2.147934381 podStartE2EDuration="2.147934381s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 01:14:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 01:14:35.1467111 +0000 UTC m=+1009.798085351" watchObservedRunningTime="2026-09-11 01:14:35.147934381 +0000 UTC m=+1009.799308630"
Sep 11 01:14:36.940617 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:14:36.940585    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-8d451c-7d7b4b9f74-xw47v"]
Sep 11 01:14:36.941086 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:14:36.940836    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-explicit-8d451c-7d7b4b9f74-xw47v" podUID="f9868c6b-59cb-4e62-9979-b3a36b4fa2e2" containerName="payload-pre-processing" containerID="cri-o://9984423a1c56e1420832ef81af056f596a5f08aced94667feae807195604db4e" gracePeriod=30
Sep 11 01:14:37.087770 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:14:37.087735    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-8d451c-b8cjp"]
Sep 11 01:14:37.090044 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:14:37.090016    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-8d451c-b8cjp"]
Sep 11 01:14:37.132965 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:14:37.132857    2569 generic.go:356] "Generic (PLEG): container finished" podID="f9868c6b-59cb-4e62-9979-b3a36b4fa2e2" containerID="9984423a1c56e1420832ef81af056f596a5f08aced94667feae807195604db4e" exitCode=0
Sep 11 01:14:37.132965 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:14:37.132920    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-8d451c-7d7b4b9f74-xw47v" event={"ID":"f9868c6b-59cb-4e62-9979-b3a36b4fa2e2","Type":"ContainerDied","Data":"9984423a1c56e1420832ef81af056f596a5f08aced94667feae807195604db4e"}
Sep 11 01:14:37.218388 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:14:37.218364    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-8d451c-7d7b4b9f74-xw47v"
Sep 11 01:14:37.276518 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:14:37.276489    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c6qg5\" (UniqueName: \"kubernetes.io/projected/f9868c6b-59cb-4e62-9979-b3a36b4fa2e2-kube-api-access-c6qg5\") pod \"f9868c6b-59cb-4e62-9979-b3a36b4fa2e2\" (UID: \"f9868c6b-59cb-4e62-9979-b3a36b4fa2e2\") "
Sep 11 01:14:37.276673 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:14:37.276586    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9868c6b-59cb-4e62-9979-b3a36b4fa2e2-plugins-config-volume\") pod \"f9868c6b-59cb-4e62-9979-b3a36b4fa2e2\" (UID: \"f9868c6b-59cb-4e62-9979-b3a36b4fa2e2\") "
Sep 11 01:14:37.276977 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:14:37.276955    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9868c6b-59cb-4e62-9979-b3a36b4fa2e2-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f9868c6b-59cb-4e62-9979-b3a36b4fa2e2" (UID: "f9868c6b-59cb-4e62-9979-b3a36b4fa2e2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 01:14:37.277168 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:14:37.277155    2569 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9868c6b-59cb-4e62-9979-b3a36b4fa2e2-plugins-config-volume\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:14:37.278824 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:14:37.278793    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9868c6b-59cb-4e62-9979-b3a36b4fa2e2-kube-api-access-c6qg5" (OuterVolumeSpecName: "kube-api-access-c6qg5") pod "f9868c6b-59cb-4e62-9979-b3a36b4fa2e2" (UID: "f9868c6b-59cb-4e62-9979-b3a36b4fa2e2"). InnerVolumeSpecName "kube-api-access-c6qg5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 01:14:37.378650 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:14:37.378619    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c6qg5\" (UniqueName: \"kubernetes.io/projected/f9868c6b-59cb-4e62-9979-b3a36b4fa2e2-kube-api-access-c6qg5\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:14:37.876769 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:14:37.876731    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="437556d2-b3e8-40c4-8cff-6571c34c2083" path="/var/lib/kubelet/pods/437556d2-b3e8-40c4-8cff-6571c34c2083/volumes"
Sep 11 01:14:38.140207 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:14:38.140118    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-8d451c-7d7b4b9f74-xw47v" event={"ID":"f9868c6b-59cb-4e62-9979-b3a36b4fa2e2","Type":"ContainerDied","Data":"34b7453f92c5a29ee2de8fe97f067ccd84621349aa446e4e717170e8c223b92d"}
Sep 11 01:14:38.140207 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:14:38.140157    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-8d451c-7d7b4b9f74-xw47v"
Sep 11 01:14:38.140207 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:14:38.140187    2569 scope.go:117] "RemoveContainer" containerID="9984423a1c56e1420832ef81af056f596a5f08aced94667feae807195604db4e"
Sep 11 01:14:38.156838 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:14:38.156812    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-8d451c-7d7b4b9f74-xw47v"]
Sep 11 01:14:38.160389 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:14:38.160366    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-8d451c-7d7b4b9f74-xw47v"]
Sep 11 01:14:39.871597 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:14:39.871564    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f9868c6b-59cb-4e62-9979-b3a36b4fa2e2" path="/var/lib/kubelet/pods/f9868c6b-59cb-4e62-9979-b3a36b4fa2e2/volumes"
Sep 11 01:14:46.125368 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:14:46.125340    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-7b8e72c7-7df96bbb58-cjwl7"
Sep 11 01:14:46.125793 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:14:46.125406    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-7b8e72c7-66f454f4bf-k6l45"
Sep 11 01:14:48.696024 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:14:48.695986    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kmx5x"]
Sep 11 01:14:59.112061 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:14:59.112023    2569 patch_prober.go:28] interesting pod/e2e-mt-7b8e72c7-openshift-default-69b8bddc8d-9t2bk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 11 01:14:59.112573 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:14:59.112096    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-7b8e72c7-openshift-default-69b8bddc8d-9t2bk" podUID="e0387913-3dec-4201-8294-9fef9626afab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 11 01:15:00.216069 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:00.216035    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-891e67-29818155-nwb22"]
Sep 11 01:15:00.216713 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:00.216690    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f9868c6b-59cb-4e62-9979-b3a36b4fa2e2" containerName="payload-pre-processing"
Sep 11 01:15:00.216806 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:00.216716    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="f9868c6b-59cb-4e62-9979-b3a36b4fa2e2" containerName="payload-pre-processing"
Sep 11 01:15:00.216864 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:00.216838    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="f9868c6b-59cb-4e62-9979-b3a36b4fa2e2" containerName="payload-pre-processing"
Sep 11 01:15:00.219971 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:00.219946    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-891e67-29818155-nwb22"
Sep 11 01:15:00.223351 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:00.223327    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-cef92e-29818155-hdjcr"]
Sep 11 01:15:00.227034 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:00.227014    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-cef92e-29818155-hdjcr"
Sep 11 01:15:00.230572 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:00.230552    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pppkt\""
Sep 11 01:15:00.231043 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:00.231026    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-891e67-29818155-nwb22"
Sep 11 01:15:00.239051 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:00.239029    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-cef92e-29818155-hdjcr"
Sep 11 01:15:00.243431 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:00.243375    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-891e67-29818155-nwb22"]
Sep 11 01:15:00.251361 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:00.251127    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-cef92e-29818155-hdjcr"]
Sep 11 01:15:00.272901 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:00.272671    2569 generic.go:356] "Generic (PLEG): container finished" podID="e0387913-3dec-4201-8294-9fef9626afab" containerID="3bce9347574786343feaffbca610f346303b227c049b424d09cdbb521d51a9f2" exitCode=0
Sep 11 01:15:00.272901 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:00.272862    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7b8e72c7-openshift-default-69b8bddc8d-9t2bk" event={"ID":"e0387913-3dec-4201-8294-9fef9626afab","Type":"ContainerDied","Data":"3bce9347574786343feaffbca610f346303b227c049b424d09cdbb521d51a9f2"}
Sep 11 01:15:00.273366 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:00.273344    2569 scope.go:117] "RemoveContainer" containerID="3bce9347574786343feaffbca610f346303b227c049b424d09cdbb521d51a9f2"
Sep 11 01:15:00.274670 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:00.274637    2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 01:15:00.274757 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:00.274712    2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 01:15:00.274757 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:00.274750    2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 01:15:00.286191 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:00.286167    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6ba087-29818155-sjmqn"]
Sep 11 01:15:00.290668 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:00.290648    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6ba087-29818155-sjmqn"
Sep 11 01:15:00.323036 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:00.321660    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6ba087-29818155-sjmqn"]
Sep 11 01:15:00.336215 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:00.336195    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6ba087-29818155-sjmqn"
Sep 11 01:15:00.536422 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:00.536392    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-891e67-29818155-nwb22"]
Sep 11 01:15:00.538891 ip-10-0-129-222 kubenswrapper[2569]: W0911 01:15:00.538844    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccb561e8_3308_4372_9e98_3244fc775320.slice/crio-d770ed2a7c6bed85c3aaccfa5414cd57b135e8cdb840178083e947c5760008eb WatchSource:0}: Error finding container d770ed2a7c6bed85c3aaccfa5414cd57b135e8cdb840178083e947c5760008eb: Status 404 returned error can't find the container with id d770ed2a7c6bed85c3aaccfa5414cd57b135e8cdb840178083e947c5760008eb
Sep 11 01:15:00.561391 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:00.561365    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-cef92e-29818155-hdjcr"]
Sep 11 01:15:00.563325 ip-10-0-129-222 kubenswrapper[2569]: W0911 01:15:00.563298    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59c4ecbf_6d82_490f_b399_a97c8e1121bb.slice/crio-b586f704fb6527dd48d7bfb2f847c119ef135dc1aa24170faa3efca34b47db31 WatchSource:0}: Error finding container b586f704fb6527dd48d7bfb2f847c119ef135dc1aa24170faa3efca34b47db31: Status 404 returned error can't find the container with id b586f704fb6527dd48d7bfb2f847c119ef135dc1aa24170faa3efca34b47db31
Sep 11 01:15:00.628714 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:00.628688    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6ba087-29818155-sjmqn"]
Sep 11 01:15:00.629857 ip-10-0-129-222 kubenswrapper[2569]: W0911 01:15:00.629836    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9081aca3_50bb_4154_af4b_e494a88ecc32.slice/crio-141b0e0ff7e5271aed778ba842f1870affa4720de80183b4d8c4f864622c0cf4 WatchSource:0}: Error finding container 141b0e0ff7e5271aed778ba842f1870affa4720de80183b4d8c4f864622c0cf4: Status 404 returned error can't find the container with id 141b0e0ff7e5271aed778ba842f1870affa4720de80183b4d8c4f864622c0cf4
Sep 11 01:15:01.281287 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:01.281188    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6ba087-29818155-sjmqn" event={"ID":"9081aca3-50bb-4154-af4b-e494a88ecc32","Type":"ContainerStarted","Data":"e9202fc50c541f08db29853d495e0f0055f5302d687e9cff66d02912a15191b8"}
Sep 11 01:15:01.281287 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:01.281250    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6ba087-29818155-sjmqn" event={"ID":"9081aca3-50bb-4154-af4b-e494a88ecc32","Type":"ContainerStarted","Data":"141b0e0ff7e5271aed778ba842f1870affa4720de80183b4d8c4f864622c0cf4"}
Sep 11 01:15:01.282860 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:01.282834    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-891e67-29818155-nwb22" event={"ID":"ccb561e8-3308-4372-9e98-3244fc775320","Type":"ContainerStarted","Data":"4925d9c3f1ad2ce0d5bd2d901c6f1627fb6f8b4b18a51e0c2ecfb09f17eb66a3"}
Sep 11 01:15:01.282979 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:01.282891    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-891e67-29818155-nwb22" event={"ID":"ccb561e8-3308-4372-9e98-3244fc775320","Type":"ContainerStarted","Data":"d770ed2a7c6bed85c3aaccfa5414cd57b135e8cdb840178083e947c5760008eb"}
Sep 11 01:15:01.284917 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:01.284889    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7b8e72c7-openshift-default-69b8bddc8d-9t2bk" event={"ID":"e0387913-3dec-4201-8294-9fef9626afab","Type":"ContainerStarted","Data":"aa919d98f76731ccd5a3fe4e33a42a5b837dbda884efe342ef59e5c250421ba6"}
Sep 11 01:15:01.286471 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:01.286449    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-cef92e-29818155-hdjcr" event={"ID":"59c4ecbf-6d82-490f-b399-a97c8e1121bb","Type":"ContainerStarted","Data":"6204b0f50e449b004329d33c38d705dd9ff1cd601d600909c69048abb9591ad6"}
Sep 11 01:15:01.286572 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:01.286493    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-cef92e-29818155-hdjcr" event={"ID":"59c4ecbf-6d82-490f-b399-a97c8e1121bb","Type":"ContainerStarted","Data":"b586f704fb6527dd48d7bfb2f847c119ef135dc1aa24170faa3efca34b47db31"}
Sep 11 01:15:01.299430 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:01.299389    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6ba087-29818155-sjmqn" podStartSLOduration=1.299374654 podStartE2EDuration="1.299374654s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 01:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 01:15:01.297994509 +0000 UTC m=+1035.949368758" watchObservedRunningTime="2026-09-11 01:15:01.299374654 +0000 UTC m=+1035.950748910"
Sep 11 01:15:01.332723 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:01.332680    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-891e67-29818155-nwb22" podStartSLOduration=1.332667185 podStartE2EDuration="1.332667185s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 01:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 01:15:01.331900522 +0000 UTC m=+1035.983274771" watchObservedRunningTime="2026-09-11 01:15:01.332667185 +0000 UTC m=+1035.984041433"
Sep 11 01:15:01.358622 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:01.358548    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-cef92e-29818155-hdjcr" podStartSLOduration=1.358525585 podStartE2EDuration="1.358525585s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 01:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 01:15:01.35579871 +0000 UTC m=+1036.007172959" watchObservedRunningTime="2026-09-11 01:15:01.358525585 +0000 UTC m=+1036.009899836"
Sep 11 01:15:02.173931 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:02.173864    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-7b8e72c7-openshift-default-69b8bddc8d-9t2bk"
Sep 11 01:15:02.174414 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:02.174391    2569 patch_prober.go:28] interesting pod/e2e-mt-7b8e72c7-openshift-default-69b8bddc8d-9t2bk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body=
Sep 11 01:15:02.174467 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:02.174442    2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7b8e72c7-openshift-default-69b8bddc8d-9t2bk" podUID="e0387913-3dec-4201-8294-9fef9626afab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused"
Sep 11 01:15:02.294390 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:02.294359    2569 generic.go:356] "Generic (PLEG): container finished" podID="9081aca3-50bb-4154-af4b-e494a88ecc32" containerID="e9202fc50c541f08db29853d495e0f0055f5302d687e9cff66d02912a15191b8" exitCode=0
Sep 11 01:15:02.294802 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:02.294432    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6ba087-29818155-sjmqn" event={"ID":"9081aca3-50bb-4154-af4b-e494a88ecc32","Type":"ContainerDied","Data":"e9202fc50c541f08db29853d495e0f0055f5302d687e9cff66d02912a15191b8"}
Sep 11 01:15:02.295822 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:02.295799    2569 generic.go:356] "Generic (PLEG): container finished" podID="ccb561e8-3308-4372-9e98-3244fc775320" containerID="4925d9c3f1ad2ce0d5bd2d901c6f1627fb6f8b4b18a51e0c2ecfb09f17eb66a3" exitCode=0
Sep 11 01:15:02.295919 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:02.295887    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-891e67-29818155-nwb22" event={"ID":"ccb561e8-3308-4372-9e98-3244fc775320","Type":"ContainerDied","Data":"4925d9c3f1ad2ce0d5bd2d901c6f1627fb6f8b4b18a51e0c2ecfb09f17eb66a3"}
Sep 11 01:15:02.297329 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:02.297305    2569 generic.go:356] "Generic (PLEG): container finished" podID="59c4ecbf-6d82-490f-b399-a97c8e1121bb" containerID="6204b0f50e449b004329d33c38d705dd9ff1cd601d600909c69048abb9591ad6" exitCode=0
Sep 11 01:15:02.297468 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:02.297377    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-cef92e-29818155-hdjcr" event={"ID":"59c4ecbf-6d82-490f-b399-a97c8e1121bb","Type":"ContainerDied","Data":"6204b0f50e449b004329d33c38d705dd9ff1cd601d600909c69048abb9591ad6"}
Sep 11 01:15:03.174188 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:03.174147    2569 patch_prober.go:28] interesting pod/e2e-mt-7b8e72c7-openshift-default-69b8bddc8d-9t2bk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body=
Sep 11 01:15:03.174363 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:03.174220    2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7b8e72c7-openshift-default-69b8bddc8d-9t2bk" podUID="e0387913-3dec-4201-8294-9fef9626afab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused"
Sep 11 01:15:03.501776 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:03.501753    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6ba087-29818155-sjmqn"
Sep 11 01:15:03.562429 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:03.562402    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-891e67-29818155-nwb22"
Sep 11 01:15:03.567501 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:03.567469    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-cef92e-29818155-hdjcr"
Sep 11 01:15:04.174299 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:04.174205    2569 patch_prober.go:28] interesting pod/e2e-mt-7b8e72c7-openshift-default-69b8bddc8d-9t2bk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body=
Sep 11 01:15:04.174299 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:04.174272    2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7b8e72c7-openshift-default-69b8bddc8d-9t2bk" podUID="e0387913-3dec-4201-8294-9fef9626afab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused"
Sep 11 01:15:04.309824 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:04.309794    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6ba087-29818155-sjmqn"
Sep 11 01:15:04.310033 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:04.309797    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6ba087-29818155-sjmqn" event={"ID":"9081aca3-50bb-4154-af4b-e494a88ecc32","Type":"ContainerDied","Data":"141b0e0ff7e5271aed778ba842f1870affa4720de80183b4d8c4f864622c0cf4"}
Sep 11 01:15:04.310033 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:04.309941    2569 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="141b0e0ff7e5271aed778ba842f1870affa4720de80183b4d8c4f864622c0cf4"
Sep 11 01:15:04.311239 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:04.311215    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-891e67-29818155-nwb22"
Sep 11 01:15:04.311357 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:04.311238    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-891e67-29818155-nwb22" event={"ID":"ccb561e8-3308-4372-9e98-3244fc775320","Type":"ContainerDied","Data":"d770ed2a7c6bed85c3aaccfa5414cd57b135e8cdb840178083e947c5760008eb"}
Sep 11 01:15:04.311357 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:04.311276    2569 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d770ed2a7c6bed85c3aaccfa5414cd57b135e8cdb840178083e947c5760008eb"
Sep 11 01:15:04.312608 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:04.312591    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-cef92e-29818155-hdjcr"
Sep 11 01:15:04.312706 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:04.312631    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-cef92e-29818155-hdjcr" event={"ID":"59c4ecbf-6d82-490f-b399-a97c8e1121bb","Type":"ContainerDied","Data":"b586f704fb6527dd48d7bfb2f847c119ef135dc1aa24170faa3efca34b47db31"}
Sep 11 01:15:04.312706 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:04.312665    2569 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b586f704fb6527dd48d7bfb2f847c119ef135dc1aa24170faa3efca34b47db31"
Sep 11 01:15:05.173463 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:05.173426    2569 patch_prober.go:28] interesting pod/e2e-mt-7b8e72c7-openshift-default-69b8bddc8d-9t2bk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body=
Sep 11 01:15:05.173935 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:05.173489    2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7b8e72c7-openshift-default-69b8bddc8d-9t2bk" podUID="e0387913-3dec-4201-8294-9fef9626afab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused"
Sep 11 01:15:06.174254 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:06.174222    2569 patch_prober.go:28] interesting pod/e2e-mt-7b8e72c7-openshift-default-69b8bddc8d-9t2bk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body=
Sep 11 01:15:06.174645 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:06.174283    2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7b8e72c7-openshift-default-69b8bddc8d-9t2bk" podUID="e0387913-3dec-4201-8294-9fef9626afab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused"
Sep 11 01:15:07.177198 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:07.177169    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-7b8e72c7-openshift-default-69b8bddc8d-9t2bk"
Sep 11 01:15:07.177634 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:07.177439    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-7b8e72c7-openshift-default-69b8bddc8d-9t2bk"
Sep 11 01:15:07.178239 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:07.178223    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-7b8e72c7-openshift-default-69b8bddc8d-9t2bk"
Sep 11 01:15:14.288806 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:14.288771    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-7b8e72c7-66f454f4bf-k6l45"]
Sep 11 01:15:14.289295 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:14.289049    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-7b8e72c7-66f454f4bf-k6l45" podUID="a926ff1c-074b-4592-9089-4b9278c44414" containerName="payload-processing" containerID="cri-o://f07ab2d05aa332a84ef9a86682eed74471beac6f39f2255c664b86f8e7ab519b" gracePeriod=30
Sep 11 01:15:14.292526 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:14.292495    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-7b8e72c7-7df96bbb58-cjwl7"]
Sep 11 01:15:14.292762 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:14.292714    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-7b8e72c7-7df96bbb58-cjwl7" podUID="39491142-6059-4835-9c24-fef3f72af78c" containerName="payload-pre-processing" containerID="cri-o://9274786b2222c6c07b810aa5e103afbe60e71d14df462e51aa5302d17fe2580b" gracePeriod=30
Sep 11 01:15:14.396589 ip-10-0-129-222 kubenswrapper[2569]: E0911 01:15:14.396556    2569 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda926ff1c_074b_4592_9089_4b9278c44414.slice/crio-conmon-f07ab2d05aa332a84ef9a86682eed74471beac6f39f2255c664b86f8e7ab519b.scope\": RecentStats: unable to find data in memory cache]"
Sep 11 01:15:14.594860 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:14.594830    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-7b8e72c7-7df96bbb58-cjwl7"
Sep 11 01:15:14.597792 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:14.597767    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-7b8e72c7-66f454f4bf-k6l45"
Sep 11 01:15:14.672667 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:14.672623    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a926ff1c-074b-4592-9089-4b9278c44414-plugins-config-volume\") pod \"a926ff1c-074b-4592-9089-4b9278c44414\" (UID: \"a926ff1c-074b-4592-9089-4b9278c44414\") "
Sep 11 01:15:14.672838 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:14.672709    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39491142-6059-4835-9c24-fef3f72af78c-plugins-config-volume\") pod \"39491142-6059-4835-9c24-fef3f72af78c\" (UID: \"39491142-6059-4835-9c24-fef3f72af78c\") "
Sep 11 01:15:14.672838 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:14.672802    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-647ng\" (UniqueName: \"kubernetes.io/projected/a926ff1c-074b-4592-9089-4b9278c44414-kube-api-access-647ng\") pod \"a926ff1c-074b-4592-9089-4b9278c44414\" (UID: \"a926ff1c-074b-4592-9089-4b9278c44414\") "
Sep 11 01:15:14.672997 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:14.672861    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m2fsd\" (UniqueName: \"kubernetes.io/projected/39491142-6059-4835-9c24-fef3f72af78c-kube-api-access-m2fsd\") pod \"39491142-6059-4835-9c24-fef3f72af78c\" (UID: \"39491142-6059-4835-9c24-fef3f72af78c\") "
Sep 11 01:15:14.673095 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:14.673062    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a926ff1c-074b-4592-9089-4b9278c44414-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a926ff1c-074b-4592-9089-4b9278c44414" (UID: "a926ff1c-074b-4592-9089-4b9278c44414"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 01:15:14.673192 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:14.673165    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39491142-6059-4835-9c24-fef3f72af78c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "39491142-6059-4835-9c24-fef3f72af78c" (UID: "39491142-6059-4835-9c24-fef3f72af78c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 01:15:14.673351 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:14.673333    2569 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a926ff1c-074b-4592-9089-4b9278c44414-plugins-config-volume\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:15:14.673420 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:14.673354    2569 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39491142-6059-4835-9c24-fef3f72af78c-plugins-config-volume\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:15:14.675204 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:14.675160    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a926ff1c-074b-4592-9089-4b9278c44414-kube-api-access-647ng" (OuterVolumeSpecName: "kube-api-access-647ng") pod "a926ff1c-074b-4592-9089-4b9278c44414" (UID: "a926ff1c-074b-4592-9089-4b9278c44414"). InnerVolumeSpecName "kube-api-access-647ng". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 01:15:14.675302 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:14.675279    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39491142-6059-4835-9c24-fef3f72af78c-kube-api-access-m2fsd" (OuterVolumeSpecName: "kube-api-access-m2fsd") pod "39491142-6059-4835-9c24-fef3f72af78c" (UID: "39491142-6059-4835-9c24-fef3f72af78c"). InnerVolumeSpecName "kube-api-access-m2fsd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 01:15:14.774170 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:14.774139    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-647ng\" (UniqueName: \"kubernetes.io/projected/a926ff1c-074b-4592-9089-4b9278c44414-kube-api-access-647ng\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:15:14.774170 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:14.774165    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m2fsd\" (UniqueName: \"kubernetes.io/projected/39491142-6059-4835-9c24-fef3f72af78c-kube-api-access-m2fsd\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:15:15.379174 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:15.379143    2569 generic.go:356] "Generic (PLEG): container finished" podID="a926ff1c-074b-4592-9089-4b9278c44414" containerID="f07ab2d05aa332a84ef9a86682eed74471beac6f39f2255c664b86f8e7ab519b" exitCode=0
Sep 11 01:15:15.379630 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:15.379183    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-7b8e72c7-66f454f4bf-k6l45"
Sep 11 01:15:15.379630 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:15.379220    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-7b8e72c7-66f454f4bf-k6l45" event={"ID":"a926ff1c-074b-4592-9089-4b9278c44414","Type":"ContainerDied","Data":"f07ab2d05aa332a84ef9a86682eed74471beac6f39f2255c664b86f8e7ab519b"}
Sep 11 01:15:15.379630 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:15.379274    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-7b8e72c7-66f454f4bf-k6l45" event={"ID":"a926ff1c-074b-4592-9089-4b9278c44414","Type":"ContainerDied","Data":"d864dc84d6d4ef05a36505f80a2b7b5adf75fd0abe47bc177954f66ed98bef57"}
Sep 11 01:15:15.379630 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:15.379305    2569 scope.go:117] "RemoveContainer" containerID="f07ab2d05aa332a84ef9a86682eed74471beac6f39f2255c664b86f8e7ab519b"
Sep 11 01:15:15.380629 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:15.380545    2569 generic.go:356] "Generic (PLEG): container finished" podID="39491142-6059-4835-9c24-fef3f72af78c" containerID="9274786b2222c6c07b810aa5e103afbe60e71d14df462e51aa5302d17fe2580b" exitCode=0
Sep 11 01:15:15.380690 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:15.380663    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-7b8e72c7-7df96bbb58-cjwl7"
Sep 11 01:15:15.380690 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:15.380679    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-7b8e72c7-7df96bbb58-cjwl7" event={"ID":"39491142-6059-4835-9c24-fef3f72af78c","Type":"ContainerDied","Data":"9274786b2222c6c07b810aa5e103afbe60e71d14df462e51aa5302d17fe2580b"}
Sep 11 01:15:15.380773 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:15.380709    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-7b8e72c7-7df96bbb58-cjwl7" event={"ID":"39491142-6059-4835-9c24-fef3f72af78c","Type":"ContainerDied","Data":"40c514fc2dfcc976b8b1a9262101c040cb42176710c820c9520c1a60c42075b0"}
Sep 11 01:15:15.390732 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:15.390717    2569 scope.go:117] "RemoveContainer" containerID="f07ab2d05aa332a84ef9a86682eed74471beac6f39f2255c664b86f8e7ab519b"
Sep 11 01:15:15.391080 ip-10-0-129-222 kubenswrapper[2569]: E0911 01:15:15.391050    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f07ab2d05aa332a84ef9a86682eed74471beac6f39f2255c664b86f8e7ab519b\": container with ID starting with f07ab2d05aa332a84ef9a86682eed74471beac6f39f2255c664b86f8e7ab519b not found: ID does not exist" containerID="f07ab2d05aa332a84ef9a86682eed74471beac6f39f2255c664b86f8e7ab519b"
Sep 11 01:15:15.391169 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:15.391090    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f07ab2d05aa332a84ef9a86682eed74471beac6f39f2255c664b86f8e7ab519b"} err="failed to get container status \"f07ab2d05aa332a84ef9a86682eed74471beac6f39f2255c664b86f8e7ab519b\": rpc error: code = NotFound desc = could not find container \"f07ab2d05aa332a84ef9a86682eed74471beac6f39f2255c664b86f8e7ab519b\": container with ID starting with f07ab2d05aa332a84ef9a86682eed74471beac6f39f2255c664b86f8e7ab519b not found: ID does not exist"
Sep 11 01:15:15.391169 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:15.391114    2569 scope.go:117] "RemoveContainer" containerID="9274786b2222c6c07b810aa5e103afbe60e71d14df462e51aa5302d17fe2580b"
Sep 11 01:15:15.399555 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:15.399537    2569 scope.go:117] "RemoveContainer" containerID="9274786b2222c6c07b810aa5e103afbe60e71d14df462e51aa5302d17fe2580b"
Sep 11 01:15:15.399825 ip-10-0-129-222 kubenswrapper[2569]: E0911 01:15:15.399802    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9274786b2222c6c07b810aa5e103afbe60e71d14df462e51aa5302d17fe2580b\": container with ID starting with 9274786b2222c6c07b810aa5e103afbe60e71d14df462e51aa5302d17fe2580b not found: ID does not exist" containerID="9274786b2222c6c07b810aa5e103afbe60e71d14df462e51aa5302d17fe2580b"
Sep 11 01:15:15.399956 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:15.399836    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9274786b2222c6c07b810aa5e103afbe60e71d14df462e51aa5302d17fe2580b"} err="failed to get container status \"9274786b2222c6c07b810aa5e103afbe60e71d14df462e51aa5302d17fe2580b\": rpc error: code = NotFound desc = could not find container \"9274786b2222c6c07b810aa5e103afbe60e71d14df462e51aa5302d17fe2580b\": container with ID starting with 9274786b2222c6c07b810aa5e103afbe60e71d14df462e51aa5302d17fe2580b not found: ID does not exist"
Sep 11 01:15:15.406307 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:15.406278    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-7b8e72c7-7df96bbb58-cjwl7"]
Sep 11 01:15:15.409850 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:15.409828    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-7b8e72c7-7df96bbb58-cjwl7"]
Sep 11 01:15:15.418605 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:15.418584    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-7b8e72c7-66f454f4bf-k6l45"]
Sep 11 01:15:15.424516 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:15.424496    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-7b8e72c7-66f454f4bf-k6l45"]
Sep 11 01:15:15.871337 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:15.871303    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="39491142-6059-4835-9c24-fef3f72af78c" path="/var/lib/kubelet/pods/39491142-6059-4835-9c24-fef3f72af78c/volumes"
Sep 11 01:15:15.871747 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:15.871730    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a926ff1c-074b-4592-9089-4b9278c44414" path="/var/lib/kubelet/pods/a926ff1c-074b-4592-9089-4b9278c44414/volumes"
Sep 11 01:15:20.298605 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:20.298570    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kmx5x"]
Sep 11 01:15:20.879345 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:20.879312    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-7b8e72c7-openshift-default-69b8bddc8d-9t2bk"]
Sep 11 01:15:20.879535 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:20.879517    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-7b8e72c7-openshift-default-69b8bddc8d-9t2bk" podUID="e0387913-3dec-4201-8294-9fef9626afab" containerName="istio-proxy" containerID="cri-o://aa919d98f76731ccd5a3fe4e33a42a5b837dbda884efe342ef59e5c250421ba6" gracePeriod=30
Sep 11 01:15:22.179595 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:22.179558    2569 patch_prober.go:28] interesting pod/e2e-mt-7b8e72c7-openshift-default-69b8bddc8d-9t2bk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 11 01:15:22.179967 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:22.179614    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-7b8e72c7-openshift-default-69b8bddc8d-9t2bk" podUID="e0387913-3dec-4201-8294-9fef9626afab" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 11 01:15:26.143736 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:26.143677    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7b8e72c7-openshift-default-69b8bddc8d-9t2bk"
Sep 11 01:15:26.287677 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:26.287636    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0387913-3dec-4201-8294-9fef9626afab-istio-podinfo\") pod \"e0387913-3dec-4201-8294-9fef9626afab\" (UID: \"e0387913-3dec-4201-8294-9fef9626afab\") "
Sep 11 01:15:26.287677 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:26.287686    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2f2n7\" (UniqueName: \"kubernetes.io/projected/e0387913-3dec-4201-8294-9fef9626afab-kube-api-access-2f2n7\") pod \"e0387913-3dec-4201-8294-9fef9626afab\" (UID: \"e0387913-3dec-4201-8294-9fef9626afab\") "
Sep 11 01:15:26.287954 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:26.287712    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0387913-3dec-4201-8294-9fef9626afab-istio-data\") pod \"e0387913-3dec-4201-8294-9fef9626afab\" (UID: \"e0387913-3dec-4201-8294-9fef9626afab\") "
Sep 11 01:15:26.287954 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:26.287741    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0387913-3dec-4201-8294-9fef9626afab-workload-certs\") pod \"e0387913-3dec-4201-8294-9fef9626afab\" (UID: \"e0387913-3dec-4201-8294-9fef9626afab\") "
Sep 11 01:15:26.287954 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:26.287758    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0387913-3dec-4201-8294-9fef9626afab-istio-envoy\") pod \"e0387913-3dec-4201-8294-9fef9626afab\" (UID: \"e0387913-3dec-4201-8294-9fef9626afab\") "
Sep 11 01:15:26.287954 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:26.287795    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0387913-3dec-4201-8294-9fef9626afab-workload-socket\") pod \"e0387913-3dec-4201-8294-9fef9626afab\" (UID: \"e0387913-3dec-4201-8294-9fef9626afab\") "
Sep 11 01:15:26.287954 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:26.287819    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0387913-3dec-4201-8294-9fef9626afab-istio-token\") pod \"e0387913-3dec-4201-8294-9fef9626afab\" (UID: \"e0387913-3dec-4201-8294-9fef9626afab\") "
Sep 11 01:15:26.287954 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:26.287845    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0387913-3dec-4201-8294-9fef9626afab-istiod-ca-cert\") pod \"e0387913-3dec-4201-8294-9fef9626afab\" (UID: \"e0387913-3dec-4201-8294-9fef9626afab\") "
Sep 11 01:15:26.287954 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:26.287920    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0387913-3dec-4201-8294-9fef9626afab-credential-socket\") pod \"e0387913-3dec-4201-8294-9fef9626afab\" (UID: \"e0387913-3dec-4201-8294-9fef9626afab\") "
Sep 11 01:15:26.288318 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:26.288140    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0387913-3dec-4201-8294-9fef9626afab-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e0387913-3dec-4201-8294-9fef9626afab" (UID: "e0387913-3dec-4201-8294-9fef9626afab"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 01:15:26.288318 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:26.288149    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0387913-3dec-4201-8294-9fef9626afab-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e0387913-3dec-4201-8294-9fef9626afab" (UID: "e0387913-3dec-4201-8294-9fef9626afab"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 01:15:26.288430 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:26.288333    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0387913-3dec-4201-8294-9fef9626afab-istio-data" (OuterVolumeSpecName: "istio-data") pod "e0387913-3dec-4201-8294-9fef9626afab" (UID: "e0387913-3dec-4201-8294-9fef9626afab"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 01:15:26.288430 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:26.288348    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0387913-3dec-4201-8294-9fef9626afab-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e0387913-3dec-4201-8294-9fef9626afab" (UID: "e0387913-3dec-4201-8294-9fef9626afab"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 01:15:26.288537 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:26.288445    2569 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0387913-3dec-4201-8294-9fef9626afab-workload-certs\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:15:26.288537 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:26.288462    2569 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0387913-3dec-4201-8294-9fef9626afab-workload-socket\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:15:26.288537 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:26.288444    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0387913-3dec-4201-8294-9fef9626afab-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e0387913-3dec-4201-8294-9fef9626afab" (UID: "e0387913-3dec-4201-8294-9fef9626afab"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 01:15:26.289960 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:26.289938    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0387913-3dec-4201-8294-9fef9626afab-kube-api-access-2f2n7" (OuterVolumeSpecName: "kube-api-access-2f2n7") pod "e0387913-3dec-4201-8294-9fef9626afab" (UID: "e0387913-3dec-4201-8294-9fef9626afab"). InnerVolumeSpecName "kube-api-access-2f2n7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 01:15:26.289960 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:26.289942    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e0387913-3dec-4201-8294-9fef9626afab-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e0387913-3dec-4201-8294-9fef9626afab" (UID: "e0387913-3dec-4201-8294-9fef9626afab"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 01:15:26.290395 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:26.290372    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0387913-3dec-4201-8294-9fef9626afab-istio-token" (OuterVolumeSpecName: "istio-token") pod "e0387913-3dec-4201-8294-9fef9626afab" (UID: "e0387913-3dec-4201-8294-9fef9626afab"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 01:15:26.290456 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:26.290370    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0387913-3dec-4201-8294-9fef9626afab-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e0387913-3dec-4201-8294-9fef9626afab" (UID: "e0387913-3dec-4201-8294-9fef9626afab"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 01:15:26.389669 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:26.389638    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2f2n7\" (UniqueName: \"kubernetes.io/projected/e0387913-3dec-4201-8294-9fef9626afab-kube-api-access-2f2n7\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:15:26.389669 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:26.389666    2569 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0387913-3dec-4201-8294-9fef9626afab-istio-data\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:15:26.389669 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:26.389675    2569 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0387913-3dec-4201-8294-9fef9626afab-istio-envoy\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:15:26.389923 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:26.389683    2569 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0387913-3dec-4201-8294-9fef9626afab-istio-token\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:15:26.389923 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:26.389692    2569 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0387913-3dec-4201-8294-9fef9626afab-istiod-ca-cert\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:15:26.389923 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:26.389700    2569 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0387913-3dec-4201-8294-9fef9626afab-credential-socket\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:15:26.389923 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:26.389711    2569 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0387913-3dec-4201-8294-9fef9626afab-istio-podinfo\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:15:26.444086 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:26.444020    2569 generic.go:356] "Generic (PLEG): container finished" podID="e0387913-3dec-4201-8294-9fef9626afab" containerID="aa919d98f76731ccd5a3fe4e33a42a5b837dbda884efe342ef59e5c250421ba6" exitCode=0
Sep 11 01:15:26.444086 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:26.444074    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7b8e72c7-openshift-default-69b8bddc8d-9t2bk"
Sep 11 01:15:26.444237 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:26.444108    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7b8e72c7-openshift-default-69b8bddc8d-9t2bk" event={"ID":"e0387913-3dec-4201-8294-9fef9626afab","Type":"ContainerDied","Data":"aa919d98f76731ccd5a3fe4e33a42a5b837dbda884efe342ef59e5c250421ba6"}
Sep 11 01:15:26.444237 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:26.444163    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7b8e72c7-openshift-default-69b8bddc8d-9t2bk" event={"ID":"e0387913-3dec-4201-8294-9fef9626afab","Type":"ContainerDied","Data":"e68d2d6301052d7e6240daa0df9a2f4c5c31d2ccafa8d58c814ade07fe8d0855"}
Sep 11 01:15:26.444237 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:26.444192    2569 scope.go:117] "RemoveContainer" containerID="aa919d98f76731ccd5a3fe4e33a42a5b837dbda884efe342ef59e5c250421ba6"
Sep 11 01:15:26.454495 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:26.454475    2569 scope.go:117] "RemoveContainer" containerID="3bce9347574786343feaffbca610f346303b227c049b424d09cdbb521d51a9f2"
Sep 11 01:15:26.463949 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:26.463925    2569 scope.go:117] "RemoveContainer" containerID="aa919d98f76731ccd5a3fe4e33a42a5b837dbda884efe342ef59e5c250421ba6"
Sep 11 01:15:26.464267 ip-10-0-129-222 kubenswrapper[2569]: E0911 01:15:26.464243    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa919d98f76731ccd5a3fe4e33a42a5b837dbda884efe342ef59e5c250421ba6\": container with ID starting with aa919d98f76731ccd5a3fe4e33a42a5b837dbda884efe342ef59e5c250421ba6 not found: ID does not exist" containerID="aa919d98f76731ccd5a3fe4e33a42a5b837dbda884efe342ef59e5c250421ba6"
Sep 11 01:15:26.464341 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:26.464277    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa919d98f76731ccd5a3fe4e33a42a5b837dbda884efe342ef59e5c250421ba6"} err="failed to get container status \"aa919d98f76731ccd5a3fe4e33a42a5b837dbda884efe342ef59e5c250421ba6\": rpc error: code = NotFound desc = could not find container \"aa919d98f76731ccd5a3fe4e33a42a5b837dbda884efe342ef59e5c250421ba6\": container with ID starting with aa919d98f76731ccd5a3fe4e33a42a5b837dbda884efe342ef59e5c250421ba6 not found: ID does not exist"
Sep 11 01:15:26.464341 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:26.464296    2569 scope.go:117] "RemoveContainer" containerID="3bce9347574786343feaffbca610f346303b227c049b424d09cdbb521d51a9f2"
Sep 11 01:15:26.464561 ip-10-0-129-222 kubenswrapper[2569]: E0911 01:15:26.464542    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3bce9347574786343feaffbca610f346303b227c049b424d09cdbb521d51a9f2\": container with ID starting with 3bce9347574786343feaffbca610f346303b227c049b424d09cdbb521d51a9f2 not found: ID does not exist" containerID="3bce9347574786343feaffbca610f346303b227c049b424d09cdbb521d51a9f2"
Sep 11 01:15:26.464601 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:26.464565    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3bce9347574786343feaffbca610f346303b227c049b424d09cdbb521d51a9f2"} err="failed to get container status \"3bce9347574786343feaffbca610f346303b227c049b424d09cdbb521d51a9f2\": rpc error: code = NotFound desc = could not find container \"3bce9347574786343feaffbca610f346303b227c049b424d09cdbb521d51a9f2\": container with ID starting with 3bce9347574786343feaffbca610f346303b227c049b424d09cdbb521d51a9f2 not found: ID does not exist"
Sep 11 01:15:26.466612 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:26.466566    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-7b8e72c7-openshift-default-69b8bddc8d-9t2bk"]
Sep 11 01:15:26.469714 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:26.469692    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-7b8e72c7-openshift-default-69b8bddc8d-9t2bk"]
Sep 11 01:15:27.871168 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:27.871125    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e0387913-3dec-4201-8294-9fef9626afab" path="/var/lib/kubelet/pods/e0387913-3dec-4201-8294-9fef9626afab/volumes"
Sep 11 01:15:34.791887 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:34.791844    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-799002-openshift-default-69d7698d8d-gfrd4"]
Sep 11 01:15:34.792382 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:34.792328    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9081aca3-50bb-4154-af4b-e494a88ecc32" containerName="cleanup"
Sep 11 01:15:34.792382 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:34.792344    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="9081aca3-50bb-4154-af4b-e494a88ecc32" containerName="cleanup"
Sep 11 01:15:34.792382 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:34.792356    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0387913-3dec-4201-8294-9fef9626afab" containerName="istio-proxy"
Sep 11 01:15:34.792382 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:34.792362    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0387913-3dec-4201-8294-9fef9626afab" containerName="istio-proxy"
Sep 11 01:15:34.792382 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:34.792379    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="39491142-6059-4835-9c24-fef3f72af78c" containerName="payload-pre-processing"
Sep 11 01:15:34.792382 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:34.792384    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="39491142-6059-4835-9c24-fef3f72af78c" containerName="payload-pre-processing"
Sep 11 01:15:34.792600 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:34.792391    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a926ff1c-074b-4592-9089-4b9278c44414" containerName="payload-processing"
Sep 11 01:15:34.792600 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:34.792397    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="a926ff1c-074b-4592-9089-4b9278c44414" containerName="payload-processing"
Sep 11 01:15:34.792600 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:34.792407    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ccb561e8-3308-4372-9e98-3244fc775320" containerName="cleanup"
Sep 11 01:15:34.792600 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:34.792415    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="ccb561e8-3308-4372-9e98-3244fc775320" containerName="cleanup"
Sep 11 01:15:34.792600 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:34.792445    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59c4ecbf-6d82-490f-b399-a97c8e1121bb" containerName="cleanup"
Sep 11 01:15:34.792600 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:34.792451    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="59c4ecbf-6d82-490f-b399-a97c8e1121bb" containerName="cleanup"
Sep 11 01:15:34.792600 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:34.792506    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="9081aca3-50bb-4154-af4b-e494a88ecc32" containerName="cleanup"
Sep 11 01:15:34.792600 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:34.792517    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="ccb561e8-3308-4372-9e98-3244fc775320" containerName="cleanup"
Sep 11 01:15:34.792600 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:34.792524    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="e0387913-3dec-4201-8294-9fef9626afab" containerName="istio-proxy"
Sep 11 01:15:34.792600 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:34.792528    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="39491142-6059-4835-9c24-fef3f72af78c" containerName="payload-pre-processing"
Sep 11 01:15:34.792600 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:34.792533    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="a926ff1c-074b-4592-9089-4b9278c44414" containerName="payload-processing"
Sep 11 01:15:34.792600 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:34.792538    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="59c4ecbf-6d82-490f-b399-a97c8e1121bb" containerName="cleanup"
Sep 11 01:15:34.792600 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:34.792544    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="e0387913-3dec-4201-8294-9fef9626afab" containerName="istio-proxy"
Sep 11 01:15:34.797190 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:34.797170    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-799002-openshift-default-69d7698d8d-gfrd4"
Sep 11 01:15:34.799440 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:34.799416    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-799002-openshift-default-dockercfg-c8qnk\""
Sep 11 01:15:34.807156 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:34.807132    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-799002-openshift-default-69d7698d8d-gfrd4"]
Sep 11 01:15:34.854650 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:34.854622    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64a075bf-600d-4fd2-8ec1-a74c1f4cedd5-workload-certs\") pod \"e2e-shared-a-w2-799002-openshift-default-69d7698d8d-gfrd4\" (UID: \"64a075bf-600d-4fd2-8ec1-a74c1f4cedd5\") " pod="openshift-ingress/e2e-shared-a-w2-799002-openshift-default-69d7698d8d-gfrd4"
Sep 11 01:15:34.854784 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:34.854659    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dsbjq\" (UniqueName: \"kubernetes.io/projected/64a075bf-600d-4fd2-8ec1-a74c1f4cedd5-kube-api-access-dsbjq\") pod \"e2e-shared-a-w2-799002-openshift-default-69d7698d8d-gfrd4\" (UID: \"64a075bf-600d-4fd2-8ec1-a74c1f4cedd5\") " pod="openshift-ingress/e2e-shared-a-w2-799002-openshift-default-69d7698d8d-gfrd4"
Sep 11 01:15:34.854784 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:34.854753    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64a075bf-600d-4fd2-8ec1-a74c1f4cedd5-istio-data\") pod \"e2e-shared-a-w2-799002-openshift-default-69d7698d8d-gfrd4\" (UID: \"64a075bf-600d-4fd2-8ec1-a74c1f4cedd5\") " pod="openshift-ingress/e2e-shared-a-w2-799002-openshift-default-69d7698d8d-gfrd4"
Sep 11 01:15:34.854862 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:34.854801    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64a075bf-600d-4fd2-8ec1-a74c1f4cedd5-credential-socket\") pod \"e2e-shared-a-w2-799002-openshift-default-69d7698d8d-gfrd4\" (UID: \"64a075bf-600d-4fd2-8ec1-a74c1f4cedd5\") " pod="openshift-ingress/e2e-shared-a-w2-799002-openshift-default-69d7698d8d-gfrd4"
Sep 11 01:15:34.854862 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:34.854819    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64a075bf-600d-4fd2-8ec1-a74c1f4cedd5-istiod-ca-cert\") pod \"e2e-shared-a-w2-799002-openshift-default-69d7698d8d-gfrd4\" (UID: \"64a075bf-600d-4fd2-8ec1-a74c1f4cedd5\") " pod="openshift-ingress/e2e-shared-a-w2-799002-openshift-default-69d7698d8d-gfrd4"
Sep 11 01:15:34.854862 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:34.854837    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64a075bf-600d-4fd2-8ec1-a74c1f4cedd5-workload-socket\") pod \"e2e-shared-a-w2-799002-openshift-default-69d7698d8d-gfrd4\" (UID: \"64a075bf-600d-4fd2-8ec1-a74c1f4cedd5\") " pod="openshift-ingress/e2e-shared-a-w2-799002-openshift-default-69d7698d8d-gfrd4"
Sep 11 01:15:34.855003 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:34.854894    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64a075bf-600d-4fd2-8ec1-a74c1f4cedd5-istio-token\") pod \"e2e-shared-a-w2-799002-openshift-default-69d7698d8d-gfrd4\" (UID: \"64a075bf-600d-4fd2-8ec1-a74c1f4cedd5\") " pod="openshift-ingress/e2e-shared-a-w2-799002-openshift-default-69d7698d8d-gfrd4"
Sep 11 01:15:34.855003 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:34.854952    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64a075bf-600d-4fd2-8ec1-a74c1f4cedd5-istio-podinfo\") pod \"e2e-shared-a-w2-799002-openshift-default-69d7698d8d-gfrd4\" (UID: \"64a075bf-600d-4fd2-8ec1-a74c1f4cedd5\") " pod="openshift-ingress/e2e-shared-a-w2-799002-openshift-default-69d7698d8d-gfrd4"
Sep 11 01:15:34.855003 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:34.854992    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64a075bf-600d-4fd2-8ec1-a74c1f4cedd5-istio-envoy\") pod \"e2e-shared-a-w2-799002-openshift-default-69d7698d8d-gfrd4\" (UID: \"64a075bf-600d-4fd2-8ec1-a74c1f4cedd5\") " pod="openshift-ingress/e2e-shared-a-w2-799002-openshift-default-69d7698d8d-gfrd4"
Sep 11 01:15:34.955689 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:34.955653    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dsbjq\" (UniqueName: \"kubernetes.io/projected/64a075bf-600d-4fd2-8ec1-a74c1f4cedd5-kube-api-access-dsbjq\") pod \"e2e-shared-a-w2-799002-openshift-default-69d7698d8d-gfrd4\" (UID: \"64a075bf-600d-4fd2-8ec1-a74c1f4cedd5\") " pod="openshift-ingress/e2e-shared-a-w2-799002-openshift-default-69d7698d8d-gfrd4"
Sep 11 01:15:34.955909 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:34.955741    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64a075bf-600d-4fd2-8ec1-a74c1f4cedd5-istio-data\") pod \"e2e-shared-a-w2-799002-openshift-default-69d7698d8d-gfrd4\" (UID: \"64a075bf-600d-4fd2-8ec1-a74c1f4cedd5\") " pod="openshift-ingress/e2e-shared-a-w2-799002-openshift-default-69d7698d8d-gfrd4"
Sep 11 01:15:34.955909 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:34.955803    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64a075bf-600d-4fd2-8ec1-a74c1f4cedd5-credential-socket\") pod \"e2e-shared-a-w2-799002-openshift-default-69d7698d8d-gfrd4\" (UID: \"64a075bf-600d-4fd2-8ec1-a74c1f4cedd5\") " pod="openshift-ingress/e2e-shared-a-w2-799002-openshift-default-69d7698d8d-gfrd4"
Sep 11 01:15:34.955909 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:34.955830    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64a075bf-600d-4fd2-8ec1-a74c1f4cedd5-istiod-ca-cert\") pod \"e2e-shared-a-w2-799002-openshift-default-69d7698d8d-gfrd4\" (UID: \"64a075bf-600d-4fd2-8ec1-a74c1f4cedd5\") " pod="openshift-ingress/e2e-shared-a-w2-799002-openshift-default-69d7698d8d-gfrd4"
Sep 11 01:15:34.955909 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:34.955861    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64a075bf-600d-4fd2-8ec1-a74c1f4cedd5-workload-socket\") pod \"e2e-shared-a-w2-799002-openshift-default-69d7698d8d-gfrd4\" (UID: \"64a075bf-600d-4fd2-8ec1-a74c1f4cedd5\") " pod="openshift-ingress/e2e-shared-a-w2-799002-openshift-default-69d7698d8d-gfrd4"
Sep 11 01:15:34.956180 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:34.955929    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64a075bf-600d-4fd2-8ec1-a74c1f4cedd5-istio-token\") pod \"e2e-shared-a-w2-799002-openshift-default-69d7698d8d-gfrd4\" (UID: \"64a075bf-600d-4fd2-8ec1-a74c1f4cedd5\") " pod="openshift-ingress/e2e-shared-a-w2-799002-openshift-default-69d7698d8d-gfrd4"
Sep 11 01:15:34.956180 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:34.955979    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64a075bf-600d-4fd2-8ec1-a74c1f4cedd5-istio-podinfo\") pod \"e2e-shared-a-w2-799002-openshift-default-69d7698d8d-gfrd4\" (UID: \"64a075bf-600d-4fd2-8ec1-a74c1f4cedd5\") " pod="openshift-ingress/e2e-shared-a-w2-799002-openshift-default-69d7698d8d-gfrd4"
Sep 11 01:15:34.956180 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:34.956010    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64a075bf-600d-4fd2-8ec1-a74c1f4cedd5-istio-envoy\") pod \"e2e-shared-a-w2-799002-openshift-default-69d7698d8d-gfrd4\" (UID: \"64a075bf-600d-4fd2-8ec1-a74c1f4cedd5\") " pod="openshift-ingress/e2e-shared-a-w2-799002-openshift-default-69d7698d8d-gfrd4"
Sep 11 01:15:34.956180 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:34.956077    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64a075bf-600d-4fd2-8ec1-a74c1f4cedd5-workload-certs\") pod \"e2e-shared-a-w2-799002-openshift-default-69d7698d8d-gfrd4\" (UID: \"64a075bf-600d-4fd2-8ec1-a74c1f4cedd5\") " pod="openshift-ingress/e2e-shared-a-w2-799002-openshift-default-69d7698d8d-gfrd4"
Sep 11 01:15:34.956180 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:34.956151    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64a075bf-600d-4fd2-8ec1-a74c1f4cedd5-istio-data\") pod \"e2e-shared-a-w2-799002-openshift-default-69d7698d8d-gfrd4\" (UID: \"64a075bf-600d-4fd2-8ec1-a74c1f4cedd5\") " pod="openshift-ingress/e2e-shared-a-w2-799002-openshift-default-69d7698d8d-gfrd4"
Sep 11 01:15:34.956403 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:34.956210    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64a075bf-600d-4fd2-8ec1-a74c1f4cedd5-credential-socket\") pod \"e2e-shared-a-w2-799002-openshift-default-69d7698d8d-gfrd4\" (UID: \"64a075bf-600d-4fd2-8ec1-a74c1f4cedd5\") " pod="openshift-ingress/e2e-shared-a-w2-799002-openshift-default-69d7698d8d-gfrd4"
Sep 11 01:15:34.956403 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:34.956257    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64a075bf-600d-4fd2-8ec1-a74c1f4cedd5-workload-socket\") pod \"e2e-shared-a-w2-799002-openshift-default-69d7698d8d-gfrd4\" (UID: \"64a075bf-600d-4fd2-8ec1-a74c1f4cedd5\") " pod="openshift-ingress/e2e-shared-a-w2-799002-openshift-default-69d7698d8d-gfrd4"
Sep 11 01:15:34.956403 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:34.956385    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64a075bf-600d-4fd2-8ec1-a74c1f4cedd5-workload-certs\") pod \"e2e-shared-a-w2-799002-openshift-default-69d7698d8d-gfrd4\" (UID: \"64a075bf-600d-4fd2-8ec1-a74c1f4cedd5\") " pod="openshift-ingress/e2e-shared-a-w2-799002-openshift-default-69d7698d8d-gfrd4"
Sep 11 01:15:34.956803 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:34.956781    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64a075bf-600d-4fd2-8ec1-a74c1f4cedd5-istiod-ca-cert\") pod \"e2e-shared-a-w2-799002-openshift-default-69d7698d8d-gfrd4\" (UID: \"64a075bf-600d-4fd2-8ec1-a74c1f4cedd5\") " pod="openshift-ingress/e2e-shared-a-w2-799002-openshift-default-69d7698d8d-gfrd4"
Sep 11 01:15:34.958384 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:34.958362    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64a075bf-600d-4fd2-8ec1-a74c1f4cedd5-istio-envoy\") pod \"e2e-shared-a-w2-799002-openshift-default-69d7698d8d-gfrd4\" (UID: \"64a075bf-600d-4fd2-8ec1-a74c1f4cedd5\") " pod="openshift-ingress/e2e-shared-a-w2-799002-openshift-default-69d7698d8d-gfrd4"
Sep 11 01:15:34.958735 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:34.958715    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64a075bf-600d-4fd2-8ec1-a74c1f4cedd5-istio-podinfo\") pod \"e2e-shared-a-w2-799002-openshift-default-69d7698d8d-gfrd4\" (UID: \"64a075bf-600d-4fd2-8ec1-a74c1f4cedd5\") " pod="openshift-ingress/e2e-shared-a-w2-799002-openshift-default-69d7698d8d-gfrd4"
Sep 11 01:15:34.963336 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:34.963311    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64a075bf-600d-4fd2-8ec1-a74c1f4cedd5-istio-token\") pod \"e2e-shared-a-w2-799002-openshift-default-69d7698d8d-gfrd4\" (UID: \"64a075bf-600d-4fd2-8ec1-a74c1f4cedd5\") " pod="openshift-ingress/e2e-shared-a-w2-799002-openshift-default-69d7698d8d-gfrd4"
Sep 11 01:15:34.963434 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:34.963416    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dsbjq\" (UniqueName: \"kubernetes.io/projected/64a075bf-600d-4fd2-8ec1-a74c1f4cedd5-kube-api-access-dsbjq\") pod \"e2e-shared-a-w2-799002-openshift-default-69d7698d8d-gfrd4\" (UID: \"64a075bf-600d-4fd2-8ec1-a74c1f4cedd5\") " pod="openshift-ingress/e2e-shared-a-w2-799002-openshift-default-69d7698d8d-gfrd4"
Sep 11 01:15:35.110098 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:35.110063    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-799002-openshift-default-69d7698d8d-gfrd4"
Sep 11 01:15:35.385983 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:35.385957    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-799002-openshift-default-69d7698d8d-gfrd4"]
Sep 11 01:15:35.387531 ip-10-0-129-222 kubenswrapper[2569]: W0911 01:15:35.387489    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64a075bf_600d_4fd2_8ec1_a74c1f4cedd5.slice/crio-ed838955582c82d0c044329134a1604789260c05efc2a782f06371792312b390 WatchSource:0}: Error finding container ed838955582c82d0c044329134a1604789260c05efc2a782f06371792312b390: Status 404 returned error can't find the container with id ed838955582c82d0c044329134a1604789260c05efc2a782f06371792312b390
Sep 11 01:15:35.389748 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:35.389717    2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 01:15:35.389818 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:35.389798    2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 01:15:35.389862 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:35.389829    2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 01:15:35.501294 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:35.501261    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-799002-openshift-default-69d7698d8d-gfrd4" event={"ID":"64a075bf-600d-4fd2-8ec1-a74c1f4cedd5","Type":"ContainerStarted","Data":"a12e5d3dec99af4d34aa5ee0ba2517c060ab0e133b424f3a23c27a531486b7fd"}
Sep 11 01:15:35.501458 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:35.501323    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-799002-openshift-default-69d7698d8d-gfrd4" event={"ID":"64a075bf-600d-4fd2-8ec1-a74c1f4cedd5","Type":"ContainerStarted","Data":"ed838955582c82d0c044329134a1604789260c05efc2a782f06371792312b390"}
Sep 11 01:15:35.521274 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:35.521218    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-799002-openshift-default-69d7698d8d-gfrd4" podStartSLOduration=1.521196421 podStartE2EDuration="1.521196421s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 01:15:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 01:15:35.518375669 +0000 UTC m=+1070.169749919" watchObservedRunningTime="2026-09-11 01:15:35.521196421 +0000 UTC m=+1070.172570671"
Sep 11 01:15:36.111102 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:36.111066    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-799002-openshift-default-69d7698d8d-gfrd4"
Sep 11 01:15:36.115971 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:36.115948    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-799002-openshift-default-69d7698d8d-gfrd4"
Sep 11 01:15:36.507670 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:36.507581    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-799002-openshift-default-69d7698d8d-gfrd4"
Sep 11 01:15:36.508527 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:36.508501    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-799002-openshift-default-69d7698d8d-gfrd4"
Sep 11 01:15:39.264337 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:39.264257    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-799002-b8d69fd78-5chn7"]
Sep 11 01:15:39.264834 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:39.264816    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0387913-3dec-4201-8294-9fef9626afab" containerName="istio-proxy"
Sep 11 01:15:39.264922 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:39.264840    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0387913-3dec-4201-8294-9fef9626afab" containerName="istio-proxy"
Sep 11 01:15:39.267760 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:39.267744    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-799002-b8d69fd78-5chn7"
Sep 11 01:15:39.270245 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:39.270223    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-799002\""
Sep 11 01:15:39.278118 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:39.278095    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-799002-b8d69fd78-5chn7"]
Sep 11 01:15:39.300201 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:39.300170    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a465f3ba-d922-48fc-aea7-76be3e6e8c16-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-799002-b8d69fd78-5chn7\" (UID: \"a465f3ba-d922-48fc-aea7-76be3e6e8c16\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-799002-b8d69fd78-5chn7"
Sep 11 01:15:39.300408 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:39.300380    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67zgk\" (UniqueName: \"kubernetes.io/projected/a465f3ba-d922-48fc-aea7-76be3e6e8c16-kube-api-access-67zgk\") pod \"payload-pre-processing-e2e-shared-a-w2-799002-b8d69fd78-5chn7\" (UID: \"a465f3ba-d922-48fc-aea7-76be3e6e8c16\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-799002-b8d69fd78-5chn7"
Sep 11 01:15:39.308281 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:39.308257    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-799002-d567ff65f-z74b7"]
Sep 11 01:15:39.311893 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:39.311861    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-799002-d567ff65f-z74b7"
Sep 11 01:15:39.314240 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:39.314222    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-799002-dockercfg-pp5c8\""
Sep 11 01:15:39.321976 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:39.321954    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-799002-d567ff65f-z74b7"]
Sep 11 01:15:39.401858 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:39.401299    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmj4t\" (UniqueName: \"kubernetes.io/projected/05bdec86-c9eb-44c0-9c76-7e44625a96c7-kube-api-access-tmj4t\") pod \"payload-processing-e2e-shared-a-w2-799002-d567ff65f-z74b7\" (UID: \"05bdec86-c9eb-44c0-9c76-7e44625a96c7\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-799002-d567ff65f-z74b7"
Sep 11 01:15:39.401858 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:39.401418    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05bdec86-c9eb-44c0-9c76-7e44625a96c7-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-799002-d567ff65f-z74b7\" (UID: \"05bdec86-c9eb-44c0-9c76-7e44625a96c7\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-799002-d567ff65f-z74b7"
Sep 11 01:15:39.401858 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:39.401476    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67zgk\" (UniqueName: \"kubernetes.io/projected/a465f3ba-d922-48fc-aea7-76be3e6e8c16-kube-api-access-67zgk\") pod \"payload-pre-processing-e2e-shared-a-w2-799002-b8d69fd78-5chn7\" (UID: \"a465f3ba-d922-48fc-aea7-76be3e6e8c16\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-799002-b8d69fd78-5chn7"
Sep 11 01:15:39.401858 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:39.401566    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a465f3ba-d922-48fc-aea7-76be3e6e8c16-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-799002-b8d69fd78-5chn7\" (UID: \"a465f3ba-d922-48fc-aea7-76be3e6e8c16\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-799002-b8d69fd78-5chn7"
Sep 11 01:15:39.402810 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:39.402786    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a465f3ba-d922-48fc-aea7-76be3e6e8c16-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-799002-b8d69fd78-5chn7\" (UID: \"a465f3ba-d922-48fc-aea7-76be3e6e8c16\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-799002-b8d69fd78-5chn7"
Sep 11 01:15:39.410268 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:39.410233    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-67zgk\" (UniqueName: \"kubernetes.io/projected/a465f3ba-d922-48fc-aea7-76be3e6e8c16-kube-api-access-67zgk\") pod \"payload-pre-processing-e2e-shared-a-w2-799002-b8d69fd78-5chn7\" (UID: \"a465f3ba-d922-48fc-aea7-76be3e6e8c16\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-799002-b8d69fd78-5chn7"
Sep 11 01:15:39.503067 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:39.503036    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmj4t\" (UniqueName: \"kubernetes.io/projected/05bdec86-c9eb-44c0-9c76-7e44625a96c7-kube-api-access-tmj4t\") pod \"payload-processing-e2e-shared-a-w2-799002-d567ff65f-z74b7\" (UID: \"05bdec86-c9eb-44c0-9c76-7e44625a96c7\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-799002-d567ff65f-z74b7"
Sep 11 01:15:39.503232 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:39.503085    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05bdec86-c9eb-44c0-9c76-7e44625a96c7-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-799002-d567ff65f-z74b7\" (UID: \"05bdec86-c9eb-44c0-9c76-7e44625a96c7\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-799002-d567ff65f-z74b7"
Sep 11 01:15:39.503749 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:39.503728    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05bdec86-c9eb-44c0-9c76-7e44625a96c7-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-799002-d567ff65f-z74b7\" (UID: \"05bdec86-c9eb-44c0-9c76-7e44625a96c7\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-799002-d567ff65f-z74b7"
Sep 11 01:15:39.510592 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:39.510562    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmj4t\" (UniqueName: \"kubernetes.io/projected/05bdec86-c9eb-44c0-9c76-7e44625a96c7-kube-api-access-tmj4t\") pod \"payload-processing-e2e-shared-a-w2-799002-d567ff65f-z74b7\" (UID: \"05bdec86-c9eb-44c0-9c76-7e44625a96c7\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-799002-d567ff65f-z74b7"
Sep 11 01:15:39.580160 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:39.580138    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-799002-b8d69fd78-5chn7"
Sep 11 01:15:39.622610 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:39.622576    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-799002-d567ff65f-z74b7"
Sep 11 01:15:39.859286 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:39.859259    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-799002-b8d69fd78-5chn7"]
Sep 11 01:15:39.861145 ip-10-0-129-222 kubenswrapper[2569]: W0911 01:15:39.861116    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda465f3ba_d922_48fc_aea7_76be3e6e8c16.slice/crio-2dcbb1d5ef59981a12c410db8b7c37f9a46c61cf1dc7dcac2248c19e1f4a809b WatchSource:0}: Error finding container 2dcbb1d5ef59981a12c410db8b7c37f9a46c61cf1dc7dcac2248c19e1f4a809b: Status 404 returned error can't find the container with id 2dcbb1d5ef59981a12c410db8b7c37f9a46c61cf1dc7dcac2248c19e1f4a809b
Sep 11 01:15:39.913252 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:39.913228    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-799002-d567ff65f-z74b7"]
Sep 11 01:15:39.916595 ip-10-0-129-222 kubenswrapper[2569]: W0911 01:15:39.916566    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05bdec86_c9eb_44c0_9c76_7e44625a96c7.slice/crio-063369b6ae684f1cc2d85801d40875a1391241dfadbf662fcbeddd123661a434 WatchSource:0}: Error finding container 063369b6ae684f1cc2d85801d40875a1391241dfadbf662fcbeddd123661a434: Status 404 returned error can't find the container with id 063369b6ae684f1cc2d85801d40875a1391241dfadbf662fcbeddd123661a434
Sep 11 01:15:40.532538 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:40.532501    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-799002-d567ff65f-z74b7" event={"ID":"05bdec86-c9eb-44c0-9c76-7e44625a96c7","Type":"ContainerStarted","Data":"a8f41554e6925ced1178920113a5b106830b363870f2a5153d1f6b19174acaa1"}
Sep 11 01:15:40.533008 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:40.532565    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-799002-d567ff65f-z74b7"
Sep 11 01:15:40.533008 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:40.532591    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-799002-d567ff65f-z74b7" event={"ID":"05bdec86-c9eb-44c0-9c76-7e44625a96c7","Type":"ContainerStarted","Data":"063369b6ae684f1cc2d85801d40875a1391241dfadbf662fcbeddd123661a434"}
Sep 11 01:15:40.534131 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:40.534103    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-799002-b8d69fd78-5chn7" event={"ID":"a465f3ba-d922-48fc-aea7-76be3e6e8c16","Type":"ContainerStarted","Data":"dd108f389896446d217922a4bb39ff98bd647e7caf07ec753b03429994414797"}
Sep 11 01:15:40.534253 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:40.534148    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-799002-b8d69fd78-5chn7"
Sep 11 01:15:40.534253 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:40.534164    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-799002-b8d69fd78-5chn7" event={"ID":"a465f3ba-d922-48fc-aea7-76be3e6e8c16","Type":"ContainerStarted","Data":"2dcbb1d5ef59981a12c410db8b7c37f9a46c61cf1dc7dcac2248c19e1f4a809b"}
Sep 11 01:15:40.549689 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:40.549637    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-799002-d567ff65f-z74b7" podStartSLOduration=1.5496174310000002 podStartE2EDuration="1.549617431s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 01:15:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 01:15:40.548662654 +0000 UTC m=+1075.200036927" watchObservedRunningTime="2026-09-11 01:15:40.549617431 +0000 UTC m=+1075.200991683"
Sep 11 01:15:40.566994 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:40.566947    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-799002-b8d69fd78-5chn7" podStartSLOduration=1.5669333509999999 podStartE2EDuration="1.566933351s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 01:15:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 01:15:40.56483827 +0000 UTC m=+1075.216212521" watchObservedRunningTime="2026-09-11 01:15:40.566933351 +0000 UTC m=+1075.218307600"
Sep 11 01:15:48.005528 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:48.005491    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kmx5x"]
Sep 11 01:15:51.545117 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:51.545078    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-799002-b8d69fd78-5chn7"
Sep 11 01:15:51.546508 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:51.546485    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-799002-d567ff65f-z74b7"
Sep 11 01:15:59.651092 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:59.651060    2569 generic.go:356] "Generic (PLEG): container finished" podID="64a075bf-600d-4fd2-8ec1-a74c1f4cedd5" containerID="a12e5d3dec99af4d34aa5ee0ba2517c060ab0e133b424f3a23c27a531486b7fd" exitCode=0
Sep 11 01:15:59.651596 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:59.651119    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-799002-openshift-default-69d7698d8d-gfrd4" event={"ID":"64a075bf-600d-4fd2-8ec1-a74c1f4cedd5","Type":"ContainerDied","Data":"a12e5d3dec99af4d34aa5ee0ba2517c060ab0e133b424f3a23c27a531486b7fd"}
Sep 11 01:15:59.651664 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:59.651636    2569 scope.go:117] "RemoveContainer" containerID="a12e5d3dec99af4d34aa5ee0ba2517c060ab0e133b424f3a23c27a531486b7fd"
Sep 11 01:15:59.652919 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:59.652891    2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 01:15:59.652994 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:59.652961    2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 01:15:59.653032 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:15:59.652991    2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 01:16:00.663050 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:00.663013    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-799002-openshift-default-69d7698d8d-gfrd4" event={"ID":"64a075bf-600d-4fd2-8ec1-a74c1f4cedd5","Type":"ContainerStarted","Data":"199dc80f63c9e8f7fee9b62d64eaec4f1104c63c77a2df2d6277226135a76a30"}
Sep 11 01:16:01.110613 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:01.110575    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-799002-openshift-default-69d7698d8d-gfrd4"
Sep 11 01:16:01.110789 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:01.110734    2569 patch_prober.go:28] interesting pod/e2e-shared-a-w2-799002-openshift-default-69d7698d8d-gfrd4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body=
Sep 11 01:16:01.110789 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:01.110776    2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-799002-openshift-default-69d7698d8d-gfrd4" podUID="64a075bf-600d-4fd2-8ec1-a74c1f4cedd5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused"
Sep 11 01:16:02.110853 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:02.110816    2569 patch_prober.go:28] interesting pod/e2e-shared-a-w2-799002-openshift-default-69d7698d8d-gfrd4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body=
Sep 11 01:16:02.111284 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:02.110897    2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-799002-openshift-default-69d7698d8d-gfrd4" podUID="64a075bf-600d-4fd2-8ec1-a74c1f4cedd5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused"
Sep 11 01:16:02.968321 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:02.968279    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-71ba6d-openshift-default-765f5dc878-82nr2"]
Sep 11 01:16:02.972917 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:02.972895    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-71ba6d-openshift-default-765f5dc878-82nr2"
Sep 11 01:16:02.975094 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:02.975069    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-71ba6d-openshift-default-dockercfg-phwfp\""
Sep 11 01:16:02.985453 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:02.985431    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-71ba6d-openshift-default-765f5dc878-82nr2"]
Sep 11 01:16:03.059922 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:03.053852    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9896000-8b68-4e4a-afd6-b3dd99e275b6-istiod-ca-cert\") pod \"e2e-shared-b-w2-71ba6d-openshift-default-765f5dc878-82nr2\" (UID: \"a9896000-8b68-4e4a-afd6-b3dd99e275b6\") " pod="openshift-ingress/e2e-shared-b-w2-71ba6d-openshift-default-765f5dc878-82nr2"
Sep 11 01:16:03.059922 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:03.053928    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9896000-8b68-4e4a-afd6-b3dd99e275b6-istio-podinfo\") pod \"e2e-shared-b-w2-71ba6d-openshift-default-765f5dc878-82nr2\" (UID: \"a9896000-8b68-4e4a-afd6-b3dd99e275b6\") " pod="openshift-ingress/e2e-shared-b-w2-71ba6d-openshift-default-765f5dc878-82nr2"
Sep 11 01:16:03.059922 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:03.053962    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9896000-8b68-4e4a-afd6-b3dd99e275b6-istio-token\") pod \"e2e-shared-b-w2-71ba6d-openshift-default-765f5dc878-82nr2\" (UID: \"a9896000-8b68-4e4a-afd6-b3dd99e275b6\") " pod="openshift-ingress/e2e-shared-b-w2-71ba6d-openshift-default-765f5dc878-82nr2"
Sep 11 01:16:03.059922 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:03.054010    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9896000-8b68-4e4a-afd6-b3dd99e275b6-istio-data\") pod \"e2e-shared-b-w2-71ba6d-openshift-default-765f5dc878-82nr2\" (UID: \"a9896000-8b68-4e4a-afd6-b3dd99e275b6\") " pod="openshift-ingress/e2e-shared-b-w2-71ba6d-openshift-default-765f5dc878-82nr2"
Sep 11 01:16:03.059922 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:03.054038    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9896000-8b68-4e4a-afd6-b3dd99e275b6-workload-certs\") pod \"e2e-shared-b-w2-71ba6d-openshift-default-765f5dc878-82nr2\" (UID: \"a9896000-8b68-4e4a-afd6-b3dd99e275b6\") " pod="openshift-ingress/e2e-shared-b-w2-71ba6d-openshift-default-765f5dc878-82nr2"
Sep 11 01:16:03.059922 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:03.054111    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vsc7\" (UniqueName: \"kubernetes.io/projected/a9896000-8b68-4e4a-afd6-b3dd99e275b6-kube-api-access-4vsc7\") pod \"e2e-shared-b-w2-71ba6d-openshift-default-765f5dc878-82nr2\" (UID: \"a9896000-8b68-4e4a-afd6-b3dd99e275b6\") " pod="openshift-ingress/e2e-shared-b-w2-71ba6d-openshift-default-765f5dc878-82nr2"
Sep 11 01:16:03.059922 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:03.054178    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9896000-8b68-4e4a-afd6-b3dd99e275b6-istio-envoy\") pod \"e2e-shared-b-w2-71ba6d-openshift-default-765f5dc878-82nr2\" (UID: \"a9896000-8b68-4e4a-afd6-b3dd99e275b6\") " pod="openshift-ingress/e2e-shared-b-w2-71ba6d-openshift-default-765f5dc878-82nr2"
Sep 11 01:16:03.059922 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:03.054226    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9896000-8b68-4e4a-afd6-b3dd99e275b6-workload-socket\") pod \"e2e-shared-b-w2-71ba6d-openshift-default-765f5dc878-82nr2\" (UID: \"a9896000-8b68-4e4a-afd6-b3dd99e275b6\") " pod="openshift-ingress/e2e-shared-b-w2-71ba6d-openshift-default-765f5dc878-82nr2"
Sep 11 01:16:03.059922 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:03.054251    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9896000-8b68-4e4a-afd6-b3dd99e275b6-credential-socket\") pod \"e2e-shared-b-w2-71ba6d-openshift-default-765f5dc878-82nr2\" (UID: \"a9896000-8b68-4e4a-afd6-b3dd99e275b6\") " pod="openshift-ingress/e2e-shared-b-w2-71ba6d-openshift-default-765f5dc878-82nr2"
Sep 11 01:16:03.110863 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:03.110832    2569 patch_prober.go:28] interesting pod/e2e-shared-a-w2-799002-openshift-default-69d7698d8d-gfrd4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body=
Sep 11 01:16:03.111050 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:03.110917    2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-799002-openshift-default-69d7698d8d-gfrd4" podUID="64a075bf-600d-4fd2-8ec1-a74c1f4cedd5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused"
Sep 11 01:16:03.157159 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:03.154997    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9896000-8b68-4e4a-afd6-b3dd99e275b6-istiod-ca-cert\") pod \"e2e-shared-b-w2-71ba6d-openshift-default-765f5dc878-82nr2\" (UID: \"a9896000-8b68-4e4a-afd6-b3dd99e275b6\") " pod="openshift-ingress/e2e-shared-b-w2-71ba6d-openshift-default-765f5dc878-82nr2"
Sep 11 01:16:03.157159 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:03.155052    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9896000-8b68-4e4a-afd6-b3dd99e275b6-istio-podinfo\") pod \"e2e-shared-b-w2-71ba6d-openshift-default-765f5dc878-82nr2\" (UID: \"a9896000-8b68-4e4a-afd6-b3dd99e275b6\") " pod="openshift-ingress/e2e-shared-b-w2-71ba6d-openshift-default-765f5dc878-82nr2"
Sep 11 01:16:03.157159 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:03.155083    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9896000-8b68-4e4a-afd6-b3dd99e275b6-istio-token\") pod \"e2e-shared-b-w2-71ba6d-openshift-default-765f5dc878-82nr2\" (UID: \"a9896000-8b68-4e4a-afd6-b3dd99e275b6\") " pod="openshift-ingress/e2e-shared-b-w2-71ba6d-openshift-default-765f5dc878-82nr2"
Sep 11 01:16:03.157159 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:03.155133    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9896000-8b68-4e4a-afd6-b3dd99e275b6-istio-data\") pod \"e2e-shared-b-w2-71ba6d-openshift-default-765f5dc878-82nr2\" (UID: \"a9896000-8b68-4e4a-afd6-b3dd99e275b6\") " pod="openshift-ingress/e2e-shared-b-w2-71ba6d-openshift-default-765f5dc878-82nr2"
Sep 11 01:16:03.157159 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:03.155161    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9896000-8b68-4e4a-afd6-b3dd99e275b6-workload-certs\") pod \"e2e-shared-b-w2-71ba6d-openshift-default-765f5dc878-82nr2\" (UID: \"a9896000-8b68-4e4a-afd6-b3dd99e275b6\") " pod="openshift-ingress/e2e-shared-b-w2-71ba6d-openshift-default-765f5dc878-82nr2"
Sep 11 01:16:03.157159 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:03.155213    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vsc7\" (UniqueName: \"kubernetes.io/projected/a9896000-8b68-4e4a-afd6-b3dd99e275b6-kube-api-access-4vsc7\") pod \"e2e-shared-b-w2-71ba6d-openshift-default-765f5dc878-82nr2\" (UID: \"a9896000-8b68-4e4a-afd6-b3dd99e275b6\") " pod="openshift-ingress/e2e-shared-b-w2-71ba6d-openshift-default-765f5dc878-82nr2"
Sep 11 01:16:03.157159 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:03.155281    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9896000-8b68-4e4a-afd6-b3dd99e275b6-istio-envoy\") pod \"e2e-shared-b-w2-71ba6d-openshift-default-765f5dc878-82nr2\" (UID: \"a9896000-8b68-4e4a-afd6-b3dd99e275b6\") " pod="openshift-ingress/e2e-shared-b-w2-71ba6d-openshift-default-765f5dc878-82nr2"
Sep 11 01:16:03.157159 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:03.155328    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9896000-8b68-4e4a-afd6-b3dd99e275b6-workload-socket\") pod \"e2e-shared-b-w2-71ba6d-openshift-default-765f5dc878-82nr2\" (UID: \"a9896000-8b68-4e4a-afd6-b3dd99e275b6\") " pod="openshift-ingress/e2e-shared-b-w2-71ba6d-openshift-default-765f5dc878-82nr2"
Sep 11 01:16:03.157159 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:03.155355    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9896000-8b68-4e4a-afd6-b3dd99e275b6-credential-socket\") pod \"e2e-shared-b-w2-71ba6d-openshift-default-765f5dc878-82nr2\" (UID: \"a9896000-8b68-4e4a-afd6-b3dd99e275b6\") " pod="openshift-ingress/e2e-shared-b-w2-71ba6d-openshift-default-765f5dc878-82nr2"
Sep 11 01:16:03.157159 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:03.155819    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9896000-8b68-4e4a-afd6-b3dd99e275b6-credential-socket\") pod \"e2e-shared-b-w2-71ba6d-openshift-default-765f5dc878-82nr2\" (UID: \"a9896000-8b68-4e4a-afd6-b3dd99e275b6\") " pod="openshift-ingress/e2e-shared-b-w2-71ba6d-openshift-default-765f5dc878-82nr2"
Sep 11 01:16:03.157159 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:03.156730    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9896000-8b68-4e4a-afd6-b3dd99e275b6-istiod-ca-cert\") pod \"e2e-shared-b-w2-71ba6d-openshift-default-765f5dc878-82nr2\" (UID: \"a9896000-8b68-4e4a-afd6-b3dd99e275b6\") " pod="openshift-ingress/e2e-shared-b-w2-71ba6d-openshift-default-765f5dc878-82nr2"
Sep 11 01:16:03.158829 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:03.158515    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9896000-8b68-4e4a-afd6-b3dd99e275b6-workload-socket\") pod \"e2e-shared-b-w2-71ba6d-openshift-default-765f5dc878-82nr2\" (UID: \"a9896000-8b68-4e4a-afd6-b3dd99e275b6\") " pod="openshift-ingress/e2e-shared-b-w2-71ba6d-openshift-default-765f5dc878-82nr2"
Sep 11 01:16:03.158829 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:03.158779    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9896000-8b68-4e4a-afd6-b3dd99e275b6-workload-certs\") pod \"e2e-shared-b-w2-71ba6d-openshift-default-765f5dc878-82nr2\" (UID: \"a9896000-8b68-4e4a-afd6-b3dd99e275b6\") " pod="openshift-ingress/e2e-shared-b-w2-71ba6d-openshift-default-765f5dc878-82nr2"
Sep 11 01:16:03.159894 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:03.159343    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9896000-8b68-4e4a-afd6-b3dd99e275b6-istio-data\") pod \"e2e-shared-b-w2-71ba6d-openshift-default-765f5dc878-82nr2\" (UID: \"a9896000-8b68-4e4a-afd6-b3dd99e275b6\") " pod="openshift-ingress/e2e-shared-b-w2-71ba6d-openshift-default-765f5dc878-82nr2"
Sep 11 01:16:03.159894 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:03.159831    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9896000-8b68-4e4a-afd6-b3dd99e275b6-istio-podinfo\") pod \"e2e-shared-b-w2-71ba6d-openshift-default-765f5dc878-82nr2\" (UID: \"a9896000-8b68-4e4a-afd6-b3dd99e275b6\") " pod="openshift-ingress/e2e-shared-b-w2-71ba6d-openshift-default-765f5dc878-82nr2"
Sep 11 01:16:03.162113 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:03.162072    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9896000-8b68-4e4a-afd6-b3dd99e275b6-istio-envoy\") pod \"e2e-shared-b-w2-71ba6d-openshift-default-765f5dc878-82nr2\" (UID: \"a9896000-8b68-4e4a-afd6-b3dd99e275b6\") " pod="openshift-ingress/e2e-shared-b-w2-71ba6d-openshift-default-765f5dc878-82nr2"
Sep 11 01:16:03.166379 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:03.166354    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9896000-8b68-4e4a-afd6-b3dd99e275b6-istio-token\") pod \"e2e-shared-b-w2-71ba6d-openshift-default-765f5dc878-82nr2\" (UID: \"a9896000-8b68-4e4a-afd6-b3dd99e275b6\") " pod="openshift-ingress/e2e-shared-b-w2-71ba6d-openshift-default-765f5dc878-82nr2"
Sep 11 01:16:03.166770 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:03.166747    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vsc7\" (UniqueName: \"kubernetes.io/projected/a9896000-8b68-4e4a-afd6-b3dd99e275b6-kube-api-access-4vsc7\") pod \"e2e-shared-b-w2-71ba6d-openshift-default-765f5dc878-82nr2\" (UID: \"a9896000-8b68-4e4a-afd6-b3dd99e275b6\") " pod="openshift-ingress/e2e-shared-b-w2-71ba6d-openshift-default-765f5dc878-82nr2"
Sep 11 01:16:03.285592 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:03.285492    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-71ba6d-openshift-default-765f5dc878-82nr2"
Sep 11 01:16:03.582231 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:03.582200    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-71ba6d-openshift-default-765f5dc878-82nr2"]
Sep 11 01:16:03.583846 ip-10-0-129-222 kubenswrapper[2569]: W0911 01:16:03.583815    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9896000_8b68_4e4a_afd6_b3dd99e275b6.slice/crio-0f1f4c29bb4053e330adae29340e4e5ccea9374ea04975a3158b4e51073b72bd WatchSource:0}: Error finding container 0f1f4c29bb4053e330adae29340e4e5ccea9374ea04975a3158b4e51073b72bd: Status 404 returned error can't find the container with id 0f1f4c29bb4053e330adae29340e4e5ccea9374ea04975a3158b4e51073b72bd
Sep 11 01:16:03.586244 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:03.586210    2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 01:16:03.586359 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:03.586274    2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 01:16:03.586359 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:03.586314    2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 01:16:03.681932 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:03.681904    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-71ba6d-openshift-default-765f5dc878-82nr2" event={"ID":"a9896000-8b68-4e4a-afd6-b3dd99e275b6","Type":"ContainerStarted","Data":"738a9435af486af08772608e7259fe29e4cb1fa841172dd3a81aef04897191cc"}
Sep 11 01:16:03.682085 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:03.681955    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-71ba6d-openshift-default-765f5dc878-82nr2" event={"ID":"a9896000-8b68-4e4a-afd6-b3dd99e275b6","Type":"ContainerStarted","Data":"0f1f4c29bb4053e330adae29340e4e5ccea9374ea04975a3158b4e51073b72bd"}
Sep 11 01:16:03.702477 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:03.702414    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-71ba6d-openshift-default-765f5dc878-82nr2" podStartSLOduration=1.70239582 podStartE2EDuration="1.70239582s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 01:16:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 01:16:03.701484053 +0000 UTC m=+1098.352858319" watchObservedRunningTime="2026-09-11 01:16:03.70239582 +0000 UTC m=+1098.353770069"
Sep 11 01:16:04.110643 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:04.110615    2569 patch_prober.go:28] interesting pod/e2e-shared-a-w2-799002-openshift-default-69d7698d8d-gfrd4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body=
Sep 11 01:16:04.110813 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:04.110665    2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-799002-openshift-default-69d7698d8d-gfrd4" podUID="64a075bf-600d-4fd2-8ec1-a74c1f4cedd5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused"
Sep 11 01:16:04.286501 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:04.286468    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-71ba6d-openshift-default-765f5dc878-82nr2"
Sep 11 01:16:04.291893 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:04.291855    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-71ba6d-openshift-default-765f5dc878-82nr2"
Sep 11 01:16:04.687391 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:04.687360    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-71ba6d-openshift-default-765f5dc878-82nr2"
Sep 11 01:16:04.688503 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:04.688474    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-71ba6d-openshift-default-765f5dc878-82nr2"
Sep 11 01:16:05.110683 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:05.110653    2569 patch_prober.go:28] interesting pod/e2e-shared-a-w2-799002-openshift-default-69d7698d8d-gfrd4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body=
Sep 11 01:16:05.110850 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:05.110703    2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-799002-openshift-default-69d7698d8d-gfrd4" podUID="64a075bf-600d-4fd2-8ec1-a74c1f4cedd5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused"
Sep 11 01:16:06.114226 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:06.114195    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-799002-openshift-default-69d7698d8d-gfrd4"
Sep 11 01:16:06.114671 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:06.114559    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-799002-openshift-default-69d7698d8d-gfrd4"
Sep 11 01:16:06.115207 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:06.115191    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-799002-openshift-default-69d7698d8d-gfrd4"
Sep 11 01:16:07.675051 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:07.675018    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-8d451c-openshift-default-7b5868fcdd-g9rzj"]
Sep 11 01:16:07.675468 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:07.675234    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-explicit-8d451c-openshift-default-7b5868fcdd-g9rzj" podUID="f6c2f5c6-7f99-4519-acf1-7af6c423bc0f" containerName="istio-proxy" containerID="cri-o://7b5de0a04a22a15504b11f4f57e4dd7aca0dab092588ff88591adc7a51571373" gracePeriod=30
Sep 11 01:16:12.925976 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:12.925953    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-8d451c-openshift-default-7b5868fcdd-g9rzj"
Sep 11 01:16:12.966423 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:12.966349    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6c2f5c6-7f99-4519-acf1-7af6c423bc0f-workload-socket\") pod \"f6c2f5c6-7f99-4519-acf1-7af6c423bc0f\" (UID: \"f6c2f5c6-7f99-4519-acf1-7af6c423bc0f\") "
Sep 11 01:16:12.966423 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:12.966420    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6c2f5c6-7f99-4519-acf1-7af6c423bc0f-istio-token\") pod \"f6c2f5c6-7f99-4519-acf1-7af6c423bc0f\" (UID: \"f6c2f5c6-7f99-4519-acf1-7af6c423bc0f\") "
Sep 11 01:16:12.966598 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:12.966524    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6c2f5c6-7f99-4519-acf1-7af6c423bc0f-workload-certs\") pod \"f6c2f5c6-7f99-4519-acf1-7af6c423bc0f\" (UID: \"f6c2f5c6-7f99-4519-acf1-7af6c423bc0f\") "
Sep 11 01:16:12.966598 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:12.966586    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6c2f5c6-7f99-4519-acf1-7af6c423bc0f-istio-podinfo\") pod \"f6c2f5c6-7f99-4519-acf1-7af6c423bc0f\" (UID: \"f6c2f5c6-7f99-4519-acf1-7af6c423bc0f\") "
Sep 11 01:16:12.966719 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:12.966603    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6c2f5c6-7f99-4519-acf1-7af6c423bc0f-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "f6c2f5c6-7f99-4519-acf1-7af6c423bc0f" (UID: "f6c2f5c6-7f99-4519-acf1-7af6c423bc0f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 01:16:12.966719 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:12.966628    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6c2f5c6-7f99-4519-acf1-7af6c423bc0f-istio-envoy\") pod \"f6c2f5c6-7f99-4519-acf1-7af6c423bc0f\" (UID: \"f6c2f5c6-7f99-4519-acf1-7af6c423bc0f\") "
Sep 11 01:16:12.966719 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:12.966659    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-24wfx\" (UniqueName: \"kubernetes.io/projected/f6c2f5c6-7f99-4519-acf1-7af6c423bc0f-kube-api-access-24wfx\") pod \"f6c2f5c6-7f99-4519-acf1-7af6c423bc0f\" (UID: \"f6c2f5c6-7f99-4519-acf1-7af6c423bc0f\") "
Sep 11 01:16:12.966719 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:12.966691    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6c2f5c6-7f99-4519-acf1-7af6c423bc0f-istiod-ca-cert\") pod \"f6c2f5c6-7f99-4519-acf1-7af6c423bc0f\" (UID: \"f6c2f5c6-7f99-4519-acf1-7af6c423bc0f\") "
Sep 11 01:16:12.966956 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:12.966734    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6c2f5c6-7f99-4519-acf1-7af6c423bc0f-credential-socket\") pod \"f6c2f5c6-7f99-4519-acf1-7af6c423bc0f\" (UID: \"f6c2f5c6-7f99-4519-acf1-7af6c423bc0f\") "
Sep 11 01:16:12.966956 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:12.966781    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6c2f5c6-7f99-4519-acf1-7af6c423bc0f-istio-data\") pod \"f6c2f5c6-7f99-4519-acf1-7af6c423bc0f\" (UID: \"f6c2f5c6-7f99-4519-acf1-7af6c423bc0f\") "
Sep 11 01:16:12.966956 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:12.966810    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6c2f5c6-7f99-4519-acf1-7af6c423bc0f-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "f6c2f5c6-7f99-4519-acf1-7af6c423bc0f" (UID: "f6c2f5c6-7f99-4519-acf1-7af6c423bc0f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 01:16:12.967508 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:12.967189    2569 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6c2f5c6-7f99-4519-acf1-7af6c423bc0f-workload-socket\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:16:12.967508 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:12.967200    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6c2f5c6-7f99-4519-acf1-7af6c423bc0f-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "f6c2f5c6-7f99-4519-acf1-7af6c423bc0f" (UID: "f6c2f5c6-7f99-4519-acf1-7af6c423bc0f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 01:16:12.967508 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:12.967215    2569 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6c2f5c6-7f99-4519-acf1-7af6c423bc0f-workload-certs\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:16:12.967508 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:12.967229    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6c2f5c6-7f99-4519-acf1-7af6c423bc0f-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "f6c2f5c6-7f99-4519-acf1-7af6c423bc0f" (UID: "f6c2f5c6-7f99-4519-acf1-7af6c423bc0f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 01:16:12.967508 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:12.967479    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6c2f5c6-7f99-4519-acf1-7af6c423bc0f-istio-data" (OuterVolumeSpecName: "istio-data") pod "f6c2f5c6-7f99-4519-acf1-7af6c423bc0f" (UID: "f6c2f5c6-7f99-4519-acf1-7af6c423bc0f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 01:16:12.968864 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:12.968837    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6c2f5c6-7f99-4519-acf1-7af6c423bc0f-kube-api-access-24wfx" (OuterVolumeSpecName: "kube-api-access-24wfx") pod "f6c2f5c6-7f99-4519-acf1-7af6c423bc0f" (UID: "f6c2f5c6-7f99-4519-acf1-7af6c423bc0f"). InnerVolumeSpecName "kube-api-access-24wfx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 01:16:12.968864 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:12.968847    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f6c2f5c6-7f99-4519-acf1-7af6c423bc0f-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "f6c2f5c6-7f99-4519-acf1-7af6c423bc0f" (UID: "f6c2f5c6-7f99-4519-acf1-7af6c423bc0f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 01:16:12.969092 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:12.969076    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6c2f5c6-7f99-4519-acf1-7af6c423bc0f-istio-token" (OuterVolumeSpecName: "istio-token") pod "f6c2f5c6-7f99-4519-acf1-7af6c423bc0f" (UID: "f6c2f5c6-7f99-4519-acf1-7af6c423bc0f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 01:16:12.969158 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:12.969139    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6c2f5c6-7f99-4519-acf1-7af6c423bc0f-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "f6c2f5c6-7f99-4519-acf1-7af6c423bc0f" (UID: "f6c2f5c6-7f99-4519-acf1-7af6c423bc0f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 01:16:13.068045 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:13.068020    2569 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6c2f5c6-7f99-4519-acf1-7af6c423bc0f-istio-token\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:16:13.068045 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:13.068043    2569 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6c2f5c6-7f99-4519-acf1-7af6c423bc0f-istio-podinfo\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:16:13.068190 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:13.068052    2569 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6c2f5c6-7f99-4519-acf1-7af6c423bc0f-istio-envoy\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:16:13.068190 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:13.068061    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-24wfx\" (UniqueName: \"kubernetes.io/projected/f6c2f5c6-7f99-4519-acf1-7af6c423bc0f-kube-api-access-24wfx\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:16:13.068190 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:13.068070    2569 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6c2f5c6-7f99-4519-acf1-7af6c423bc0f-istiod-ca-cert\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:16:13.068190 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:13.068078    2569 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6c2f5c6-7f99-4519-acf1-7af6c423bc0f-credential-socket\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:16:13.068190 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:13.068087    2569 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6c2f5c6-7f99-4519-acf1-7af6c423bc0f-istio-data\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:16:13.749692 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:13.749658    2569 generic.go:356] "Generic (PLEG): container finished" podID="f6c2f5c6-7f99-4519-acf1-7af6c423bc0f" containerID="7b5de0a04a22a15504b11f4f57e4dd7aca0dab092588ff88591adc7a51571373" exitCode=0
Sep 11 01:16:13.749920 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:13.749712    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-8d451c-openshift-default-7b5868fcdd-g9rzj"
Sep 11 01:16:13.749920 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:13.749739    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-8d451c-openshift-default-7b5868fcdd-g9rzj" event={"ID":"f6c2f5c6-7f99-4519-acf1-7af6c423bc0f","Type":"ContainerDied","Data":"7b5de0a04a22a15504b11f4f57e4dd7aca0dab092588ff88591adc7a51571373"}
Sep 11 01:16:13.749920 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:13.749798    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-8d451c-openshift-default-7b5868fcdd-g9rzj" event={"ID":"f6c2f5c6-7f99-4519-acf1-7af6c423bc0f","Type":"ContainerDied","Data":"8ddc4e2ebfb667d4d21fb6df54fc539ac819aef5142a14942b9c1a636b0f8758"}
Sep 11 01:16:13.749920 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:13.749824    2569 scope.go:117] "RemoveContainer" containerID="7b5de0a04a22a15504b11f4f57e4dd7aca0dab092588ff88591adc7a51571373"
Sep 11 01:16:13.759493 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:13.759477    2569 scope.go:117] "RemoveContainer" containerID="17c42f6abbe8776859d57d14880e986f7c0b9a8b71ee157f8440c0109b535f1a"
Sep 11 01:16:13.767560 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:13.767544    2569 scope.go:117] "RemoveContainer" containerID="7b5de0a04a22a15504b11f4f57e4dd7aca0dab092588ff88591adc7a51571373"
Sep 11 01:16:13.767837 ip-10-0-129-222 kubenswrapper[2569]: E0911 01:16:13.767816    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b5de0a04a22a15504b11f4f57e4dd7aca0dab092588ff88591adc7a51571373\": container with ID starting with 7b5de0a04a22a15504b11f4f57e4dd7aca0dab092588ff88591adc7a51571373 not found: ID does not exist" containerID="7b5de0a04a22a15504b11f4f57e4dd7aca0dab092588ff88591adc7a51571373"
Sep 11 01:16:13.767908 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:13.767845    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b5de0a04a22a15504b11f4f57e4dd7aca0dab092588ff88591adc7a51571373"} err="failed to get container status \"7b5de0a04a22a15504b11f4f57e4dd7aca0dab092588ff88591adc7a51571373\": rpc error: code = NotFound desc = could not find container \"7b5de0a04a22a15504b11f4f57e4dd7aca0dab092588ff88591adc7a51571373\": container with ID starting with 7b5de0a04a22a15504b11f4f57e4dd7aca0dab092588ff88591adc7a51571373 not found: ID does not exist"
Sep 11 01:16:13.767908 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:13.767864    2569 scope.go:117] "RemoveContainer" containerID="17c42f6abbe8776859d57d14880e986f7c0b9a8b71ee157f8440c0109b535f1a"
Sep 11 01:16:13.768169 ip-10-0-129-222 kubenswrapper[2569]: E0911 01:16:13.768143    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17c42f6abbe8776859d57d14880e986f7c0b9a8b71ee157f8440c0109b535f1a\": container with ID starting with 17c42f6abbe8776859d57d14880e986f7c0b9a8b71ee157f8440c0109b535f1a not found: ID does not exist" containerID="17c42f6abbe8776859d57d14880e986f7c0b9a8b71ee157f8440c0109b535f1a"
Sep 11 01:16:13.768263 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:13.768176    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17c42f6abbe8776859d57d14880e986f7c0b9a8b71ee157f8440c0109b535f1a"} err="failed to get container status \"17c42f6abbe8776859d57d14880e986f7c0b9a8b71ee157f8440c0109b535f1a\": rpc error: code = NotFound desc = could not find container \"17c42f6abbe8776859d57d14880e986f7c0b9a8b71ee157f8440c0109b535f1a\": container with ID starting with 17c42f6abbe8776859d57d14880e986f7c0b9a8b71ee157f8440c0109b535f1a not found: ID does not exist"
Sep 11 01:16:13.773513 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:13.773490    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-8d451c-openshift-default-7b5868fcdd-g9rzj"]
Sep 11 01:16:13.777083 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:13.777059    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-explicit-8d451c-openshift-default-7b5868fcdd-g9rzj"]
Sep 11 01:16:13.871143 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:13.871109    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f6c2f5c6-7f99-4519-acf1-7af6c423bc0f" path="/var/lib/kubelet/pods/f6c2f5c6-7f99-4519-acf1-7af6c423bc0f/volumes"
Sep 11 01:16:20.708277 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:20.708240    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kmx5x"]
Sep 11 01:16:28.276704 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:28.276656    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-cef92e-njt6d"]
Sep 11 01:16:28.277320 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:28.277304    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f6c2f5c6-7f99-4519-acf1-7af6c423bc0f" containerName="istio-proxy"
Sep 11 01:16:28.277365 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:28.277323    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="f6c2f5c6-7f99-4519-acf1-7af6c423bc0f" containerName="istio-proxy"
Sep 11 01:16:28.277365 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:28.277352    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f6c2f5c6-7f99-4519-acf1-7af6c423bc0f" containerName="istio-proxy"
Sep 11 01:16:28.277365 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:28.277359    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="f6c2f5c6-7f99-4519-acf1-7af6c423bc0f" containerName="istio-proxy"
Sep 11 01:16:28.277482 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:28.277471    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="f6c2f5c6-7f99-4519-acf1-7af6c423bc0f" containerName="istio-proxy"
Sep 11 01:16:28.277519 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:28.277486    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="f6c2f5c6-7f99-4519-acf1-7af6c423bc0f" containerName="istio-proxy"
Sep 11 01:16:28.282556 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:28.282538    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-cef92e-njt6d"
Sep 11 01:16:28.288679 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:28.287927    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 01:16:28.288679 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:28.288176    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pppkt\""
Sep 11 01:16:28.291155 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:28.291088    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-cef92e-njt6d"]
Sep 11 01:16:28.421954 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:28.421915    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/480eaa89-2544-481a-b34a-06e2ed96dbdb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-cef92e-njt6d\" (UID: \"480eaa89-2544-481a-b34a-06e2ed96dbdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-cef92e-njt6d"
Sep 11 01:16:28.523389 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:28.523349    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/480eaa89-2544-481a-b34a-06e2ed96dbdb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-cef92e-njt6d\" (UID: \"480eaa89-2544-481a-b34a-06e2ed96dbdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-cef92e-njt6d"
Sep 11 01:16:28.524278 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:28.524253    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/480eaa89-2544-481a-b34a-06e2ed96dbdb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-cef92e-njt6d\" (UID: \"480eaa89-2544-481a-b34a-06e2ed96dbdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-cef92e-njt6d"
Sep 11 01:16:28.598060 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:28.598024    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-cef92e-njt6d"
Sep 11 01:16:28.882826 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:28.882799    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-cef92e-njt6d"]
Sep 11 01:16:28.885203 ip-10-0-129-222 kubenswrapper[2569]: W0911 01:16:28.885175    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod480eaa89_2544_481a_b34a_06e2ed96dbdb.slice/crio-5e9f5cc33d88eaafc45736ee3775a111d4af0099866795855bfc4490398a7e67 WatchSource:0}: Error finding container 5e9f5cc33d88eaafc45736ee3775a111d4af0099866795855bfc4490398a7e67: Status 404 returned error can't find the container with id 5e9f5cc33d88eaafc45736ee3775a111d4af0099866795855bfc4490398a7e67
Sep 11 01:16:29.850648 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:29.850614    2569 generic.go:356] "Generic (PLEG): container finished" podID="480eaa89-2544-481a-b34a-06e2ed96dbdb" containerID="43267619a78852ec4d2b1be2887a4f0b31c44c2012ca8ea24d83485b8e99a194" exitCode=0
Sep 11 01:16:29.851046 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:29.850695    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-cef92e-njt6d" event={"ID":"480eaa89-2544-481a-b34a-06e2ed96dbdb","Type":"ContainerDied","Data":"43267619a78852ec4d2b1be2887a4f0b31c44c2012ca8ea24d83485b8e99a194"}
Sep 11 01:16:29.851046 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:29.850753    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-cef92e-njt6d" event={"ID":"480eaa89-2544-481a-b34a-06e2ed96dbdb","Type":"ContainerStarted","Data":"5e9f5cc33d88eaafc45736ee3775a111d4af0099866795855bfc4490398a7e67"}
Sep 11 01:16:30.857641 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:30.857612    2569 generic.go:356] "Generic (PLEG): container finished" podID="a9896000-8b68-4e4a-afd6-b3dd99e275b6" containerID="738a9435af486af08772608e7259fe29e4cb1fa841172dd3a81aef04897191cc" exitCode=0
Sep 11 01:16:30.858132 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:30.857688    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-71ba6d-openshift-default-765f5dc878-82nr2" event={"ID":"a9896000-8b68-4e4a-afd6-b3dd99e275b6","Type":"ContainerDied","Data":"738a9435af486af08772608e7259fe29e4cb1fa841172dd3a81aef04897191cc"}
Sep 11 01:16:30.858661 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:30.858322    2569 scope.go:117] "RemoveContainer" containerID="738a9435af486af08772608e7259fe29e4cb1fa841172dd3a81aef04897191cc"
Sep 11 01:16:30.859557 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:30.859528    2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 01:16:30.859657 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:30.859613    2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 01:16:30.859722 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:30.859659    2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 01:16:31.025751 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:31.025727    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-cef92e-njt6d"
Sep 11 01:16:31.153372 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:31.153279    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/480eaa89-2544-481a-b34a-06e2ed96dbdb-maas-api-service-ca\") pod \"480eaa89-2544-481a-b34a-06e2ed96dbdb\" (UID: \"480eaa89-2544-481a-b34a-06e2ed96dbdb\") "
Sep 11 01:16:31.153611 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:31.153589    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/480eaa89-2544-481a-b34a-06e2ed96dbdb-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "480eaa89-2544-481a-b34a-06e2ed96dbdb" (UID: "480eaa89-2544-481a-b34a-06e2ed96dbdb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 01:16:31.153718 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:31.153699    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/480eaa89-2544-481a-b34a-06e2ed96dbdb-maas-api-service-ca\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:16:31.351694 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:31.351664    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7688-fccb11"]
Sep 11 01:16:31.352128 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:31.352114    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="480eaa89-2544-481a-b34a-06e2ed96dbdb" containerName="revoke-keys"
Sep 11 01:16:31.352128 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:31.352129    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="480eaa89-2544-481a-b34a-06e2ed96dbdb" containerName="revoke-keys"
Sep 11 01:16:31.352223 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:31.352205    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="480eaa89-2544-481a-b34a-06e2ed96dbdb" containerName="revoke-keys"
Sep 11 01:16:31.355187 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:31.355172    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7688-fccb11"
Sep 11 01:16:31.357252 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:31.357233    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-d5ldp\""
Sep 11 01:16:31.360928 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:31.360906    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7688-fccb11"]
Sep 11 01:16:31.456390 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:31.456320    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvpbl\" (UniqueName: \"kubernetes.io/projected/2103810c-bdb2-49a4-a8e8-96f6a244063b-kube-api-access-fvpbl\") pod \"test-curl-7688-fccb11\" (UID: \"2103810c-bdb2-49a4-a8e8-96f6a244063b\") " pod="opendatahub/test-curl-7688-fccb11"
Sep 11 01:16:31.557075 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:31.557046    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvpbl\" (UniqueName: \"kubernetes.io/projected/2103810c-bdb2-49a4-a8e8-96f6a244063b-kube-api-access-fvpbl\") pod \"test-curl-7688-fccb11\" (UID: \"2103810c-bdb2-49a4-a8e8-96f6a244063b\") " pod="opendatahub/test-curl-7688-fccb11"
Sep 11 01:16:31.564653 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:31.564626    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvpbl\" (UniqueName: \"kubernetes.io/projected/2103810c-bdb2-49a4-a8e8-96f6a244063b-kube-api-access-fvpbl\") pod \"test-curl-7688-fccb11\" (UID: \"2103810c-bdb2-49a4-a8e8-96f6a244063b\") " pod="opendatahub/test-curl-7688-fccb11"
Sep 11 01:16:31.666259 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:31.666231    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7688-fccb11"
Sep 11 01:16:31.868285 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:31.868256    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-cef92e-njt6d"
Sep 11 01:16:31.879857 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:31.873858    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-cef92e-njt6d" event={"ID":"480eaa89-2544-481a-b34a-06e2ed96dbdb","Type":"ContainerDied","Data":"5e9f5cc33d88eaafc45736ee3775a111d4af0099866795855bfc4490398a7e67"}
Sep 11 01:16:31.879857 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:31.873963    2569 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5e9f5cc33d88eaafc45736ee3775a111d4af0099866795855bfc4490398a7e67"
Sep 11 01:16:31.879857 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:31.873980    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-71ba6d-openshift-default-765f5dc878-82nr2" event={"ID":"a9896000-8b68-4e4a-afd6-b3dd99e275b6","Type":"ContainerStarted","Data":"82a222ba16c60d65879eccc766539947eaf362e3b5e513b38720cfb54d71a53a"}
Sep 11 01:16:31.943687 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:31.940818    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7688-fccb11"]
Sep 11 01:16:32.285794 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:32.285686    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-71ba6d-openshift-default-765f5dc878-82nr2"
Sep 11 01:16:32.286811 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:32.286782    2569 patch_prober.go:28] interesting pod/e2e-shared-b-w2-71ba6d-openshift-default-765f5dc878-82nr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body=
Sep 11 01:16:32.286956 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:32.286847    2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-71ba6d-openshift-default-765f5dc878-82nr2" podUID="a9896000-8b68-4e4a-afd6-b3dd99e275b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused"
Sep 11 01:16:32.878707 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:32.878668    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7688-fccb11" event={"ID":"2103810c-bdb2-49a4-a8e8-96f6a244063b","Type":"ContainerStarted","Data":"60c6424a0813c61fa40a9a5bb08fa421319a5a1d7ab8e98f51dc877f02b87803"}
Sep 11 01:16:32.879134 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:32.878720    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7688-fccb11" event={"ID":"2103810c-bdb2-49a4-a8e8-96f6a244063b","Type":"ContainerStarted","Data":"892ec0ff50a23e14e8aaa507faf9bee6eb4f342dcb9b993ce38ad4bf0ba5a683"}
Sep 11 01:16:32.901433 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:32.901378    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7688-fccb11" podStartSLOduration=1.9013603479999999 podStartE2EDuration="1.901360348s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 01:16:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 01:16:32.900962037 +0000 UTC m=+1127.552336291" watchObservedRunningTime="2026-09-11 01:16:32.901360348 +0000 UTC m=+1127.552734601"
Sep 11 01:16:33.286554 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:33.286474    2569 patch_prober.go:28] interesting pod/e2e-shared-b-w2-71ba6d-openshift-default-765f5dc878-82nr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body=
Sep 11 01:16:33.286554 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:33.286526    2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-71ba6d-openshift-default-765f5dc878-82nr2" podUID="a9896000-8b68-4e4a-afd6-b3dd99e275b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused"
Sep 11 01:16:33.521641 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:33.521607    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7688-fccb11"]
Sep 11 01:16:33.525639 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:33.525614    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7688-fccb11"]
Sep 11 01:16:33.662081 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:33.662021    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7688-f2a5c5"]
Sep 11 01:16:33.663252 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:33.663225    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2103810c-bdb2-49a4-a8e8-96f6a244063b" containerName="test-curl-7688-fccb11"
Sep 11 01:16:33.663394 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:33.663383    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="2103810c-bdb2-49a4-a8e8-96f6a244063b" containerName="test-curl-7688-fccb11"
Sep 11 01:16:33.664015 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:33.663994    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="2103810c-bdb2-49a4-a8e8-96f6a244063b" containerName="test-curl-7688-fccb11"
Sep 11 01:16:33.667939 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:33.667916    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7688-f2a5c5"]
Sep 11 01:16:33.668036 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:33.667993    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7688-f2a5c5"
Sep 11 01:16:33.783542 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:33.783501    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmcbm\" (UniqueName: \"kubernetes.io/projected/0982daaf-1808-442e-9c39-5348f8f41b25-kube-api-access-gmcbm\") pod \"test-curl-7688-f2a5c5\" (UID: \"0982daaf-1808-442e-9c39-5348f8f41b25\") " pod="opendatahub/test-curl-7688-f2a5c5"
Sep 11 01:16:33.884251 ip-10-0-129-222 kubenswrapper[2569]: E0911 01:16:33.884220    2569 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7688-fccb11" container="test-curl-7688-fccb11"
Sep 11 01:16:33.884593 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:33.884410    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmcbm\" (UniqueName: \"kubernetes.io/projected/0982daaf-1808-442e-9c39-5348f8f41b25-kube-api-access-gmcbm\") pod \"test-curl-7688-f2a5c5\" (UID: \"0982daaf-1808-442e-9c39-5348f8f41b25\") " pod="opendatahub/test-curl-7688-f2a5c5"
Sep 11 01:16:33.897860 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:33.897832    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmcbm\" (UniqueName: \"kubernetes.io/projected/0982daaf-1808-442e-9c39-5348f8f41b25-kube-api-access-gmcbm\") pod \"test-curl-7688-f2a5c5\" (UID: \"0982daaf-1808-442e-9c39-5348f8f41b25\") " pod="opendatahub/test-curl-7688-f2a5c5"
Sep 11 01:16:33.976980 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:33.976898    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7688-f2a5c5"
Sep 11 01:16:34.249553 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:34.249464    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7688-f2a5c5"]
Sep 11 01:16:34.258754 ip-10-0-129-222 kubenswrapper[2569]: W0911 01:16:34.258702    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0982daaf_1808_442e_9c39_5348f8f41b25.slice/crio-3a1c2e065d5d5798efd720661c34d118004e294f8a5b0e5f30c3f1efedbd0c3d WatchSource:0}: Error finding container 3a1c2e065d5d5798efd720661c34d118004e294f8a5b0e5f30c3f1efedbd0c3d: Status 404 returned error can't find the container with id 3a1c2e065d5d5798efd720661c34d118004e294f8a5b0e5f30c3f1efedbd0c3d
Sep 11 01:16:34.286001 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:34.285977    2569 patch_prober.go:28] interesting pod/e2e-shared-b-w2-71ba6d-openshift-default-765f5dc878-82nr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body=
Sep 11 01:16:34.286109 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:34.286027    2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-71ba6d-openshift-default-765f5dc878-82nr2" podUID="a9896000-8b68-4e4a-afd6-b3dd99e275b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused"
Sep 11 01:16:34.688058 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:34.688011    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-71ba6d-openshift-default-765f5dc878-82nr2"
Sep 11 01:16:34.891371 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:34.891338    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7688-f2a5c5" event={"ID":"0982daaf-1808-442e-9c39-5348f8f41b25","Type":"ContainerStarted","Data":"3b6815c61b75a021825edf31a047c8ae40627811da1afb80a3b28141a6fedaa9"}
Sep 11 01:16:34.891790 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:34.891378    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7688-f2a5c5" event={"ID":"0982daaf-1808-442e-9c39-5348f8f41b25","Type":"ContainerStarted","Data":"3a1c2e065d5d5798efd720661c34d118004e294f8a5b0e5f30c3f1efedbd0c3d"}
Sep 11 01:16:34.891790 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:34.891441    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7688-fccb11" podUID="2103810c-bdb2-49a4-a8e8-96f6a244063b" containerName="test-curl-7688-fccb11" containerID="cri-o://60c6424a0813c61fa40a9a5bb08fa421319a5a1d7ab8e98f51dc877f02b87803" gracePeriod=2
Sep 11 01:16:34.906418 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:34.906365    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7688-f2a5c5" podStartSLOduration=1.906349168 podStartE2EDuration="1.906349168s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 01:16:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 01:16:34.904896346 +0000 UTC m=+1129.556270589" watchObservedRunningTime="2026-09-11 01:16:34.906349168 +0000 UTC m=+1129.557723421"
Sep 11 01:16:35.285916 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:35.285892    2569 patch_prober.go:28] interesting pod/e2e-shared-b-w2-71ba6d-openshift-default-765f5dc878-82nr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body=
Sep 11 01:16:35.286009 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:35.285942    2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-71ba6d-openshift-default-765f5dc878-82nr2" podUID="a9896000-8b68-4e4a-afd6-b3dd99e275b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused"
Sep 11 01:16:35.576327 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:35.576291    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7688-f2a5c5"]
Sep 11 01:16:35.583606 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:35.583582    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7688-f2a5c5"]
Sep 11 01:16:35.702357 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:35.702322    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7688-776152"]
Sep 11 01:16:35.702796 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:35.702783    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0982daaf-1808-442e-9c39-5348f8f41b25" containerName="test-curl-7688-f2a5c5"
Sep 11 01:16:35.702840 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:35.702798    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="0982daaf-1808-442e-9c39-5348f8f41b25" containerName="test-curl-7688-f2a5c5"
Sep 11 01:16:35.702900 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:35.702893    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="0982daaf-1808-442e-9c39-5348f8f41b25" containerName="test-curl-7688-f2a5c5"
Sep 11 01:16:35.706157 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:35.706142    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7688-776152"
Sep 11 01:16:35.710036 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:35.710013    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7688-776152"]
Sep 11 01:16:35.805433 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:35.805400    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lph2p\" (UniqueName: \"kubernetes.io/projected/0053c0e1-63bf-4f4e-a4ad-556d91f2b010-kube-api-access-lph2p\") pod \"test-curl-7688-776152\" (UID: \"0053c0e1-63bf-4f4e-a4ad-556d91f2b010\") " pod="opendatahub/test-curl-7688-776152"
Sep 11 01:16:35.896563 ip-10-0-129-222 kubenswrapper[2569]: E0911 01:16:35.896478    2569 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7688-f2a5c5" container="test-curl-7688-f2a5c5"
Sep 11 01:16:35.906733 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:35.906705    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lph2p\" (UniqueName: \"kubernetes.io/projected/0053c0e1-63bf-4f4e-a4ad-556d91f2b010-kube-api-access-lph2p\") pod \"test-curl-7688-776152\" (UID: \"0053c0e1-63bf-4f4e-a4ad-556d91f2b010\") " pod="opendatahub/test-curl-7688-776152"
Sep 11 01:16:35.915208 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:35.915178    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lph2p\" (UniqueName: \"kubernetes.io/projected/0053c0e1-63bf-4f4e-a4ad-556d91f2b010-kube-api-access-lph2p\") pod \"test-curl-7688-776152\" (UID: \"0053c0e1-63bf-4f4e-a4ad-556d91f2b010\") " pod="opendatahub/test-curl-7688-776152"
Sep 11 01:16:36.016754 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:36.016718    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7688-776152"
Sep 11 01:16:36.286560 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:36.286476    2569 patch_prober.go:28] interesting pod/e2e-shared-b-w2-71ba6d-openshift-default-765f5dc878-82nr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body=
Sep 11 01:16:36.286560 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:36.286541    2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-71ba6d-openshift-default-765f5dc878-82nr2" podUID="a9896000-8b68-4e4a-afd6-b3dd99e275b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused"
Sep 11 01:16:36.311712 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:36.311687    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7688-776152"]
Sep 11 01:16:36.314022 ip-10-0-129-222 kubenswrapper[2569]: W0911 01:16:36.313991    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0053c0e1_63bf_4f4e_a4ad_556d91f2b010.slice/crio-de5f7cf513beca75186b259baa00e47299cf74852fdb51521e1ea886af253007 WatchSource:0}: Error finding container de5f7cf513beca75186b259baa00e47299cf74852fdb51521e1ea886af253007: Status 404 returned error can't find the container with id de5f7cf513beca75186b259baa00e47299cf74852fdb51521e1ea886af253007
Sep 11 01:16:36.904267 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:36.904239    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7688-776152" event={"ID":"0053c0e1-63bf-4f4e-a4ad-556d91f2b010","Type":"ContainerStarted","Data":"2b1b2fed38355e9c35033f5e24ca165a6e49e9a8e8c41ffaae8da0be1c0e4663"}
Sep 11 01:16:36.904595 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:36.904288    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7688-776152" event={"ID":"0053c0e1-63bf-4f4e-a4ad-556d91f2b010","Type":"ContainerStarted","Data":"de5f7cf513beca75186b259baa00e47299cf74852fdb51521e1ea886af253007"}
Sep 11 01:16:36.904595 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:36.904357    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7688-f2a5c5" podUID="0982daaf-1808-442e-9c39-5348f8f41b25" containerName="test-curl-7688-f2a5c5" containerID="cri-o://3b6815c61b75a021825edf31a047c8ae40627811da1afb80a3b28141a6fedaa9" gracePeriod=2
Sep 11 01:16:36.918457 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:36.918408    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7688-776152" podStartSLOduration=1.918394588 podStartE2EDuration="1.918394588s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 01:16:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 01:16:36.917531771 +0000 UTC m=+1131.568906023" watchObservedRunningTime="2026-09-11 01:16:36.918394588 +0000 UTC m=+1131.569768837"
Sep 11 01:16:37.111363 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:37.111341    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7688-fccb11"
Sep 11 01:16:37.219196 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:37.219122    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fvpbl\" (UniqueName: \"kubernetes.io/projected/2103810c-bdb2-49a4-a8e8-96f6a244063b-kube-api-access-fvpbl\") pod \"2103810c-bdb2-49a4-a8e8-96f6a244063b\" (UID: \"2103810c-bdb2-49a4-a8e8-96f6a244063b\") "
Sep 11 01:16:37.221028 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:37.221001    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2103810c-bdb2-49a4-a8e8-96f6a244063b-kube-api-access-fvpbl" (OuterVolumeSpecName: "kube-api-access-fvpbl") pod "2103810c-bdb2-49a4-a8e8-96f6a244063b" (UID: "2103810c-bdb2-49a4-a8e8-96f6a244063b"). InnerVolumeSpecName "kube-api-access-fvpbl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 01:16:37.286226 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:37.286182    2569 patch_prober.go:28] interesting pod/e2e-shared-b-w2-71ba6d-openshift-default-765f5dc878-82nr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body=
Sep 11 01:16:37.286407 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:37.286265    2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-71ba6d-openshift-default-765f5dc878-82nr2" podUID="a9896000-8b68-4e4a-afd6-b3dd99e275b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused"
Sep 11 01:16:37.321116 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:37.321083    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fvpbl\" (UniqueName: \"kubernetes.io/projected/2103810c-bdb2-49a4-a8e8-96f6a244063b-kube-api-access-fvpbl\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:16:37.529067 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:37.528979    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7688-776152"]
Sep 11 01:16:37.531591 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:37.531565    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7688-776152"]
Sep 11 01:16:37.641831 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:37.641800    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7688-d88d3b"]
Sep 11 01:16:37.642273 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:37.642260    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0053c0e1-63bf-4f4e-a4ad-556d91f2b010" containerName="test-curl-7688-776152"
Sep 11 01:16:37.642333 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:37.642275    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="0053c0e1-63bf-4f4e-a4ad-556d91f2b010" containerName="test-curl-7688-776152"
Sep 11 01:16:37.642370 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:37.642349    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="0053c0e1-63bf-4f4e-a4ad-556d91f2b010" containerName="test-curl-7688-776152"
Sep 11 01:16:37.645482 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:37.645466    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7688-d88d3b"
Sep 11 01:16:37.651345 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:37.651321    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7688-d88d3b"]
Sep 11 01:16:37.725176 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:37.725145    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b58bm\" (UniqueName: \"kubernetes.io/projected/9c4660de-95c8-4829-a6de-3287066c9b42-kube-api-access-b58bm\") pod \"test-curl-7688-d88d3b\" (UID: \"9c4660de-95c8-4829-a6de-3287066c9b42\") " pod="opendatahub/test-curl-7688-d88d3b"
Sep 11 01:16:37.826121 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:37.826096    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b58bm\" (UniqueName: \"kubernetes.io/projected/9c4660de-95c8-4829-a6de-3287066c9b42-kube-api-access-b58bm\") pod \"test-curl-7688-d88d3b\" (UID: \"9c4660de-95c8-4829-a6de-3287066c9b42\") " pod="opendatahub/test-curl-7688-d88d3b"
Sep 11 01:16:37.833954 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:37.833926    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b58bm\" (UniqueName: \"kubernetes.io/projected/9c4660de-95c8-4829-a6de-3287066c9b42-kube-api-access-b58bm\") pod \"test-curl-7688-d88d3b\" (UID: \"9c4660de-95c8-4829-a6de-3287066c9b42\") " pod="opendatahub/test-curl-7688-d88d3b"
Sep 11 01:16:37.871815 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:37.871790    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2103810c-bdb2-49a4-a8e8-96f6a244063b" path="/var/lib/kubelet/pods/2103810c-bdb2-49a4-a8e8-96f6a244063b/volumes"
Sep 11 01:16:37.912110 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:37.912086    2569 generic.go:356] "Generic (PLEG): container finished" podID="2103810c-bdb2-49a4-a8e8-96f6a244063b" containerID="60c6424a0813c61fa40a9a5bb08fa421319a5a1d7ab8e98f51dc877f02b87803" exitCode=137
Sep 11 01:16:37.912467 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:37.912121    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7688-fccb11"
Sep 11 01:16:37.912467 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:37.912134    2569 scope.go:117] "RemoveContainer" containerID="60c6424a0813c61fa40a9a5bb08fa421319a5a1d7ab8e98f51dc877f02b87803"
Sep 11 01:16:37.912467 ip-10-0-129-222 kubenswrapper[2569]: E0911 01:16:37.912278    2569 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7688-776152" container="test-curl-7688-776152"
Sep 11 01:16:37.921084 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:37.921067    2569 scope.go:117] "RemoveContainer" containerID="60c6424a0813c61fa40a9a5bb08fa421319a5a1d7ab8e98f51dc877f02b87803"
Sep 11 01:16:37.921356 ip-10-0-129-222 kubenswrapper[2569]: E0911 01:16:37.921337    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60c6424a0813c61fa40a9a5bb08fa421319a5a1d7ab8e98f51dc877f02b87803\": container with ID starting with 60c6424a0813c61fa40a9a5bb08fa421319a5a1d7ab8e98f51dc877f02b87803 not found: ID does not exist" containerID="60c6424a0813c61fa40a9a5bb08fa421319a5a1d7ab8e98f51dc877f02b87803"
Sep 11 01:16:37.921422 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:37.921363    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60c6424a0813c61fa40a9a5bb08fa421319a5a1d7ab8e98f51dc877f02b87803"} err="failed to get container status \"60c6424a0813c61fa40a9a5bb08fa421319a5a1d7ab8e98f51dc877f02b87803\": rpc error: code = NotFound desc = could not find container \"60c6424a0813c61fa40a9a5bb08fa421319a5a1d7ab8e98f51dc877f02b87803\": container with ID starting with 60c6424a0813c61fa40a9a5bb08fa421319a5a1d7ab8e98f51dc877f02b87803 not found: ID does not exist"
Sep 11 01:16:37.957650 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:37.957632    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7688-d88d3b"
Sep 11 01:16:38.228391 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:38.228365    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7688-d88d3b"]
Sep 11 01:16:38.229977 ip-10-0-129-222 kubenswrapper[2569]: W0911 01:16:38.229948    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9c4660de_95c8_4829_a6de_3287066c9b42.slice/crio-66ff98e4724e1b9ca36cbf6c818321958fd1215565883b76d95d552ec2a677b8 WatchSource:0}: Error finding container 66ff98e4724e1b9ca36cbf6c818321958fd1215565883b76d95d552ec2a677b8: Status 404 returned error can't find the container with id 66ff98e4724e1b9ca36cbf6c818321958fd1215565883b76d95d552ec2a677b8
Sep 11 01:16:38.290764 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:38.290739    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-71ba6d-openshift-default-765f5dc878-82nr2"
Sep 11 01:16:38.291920 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:38.291903    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-71ba6d-openshift-default-765f5dc878-82nr2"
Sep 11 01:16:38.919736 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:38.919705    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7688-d88d3b" event={"ID":"9c4660de-95c8-4829-a6de-3287066c9b42","Type":"ContainerStarted","Data":"d9451874ae569868f33c606216056c94433e284bc5865052047a63432e1a1a46"}
Sep 11 01:16:38.920260 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:38.919749    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7688-d88d3b" event={"ID":"9c4660de-95c8-4829-a6de-3287066c9b42","Type":"ContainerStarted","Data":"66ff98e4724e1b9ca36cbf6c818321958fd1215565883b76d95d552ec2a677b8"}
Sep 11 01:16:38.920836 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:38.920813    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7688-776152" podUID="0053c0e1-63bf-4f4e-a4ad-556d91f2b010" containerName="test-curl-7688-776152" containerID="cri-o://2b1b2fed38355e9c35033f5e24ca165a6e49e9a8e8c41ffaae8da0be1c0e4663" gracePeriod=2
Sep 11 01:16:38.935006 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:38.934968    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7688-d88d3b" podStartSLOduration=1.934954482 podStartE2EDuration="1.934954482s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 01:16:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 01:16:38.934176957 +0000 UTC m=+1133.585551207" watchObservedRunningTime="2026-09-11 01:16:38.934954482 +0000 UTC m=+1133.586328731"
Sep 11 01:16:39.006575 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:39.006537    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-cef92e-29818155-hdjcr"]
Sep 11 01:16:39.008808 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:39.008777    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-cef92e-29818155-hdjcr"]
Sep 11 01:16:39.112188 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:39.112161    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7688-f2a5c5"
Sep 11 01:16:39.138711 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:39.138683    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gmcbm\" (UniqueName: \"kubernetes.io/projected/0982daaf-1808-442e-9c39-5348f8f41b25-kube-api-access-gmcbm\") pod \"0982daaf-1808-442e-9c39-5348f8f41b25\" (UID: \"0982daaf-1808-442e-9c39-5348f8f41b25\") "
Sep 11 01:16:39.140653 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:39.140618    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0982daaf-1808-442e-9c39-5348f8f41b25-kube-api-access-gmcbm" (OuterVolumeSpecName: "kube-api-access-gmcbm") pod "0982daaf-1808-442e-9c39-5348f8f41b25" (UID: "0982daaf-1808-442e-9c39-5348f8f41b25"). InnerVolumeSpecName "kube-api-access-gmcbm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 01:16:39.194638 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:39.194539    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-cef92e-njt6d"]
Sep 11 01:16:39.197644 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:39.197621    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-cef92e-njt6d"]
Sep 11 01:16:39.239857 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:39.239835    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gmcbm\" (UniqueName: \"kubernetes.io/projected/0982daaf-1808-442e-9c39-5348f8f41b25-kube-api-access-gmcbm\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:16:39.554849 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:39.554816    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7688-d88d3b"]
Sep 11 01:16:39.560390 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:39.560363    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7688-d88d3b"]
Sep 11 01:16:39.871526 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:39.871446    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0982daaf-1808-442e-9c39-5348f8f41b25" path="/var/lib/kubelet/pods/0982daaf-1808-442e-9c39-5348f8f41b25/volumes"
Sep 11 01:16:39.871793 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:39.871780    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="480eaa89-2544-481a-b34a-06e2ed96dbdb" path="/var/lib/kubelet/pods/480eaa89-2544-481a-b34a-06e2ed96dbdb/volumes"
Sep 11 01:16:39.872055 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:39.872043    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="59c4ecbf-6d82-490f-b399-a97c8e1121bb" path="/var/lib/kubelet/pods/59c4ecbf-6d82-490f-b399-a97c8e1121bb/volumes"
Sep 11 01:16:39.928057 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:39.928030    2569 generic.go:356] "Generic (PLEG): container finished" podID="0982daaf-1808-442e-9c39-5348f8f41b25" containerID="3b6815c61b75a021825edf31a047c8ae40627811da1afb80a3b28141a6fedaa9" exitCode=137
Sep 11 01:16:39.928466 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:39.928067    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7688-f2a5c5"
Sep 11 01:16:39.928466 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:39.928125    2569 scope.go:117] "RemoveContainer" containerID="3b6815c61b75a021825edf31a047c8ae40627811da1afb80a3b28141a6fedaa9"
Sep 11 01:16:39.928466 ip-10-0-129-222 kubenswrapper[2569]: E0911 01:16:39.928348    2569 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7688-d88d3b" container="test-curl-7688-d88d3b"
Sep 11 01:16:39.937452 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:39.937433    2569 scope.go:117] "RemoveContainer" containerID="3b6815c61b75a021825edf31a047c8ae40627811da1afb80a3b28141a6fedaa9"
Sep 11 01:16:39.937722 ip-10-0-129-222 kubenswrapper[2569]: E0911 01:16:39.937704    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b6815c61b75a021825edf31a047c8ae40627811da1afb80a3b28141a6fedaa9\": container with ID starting with 3b6815c61b75a021825edf31a047c8ae40627811da1afb80a3b28141a6fedaa9 not found: ID does not exist" containerID="3b6815c61b75a021825edf31a047c8ae40627811da1afb80a3b28141a6fedaa9"
Sep 11 01:16:39.937790 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:39.937731    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b6815c61b75a021825edf31a047c8ae40627811da1afb80a3b28141a6fedaa9"} err="failed to get container status \"3b6815c61b75a021825edf31a047c8ae40627811da1afb80a3b28141a6fedaa9\": rpc error: code = NotFound desc = could not find container \"3b6815c61b75a021825edf31a047c8ae40627811da1afb80a3b28141a6fedaa9\": container with ID starting with 3b6815c61b75a021825edf31a047c8ae40627811da1afb80a3b28141a6fedaa9 not found: ID does not exist"
Sep 11 01:16:40.420393 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:40.420359    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-194eb64c-openshift-default-779f4656cb-lcpcg"]
Sep 11 01:16:40.420934 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:40.420914    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c4660de-95c8-4829-a6de-3287066c9b42" containerName="test-curl-7688-d88d3b"
Sep 11 01:16:40.420934 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:40.420934    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c4660de-95c8-4829-a6de-3287066c9b42" containerName="test-curl-7688-d88d3b"
Sep 11 01:16:40.421081 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:40.421015    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="9c4660de-95c8-4829-a6de-3287066c9b42" containerName="test-curl-7688-d88d3b"
Sep 11 01:16:40.425398 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:40.425378    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-194eb64c-openshift-default-779f4656cb-lcpcg"
Sep 11 01:16:40.428316 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:40.428297    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-194eb64c-openshift-default-dockercfg-xwxxd\""
Sep 11 01:16:40.443035 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:40.443011    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-194eb64c-openshift-default-779f4656cb-lcpcg"]
Sep 11 01:16:40.450713 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:40.450690    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41517eb2-ec7f-488c-86b9-fc70f16bcd3d-istio-podinfo\") pod \"e2e-mt-194eb64c-openshift-default-779f4656cb-lcpcg\" (UID: \"41517eb2-ec7f-488c-86b9-fc70f16bcd3d\") " pod="openshift-ingress/e2e-mt-194eb64c-openshift-default-779f4656cb-lcpcg"
Sep 11 01:16:40.450824 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:40.450723    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5kw7f\" (UniqueName: \"kubernetes.io/projected/41517eb2-ec7f-488c-86b9-fc70f16bcd3d-kube-api-access-5kw7f\") pod \"e2e-mt-194eb64c-openshift-default-779f4656cb-lcpcg\" (UID: \"41517eb2-ec7f-488c-86b9-fc70f16bcd3d\") " pod="openshift-ingress/e2e-mt-194eb64c-openshift-default-779f4656cb-lcpcg"
Sep 11 01:16:40.450824 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:40.450743    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41517eb2-ec7f-488c-86b9-fc70f16bcd3d-istiod-ca-cert\") pod \"e2e-mt-194eb64c-openshift-default-779f4656cb-lcpcg\" (UID: \"41517eb2-ec7f-488c-86b9-fc70f16bcd3d\") " pod="openshift-ingress/e2e-mt-194eb64c-openshift-default-779f4656cb-lcpcg"
Sep 11 01:16:40.450824 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:40.450764    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41517eb2-ec7f-488c-86b9-fc70f16bcd3d-istio-data\") pod \"e2e-mt-194eb64c-openshift-default-779f4656cb-lcpcg\" (UID: \"41517eb2-ec7f-488c-86b9-fc70f16bcd3d\") " pod="openshift-ingress/e2e-mt-194eb64c-openshift-default-779f4656cb-lcpcg"
Sep 11 01:16:40.450824 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:40.450804    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41517eb2-ec7f-488c-86b9-fc70f16bcd3d-credential-socket\") pod \"e2e-mt-194eb64c-openshift-default-779f4656cb-lcpcg\" (UID: \"41517eb2-ec7f-488c-86b9-fc70f16bcd3d\") " pod="openshift-ingress/e2e-mt-194eb64c-openshift-default-779f4656cb-lcpcg"
Sep 11 01:16:40.451252 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:40.450883    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41517eb2-ec7f-488c-86b9-fc70f16bcd3d-istio-envoy\") pod \"e2e-mt-194eb64c-openshift-default-779f4656cb-lcpcg\" (UID: \"41517eb2-ec7f-488c-86b9-fc70f16bcd3d\") " pod="openshift-ingress/e2e-mt-194eb64c-openshift-default-779f4656cb-lcpcg"
Sep 11 01:16:40.451252 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:40.450957    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41517eb2-ec7f-488c-86b9-fc70f16bcd3d-workload-certs\") pod \"e2e-mt-194eb64c-openshift-default-779f4656cb-lcpcg\" (UID: \"41517eb2-ec7f-488c-86b9-fc70f16bcd3d\") " pod="openshift-ingress/e2e-mt-194eb64c-openshift-default-779f4656cb-lcpcg"
Sep 11 01:16:40.451252 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:40.451063    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41517eb2-ec7f-488c-86b9-fc70f16bcd3d-workload-socket\") pod \"e2e-mt-194eb64c-openshift-default-779f4656cb-lcpcg\" (UID: \"41517eb2-ec7f-488c-86b9-fc70f16bcd3d\") " pod="openshift-ingress/e2e-mt-194eb64c-openshift-default-779f4656cb-lcpcg"
Sep 11 01:16:40.451252 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:40.451091    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41517eb2-ec7f-488c-86b9-fc70f16bcd3d-istio-token\") pod \"e2e-mt-194eb64c-openshift-default-779f4656cb-lcpcg\" (UID: \"41517eb2-ec7f-488c-86b9-fc70f16bcd3d\") " pod="openshift-ingress/e2e-mt-194eb64c-openshift-default-779f4656cb-lcpcg"
Sep 11 01:16:40.551792 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:40.551744    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41517eb2-ec7f-488c-86b9-fc70f16bcd3d-istio-podinfo\") pod \"e2e-mt-194eb64c-openshift-default-779f4656cb-lcpcg\" (UID: \"41517eb2-ec7f-488c-86b9-fc70f16bcd3d\") " pod="openshift-ingress/e2e-mt-194eb64c-openshift-default-779f4656cb-lcpcg"
Sep 11 01:16:40.552003 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:40.551801    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kw7f\" (UniqueName: \"kubernetes.io/projected/41517eb2-ec7f-488c-86b9-fc70f16bcd3d-kube-api-access-5kw7f\") pod \"e2e-mt-194eb64c-openshift-default-779f4656cb-lcpcg\" (UID: \"41517eb2-ec7f-488c-86b9-fc70f16bcd3d\") " pod="openshift-ingress/e2e-mt-194eb64c-openshift-default-779f4656cb-lcpcg"
Sep 11 01:16:40.552003 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:40.551832    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41517eb2-ec7f-488c-86b9-fc70f16bcd3d-istiod-ca-cert\") pod \"e2e-mt-194eb64c-openshift-default-779f4656cb-lcpcg\" (UID: \"41517eb2-ec7f-488c-86b9-fc70f16bcd3d\") " pod="openshift-ingress/e2e-mt-194eb64c-openshift-default-779f4656cb-lcpcg"
Sep 11 01:16:40.552003 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:40.551867    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41517eb2-ec7f-488c-86b9-fc70f16bcd3d-istio-data\") pod \"e2e-mt-194eb64c-openshift-default-779f4656cb-lcpcg\" (UID: \"41517eb2-ec7f-488c-86b9-fc70f16bcd3d\") " pod="openshift-ingress/e2e-mt-194eb64c-openshift-default-779f4656cb-lcpcg"
Sep 11 01:16:40.552003 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:40.551921    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41517eb2-ec7f-488c-86b9-fc70f16bcd3d-credential-socket\") pod \"e2e-mt-194eb64c-openshift-default-779f4656cb-lcpcg\" (UID: \"41517eb2-ec7f-488c-86b9-fc70f16bcd3d\") " pod="openshift-ingress/e2e-mt-194eb64c-openshift-default-779f4656cb-lcpcg"
Sep 11 01:16:40.552003 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:40.551947    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41517eb2-ec7f-488c-86b9-fc70f16bcd3d-istio-envoy\") pod \"e2e-mt-194eb64c-openshift-default-779f4656cb-lcpcg\" (UID: \"41517eb2-ec7f-488c-86b9-fc70f16bcd3d\") " pod="openshift-ingress/e2e-mt-194eb64c-openshift-default-779f4656cb-lcpcg"
Sep 11 01:16:40.552003 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:40.551997    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41517eb2-ec7f-488c-86b9-fc70f16bcd3d-workload-certs\") pod \"e2e-mt-194eb64c-openshift-default-779f4656cb-lcpcg\" (UID: \"41517eb2-ec7f-488c-86b9-fc70f16bcd3d\") " pod="openshift-ingress/e2e-mt-194eb64c-openshift-default-779f4656cb-lcpcg"
Sep 11 01:16:40.552375 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:40.552089    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41517eb2-ec7f-488c-86b9-fc70f16bcd3d-workload-socket\") pod \"e2e-mt-194eb64c-openshift-default-779f4656cb-lcpcg\" (UID: \"41517eb2-ec7f-488c-86b9-fc70f16bcd3d\") " pod="openshift-ingress/e2e-mt-194eb64c-openshift-default-779f4656cb-lcpcg"
Sep 11 01:16:40.552375 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:40.552116    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41517eb2-ec7f-488c-86b9-fc70f16bcd3d-istio-token\") pod \"e2e-mt-194eb64c-openshift-default-779f4656cb-lcpcg\" (UID: \"41517eb2-ec7f-488c-86b9-fc70f16bcd3d\") " pod="openshift-ingress/e2e-mt-194eb64c-openshift-default-779f4656cb-lcpcg"
Sep 11 01:16:40.552945 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:40.552839    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41517eb2-ec7f-488c-86b9-fc70f16bcd3d-credential-socket\") pod \"e2e-mt-194eb64c-openshift-default-779f4656cb-lcpcg\" (UID: \"41517eb2-ec7f-488c-86b9-fc70f16bcd3d\") " pod="openshift-ingress/e2e-mt-194eb64c-openshift-default-779f4656cb-lcpcg"
Sep 11 01:16:40.552945 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:40.552856    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41517eb2-ec7f-488c-86b9-fc70f16bcd3d-workload-certs\") pod \"e2e-mt-194eb64c-openshift-default-779f4656cb-lcpcg\" (UID: \"41517eb2-ec7f-488c-86b9-fc70f16bcd3d\") " pod="openshift-ingress/e2e-mt-194eb64c-openshift-default-779f4656cb-lcpcg"
Sep 11 01:16:40.553539 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:40.553505    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41517eb2-ec7f-488c-86b9-fc70f16bcd3d-workload-socket\") pod \"e2e-mt-194eb64c-openshift-default-779f4656cb-lcpcg\" (UID: \"41517eb2-ec7f-488c-86b9-fc70f16bcd3d\") " pod="openshift-ingress/e2e-mt-194eb64c-openshift-default-779f4656cb-lcpcg"
Sep 11 01:16:40.553805 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:40.553777    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41517eb2-ec7f-488c-86b9-fc70f16bcd3d-istiod-ca-cert\") pod \"e2e-mt-194eb64c-openshift-default-779f4656cb-lcpcg\" (UID: \"41517eb2-ec7f-488c-86b9-fc70f16bcd3d\") " pod="openshift-ingress/e2e-mt-194eb64c-openshift-default-779f4656cb-lcpcg"
Sep 11 01:16:40.554391 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:40.554023    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41517eb2-ec7f-488c-86b9-fc70f16bcd3d-istio-data\") pod \"e2e-mt-194eb64c-openshift-default-779f4656cb-lcpcg\" (UID: \"41517eb2-ec7f-488c-86b9-fc70f16bcd3d\") " pod="openshift-ingress/e2e-mt-194eb64c-openshift-default-779f4656cb-lcpcg"
Sep 11 01:16:40.558247 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:40.556166    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41517eb2-ec7f-488c-86b9-fc70f16bcd3d-istio-envoy\") pod \"e2e-mt-194eb64c-openshift-default-779f4656cb-lcpcg\" (UID: \"41517eb2-ec7f-488c-86b9-fc70f16bcd3d\") " pod="openshift-ingress/e2e-mt-194eb64c-openshift-default-779f4656cb-lcpcg"
Sep 11 01:16:40.558247 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:40.556641    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41517eb2-ec7f-488c-86b9-fc70f16bcd3d-istio-podinfo\") pod \"e2e-mt-194eb64c-openshift-default-779f4656cb-lcpcg\" (UID: \"41517eb2-ec7f-488c-86b9-fc70f16bcd3d\") " pod="openshift-ingress/e2e-mt-194eb64c-openshift-default-779f4656cb-lcpcg"
Sep 11 01:16:40.561656 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:40.561632    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41517eb2-ec7f-488c-86b9-fc70f16bcd3d-istio-token\") pod \"e2e-mt-194eb64c-openshift-default-779f4656cb-lcpcg\" (UID: \"41517eb2-ec7f-488c-86b9-fc70f16bcd3d\") " pod="openshift-ingress/e2e-mt-194eb64c-openshift-default-779f4656cb-lcpcg"
Sep 11 01:16:40.562082 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:40.562060    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kw7f\" (UniqueName: \"kubernetes.io/projected/41517eb2-ec7f-488c-86b9-fc70f16bcd3d-kube-api-access-5kw7f\") pod \"e2e-mt-194eb64c-openshift-default-779f4656cb-lcpcg\" (UID: \"41517eb2-ec7f-488c-86b9-fc70f16bcd3d\") " pod="openshift-ingress/e2e-mt-194eb64c-openshift-default-779f4656cb-lcpcg"
Sep 11 01:16:40.736012 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:40.735914    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-194eb64c-openshift-default-779f4656cb-lcpcg"
Sep 11 01:16:40.935002 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:40.934959    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7688-d88d3b" podUID="9c4660de-95c8-4829-a6de-3287066c9b42" containerName="test-curl-7688-d88d3b" containerID="cri-o://d9451874ae569868f33c606216056c94433e284bc5865052047a63432e1a1a46" gracePeriod=2
Sep 11 01:16:41.014389 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:41.014296    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-194eb64c-openshift-default-779f4656cb-lcpcg"]
Sep 11 01:16:41.016528 ip-10-0-129-222 kubenswrapper[2569]: W0911 01:16:41.016502    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41517eb2_ec7f_488c_86b9_fc70f16bcd3d.slice/crio-cb914d91a4fbf94561520490fa962226e737913ab9270d170953942a6bbbe698 WatchSource:0}: Error finding container cb914d91a4fbf94561520490fa962226e737913ab9270d170953942a6bbbe698: Status 404 returned error can't find the container with id cb914d91a4fbf94561520490fa962226e737913ab9270d170953942a6bbbe698
Sep 11 01:16:41.019178 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:41.019139    2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 01:16:41.019242 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:41.019225    2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 01:16:41.019287 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:41.019270    2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 01:16:41.107136 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:41.107108    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7688-776152"
Sep 11 01:16:41.156437 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:41.156405    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lph2p\" (UniqueName: \"kubernetes.io/projected/0053c0e1-63bf-4f4e-a4ad-556d91f2b010-kube-api-access-lph2p\") pod \"0053c0e1-63bf-4f4e-a4ad-556d91f2b010\" (UID: \"0053c0e1-63bf-4f4e-a4ad-556d91f2b010\") "
Sep 11 01:16:41.158426 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:41.158394    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0053c0e1-63bf-4f4e-a4ad-556d91f2b010-kube-api-access-lph2p" (OuterVolumeSpecName: "kube-api-access-lph2p") pod "0053c0e1-63bf-4f4e-a4ad-556d91f2b010" (UID: "0053c0e1-63bf-4f4e-a4ad-556d91f2b010"). InnerVolumeSpecName "kube-api-access-lph2p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 01:16:41.257252 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:41.257166    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lph2p\" (UniqueName: \"kubernetes.io/projected/0053c0e1-63bf-4f4e-a4ad-556d91f2b010-kube-api-access-lph2p\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:16:41.872031 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:41.871947    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0053c0e1-63bf-4f4e-a4ad-556d91f2b010" path="/var/lib/kubelet/pods/0053c0e1-63bf-4f4e-a4ad-556d91f2b010/volumes"
Sep 11 01:16:41.942590 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:41.942554    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-194eb64c-openshift-default-779f4656cb-lcpcg" event={"ID":"41517eb2-ec7f-488c-86b9-fc70f16bcd3d","Type":"ContainerStarted","Data":"03a963847114368a0ce55ad6877cdc12878a8ebaf8b16218389dfd17006b4e3b"}
Sep 11 01:16:41.942989 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:41.942612    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-194eb64c-openshift-default-779f4656cb-lcpcg" event={"ID":"41517eb2-ec7f-488c-86b9-fc70f16bcd3d","Type":"ContainerStarted","Data":"cb914d91a4fbf94561520490fa962226e737913ab9270d170953942a6bbbe698"}
Sep 11 01:16:41.944088 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:41.944062    2569 generic.go:356] "Generic (PLEG): container finished" podID="0053c0e1-63bf-4f4e-a4ad-556d91f2b010" containerID="2b1b2fed38355e9c35033f5e24ca165a6e49e9a8e8c41ffaae8da0be1c0e4663" exitCode=137
Sep 11 01:16:41.944206 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:41.944093    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7688-776152"
Sep 11 01:16:41.944206 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:41.944155    2569 scope.go:117] "RemoveContainer" containerID="2b1b2fed38355e9c35033f5e24ca165a6e49e9a8e8c41ffaae8da0be1c0e4663"
Sep 11 01:16:41.954022 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:41.954003    2569 scope.go:117] "RemoveContainer" containerID="2b1b2fed38355e9c35033f5e24ca165a6e49e9a8e8c41ffaae8da0be1c0e4663"
Sep 11 01:16:41.954285 ip-10-0-129-222 kubenswrapper[2569]: E0911 01:16:41.954267    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b1b2fed38355e9c35033f5e24ca165a6e49e9a8e8c41ffaae8da0be1c0e4663\": container with ID starting with 2b1b2fed38355e9c35033f5e24ca165a6e49e9a8e8c41ffaae8da0be1c0e4663 not found: ID does not exist" containerID="2b1b2fed38355e9c35033f5e24ca165a6e49e9a8e8c41ffaae8da0be1c0e4663"
Sep 11 01:16:41.954370 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:41.954306    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b1b2fed38355e9c35033f5e24ca165a6e49e9a8e8c41ffaae8da0be1c0e4663"} err="failed to get container status \"2b1b2fed38355e9c35033f5e24ca165a6e49e9a8e8c41ffaae8da0be1c0e4663\": rpc error: code = NotFound desc = could not find container \"2b1b2fed38355e9c35033f5e24ca165a6e49e9a8e8c41ffaae8da0be1c0e4663\": container with ID starting with 2b1b2fed38355e9c35033f5e24ca165a6e49e9a8e8c41ffaae8da0be1c0e4663 not found: ID does not exist"
Sep 11 01:16:41.961530 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:41.961345    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-194eb64c-openshift-default-779f4656cb-lcpcg" podStartSLOduration=1.9613276480000001 podStartE2EDuration="1.961327648s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 01:16:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 01:16:41.960836251 +0000 UTC m=+1136.612210501" watchObservedRunningTime="2026-09-11 01:16:41.961327648 +0000 UTC m=+1136.612701900"
Sep 11 01:16:42.737032 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:42.736999    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-194eb64c-openshift-default-779f4656cb-lcpcg"
Sep 11 01:16:42.742188 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:42.742164    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-194eb64c-openshift-default-779f4656cb-lcpcg"
Sep 11 01:16:42.952290 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:42.952262    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-194eb64c-openshift-default-779f4656cb-lcpcg"
Sep 11 01:16:42.953224 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:42.953207    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-194eb64c-openshift-default-779f4656cb-lcpcg"
Sep 11 01:16:43.095764 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:43.095737    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7688-d88d3b"
Sep 11 01:16:43.175071 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:43.175042    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b58bm\" (UniqueName: \"kubernetes.io/projected/9c4660de-95c8-4829-a6de-3287066c9b42-kube-api-access-b58bm\") pod \"9c4660de-95c8-4829-a6de-3287066c9b42\" (UID: \"9c4660de-95c8-4829-a6de-3287066c9b42\") "
Sep 11 01:16:43.177008 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:43.176974    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c4660de-95c8-4829-a6de-3287066c9b42-kube-api-access-b58bm" (OuterVolumeSpecName: "kube-api-access-b58bm") pod "9c4660de-95c8-4829-a6de-3287066c9b42" (UID: "9c4660de-95c8-4829-a6de-3287066c9b42"). InnerVolumeSpecName "kube-api-access-b58bm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 01:16:43.276252 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:43.276182    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b58bm\" (UniqueName: \"kubernetes.io/projected/9c4660de-95c8-4829-a6de-3287066c9b42-kube-api-access-b58bm\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:16:43.871130 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:43.871095    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9c4660de-95c8-4829-a6de-3287066c9b42" path="/var/lib/kubelet/pods/9c4660de-95c8-4829-a6de-3287066c9b42/volumes"
Sep 11 01:16:43.958575 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:43.958541    2569 generic.go:356] "Generic (PLEG): container finished" podID="9c4660de-95c8-4829-a6de-3287066c9b42" containerID="d9451874ae569868f33c606216056c94433e284bc5865052047a63432e1a1a46" exitCode=137
Sep 11 01:16:43.958575 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:43.958580    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7688-d88d3b"
Sep 11 01:16:43.959052 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:43.958631    2569 scope.go:117] "RemoveContainer" containerID="d9451874ae569868f33c606216056c94433e284bc5865052047a63432e1a1a46"
Sep 11 01:16:43.967829 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:43.967810    2569 scope.go:117] "RemoveContainer" containerID="d9451874ae569868f33c606216056c94433e284bc5865052047a63432e1a1a46"
Sep 11 01:16:43.968123 ip-10-0-129-222 kubenswrapper[2569]: E0911 01:16:43.968103    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9451874ae569868f33c606216056c94433e284bc5865052047a63432e1a1a46\": container with ID starting with d9451874ae569868f33c606216056c94433e284bc5865052047a63432e1a1a46 not found: ID does not exist" containerID="d9451874ae569868f33c606216056c94433e284bc5865052047a63432e1a1a46"
Sep 11 01:16:43.968212 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:43.968132    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9451874ae569868f33c606216056c94433e284bc5865052047a63432e1a1a46"} err="failed to get container status \"d9451874ae569868f33c606216056c94433e284bc5865052047a63432e1a1a46\": rpc error: code = NotFound desc = could not find container \"d9451874ae569868f33c606216056c94433e284bc5865052047a63432e1a1a46\": container with ID starting with d9451874ae569868f33c606216056c94433e284bc5865052047a63432e1a1a46 not found: ID does not exist"
Sep 11 01:16:45.779996 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:45.779965    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-194eb64c-7f5bc6c478-fhdj6"]
Sep 11 01:16:45.784756 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:45.784739    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-194eb64c-7f5bc6c478-fhdj6"
Sep 11 01:16:45.787083 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:45.787056    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-194eb64c\""
Sep 11 01:16:45.795044 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:45.794914    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-194eb64c-7f5bc6c478-fhdj6"]
Sep 11 01:16:45.801508 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:45.801466    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3bccd1bc-710f-45ce-a974-985963822797-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-194eb64c-7f5bc6c478-fhdj6\" (UID: \"3bccd1bc-710f-45ce-a974-985963822797\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-194eb64c-7f5bc6c478-fhdj6"
Sep 11 01:16:45.801641 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:45.801552    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qnc8\" (UniqueName: \"kubernetes.io/projected/3bccd1bc-710f-45ce-a974-985963822797-kube-api-access-6qnc8\") pod \"payload-pre-processing-e2e-mt-194eb64c-7f5bc6c478-fhdj6\" (UID: \"3bccd1bc-710f-45ce-a974-985963822797\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-194eb64c-7f5bc6c478-fhdj6"
Sep 11 01:16:45.903410 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:45.903372    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qnc8\" (UniqueName: \"kubernetes.io/projected/3bccd1bc-710f-45ce-a974-985963822797-kube-api-access-6qnc8\") pod \"payload-pre-processing-e2e-mt-194eb64c-7f5bc6c478-fhdj6\" (UID: \"3bccd1bc-710f-45ce-a974-985963822797\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-194eb64c-7f5bc6c478-fhdj6"
Sep 11 01:16:45.903829 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:45.903804    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3bccd1bc-710f-45ce-a974-985963822797-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-194eb64c-7f5bc6c478-fhdj6\" (UID: \"3bccd1bc-710f-45ce-a974-985963822797\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-194eb64c-7f5bc6c478-fhdj6"
Sep 11 01:16:45.904730 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:45.904705    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3bccd1bc-710f-45ce-a974-985963822797-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-194eb64c-7f5bc6c478-fhdj6\" (UID: \"3bccd1bc-710f-45ce-a974-985963822797\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-194eb64c-7f5bc6c478-fhdj6"
Sep 11 01:16:45.924498 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:45.924466    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qnc8\" (UniqueName: \"kubernetes.io/projected/3bccd1bc-710f-45ce-a974-985963822797-kube-api-access-6qnc8\") pod \"payload-pre-processing-e2e-mt-194eb64c-7f5bc6c478-fhdj6\" (UID: \"3bccd1bc-710f-45ce-a974-985963822797\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-194eb64c-7f5bc6c478-fhdj6"
Sep 11 01:16:45.989186 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:45.989155    2569 scope.go:117] "RemoveContainer" containerID="a10f093132f3d6707bd15636c1bfcea4dfd863d26e754f8ca415924dfc1b368c"
Sep 11 01:16:46.097529 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:46.097501    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-194eb64c-7f5bc6c478-fhdj6"
Sep 11 01:16:46.373115 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:46.373087    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-194eb64c-7f5bc6c478-fhdj6"]
Sep 11 01:16:46.374805 ip-10-0-129-222 kubenswrapper[2569]: W0911 01:16:46.374776    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bccd1bc_710f_45ce_a974_985963822797.slice/crio-5b93f30edb2a8a46e17eeb309d31ce506cf850edd31518186abc3fe41c1e8a40 WatchSource:0}: Error finding container 5b93f30edb2a8a46e17eeb309d31ce506cf850edd31518186abc3fe41c1e8a40: Status 404 returned error can't find the container with id 5b93f30edb2a8a46e17eeb309d31ce506cf850edd31518186abc3fe41c1e8a40
Sep 11 01:16:46.979947 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:46.979908    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-194eb64c-7f5bc6c478-fhdj6" event={"ID":"3bccd1bc-710f-45ce-a974-985963822797","Type":"ContainerStarted","Data":"9d050b8a352d601182b0fb391d8406a08a69282d85e5ac345160d33653f87c1c"}
Sep 11 01:16:46.980322 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:46.979968    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-194eb64c-7f5bc6c478-fhdj6" event={"ID":"3bccd1bc-710f-45ce-a974-985963822797","Type":"ContainerStarted","Data":"5b93f30edb2a8a46e17eeb309d31ce506cf850edd31518186abc3fe41c1e8a40"}
Sep 11 01:16:46.980322 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:46.980086    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-194eb64c-7f5bc6c478-fhdj6"
Sep 11 01:16:46.996921 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:46.996865    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-194eb64c-7f5bc6c478-fhdj6" podStartSLOduration=1.996851707 podStartE2EDuration="1.996851707s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 01:16:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 01:16:46.994640996 +0000 UTC m=+1141.646015259" watchObservedRunningTime="2026-09-11 01:16:46.996851707 +0000 UTC m=+1141.648225955"
Sep 11 01:16:53.907745 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:53.907710    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kmx5x"]
Sep 11 01:16:57.988490 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:16:57.988460    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-194eb64c-7f5bc6c478-fhdj6"
Sep 11 01:17:06.104350 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:17:06.104317    2569 generic.go:356] "Generic (PLEG): container finished" podID="41517eb2-ec7f-488c-86b9-fc70f16bcd3d" containerID="03a963847114368a0ce55ad6877cdc12878a8ebaf8b16218389dfd17006b4e3b" exitCode=0
Sep 11 01:17:06.104787 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:17:06.104388    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-194eb64c-openshift-default-779f4656cb-lcpcg" event={"ID":"41517eb2-ec7f-488c-86b9-fc70f16bcd3d","Type":"ContainerDied","Data":"03a963847114368a0ce55ad6877cdc12878a8ebaf8b16218389dfd17006b4e3b"}
Sep 11 01:17:06.104834 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:17:06.104819    2569 scope.go:117] "RemoveContainer" containerID="03a963847114368a0ce55ad6877cdc12878a8ebaf8b16218389dfd17006b4e3b"
Sep 11 01:17:06.105986 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:17:06.105957    2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 01:17:06.106056 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:17:06.106026    2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 01:17:06.106093 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:17:06.106055    2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 01:17:07.113255 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:17:07.113222    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-194eb64c-openshift-default-779f4656cb-lcpcg" event={"ID":"41517eb2-ec7f-488c-86b9-fc70f16bcd3d","Type":"ContainerStarted","Data":"68d0baa36cb739257ed1f28827c5737ac1b884a9f72ea4ff247da6b54b0eed1e"}
Sep 11 01:17:07.736193 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:17:07.736158    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-194eb64c-openshift-default-779f4656cb-lcpcg"
Sep 11 01:17:07.736436 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:17:07.736403    2569 patch_prober.go:28] interesting pod/e2e-mt-194eb64c-openshift-default-779f4656cb-lcpcg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body=
Sep 11 01:17:07.736527 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:17:07.736499    2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-194eb64c-openshift-default-779f4656cb-lcpcg" podUID="41517eb2-ec7f-488c-86b9-fc70f16bcd3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused"
Sep 11 01:17:08.736822 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:17:08.736777    2569 patch_prober.go:28] interesting pod/e2e-mt-194eb64c-openshift-default-779f4656cb-lcpcg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body=
Sep 11 01:17:08.737343 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:17:08.736851    2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-194eb64c-openshift-default-779f4656cb-lcpcg" podUID="41517eb2-ec7f-488c-86b9-fc70f16bcd3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused"
Sep 11 01:17:09.736940 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:17:09.736900    2569 patch_prober.go:28] interesting pod/e2e-mt-194eb64c-openshift-default-779f4656cb-lcpcg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body=
Sep 11 01:17:09.737422 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:17:09.736986    2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-194eb64c-openshift-default-779f4656cb-lcpcg" podUID="41517eb2-ec7f-488c-86b9-fc70f16bcd3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused"
Sep 11 01:17:10.736315 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:17:10.736277    2569 patch_prober.go:28] interesting pod/e2e-mt-194eb64c-openshift-default-779f4656cb-lcpcg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body=
Sep 11 01:17:10.736625 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:17:10.736596    2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-194eb64c-openshift-default-779f4656cb-lcpcg" podUID="41517eb2-ec7f-488c-86b9-fc70f16bcd3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused"
Sep 11 01:17:11.736762 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:17:11.736718    2569 patch_prober.go:28] interesting pod/e2e-mt-194eb64c-openshift-default-779f4656cb-lcpcg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body=
Sep 11 01:17:11.737268 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:17:11.736799    2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-194eb64c-openshift-default-779f4656cb-lcpcg" podUID="41517eb2-ec7f-488c-86b9-fc70f16bcd3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused"
Sep 11 01:17:12.740840 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:17:12.740809    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-194eb64c-openshift-default-779f4656cb-lcpcg"
Sep 11 01:17:12.741324 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:17:12.741215    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-194eb64c-openshift-default-779f4656cb-lcpcg"
Sep 11 01:17:12.741987 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:17:12.741966    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-194eb64c-openshift-default-779f4656cb-lcpcg"
Sep 11 01:17:35.395689 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:17:35.395654    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kmx5x"]
Sep 11 01:17:46.032659 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:17:46.032615    2569 scope.go:117] "RemoveContainer" containerID="f10b765d3c713facd1626eb056fe21a57fdd621053f68b5a10b72293b351b9fa"
Sep 11 01:18:09.300652 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:18:09.300620    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kmx5x"]
Sep 11 01:18:14.247681 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:18:14.247641    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6ba087-29818155-sjmqn"]
Sep 11 01:18:14.249736 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:18:14.249715    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6ba087-29818155-sjmqn"]
Sep 11 01:18:14.276357 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:18:14.276321    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-6ba087-bdcc845c4-cqzxs"]
Sep 11 01:18:14.276613 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:18:14.276581    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-6ba087-bdcc845c4-cqzxs" podUID="a7b5c45d-9578-4b0c-836b-e6d9b2d8dac2" containerName="payload-pre-processing" containerID="cri-o://69b9e8ae3d1252ce4d2afccd74286028dca0d451ddc16c6944f47e24c4c40f19" gracePeriod=30
Sep 11 01:18:14.500966 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:18:14.500854    2569 generic.go:356] "Generic (PLEG): container finished" podID="a7b5c45d-9578-4b0c-836b-e6d9b2d8dac2" containerID="69b9e8ae3d1252ce4d2afccd74286028dca0d451ddc16c6944f47e24c4c40f19" exitCode=0
Sep 11 01:18:14.500966 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:18:14.500924    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-6ba087-bdcc845c4-cqzxs" event={"ID":"a7b5c45d-9578-4b0c-836b-e6d9b2d8dac2","Type":"ContainerDied","Data":"69b9e8ae3d1252ce4d2afccd74286028dca0d451ddc16c6944f47e24c4c40f19"}
Sep 11 01:18:14.547811 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:18:14.547786    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-6ba087-bdcc845c4-cqzxs"
Sep 11 01:18:14.695415 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:18:14.695379    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gmkl4\" (UniqueName: \"kubernetes.io/projected/a7b5c45d-9578-4b0c-836b-e6d9b2d8dac2-kube-api-access-gmkl4\") pod \"a7b5c45d-9578-4b0c-836b-e6d9b2d8dac2\" (UID: \"a7b5c45d-9578-4b0c-836b-e6d9b2d8dac2\") "
Sep 11 01:18:14.695615 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:18:14.695447    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7b5c45d-9578-4b0c-836b-e6d9b2d8dac2-plugins-config-volume\") pod \"a7b5c45d-9578-4b0c-836b-e6d9b2d8dac2\" (UID: \"a7b5c45d-9578-4b0c-836b-e6d9b2d8dac2\") "
Sep 11 01:18:14.695830 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:18:14.695803    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7b5c45d-9578-4b0c-836b-e6d9b2d8dac2-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a7b5c45d-9578-4b0c-836b-e6d9b2d8dac2" (UID: "a7b5c45d-9578-4b0c-836b-e6d9b2d8dac2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 01:18:14.697653 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:18:14.697629    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7b5c45d-9578-4b0c-836b-e6d9b2d8dac2-kube-api-access-gmkl4" (OuterVolumeSpecName: "kube-api-access-gmkl4") pod "a7b5c45d-9578-4b0c-836b-e6d9b2d8dac2" (UID: "a7b5c45d-9578-4b0c-836b-e6d9b2d8dac2"). InnerVolumeSpecName "kube-api-access-gmkl4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 01:18:14.797115 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:18:14.797045    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gmkl4\" (UniqueName: \"kubernetes.io/projected/a7b5c45d-9578-4b0c-836b-e6d9b2d8dac2-kube-api-access-gmkl4\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:18:14.797115 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:18:14.797070    2569 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7b5c45d-9578-4b0c-836b-e6d9b2d8dac2-plugins-config-volume\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:18:15.507663 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:18:15.507633    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-6ba087-bdcc845c4-cqzxs"
Sep 11 01:18:15.507663 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:18:15.507651    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-6ba087-bdcc845c4-cqzxs" event={"ID":"a7b5c45d-9578-4b0c-836b-e6d9b2d8dac2","Type":"ContainerDied","Data":"367f743f198cb692ec1fb82d9b18b93285fc6645123ee0b660d18169aa454e2a"}
Sep 11 01:18:15.508152 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:18:15.507710    2569 scope.go:117] "RemoveContainer" containerID="69b9e8ae3d1252ce4d2afccd74286028dca0d451ddc16c6944f47e24c4c40f19"
Sep 11 01:18:15.531591 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:18:15.531563    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-6ba087-bdcc845c4-cqzxs"]
Sep 11 01:18:15.540365 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:18:15.540344    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-6ba087-bdcc845c4-cqzxs"]
Sep 11 01:18:15.872089 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:18:15.872052    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9081aca3-50bb-4154-af4b-e494a88ecc32" path="/var/lib/kubelet/pods/9081aca3-50bb-4154-af4b-e494a88ecc32/volumes"
Sep 11 01:18:15.872472 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:18:15.872453    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a7b5c45d-9578-4b0c-836b-e6d9b2d8dac2" path="/var/lib/kubelet/pods/a7b5c45d-9578-4b0c-836b-e6d9b2d8dac2/volumes"
Sep 11 01:18:20.725345 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:18:20.725312    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-891e67-29818155-nwb22"]
Sep 11 01:18:20.728537 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:18:20.728511    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-891e67-29818155-nwb22"]
Sep 11 01:18:20.753614 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:18:20.753589    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-891e67-55d8c8cf55-j7jmn"]
Sep 11 01:18:20.753887 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:18:20.753828    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-891e67-55d8c8cf55-j7jmn" podUID="67a94c83-7633-4bec-8ad9-37f452ad8f8f" containerName="payload-processing" containerID="cri-o://e214384c4fb05c9d4e21f0087cc5cfdf5f465b72d8c51dcd53d65e70746d1f7f" gracePeriod=30
Sep 11 01:18:20.760267 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:18:20.760232    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-891e67-fc5bccbf4-4w5pk"]
Sep 11 01:18:20.760470 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:18:20.760446    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-891e67-fc5bccbf4-4w5pk" podUID="70a044e1-e833-4cb8-a58d-2e41dc53ec40" containerName="payload-pre-processing" containerID="cri-o://c1a4e6d05f2374204cf0932b9dbb4c9ab0cb960f5d04d75c69806964415b9fcc" gracePeriod=30
Sep 11 01:18:20.876727 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:18:20.876684    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-6ba087-openshift-default-85464b8f68-b5knz"]
Sep 11 01:18:20.877128 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:18:20.876988    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-6ba087-openshift-default-85464b8f68-b5knz" podUID="e147aad2-95bd-4135-82aa-7f015f6996a4" containerName="istio-proxy" containerID="cri-o://25f4d1bac28bf3430c1ea1d494cee8a6f0f57677871637a577bf47c2eabf5066" gracePeriod=30
Sep 11 01:18:21.045120 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:18:21.045097    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-891e67-55d8c8cf55-j7jmn"
Sep 11 01:18:21.048408 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:18:21.048385    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-891e67-fc5bccbf4-4w5pk"
Sep 11 01:18:21.155803 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:18:21.155767    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-28rcb\" (UniqueName: \"kubernetes.io/projected/70a044e1-e833-4cb8-a58d-2e41dc53ec40-kube-api-access-28rcb\") pod \"70a044e1-e833-4cb8-a58d-2e41dc53ec40\" (UID: \"70a044e1-e833-4cb8-a58d-2e41dc53ec40\") "
Sep 11 01:18:21.155987 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:18:21.155833    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70a044e1-e833-4cb8-a58d-2e41dc53ec40-plugins-config-volume\") pod \"70a044e1-e833-4cb8-a58d-2e41dc53ec40\" (UID: \"70a044e1-e833-4cb8-a58d-2e41dc53ec40\") "
Sep 11 01:18:21.155987 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:18:21.155935    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67a94c83-7633-4bec-8ad9-37f452ad8f8f-plugins-config-volume\") pod \"67a94c83-7633-4bec-8ad9-37f452ad8f8f\" (UID: \"67a94c83-7633-4bec-8ad9-37f452ad8f8f\") "
Sep 11 01:18:21.156085 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:18:21.155984    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d5f59\" (UniqueName: \"kubernetes.io/projected/67a94c83-7633-4bec-8ad9-37f452ad8f8f-kube-api-access-d5f59\") pod \"67a94c83-7633-4bec-8ad9-37f452ad8f8f\" (UID: \"67a94c83-7633-4bec-8ad9-37f452ad8f8f\") "
Sep 11 01:18:21.156247 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:18:21.156223    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70a044e1-e833-4cb8-a58d-2e41dc53ec40-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "70a044e1-e833-4cb8-a58d-2e41dc53ec40" (UID: "70a044e1-e833-4cb8-a58d-2e41dc53ec40"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 01:18:21.156323 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:18:21.156294    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67a94c83-7633-4bec-8ad9-37f452ad8f8f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "67a94c83-7633-4bec-8ad9-37f452ad8f8f" (UID: "67a94c83-7633-4bec-8ad9-37f452ad8f8f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 01:18:21.156477 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:18:21.156461    2569 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70a044e1-e833-4cb8-a58d-2e41dc53ec40-plugins-config-volume\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:18:21.156524 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:18:21.156481    2569 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67a94c83-7633-4bec-8ad9-37f452ad8f8f-plugins-config-volume\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:18:21.158099 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:18:21.158069    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67a94c83-7633-4bec-8ad9-37f452ad8f8f-kube-api-access-d5f59" (OuterVolumeSpecName: "kube-api-access-d5f59") pod "67a94c83-7633-4bec-8ad9-37f452ad8f8f" (UID: "67a94c83-7633-4bec-8ad9-37f452ad8f8f"). InnerVolumeSpecName "kube-api-access-d5f59". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 01:18:21.158203 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:18:21.158129    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70a044e1-e833-4cb8-a58d-2e41dc53ec40-kube-api-access-28rcb" (OuterVolumeSpecName: "kube-api-access-28rcb") pod "70a044e1-e833-4cb8-a58d-2e41dc53ec40" (UID: "70a044e1-e833-4cb8-a58d-2e41dc53ec40"). InnerVolumeSpecName "kube-api-access-28rcb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 01:18:21.257726 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:18:21.257695    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d5f59\" (UniqueName: \"kubernetes.io/projected/67a94c83-7633-4bec-8ad9-37f452ad8f8f-kube-api-access-d5f59\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:18:21.257726 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:18:21.257720    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-28rcb\" (UniqueName: \"kubernetes.io/projected/70a044e1-e833-4cb8-a58d-2e41dc53ec40-kube-api-access-28rcb\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:18:21.543800 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:18:21.543769    2569 generic.go:356] "Generic (PLEG): container finished" podID="67a94c83-7633-4bec-8ad9-37f452ad8f8f" containerID="e214384c4fb05c9d4e21f0087cc5cfdf5f465b72d8c51dcd53d65e70746d1f7f" exitCode=0
Sep 11 01:18:21.543998 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:18:21.543811    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-891e67-55d8c8cf55-j7jmn"
Sep 11 01:18:21.543998 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:18:21.543849    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-891e67-55d8c8cf55-j7jmn" event={"ID":"67a94c83-7633-4bec-8ad9-37f452ad8f8f","Type":"ContainerDied","Data":"e214384c4fb05c9d4e21f0087cc5cfdf5f465b72d8c51dcd53d65e70746d1f7f"}
Sep 11 01:18:21.543998 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:18:21.543921    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-891e67-55d8c8cf55-j7jmn" event={"ID":"67a94c83-7633-4bec-8ad9-37f452ad8f8f","Type":"ContainerDied","Data":"24ef9e102909e3bf0ff9fcc2654c2a8a13fc6534ca483de3ccccf203d945457e"}
Sep 11 01:18:21.543998 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:18:21.543946    2569 scope.go:117] "RemoveContainer" containerID="e214384c4fb05c9d4e21f0087cc5cfdf5f465b72d8c51dcd53d65e70746d1f7f"
Sep 11 01:18:21.545353 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:18:21.545330    2569 generic.go:356] "Generic (PLEG): container finished" podID="70a044e1-e833-4cb8-a58d-2e41dc53ec40" containerID="c1a4e6d05f2374204cf0932b9dbb4c9ab0cb960f5d04d75c69806964415b9fcc" exitCode=0
Sep 11 01:18:21.545482 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:18:21.545380    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-891e67-fc5bccbf4-4w5pk"
Sep 11 01:18:21.545482 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:18:21.545421    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-891e67-fc5bccbf4-4w5pk" event={"ID":"70a044e1-e833-4cb8-a58d-2e41dc53ec40","Type":"ContainerDied","Data":"c1a4e6d05f2374204cf0932b9dbb4c9ab0cb960f5d04d75c69806964415b9fcc"}
Sep 11 01:18:21.545482 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:18:21.545474    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-891e67-fc5bccbf4-4w5pk" event={"ID":"70a044e1-e833-4cb8-a58d-2e41dc53ec40","Type":"ContainerDied","Data":"1519da419f76e363216ca942aba5394ab3d69dbb8fa346145a79cb286b89b25e"}
Sep 11 01:18:21.555444 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:18:21.555423    2569 scope.go:117] "RemoveContainer" containerID="e214384c4fb05c9d4e21f0087cc5cfdf5f465b72d8c51dcd53d65e70746d1f7f"
Sep 11 01:18:21.555764 ip-10-0-129-222 kubenswrapper[2569]: E0911 01:18:21.555737    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e214384c4fb05c9d4e21f0087cc5cfdf5f465b72d8c51dcd53d65e70746d1f7f\": container with ID starting with e214384c4fb05c9d4e21f0087cc5cfdf5f465b72d8c51dcd53d65e70746d1f7f not found: ID does not exist" containerID="e214384c4fb05c9d4e21f0087cc5cfdf5f465b72d8c51dcd53d65e70746d1f7f"
Sep 11 01:18:21.555861 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:18:21.555773    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e214384c4fb05c9d4e21f0087cc5cfdf5f465b72d8c51dcd53d65e70746d1f7f"} err="failed to get container status \"e214384c4fb05c9d4e21f0087cc5cfdf5f465b72d8c51dcd53d65e70746d1f7f\": rpc error: code = NotFound desc = could not find container \"e214384c4fb05c9d4e21f0087cc5cfdf5f465b72d8c51dcd53d65e70746d1f7f\": container with ID starting with e214384c4fb05c9d4e21f0087cc5cfdf5f465b72d8c51dcd53d65e70746d1f7f not found: ID does not exist"
Sep 11 01:18:21.555861 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:18:21.555797    2569 scope.go:117] "RemoveContainer" containerID="c1a4e6d05f2374204cf0932b9dbb4c9ab0cb960f5d04d75c69806964415b9fcc"
Sep 11 01:18:21.564779 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:18:21.564761    2569 scope.go:117] "RemoveContainer" containerID="c1a4e6d05f2374204cf0932b9dbb4c9ab0cb960f5d04d75c69806964415b9fcc"
Sep 11 01:18:21.565116 ip-10-0-129-222 kubenswrapper[2569]: E0911 01:18:21.565096    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1a4e6d05f2374204cf0932b9dbb4c9ab0cb960f5d04d75c69806964415b9fcc\": container with ID starting with c1a4e6d05f2374204cf0932b9dbb4c9ab0cb960f5d04d75c69806964415b9fcc not found: ID does not exist" containerID="c1a4e6d05f2374204cf0932b9dbb4c9ab0cb960f5d04d75c69806964415b9fcc"
Sep 11 01:18:21.565180 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:18:21.565124    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1a4e6d05f2374204cf0932b9dbb4c9ab0cb960f5d04d75c69806964415b9fcc"} err="failed to get container status \"c1a4e6d05f2374204cf0932b9dbb4c9ab0cb960f5d04d75c69806964415b9fcc\": rpc error: code = NotFound desc = could not find container \"c1a4e6d05f2374204cf0932b9dbb4c9ab0cb960f5d04d75c69806964415b9fcc\": container with ID starting with c1a4e6d05f2374204cf0932b9dbb4c9ab0cb960f5d04d75c69806964415b9fcc not found: ID does not exist"
Sep 11 01:18:21.570700 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:18:21.570679    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-891e67-fc5bccbf4-4w5pk"]
Sep 11 01:18:21.574162 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:18:21.574141    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-891e67-fc5bccbf4-4w5pk"]
Sep 11 01:18:21.587448 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:18:21.587422    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-891e67-55d8c8cf55-j7jmn"]
Sep 11 01:18:21.594957 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:18:21.591649    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-891e67-55d8c8cf55-j7jmn"]
Sep 11 01:18:21.870948 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:18:21.870918    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="67a94c83-7633-4bec-8ad9-37f452ad8f8f" path="/var/lib/kubelet/pods/67a94c83-7633-4bec-8ad9-37f452ad8f8f/volumes"
Sep 11 01:18:21.871297 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:18:21.871277    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="70a044e1-e833-4cb8-a58d-2e41dc53ec40" path="/var/lib/kubelet/pods/70a044e1-e833-4cb8-a58d-2e41dc53ec40/volumes"
Sep 11 01:18:21.871527 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:18:21.871514    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ccb561e8-3308-4372-9e98-3244fc775320" path="/var/lib/kubelet/pods/ccb561e8-3308-4372-9e98-3244fc775320/volumes"
Sep 11 01:18:26.140107 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:18:26.140036    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-6ba087-openshift-default-85464b8f68-b5knz"
Sep 11 01:18:26.203588 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:18:26.203558    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e147aad2-95bd-4135-82aa-7f015f6996a4-istiod-ca-cert\") pod \"e147aad2-95bd-4135-82aa-7f015f6996a4\" (UID: \"e147aad2-95bd-4135-82aa-7f015f6996a4\") "
Sep 11 01:18:26.203813 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:18:26.203626    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e147aad2-95bd-4135-82aa-7f015f6996a4-istio-envoy\") pod \"e147aad2-95bd-4135-82aa-7f015f6996a4\" (UID: \"e147aad2-95bd-4135-82aa-7f015f6996a4\") "
Sep 11 01:18:26.203813 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:18:26.203664    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e147aad2-95bd-4135-82aa-7f015f6996a4-istio-data\") pod \"e147aad2-95bd-4135-82aa-7f015f6996a4\" (UID: \"e147aad2-95bd-4135-82aa-7f015f6996a4\") "
Sep 11 01:18:26.203813 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:18:26.203745    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e147aad2-95bd-4135-82aa-7f015f6996a4-istio-token\") pod \"e147aad2-95bd-4135-82aa-7f015f6996a4\" (UID: \"e147aad2-95bd-4135-82aa-7f015f6996a4\") "
Sep 11 01:18:26.203813 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:18:26.203772    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e147aad2-95bd-4135-82aa-7f015f6996a4-istio-podinfo\") pod \"e147aad2-95bd-4135-82aa-7f015f6996a4\" (UID: \"e147aad2-95bd-4135-82aa-7f015f6996a4\") "
Sep 11 01:18:26.203813 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:18:26.203810    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kqvjl\" (UniqueName: \"kubernetes.io/projected/e147aad2-95bd-4135-82aa-7f015f6996a4-kube-api-access-kqvjl\") pod \"e147aad2-95bd-4135-82aa-7f015f6996a4\" (UID: \"e147aad2-95bd-4135-82aa-7f015f6996a4\") "
Sep 11 01:18:26.204126 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:18:26.203847    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e147aad2-95bd-4135-82aa-7f015f6996a4-workload-socket\") pod \"e147aad2-95bd-4135-82aa-7f015f6996a4\" (UID: \"e147aad2-95bd-4135-82aa-7f015f6996a4\") "
Sep 11 01:18:26.204126 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:18:26.203897    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e147aad2-95bd-4135-82aa-7f015f6996a4-credential-socket\") pod \"e147aad2-95bd-4135-82aa-7f015f6996a4\" (UID: \"e147aad2-95bd-4135-82aa-7f015f6996a4\") "
Sep 11 01:18:26.204126 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:18:26.203932    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e147aad2-95bd-4135-82aa-7f015f6996a4-workload-certs\") pod \"e147aad2-95bd-4135-82aa-7f015f6996a4\" (UID: \"e147aad2-95bd-4135-82aa-7f015f6996a4\") "
Sep 11 01:18:26.204126 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:18:26.203993    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e147aad2-95bd-4135-82aa-7f015f6996a4-istio-data" (OuterVolumeSpecName: "istio-data") pod "e147aad2-95bd-4135-82aa-7f015f6996a4" (UID: "e147aad2-95bd-4135-82aa-7f015f6996a4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 01:18:26.204126 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:18:26.204034    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e147aad2-95bd-4135-82aa-7f015f6996a4-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e147aad2-95bd-4135-82aa-7f015f6996a4" (UID: "e147aad2-95bd-4135-82aa-7f015f6996a4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 01:18:26.204402 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:18:26.204211    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e147aad2-95bd-4135-82aa-7f015f6996a4-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e147aad2-95bd-4135-82aa-7f015f6996a4" (UID: "e147aad2-95bd-4135-82aa-7f015f6996a4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 01:18:26.204402 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:18:26.204326    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e147aad2-95bd-4135-82aa-7f015f6996a4-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e147aad2-95bd-4135-82aa-7f015f6996a4" (UID: "e147aad2-95bd-4135-82aa-7f015f6996a4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 01:18:26.204402 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:18:26.204357    2569 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e147aad2-95bd-4135-82aa-7f015f6996a4-workload-socket\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:18:26.204402 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:18:26.204376    2569 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e147aad2-95bd-4135-82aa-7f015f6996a4-istiod-ca-cert\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:18:26.204402 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:18:26.204387    2569 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e147aad2-95bd-4135-82aa-7f015f6996a4-istio-data\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:18:26.204675 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:18:26.204437    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e147aad2-95bd-4135-82aa-7f015f6996a4-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e147aad2-95bd-4135-82aa-7f015f6996a4" (UID: "e147aad2-95bd-4135-82aa-7f015f6996a4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 01:18:26.206125 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:18:26.206100    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e147aad2-95bd-4135-82aa-7f015f6996a4-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e147aad2-95bd-4135-82aa-7f015f6996a4" (UID: "e147aad2-95bd-4135-82aa-7f015f6996a4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 01:18:26.206373 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:18:26.206345    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e147aad2-95bd-4135-82aa-7f015f6996a4-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e147aad2-95bd-4135-82aa-7f015f6996a4" (UID: "e147aad2-95bd-4135-82aa-7f015f6996a4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 01:18:26.206719 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:18:26.206695    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e147aad2-95bd-4135-82aa-7f015f6996a4-kube-api-access-kqvjl" (OuterVolumeSpecName: "kube-api-access-kqvjl") pod "e147aad2-95bd-4135-82aa-7f015f6996a4" (UID: "e147aad2-95bd-4135-82aa-7f015f6996a4"). InnerVolumeSpecName "kube-api-access-kqvjl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 01:18:26.206719 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:18:26.206706    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e147aad2-95bd-4135-82aa-7f015f6996a4-istio-token" (OuterVolumeSpecName: "istio-token") pod "e147aad2-95bd-4135-82aa-7f015f6996a4" (UID: "e147aad2-95bd-4135-82aa-7f015f6996a4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 01:18:26.305714 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:18:26.305685    2569 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e147aad2-95bd-4135-82aa-7f015f6996a4-istio-envoy\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:18:26.305714 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:18:26.305715    2569 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e147aad2-95bd-4135-82aa-7f015f6996a4-istio-token\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:18:26.305909 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:18:26.305726    2569 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e147aad2-95bd-4135-82aa-7f015f6996a4-istio-podinfo\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:18:26.305909 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:18:26.305737    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kqvjl\" (UniqueName: \"kubernetes.io/projected/e147aad2-95bd-4135-82aa-7f015f6996a4-kube-api-access-kqvjl\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:18:26.305909 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:18:26.305750    2569 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e147aad2-95bd-4135-82aa-7f015f6996a4-credential-socket\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:18:26.305909 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:18:26.305762    2569 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e147aad2-95bd-4135-82aa-7f015f6996a4-workload-certs\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:18:26.577495 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:18:26.577466    2569 generic.go:356] "Generic (PLEG): container finished" podID="e147aad2-95bd-4135-82aa-7f015f6996a4" containerID="25f4d1bac28bf3430c1ea1d494cee8a6f0f57677871637a577bf47c2eabf5066" exitCode=0
Sep 11 01:18:26.577678 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:18:26.577516    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-6ba087-openshift-default-85464b8f68-b5knz"
Sep 11 01:18:26.577678 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:18:26.577555    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-6ba087-openshift-default-85464b8f68-b5knz" event={"ID":"e147aad2-95bd-4135-82aa-7f015f6996a4","Type":"ContainerDied","Data":"25f4d1bac28bf3430c1ea1d494cee8a6f0f57677871637a577bf47c2eabf5066"}
Sep 11 01:18:26.577678 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:18:26.577617    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-6ba087-openshift-default-85464b8f68-b5knz" event={"ID":"e147aad2-95bd-4135-82aa-7f015f6996a4","Type":"ContainerDied","Data":"af43b9ffa6f26f9491670781c27c3782857131dcfaff922f9e5fd009662a8f61"}
Sep 11 01:18:26.577678 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:18:26.577645    2569 scope.go:117] "RemoveContainer" containerID="25f4d1bac28bf3430c1ea1d494cee8a6f0f57677871637a577bf47c2eabf5066"
Sep 11 01:18:26.587408 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:18:26.587391    2569 scope.go:117] "RemoveContainer" containerID="7dbec7d9e178cfcf55fd46d3858b4e3f1c87306bc6870f7adb05665f9802cd83"
Sep 11 01:18:26.595974 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:18:26.595950    2569 scope.go:117] "RemoveContainer" containerID="25f4d1bac28bf3430c1ea1d494cee8a6f0f57677871637a577bf47c2eabf5066"
Sep 11 01:18:26.596243 ip-10-0-129-222 kubenswrapper[2569]: E0911 01:18:26.596223    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25f4d1bac28bf3430c1ea1d494cee8a6f0f57677871637a577bf47c2eabf5066\": container with ID starting with 25f4d1bac28bf3430c1ea1d494cee8a6f0f57677871637a577bf47c2eabf5066 not found: ID does not exist" containerID="25f4d1bac28bf3430c1ea1d494cee8a6f0f57677871637a577bf47c2eabf5066"
Sep 11 01:18:26.596320 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:18:26.596249    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25f4d1bac28bf3430c1ea1d494cee8a6f0f57677871637a577bf47c2eabf5066"} err="failed to get container status \"25f4d1bac28bf3430c1ea1d494cee8a6f0f57677871637a577bf47c2eabf5066\": rpc error: code = NotFound desc = could not find container \"25f4d1bac28bf3430c1ea1d494cee8a6f0f57677871637a577bf47c2eabf5066\": container with ID starting with 25f4d1bac28bf3430c1ea1d494cee8a6f0f57677871637a577bf47c2eabf5066 not found: ID does not exist"
Sep 11 01:18:26.596320 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:18:26.596268    2569 scope.go:117] "RemoveContainer" containerID="7dbec7d9e178cfcf55fd46d3858b4e3f1c87306bc6870f7adb05665f9802cd83"
Sep 11 01:18:26.596539 ip-10-0-129-222 kubenswrapper[2569]: E0911 01:18:26.596518    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7dbec7d9e178cfcf55fd46d3858b4e3f1c87306bc6870f7adb05665f9802cd83\": container with ID starting with 7dbec7d9e178cfcf55fd46d3858b4e3f1c87306bc6870f7adb05665f9802cd83 not found: ID does not exist" containerID="7dbec7d9e178cfcf55fd46d3858b4e3f1c87306bc6870f7adb05665f9802cd83"
Sep 11 01:18:26.596632 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:18:26.596549    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7dbec7d9e178cfcf55fd46d3858b4e3f1c87306bc6870f7adb05665f9802cd83"} err="failed to get container status \"7dbec7d9e178cfcf55fd46d3858b4e3f1c87306bc6870f7adb05665f9802cd83\": rpc error: code = NotFound desc = could not find container \"7dbec7d9e178cfcf55fd46d3858b4e3f1c87306bc6870f7adb05665f9802cd83\": container with ID starting with 7dbec7d9e178cfcf55fd46d3858b4e3f1c87306bc6870f7adb05665f9802cd83 not found: ID does not exist"
Sep 11 01:18:26.616774 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:18:26.616749    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-6ba087-openshift-default-85464b8f68-b5knz"]
Sep 11 01:18:26.625968 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:18:26.625945    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-6ba087-openshift-default-85464b8f68-b5knz"]
Sep 11 01:18:27.871167 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:18:27.871130    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e147aad2-95bd-4135-82aa-7f015f6996a4" path="/var/lib/kubelet/pods/e147aad2-95bd-4135-82aa-7f015f6996a4/volumes"
Sep 11 01:18:42.793089 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:18:42.793004    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kmx5x"]
Sep 11 01:18:46.056924 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:18:46.056891    2569 scope.go:117] "RemoveContainer" containerID="c005a9d9db59f4d32357a4db73232df4aed5442391d7b3668253aae3a64d6712"
Sep 11 01:19:18.242076 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:19:18.242041    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-891e67-openshift-default-564fcdf9fb-xw8kt"]
Sep 11 01:19:18.242597 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:19:18.242371    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-891e67-openshift-default-564fcdf9fb-xw8kt" podUID="2b6bb107-1129-43c9-aa58-25cd7c70bfbf" containerName="istio-proxy" containerID="cri-o://d9cf58c9d35182fe0447734cffe15a3d7d83220501297c6b56bb6f36589f5666" gracePeriod=30
Sep 11 01:19:19.112130 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:19:19.112092    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-2482ac-openshift-default-75b5d5c9b8-zfr9t"]
Sep 11 01:19:19.112781 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:19:19.112757    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e147aad2-95bd-4135-82aa-7f015f6996a4" containerName="istio-proxy"
Sep 11 01:19:19.112781 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:19:19.112784    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="e147aad2-95bd-4135-82aa-7f015f6996a4" containerName="istio-proxy"
Sep 11 01:19:19.113041 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:19:19.112798    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67a94c83-7633-4bec-8ad9-37f452ad8f8f" containerName="payload-processing"
Sep 11 01:19:19.113041 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:19:19.112807    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="67a94c83-7633-4bec-8ad9-37f452ad8f8f" containerName="payload-processing"
Sep 11 01:19:19.113041 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:19:19.112822    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e147aad2-95bd-4135-82aa-7f015f6996a4" containerName="istio-proxy"
Sep 11 01:19:19.113041 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:19:19.112831    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="e147aad2-95bd-4135-82aa-7f015f6996a4" containerName="istio-proxy"
Sep 11 01:19:19.113041 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:19:19.112842    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7b5c45d-9578-4b0c-836b-e6d9b2d8dac2" containerName="payload-pre-processing"
Sep 11 01:19:19.113041 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:19:19.112851    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7b5c45d-9578-4b0c-836b-e6d9b2d8dac2" containerName="payload-pre-processing"
Sep 11 01:19:19.113041 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:19:19.112919    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="70a044e1-e833-4cb8-a58d-2e41dc53ec40" containerName="payload-pre-processing"
Sep 11 01:19:19.113041 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:19:19.112926    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="70a044e1-e833-4cb8-a58d-2e41dc53ec40" containerName="payload-pre-processing"
Sep 11 01:19:19.113041 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:19:19.113009    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="e147aad2-95bd-4135-82aa-7f015f6996a4" containerName="istio-proxy"
Sep 11 01:19:19.113041 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:19:19.113016    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="e147aad2-95bd-4135-82aa-7f015f6996a4" containerName="istio-proxy"
Sep 11 01:19:19.113041 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:19:19.113025    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="a7b5c45d-9578-4b0c-836b-e6d9b2d8dac2" containerName="payload-pre-processing"
Sep 11 01:19:19.113041 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:19:19.113032    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="70a044e1-e833-4cb8-a58d-2e41dc53ec40" containerName="payload-pre-processing"
Sep 11 01:19:19.113041 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:19:19.113038    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="67a94c83-7633-4bec-8ad9-37f452ad8f8f" containerName="payload-processing"
Sep 11 01:19:19.115210 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:19:19.115188    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-2482ac-openshift-default-75b5d5c9b8-zfr9t"
Sep 11 01:19:19.118134 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:19:19.118114    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-2482ac-openshift-default-dockercfg-7ccdv\""
Sep 11 01:19:19.128526 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:19:19.128491    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-2482ac-openshift-default-75b5d5c9b8-zfr9t"]
Sep 11 01:19:19.185392 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:19:19.185365    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4wxl\" (UniqueName: \"kubernetes.io/projected/7a92f15b-fd83-4f83-be73-8e5249d0686a-kube-api-access-s4wxl\") pod \"e2e-shared-a-w3-2482ac-openshift-default-75b5d5c9b8-zfr9t\" (UID: \"7a92f15b-fd83-4f83-be73-8e5249d0686a\") " pod="openshift-ingress/e2e-shared-a-w3-2482ac-openshift-default-75b5d5c9b8-zfr9t"
Sep 11 01:19:19.185518 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:19:19.185399    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a92f15b-fd83-4f83-be73-8e5249d0686a-workload-certs\") pod \"e2e-shared-a-w3-2482ac-openshift-default-75b5d5c9b8-zfr9t\" (UID: \"7a92f15b-fd83-4f83-be73-8e5249d0686a\") " pod="openshift-ingress/e2e-shared-a-w3-2482ac-openshift-default-75b5d5c9b8-zfr9t"
Sep 11 01:19:19.185518 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:19:19.185448    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a92f15b-fd83-4f83-be73-8e5249d0686a-istio-data\") pod \"e2e-shared-a-w3-2482ac-openshift-default-75b5d5c9b8-zfr9t\" (UID: \"7a92f15b-fd83-4f83-be73-8e5249d0686a\") " pod="openshift-ingress/e2e-shared-a-w3-2482ac-openshift-default-75b5d5c9b8-zfr9t"
Sep 11 01:19:19.185518 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:19:19.185465    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a92f15b-fd83-4f83-be73-8e5249d0686a-istio-envoy\") pod \"e2e-shared-a-w3-2482ac-openshift-default-75b5d5c9b8-zfr9t\" (UID: \"7a92f15b-fd83-4f83-be73-8e5249d0686a\") " pod="openshift-ingress/e2e-shared-a-w3-2482ac-openshift-default-75b5d5c9b8-zfr9t"
Sep 11 01:19:19.185518 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:19:19.185494    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a92f15b-fd83-4f83-be73-8e5249d0686a-credential-socket\") pod \"e2e-shared-a-w3-2482ac-openshift-default-75b5d5c9b8-zfr9t\" (UID: \"7a92f15b-fd83-4f83-be73-8e5249d0686a\") " pod="openshift-ingress/e2e-shared-a-w3-2482ac-openshift-default-75b5d5c9b8-zfr9t"
Sep 11 01:19:19.185658 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:19:19.185557    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a92f15b-fd83-4f83-be73-8e5249d0686a-istio-token\") pod \"e2e-shared-a-w3-2482ac-openshift-default-75b5d5c9b8-zfr9t\" (UID: \"7a92f15b-fd83-4f83-be73-8e5249d0686a\") " pod="openshift-ingress/e2e-shared-a-w3-2482ac-openshift-default-75b5d5c9b8-zfr9t"
Sep 11 01:19:19.185658 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:19:19.185594    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a92f15b-fd83-4f83-be73-8e5249d0686a-istiod-ca-cert\") pod \"e2e-shared-a-w3-2482ac-openshift-default-75b5d5c9b8-zfr9t\" (UID: \"7a92f15b-fd83-4f83-be73-8e5249d0686a\") " pod="openshift-ingress/e2e-shared-a-w3-2482ac-openshift-default-75b5d5c9b8-zfr9t"
Sep 11 01:19:19.185722 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:19:19.185686    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a92f15b-fd83-4f83-be73-8e5249d0686a-workload-socket\") pod \"e2e-shared-a-w3-2482ac-openshift-default-75b5d5c9b8-zfr9t\" (UID: \"7a92f15b-fd83-4f83-be73-8e5249d0686a\") " pod="openshift-ingress/e2e-shared-a-w3-2482ac-openshift-default-75b5d5c9b8-zfr9t"
Sep 11 01:19:19.185722 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:19:19.185715    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a92f15b-fd83-4f83-be73-8e5249d0686a-istio-podinfo\") pod \"e2e-shared-a-w3-2482ac-openshift-default-75b5d5c9b8-zfr9t\" (UID: \"7a92f15b-fd83-4f83-be73-8e5249d0686a\") " pod="openshift-ingress/e2e-shared-a-w3-2482ac-openshift-default-75b5d5c9b8-zfr9t"
Sep 11 01:19:19.286625 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:19:19.286593    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a92f15b-fd83-4f83-be73-8e5249d0686a-workload-certs\") pod \"e2e-shared-a-w3-2482ac-openshift-default-75b5d5c9b8-zfr9t\" (UID: \"7a92f15b-fd83-4f83-be73-8e5249d0686a\") " pod="openshift-ingress/e2e-shared-a-w3-2482ac-openshift-default-75b5d5c9b8-zfr9t"
Sep 11 01:19:19.287092 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:19:19.286643    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a92f15b-fd83-4f83-be73-8e5249d0686a-istio-data\") pod \"e2e-shared-a-w3-2482ac-openshift-default-75b5d5c9b8-zfr9t\" (UID: \"7a92f15b-fd83-4f83-be73-8e5249d0686a\") " pod="openshift-ingress/e2e-shared-a-w3-2482ac-openshift-default-75b5d5c9b8-zfr9t"
Sep 11 01:19:19.287092 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:19:19.286660    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a92f15b-fd83-4f83-be73-8e5249d0686a-istio-envoy\") pod \"e2e-shared-a-w3-2482ac-openshift-default-75b5d5c9b8-zfr9t\" (UID: \"7a92f15b-fd83-4f83-be73-8e5249d0686a\") " pod="openshift-ingress/e2e-shared-a-w3-2482ac-openshift-default-75b5d5c9b8-zfr9t"
Sep 11 01:19:19.287092 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:19:19.286686    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a92f15b-fd83-4f83-be73-8e5249d0686a-credential-socket\") pod \"e2e-shared-a-w3-2482ac-openshift-default-75b5d5c9b8-zfr9t\" (UID: \"7a92f15b-fd83-4f83-be73-8e5249d0686a\") " pod="openshift-ingress/e2e-shared-a-w3-2482ac-openshift-default-75b5d5c9b8-zfr9t"
Sep 11 01:19:19.287092 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:19:19.286708    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a92f15b-fd83-4f83-be73-8e5249d0686a-istio-token\") pod \"e2e-shared-a-w3-2482ac-openshift-default-75b5d5c9b8-zfr9t\" (UID: \"7a92f15b-fd83-4f83-be73-8e5249d0686a\") " pod="openshift-ingress/e2e-shared-a-w3-2482ac-openshift-default-75b5d5c9b8-zfr9t"
Sep 11 01:19:19.287092 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:19:19.286728    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a92f15b-fd83-4f83-be73-8e5249d0686a-istiod-ca-cert\") pod \"e2e-shared-a-w3-2482ac-openshift-default-75b5d5c9b8-zfr9t\" (UID: \"7a92f15b-fd83-4f83-be73-8e5249d0686a\") " pod="openshift-ingress/e2e-shared-a-w3-2482ac-openshift-default-75b5d5c9b8-zfr9t"
Sep 11 01:19:19.287092 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:19:19.286815    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a92f15b-fd83-4f83-be73-8e5249d0686a-workload-socket\") pod \"e2e-shared-a-w3-2482ac-openshift-default-75b5d5c9b8-zfr9t\" (UID: \"7a92f15b-fd83-4f83-be73-8e5249d0686a\") " pod="openshift-ingress/e2e-shared-a-w3-2482ac-openshift-default-75b5d5c9b8-zfr9t"
Sep 11 01:19:19.287092 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:19:19.286847    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a92f15b-fd83-4f83-be73-8e5249d0686a-istio-podinfo\") pod \"e2e-shared-a-w3-2482ac-openshift-default-75b5d5c9b8-zfr9t\" (UID: \"7a92f15b-fd83-4f83-be73-8e5249d0686a\") " pod="openshift-ingress/e2e-shared-a-w3-2482ac-openshift-default-75b5d5c9b8-zfr9t"
Sep 11 01:19:19.287092 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:19:19.286904    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4wxl\" (UniqueName: \"kubernetes.io/projected/7a92f15b-fd83-4f83-be73-8e5249d0686a-kube-api-access-s4wxl\") pod \"e2e-shared-a-w3-2482ac-openshift-default-75b5d5c9b8-zfr9t\" (UID: \"7a92f15b-fd83-4f83-be73-8e5249d0686a\") " pod="openshift-ingress/e2e-shared-a-w3-2482ac-openshift-default-75b5d5c9b8-zfr9t"
Sep 11 01:19:19.287498 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:19:19.287193    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a92f15b-fd83-4f83-be73-8e5249d0686a-istio-data\") pod \"e2e-shared-a-w3-2482ac-openshift-default-75b5d5c9b8-zfr9t\" (UID: \"7a92f15b-fd83-4f83-be73-8e5249d0686a\") " pod="openshift-ingress/e2e-shared-a-w3-2482ac-openshift-default-75b5d5c9b8-zfr9t"
Sep 11 01:19:19.287498 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:19:19.287365    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a92f15b-fd83-4f83-be73-8e5249d0686a-workload-certs\") pod \"e2e-shared-a-w3-2482ac-openshift-default-75b5d5c9b8-zfr9t\" (UID: \"7a92f15b-fd83-4f83-be73-8e5249d0686a\") " pod="openshift-ingress/e2e-shared-a-w3-2482ac-openshift-default-75b5d5c9b8-zfr9t"
Sep 11 01:19:19.287498 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:19:19.287448    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a92f15b-fd83-4f83-be73-8e5249d0686a-credential-socket\") pod \"e2e-shared-a-w3-2482ac-openshift-default-75b5d5c9b8-zfr9t\" (UID: \"7a92f15b-fd83-4f83-be73-8e5249d0686a\") " pod="openshift-ingress/e2e-shared-a-w3-2482ac-openshift-default-75b5d5c9b8-zfr9t"
Sep 11 01:19:19.287498 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:19:19.287471    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a92f15b-fd83-4f83-be73-8e5249d0686a-workload-socket\") pod \"e2e-shared-a-w3-2482ac-openshift-default-75b5d5c9b8-zfr9t\" (UID: \"7a92f15b-fd83-4f83-be73-8e5249d0686a\") " pod="openshift-ingress/e2e-shared-a-w3-2482ac-openshift-default-75b5d5c9b8-zfr9t"
Sep 11 01:19:19.287740 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:19:19.287718    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a92f15b-fd83-4f83-be73-8e5249d0686a-istiod-ca-cert\") pod \"e2e-shared-a-w3-2482ac-openshift-default-75b5d5c9b8-zfr9t\" (UID: \"7a92f15b-fd83-4f83-be73-8e5249d0686a\") " pod="openshift-ingress/e2e-shared-a-w3-2482ac-openshift-default-75b5d5c9b8-zfr9t"
Sep 11 01:19:19.289221 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:19:19.289201    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a92f15b-fd83-4f83-be73-8e5249d0686a-istio-envoy\") pod \"e2e-shared-a-w3-2482ac-openshift-default-75b5d5c9b8-zfr9t\" (UID: \"7a92f15b-fd83-4f83-be73-8e5249d0686a\") " pod="openshift-ingress/e2e-shared-a-w3-2482ac-openshift-default-75b5d5c9b8-zfr9t"
Sep 11 01:19:19.289378 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:19:19.289360    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a92f15b-fd83-4f83-be73-8e5249d0686a-istio-podinfo\") pod \"e2e-shared-a-w3-2482ac-openshift-default-75b5d5c9b8-zfr9t\" (UID: \"7a92f15b-fd83-4f83-be73-8e5249d0686a\") " pod="openshift-ingress/e2e-shared-a-w3-2482ac-openshift-default-75b5d5c9b8-zfr9t"
Sep 11 01:19:19.295796 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:19:19.295775    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4wxl\" (UniqueName: \"kubernetes.io/projected/7a92f15b-fd83-4f83-be73-8e5249d0686a-kube-api-access-s4wxl\") pod \"e2e-shared-a-w3-2482ac-openshift-default-75b5d5c9b8-zfr9t\" (UID: \"7a92f15b-fd83-4f83-be73-8e5249d0686a\") " pod="openshift-ingress/e2e-shared-a-w3-2482ac-openshift-default-75b5d5c9b8-zfr9t"
Sep 11 01:19:19.295957 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:19:19.295852    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a92f15b-fd83-4f83-be73-8e5249d0686a-istio-token\") pod \"e2e-shared-a-w3-2482ac-openshift-default-75b5d5c9b8-zfr9t\" (UID: \"7a92f15b-fd83-4f83-be73-8e5249d0686a\") " pod="openshift-ingress/e2e-shared-a-w3-2482ac-openshift-default-75b5d5c9b8-zfr9t"
Sep 11 01:19:19.425896 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:19:19.425770    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-2482ac-openshift-default-75b5d5c9b8-zfr9t"
Sep 11 01:19:19.699822 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:19:19.699795    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-2482ac-openshift-default-75b5d5c9b8-zfr9t"]
Sep 11 01:19:19.701746 ip-10-0-129-222 kubenswrapper[2569]: W0911 01:19:19.701720    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a92f15b_fd83_4f83_be73_8e5249d0686a.slice/crio-754e10168915541382b90102b415e453f74411d1c13391e225dfb078ac06b286 WatchSource:0}: Error finding container 754e10168915541382b90102b415e453f74411d1c13391e225dfb078ac06b286: Status 404 returned error can't find the container with id 754e10168915541382b90102b415e453f74411d1c13391e225dfb078ac06b286
Sep 11 01:19:19.703977 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:19:19.703945    2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 01:19:19.704042 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:19:19.704012    2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 01:19:19.704080 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:19:19.704040    2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 01:19:19.890203 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:19:19.890164    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-2482ac-openshift-default-75b5d5c9b8-zfr9t" event={"ID":"7a92f15b-fd83-4f83-be73-8e5249d0686a","Type":"ContainerStarted","Data":"3c76a24d83b28546610a57f01eeca3cc81db1bcb2d63887e4236e00b175380a5"}
Sep 11 01:19:19.890377 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:19:19.890229    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-2482ac-openshift-default-75b5d5c9b8-zfr9t" event={"ID":"7a92f15b-fd83-4f83-be73-8e5249d0686a","Type":"ContainerStarted","Data":"754e10168915541382b90102b415e453f74411d1c13391e225dfb078ac06b286"}
Sep 11 01:19:19.914100 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:19:19.914047    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-2482ac-openshift-default-75b5d5c9b8-zfr9t" podStartSLOduration=0.91403244 podStartE2EDuration="914.03244ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 01:19:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 01:19:19.911275191 +0000 UTC m=+1294.562649442" watchObservedRunningTime="2026-09-11 01:19:19.91403244 +0000 UTC m=+1294.565406688"
Sep 11 01:19:20.426283 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:19:20.426248    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-2482ac-openshift-default-75b5d5c9b8-zfr9t"
Sep 11 01:19:20.431185 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:19:20.431161    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-2482ac-openshift-default-75b5d5c9b8-zfr9t"
Sep 11 01:19:20.895671 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:19:20.895644    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-2482ac-openshift-default-75b5d5c9b8-zfr9t"
Sep 11 01:19:20.896791 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:19:20.896771    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-2482ac-openshift-default-75b5d5c9b8-zfr9t"
Sep 11 01:19:23.503744 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:19:23.503694    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-891e67-openshift-default-564fcdf9fb-xw8kt"
Sep 11 01:19:23.629381 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:19:23.629350    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b6bb107-1129-43c9-aa58-25cd7c70bfbf-credential-socket\") pod \"2b6bb107-1129-43c9-aa58-25cd7c70bfbf\" (UID: \"2b6bb107-1129-43c9-aa58-25cd7c70bfbf\") "
Sep 11 01:19:23.629555 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:19:23.629438    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b6bb107-1129-43c9-aa58-25cd7c70bfbf-istio-data\") pod \"2b6bb107-1129-43c9-aa58-25cd7c70bfbf\" (UID: \"2b6bb107-1129-43c9-aa58-25cd7c70bfbf\") "
Sep 11 01:19:23.629555 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:19:23.629480    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b6bb107-1129-43c9-aa58-25cd7c70bfbf-workload-socket\") pod \"2b6bb107-1129-43c9-aa58-25cd7c70bfbf\" (UID: \"2b6bb107-1129-43c9-aa58-25cd7c70bfbf\") "
Sep 11 01:19:23.629555 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:19:23.629536    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b6bb107-1129-43c9-aa58-25cd7c70bfbf-istio-podinfo\") pod \"2b6bb107-1129-43c9-aa58-25cd7c70bfbf\" (UID: \"2b6bb107-1129-43c9-aa58-25cd7c70bfbf\") "
Sep 11 01:19:23.629729 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:19:23.629569    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b6bb107-1129-43c9-aa58-25cd7c70bfbf-istio-envoy\") pod \"2b6bb107-1129-43c9-aa58-25cd7c70bfbf\" (UID: \"2b6bb107-1129-43c9-aa58-25cd7c70bfbf\") "
Sep 11 01:19:23.629729 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:19:23.629605    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b6bb107-1129-43c9-aa58-25cd7c70bfbf-istio-token\") pod \"2b6bb107-1129-43c9-aa58-25cd7c70bfbf\" (UID: \"2b6bb107-1129-43c9-aa58-25cd7c70bfbf\") "
Sep 11 01:19:23.629729 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:19:23.629604    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b6bb107-1129-43c9-aa58-25cd7c70bfbf-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "2b6bb107-1129-43c9-aa58-25cd7c70bfbf" (UID: "2b6bb107-1129-43c9-aa58-25cd7c70bfbf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 01:19:23.629729 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:19:23.629671    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b6bb107-1129-43c9-aa58-25cd7c70bfbf-workload-certs\") pod \"2b6bb107-1129-43c9-aa58-25cd7c70bfbf\" (UID: \"2b6bb107-1129-43c9-aa58-25cd7c70bfbf\") "
Sep 11 01:19:23.629729 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:19:23.629704    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xtssr\" (UniqueName: \"kubernetes.io/projected/2b6bb107-1129-43c9-aa58-25cd7c70bfbf-kube-api-access-xtssr\") pod \"2b6bb107-1129-43c9-aa58-25cd7c70bfbf\" (UID: \"2b6bb107-1129-43c9-aa58-25cd7c70bfbf\") "
Sep 11 01:19:23.630005 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:19:23.629730    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b6bb107-1129-43c9-aa58-25cd7c70bfbf-istio-data" (OuterVolumeSpecName: "istio-data") pod "2b6bb107-1129-43c9-aa58-25cd7c70bfbf" (UID: "2b6bb107-1129-43c9-aa58-25cd7c70bfbf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 01:19:23.630005 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:19:23.629744    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b6bb107-1129-43c9-aa58-25cd7c70bfbf-istiod-ca-cert\") pod \"2b6bb107-1129-43c9-aa58-25cd7c70bfbf\" (UID: \"2b6bb107-1129-43c9-aa58-25cd7c70bfbf\") "
Sep 11 01:19:23.630005 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:19:23.629855    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b6bb107-1129-43c9-aa58-25cd7c70bfbf-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "2b6bb107-1129-43c9-aa58-25cd7c70bfbf" (UID: "2b6bb107-1129-43c9-aa58-25cd7c70bfbf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 01:19:23.630152 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:19:23.630100    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b6bb107-1129-43c9-aa58-25cd7c70bfbf-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "2b6bb107-1129-43c9-aa58-25cd7c70bfbf" (UID: "2b6bb107-1129-43c9-aa58-25cd7c70bfbf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 01:19:23.630152 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:19:23.630144    2569 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b6bb107-1129-43c9-aa58-25cd7c70bfbf-istio-data\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:19:23.630245 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:19:23.630163    2569 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b6bb107-1129-43c9-aa58-25cd7c70bfbf-workload-socket\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:19:23.630245 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:19:23.630178    2569 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b6bb107-1129-43c9-aa58-25cd7c70bfbf-credential-socket\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:19:23.630245 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:19:23.630212    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b6bb107-1129-43c9-aa58-25cd7c70bfbf-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "2b6bb107-1129-43c9-aa58-25cd7c70bfbf" (UID: "2b6bb107-1129-43c9-aa58-25cd7c70bfbf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 01:19:23.631892 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:19:23.631850    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2b6bb107-1129-43c9-aa58-25cd7c70bfbf-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "2b6bb107-1129-43c9-aa58-25cd7c70bfbf" (UID: "2b6bb107-1129-43c9-aa58-25cd7c70bfbf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 01:19:23.632014 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:19:23.631856    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b6bb107-1129-43c9-aa58-25cd7c70bfbf-kube-api-access-xtssr" (OuterVolumeSpecName: "kube-api-access-xtssr") pod "2b6bb107-1129-43c9-aa58-25cd7c70bfbf" (UID: "2b6bb107-1129-43c9-aa58-25cd7c70bfbf"). InnerVolumeSpecName "kube-api-access-xtssr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 01:19:23.632211 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:19:23.632188    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b6bb107-1129-43c9-aa58-25cd7c70bfbf-istio-token" (OuterVolumeSpecName: "istio-token") pod "2b6bb107-1129-43c9-aa58-25cd7c70bfbf" (UID: "2b6bb107-1129-43c9-aa58-25cd7c70bfbf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 01:19:23.632322 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:19:23.632220    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b6bb107-1129-43c9-aa58-25cd7c70bfbf-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "2b6bb107-1129-43c9-aa58-25cd7c70bfbf" (UID: "2b6bb107-1129-43c9-aa58-25cd7c70bfbf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 01:19:23.729340 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:19:23.729308    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2482ac-75f7b999d6-bjvlg"]
Sep 11 01:19:23.729845 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:19:23.729827    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b6bb107-1129-43c9-aa58-25cd7c70bfbf" containerName="istio-proxy"
Sep 11 01:19:23.729845 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:19:23.729847    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b6bb107-1129-43c9-aa58-25cd7c70bfbf" containerName="istio-proxy"
Sep 11 01:19:23.730033 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:19:23.730020    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="2b6bb107-1129-43c9-aa58-25cd7c70bfbf" containerName="istio-proxy"
Sep 11 01:19:23.730091 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:19:23.730040    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="2b6bb107-1129-43c9-aa58-25cd7c70bfbf" containerName="istio-proxy"
Sep 11 01:19:23.731305 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:19:23.731283    2569 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b6bb107-1129-43c9-aa58-25cd7c70bfbf-istio-podinfo\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:19:23.731305 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:19:23.731305    2569 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b6bb107-1129-43c9-aa58-25cd7c70bfbf-istio-envoy\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:19:23.731495 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:19:23.731314    2569 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b6bb107-1129-43c9-aa58-25cd7c70bfbf-istio-token\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:19:23.731495 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:19:23.731322    2569 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b6bb107-1129-43c9-aa58-25cd7c70bfbf-workload-certs\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:19:23.731495 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:19:23.731330    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xtssr\" (UniqueName: \"kubernetes.io/projected/2b6bb107-1129-43c9-aa58-25cd7c70bfbf-kube-api-access-xtssr\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:19:23.731495 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:19:23.731339    2569 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b6bb107-1129-43c9-aa58-25cd7c70bfbf-istiod-ca-cert\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:19:23.732156 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:19:23.732137    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2482ac-75f7b999d6-bjvlg"
Sep 11 01:19:23.734346 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:19:23.734318    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-2482ac\""
Sep 11 01:19:23.734481 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:19:23.734348    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-2482ac-dockercfg-mgnjh\""
Sep 11 01:19:23.742721 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:19:23.742697    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2482ac-75f7b999d6-bjvlg"]
Sep 11 01:19:23.771748 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:19:23.771694    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-2482ac-6c4ddccb95-xs7tc"]
Sep 11 01:19:23.772133 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:19:23.772120    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b6bb107-1129-43c9-aa58-25cd7c70bfbf" containerName="istio-proxy"
Sep 11 01:19:23.772225 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:19:23.772135    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b6bb107-1129-43c9-aa58-25cd7c70bfbf" containerName="istio-proxy"
Sep 11 01:19:23.774116 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:19:23.774102    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2482ac-6c4ddccb95-xs7tc"
Sep 11 01:19:23.785437 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:19:23.785414    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-2482ac-6c4ddccb95-xs7tc"]
Sep 11 01:19:23.832002 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:19:23.831964    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ebb5ee7b-b3c8-45e8-a361-c2bdf98ccd20-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-2482ac-6c4ddccb95-xs7tc\" (UID: \"ebb5ee7b-b3c8-45e8-a361-c2bdf98ccd20\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2482ac-6c4ddccb95-xs7tc"
Sep 11 01:19:23.832194 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:19:23.832073    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znwjf\" (UniqueName: \"kubernetes.io/projected/7b5dbde6-6b81-444e-822e-0ee063ef4065-kube-api-access-znwjf\") pod \"payload-pre-processing-e2e-shared-a-w3-2482ac-75f7b999d6-bjvlg\" (UID: \"7b5dbde6-6b81-444e-822e-0ee063ef4065\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2482ac-75f7b999d6-bjvlg"
Sep 11 01:19:23.832194 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:19:23.832184    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zzcwp\" (UniqueName: \"kubernetes.io/projected/ebb5ee7b-b3c8-45e8-a361-c2bdf98ccd20-kube-api-access-zzcwp\") pod \"payload-processing-e2e-shared-a-w3-2482ac-6c4ddccb95-xs7tc\" (UID: \"ebb5ee7b-b3c8-45e8-a361-c2bdf98ccd20\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2482ac-6c4ddccb95-xs7tc"
Sep 11 01:19:23.832319 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:19:23.832211    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b5dbde6-6b81-444e-822e-0ee063ef4065-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-2482ac-75f7b999d6-bjvlg\" (UID: \"7b5dbde6-6b81-444e-822e-0ee063ef4065\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2482ac-75f7b999d6-bjvlg"
Sep 11 01:19:23.912292 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:19:23.912261    2569 generic.go:356] "Generic (PLEG): container finished" podID="2b6bb107-1129-43c9-aa58-25cd7c70bfbf" containerID="d9cf58c9d35182fe0447734cffe15a3d7d83220501297c6b56bb6f36589f5666" exitCode=0
Sep 11 01:19:23.912449 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:19:23.912317    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-891e67-openshift-default-564fcdf9fb-xw8kt"
Sep 11 01:19:23.912449 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:19:23.912313    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-891e67-openshift-default-564fcdf9fb-xw8kt" event={"ID":"2b6bb107-1129-43c9-aa58-25cd7c70bfbf","Type":"ContainerDied","Data":"d9cf58c9d35182fe0447734cffe15a3d7d83220501297c6b56bb6f36589f5666"}
Sep 11 01:19:23.912449 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:19:23.912443    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-891e67-openshift-default-564fcdf9fb-xw8kt" event={"ID":"2b6bb107-1129-43c9-aa58-25cd7c70bfbf","Type":"ContainerDied","Data":"5a44b163ff36da0150500bee5427dda3ef62d30b43f7f3822a847cd0b7371941"}
Sep 11 01:19:23.912606 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:19:23.912467    2569 scope.go:117] "RemoveContainer" containerID="d9cf58c9d35182fe0447734cffe15a3d7d83220501297c6b56bb6f36589f5666"
Sep 11 01:19:23.921957 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:19:23.921942    2569 scope.go:117] "RemoveContainer" containerID="8a84288b9a80622c7d651d301fd4c8a9eb292140b04df4f1b33521cb7801710f"
Sep 11 01:19:23.929625 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:19:23.929586    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-891e67-openshift-default-564fcdf9fb-xw8kt"]
Sep 11 01:19:23.930609 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:19:23.930502    2569 scope.go:117] "RemoveContainer" containerID="d9cf58c9d35182fe0447734cffe15a3d7d83220501297c6b56bb6f36589f5666"
Sep 11 01:19:23.930893 ip-10-0-129-222 kubenswrapper[2569]: E0911 01:19:23.930852    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9cf58c9d35182fe0447734cffe15a3d7d83220501297c6b56bb6f36589f5666\": container with ID starting with d9cf58c9d35182fe0447734cffe15a3d7d83220501297c6b56bb6f36589f5666 not found: ID does not exist" containerID="d9cf58c9d35182fe0447734cffe15a3d7d83220501297c6b56bb6f36589f5666"
Sep 11 01:19:23.930989 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:19:23.930894    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9cf58c9d35182fe0447734cffe15a3d7d83220501297c6b56bb6f36589f5666"} err="failed to get container status \"d9cf58c9d35182fe0447734cffe15a3d7d83220501297c6b56bb6f36589f5666\": rpc error: code = NotFound desc = could not find container \"d9cf58c9d35182fe0447734cffe15a3d7d83220501297c6b56bb6f36589f5666\": container with ID starting with d9cf58c9d35182fe0447734cffe15a3d7d83220501297c6b56bb6f36589f5666 not found: ID does not exist"
Sep 11 01:19:23.930989 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:19:23.930911    2569 scope.go:117] "RemoveContainer" containerID="8a84288b9a80622c7d651d301fd4c8a9eb292140b04df4f1b33521cb7801710f"
Sep 11 01:19:23.931203 ip-10-0-129-222 kubenswrapper[2569]: E0911 01:19:23.931183    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a84288b9a80622c7d651d301fd4c8a9eb292140b04df4f1b33521cb7801710f\": container with ID starting with 8a84288b9a80622c7d651d301fd4c8a9eb292140b04df4f1b33521cb7801710f not found: ID does not exist" containerID="8a84288b9a80622c7d651d301fd4c8a9eb292140b04df4f1b33521cb7801710f"
Sep 11 01:19:23.931268 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:19:23.931207    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a84288b9a80622c7d651d301fd4c8a9eb292140b04df4f1b33521cb7801710f"} err="failed to get container status \"8a84288b9a80622c7d651d301fd4c8a9eb292140b04df4f1b33521cb7801710f\": rpc error: code = NotFound desc = could not find container \"8a84288b9a80622c7d651d301fd4c8a9eb292140b04df4f1b33521cb7801710f\": container with ID starting with 8a84288b9a80622c7d651d301fd4c8a9eb292140b04df4f1b33521cb7801710f not found: ID does not exist"
Sep 11 01:19:23.933130 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:19:23.933108    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-891e67-openshift-default-564fcdf9fb-xw8kt"]
Sep 11 01:19:23.933532 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:19:23.933512    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzcwp\" (UniqueName: \"kubernetes.io/projected/ebb5ee7b-b3c8-45e8-a361-c2bdf98ccd20-kube-api-access-zzcwp\") pod \"payload-processing-e2e-shared-a-w3-2482ac-6c4ddccb95-xs7tc\" (UID: \"ebb5ee7b-b3c8-45e8-a361-c2bdf98ccd20\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2482ac-6c4ddccb95-xs7tc"
Sep 11 01:19:23.933615 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:19:23.933545    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b5dbde6-6b81-444e-822e-0ee063ef4065-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-2482ac-75f7b999d6-bjvlg\" (UID: \"7b5dbde6-6b81-444e-822e-0ee063ef4065\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2482ac-75f7b999d6-bjvlg"
Sep 11 01:19:23.933776 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:19:23.933754    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ebb5ee7b-b3c8-45e8-a361-c2bdf98ccd20-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-2482ac-6c4ddccb95-xs7tc\" (UID: \"ebb5ee7b-b3c8-45e8-a361-c2bdf98ccd20\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2482ac-6c4ddccb95-xs7tc"
Sep 11 01:19:23.933852 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:19:23.933828    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znwjf\" (UniqueName: \"kubernetes.io/projected/7b5dbde6-6b81-444e-822e-0ee063ef4065-kube-api-access-znwjf\") pod \"payload-pre-processing-e2e-shared-a-w3-2482ac-75f7b999d6-bjvlg\" (UID: \"7b5dbde6-6b81-444e-822e-0ee063ef4065\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2482ac-75f7b999d6-bjvlg"
Sep 11 01:19:23.934267 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:19:23.934247    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b5dbde6-6b81-444e-822e-0ee063ef4065-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-2482ac-75f7b999d6-bjvlg\" (UID: \"7b5dbde6-6b81-444e-822e-0ee063ef4065\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2482ac-75f7b999d6-bjvlg"
Sep 11 01:19:23.934490 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:19:23.934473    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ebb5ee7b-b3c8-45e8-a361-c2bdf98ccd20-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-2482ac-6c4ddccb95-xs7tc\" (UID: \"ebb5ee7b-b3c8-45e8-a361-c2bdf98ccd20\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2482ac-6c4ddccb95-xs7tc"
Sep 11 01:19:23.941341 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:19:23.941320    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzcwp\" (UniqueName: \"kubernetes.io/projected/ebb5ee7b-b3c8-45e8-a361-c2bdf98ccd20-kube-api-access-zzcwp\") pod \"payload-processing-e2e-shared-a-w3-2482ac-6c4ddccb95-xs7tc\" (UID: \"ebb5ee7b-b3c8-45e8-a361-c2bdf98ccd20\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2482ac-6c4ddccb95-xs7tc"
Sep 11 01:19:23.941477 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:19:23.941458    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-znwjf\" (UniqueName: \"kubernetes.io/projected/7b5dbde6-6b81-444e-822e-0ee063ef4065-kube-api-access-znwjf\") pod \"payload-pre-processing-e2e-shared-a-w3-2482ac-75f7b999d6-bjvlg\" (UID: \"7b5dbde6-6b81-444e-822e-0ee063ef4065\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2482ac-75f7b999d6-bjvlg"
Sep 11 01:19:24.043286 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:19:24.043229    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2482ac-75f7b999d6-bjvlg"
Sep 11 01:19:24.084677 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:19:24.084649    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2482ac-6c4ddccb95-xs7tc"
Sep 11 01:19:24.314162 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:19:24.314134    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2482ac-75f7b999d6-bjvlg"]
Sep 11 01:19:24.315538 ip-10-0-129-222 kubenswrapper[2569]: W0911 01:19:24.315510    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b5dbde6_6b81_444e_822e_0ee063ef4065.slice/crio-631b9da27dd4aa8dda424b464a3319e0138da1e0a8b0ed891fb54845a53b17c6 WatchSource:0}: Error finding container 631b9da27dd4aa8dda424b464a3319e0138da1e0a8b0ed891fb54845a53b17c6: Status 404 returned error can't find the container with id 631b9da27dd4aa8dda424b464a3319e0138da1e0a8b0ed891fb54845a53b17c6
Sep 11 01:19:24.369298 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:19:24.369276    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-2482ac-6c4ddccb95-xs7tc"]
Sep 11 01:19:24.371206 ip-10-0-129-222 kubenswrapper[2569]: W0911 01:19:24.371182    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebb5ee7b_b3c8_45e8_a361_c2bdf98ccd20.slice/crio-ef93a51ba96963f5ab01a90ef40a04a259eaadd1b48baa2ba057be3b340fb2e1 WatchSource:0}: Error finding container ef93a51ba96963f5ab01a90ef40a04a259eaadd1b48baa2ba057be3b340fb2e1: Status 404 returned error can't find the container with id ef93a51ba96963f5ab01a90ef40a04a259eaadd1b48baa2ba057be3b340fb2e1
Sep 11 01:19:24.932086 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:19:24.932045    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2482ac-6c4ddccb95-xs7tc" event={"ID":"ebb5ee7b-b3c8-45e8-a361-c2bdf98ccd20","Type":"ContainerStarted","Data":"490e0c9833c52b9ade373352235c2c9828b989b7d1619580e247281e8f7c0501"}
Sep 11 01:19:24.932536 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:19:24.932150    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2482ac-6c4ddccb95-xs7tc"
Sep 11 01:19:24.932536 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:19:24.932216    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2482ac-6c4ddccb95-xs7tc" event={"ID":"ebb5ee7b-b3c8-45e8-a361-c2bdf98ccd20","Type":"ContainerStarted","Data":"ef93a51ba96963f5ab01a90ef40a04a259eaadd1b48baa2ba057be3b340fb2e1"}
Sep 11 01:19:24.934503 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:19:24.934478    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2482ac-75f7b999d6-bjvlg" event={"ID":"7b5dbde6-6b81-444e-822e-0ee063ef4065","Type":"ContainerStarted","Data":"ddd62c8ba49c95c613a2ef0b52588beb9555bd22b59ab9e62cb3f5abb4e28ca8"}
Sep 11 01:19:24.934619 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:19:24.934519    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2482ac-75f7b999d6-bjvlg" event={"ID":"7b5dbde6-6b81-444e-822e-0ee063ef4065","Type":"ContainerStarted","Data":"631b9da27dd4aa8dda424b464a3319e0138da1e0a8b0ed891fb54845a53b17c6"}
Sep 11 01:19:24.934677 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:19:24.934623    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2482ac-75f7b999d6-bjvlg"
Sep 11 01:19:24.950300 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:19:24.950255    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2482ac-6c4ddccb95-xs7tc" podStartSLOduration=1.950240245 podStartE2EDuration="1.950240245s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 01:19:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 01:19:24.949949201 +0000 UTC m=+1299.601323453" watchObservedRunningTime="2026-09-11 01:19:24.950240245 +0000 UTC m=+1299.601614496"
Sep 11 01:19:24.966986 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:19:24.966934    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2482ac-75f7b999d6-bjvlg" podStartSLOduration=1.966917748 podStartE2EDuration="1.966917748s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 01:19:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 01:19:24.96566754 +0000 UTC m=+1299.617041788" watchObservedRunningTime="2026-09-11 01:19:24.966917748 +0000 UTC m=+1299.618291999"
Sep 11 01:19:25.871827 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:19:25.871791    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2b6bb107-1129-43c9-aa58-25cd7c70bfbf" path="/var/lib/kubelet/pods/2b6bb107-1129-43c9-aa58-25cd7c70bfbf/volumes"
Sep 11 01:19:35.942196 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:19:35.942167    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2482ac-6c4ddccb95-xs7tc"
Sep 11 01:19:35.942601 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:19:35.942231    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2482ac-75f7b999d6-bjvlg"
Sep 11 01:19:37.609257 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:19:37.609211    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kmx5x"]
Sep 11 01:19:48.066592 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:19:48.066559    2569 generic.go:356] "Generic (PLEG): container finished" podID="7a92f15b-fd83-4f83-be73-8e5249d0686a" containerID="3c76a24d83b28546610a57f01eeca3cc81db1bcb2d63887e4236e00b175380a5" exitCode=0
Sep 11 01:19:48.067118 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:19:48.066637    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-2482ac-openshift-default-75b5d5c9b8-zfr9t" event={"ID":"7a92f15b-fd83-4f83-be73-8e5249d0686a","Type":"ContainerDied","Data":"3c76a24d83b28546610a57f01eeca3cc81db1bcb2d63887e4236e00b175380a5"}
Sep 11 01:19:48.067118 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:19:48.067091    2569 scope.go:117] "RemoveContainer" containerID="3c76a24d83b28546610a57f01eeca3cc81db1bcb2d63887e4236e00b175380a5"
Sep 11 01:19:48.068362 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:19:48.068333    2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 01:19:48.068453 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:19:48.068407    2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 01:19:48.068453 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:19:48.068444    2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 01:19:49.074855 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:19:49.074819    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-2482ac-openshift-default-75b5d5c9b8-zfr9t" event={"ID":"7a92f15b-fd83-4f83-be73-8e5249d0686a","Type":"ContainerStarted","Data":"b6c252e9e6eb12d104f6189d25627a1b7c4fef8634f53140657c802031fdd77a"}
Sep 11 01:19:49.426968 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:19:49.426887    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-2482ac-openshift-default-75b5d5c9b8-zfr9t"
Sep 11 01:19:49.428219 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:19:49.428185    2569 patch_prober.go:28] interesting pod/e2e-shared-a-w3-2482ac-openshift-default-75b5d5c9b8-zfr9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body=
Sep 11 01:19:49.428356 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:19:49.428248    2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-2482ac-openshift-default-75b5d5c9b8-zfr9t" podUID="7a92f15b-fd83-4f83-be73-8e5249d0686a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused"
Sep 11 01:19:50.426707 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:19:50.426671    2569 patch_prober.go:28] interesting pod/e2e-shared-a-w3-2482ac-openshift-default-75b5d5c9b8-zfr9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body=
Sep 11 01:19:50.427096 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:19:50.426724    2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-2482ac-openshift-default-75b5d5c9b8-zfr9t" podUID="7a92f15b-fd83-4f83-be73-8e5249d0686a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused"
Sep 11 01:19:50.895828 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:19:50.895788    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-2482ac-openshift-default-75b5d5c9b8-zfr9t"
Sep 11 01:19:51.426315 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:19:51.426273    2569 patch_prober.go:28] interesting pod/e2e-shared-a-w3-2482ac-openshift-default-75b5d5c9b8-zfr9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body=
Sep 11 01:19:51.426522 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:19:51.426340    2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-2482ac-openshift-default-75b5d5c9b8-zfr9t" podUID="7a92f15b-fd83-4f83-be73-8e5249d0686a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused"
Sep 11 01:19:52.425774 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:19:52.425740    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-9433b9-6b64d44946-bt9h7"]
Sep 11 01:19:52.426553 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:19:52.426530    2569 patch_prober.go:28] interesting pod/e2e-shared-a-w3-2482ac-openshift-default-75b5d5c9b8-zfr9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body=
Sep 11 01:19:52.426626 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:19:52.426583    2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-2482ac-openshift-default-75b5d5c9b8-zfr9t" podUID="7a92f15b-fd83-4f83-be73-8e5249d0686a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused"
Sep 11 01:19:52.429978 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:19:52.429959    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-9433b9-6b64d44946-bt9h7"
Sep 11 01:19:52.432451 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:19:52.432432    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-9433b9\""
Sep 11 01:19:52.453894 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:19:52.448487    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-9433b9-6b64d44946-bt9h7"]
Sep 11 01:19:52.460893 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:19:52.455782    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-9433b9-6746686dd5-5b2nd"]
Sep 11 01:19:52.462818 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:19:52.462799    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-9433b9-6746686dd5-5b2nd"
Sep 11 01:19:52.468154 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:19:52.468132    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-9433b9-dockercfg-tl6xr\""
Sep 11 01:19:52.473214 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:19:52.473190    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-9433b9-6746686dd5-5b2nd"]
Sep 11 01:19:52.519951 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:19:52.519921    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84658a03-47b0-4f32-8385-4d36f59cdd8b-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-9433b9-6746686dd5-5b2nd\" (UID: \"84658a03-47b0-4f32-8385-4d36f59cdd8b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-9433b9-6746686dd5-5b2nd"
Sep 11 01:19:52.520102 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:19:52.520015    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gkqxl\" (UniqueName: \"kubernetes.io/projected/84658a03-47b0-4f32-8385-4d36f59cdd8b-kube-api-access-gkqxl\") pod \"payload-processing-e2e-shared-b-w3-9433b9-6746686dd5-5b2nd\" (UID: \"84658a03-47b0-4f32-8385-4d36f59cdd8b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-9433b9-6746686dd5-5b2nd"
Sep 11 01:19:52.520102 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:19:52.520055    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkw5k\" (UniqueName: \"kubernetes.io/projected/aad0fca5-b959-47f7-8a46-a8da24458e5b-kube-api-access-kkw5k\") pod \"payload-pre-processing-e2e-shared-b-w3-9433b9-6b64d44946-bt9h7\" (UID: \"aad0fca5-b959-47f7-8a46-a8da24458e5b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-9433b9-6b64d44946-bt9h7"
Sep 11 01:19:52.520210 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:19:52.520122    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aad0fca5-b959-47f7-8a46-a8da24458e5b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-9433b9-6b64d44946-bt9h7\" (UID: \"aad0fca5-b959-47f7-8a46-a8da24458e5b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-9433b9-6b64d44946-bt9h7"
Sep 11 01:19:52.621642 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:19:52.621603    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aad0fca5-b959-47f7-8a46-a8da24458e5b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-9433b9-6b64d44946-bt9h7\" (UID: \"aad0fca5-b959-47f7-8a46-a8da24458e5b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-9433b9-6b64d44946-bt9h7"
Sep 11 01:19:52.621935 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:19:52.621684    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84658a03-47b0-4f32-8385-4d36f59cdd8b-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-9433b9-6746686dd5-5b2nd\" (UID: \"84658a03-47b0-4f32-8385-4d36f59cdd8b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-9433b9-6746686dd5-5b2nd"
Sep 11 01:19:52.621935 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:19:52.621742    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkqxl\" (UniqueName: \"kubernetes.io/projected/84658a03-47b0-4f32-8385-4d36f59cdd8b-kube-api-access-gkqxl\") pod \"payload-processing-e2e-shared-b-w3-9433b9-6746686dd5-5b2nd\" (UID: \"84658a03-47b0-4f32-8385-4d36f59cdd8b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-9433b9-6746686dd5-5b2nd"
Sep 11 01:19:52.621935 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:19:52.621764    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkw5k\" (UniqueName: \"kubernetes.io/projected/aad0fca5-b959-47f7-8a46-a8da24458e5b-kube-api-access-kkw5k\") pod \"payload-pre-processing-e2e-shared-b-w3-9433b9-6b64d44946-bt9h7\" (UID: \"aad0fca5-b959-47f7-8a46-a8da24458e5b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-9433b9-6b64d44946-bt9h7"
Sep 11 01:19:52.622446 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:19:52.622423    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84658a03-47b0-4f32-8385-4d36f59cdd8b-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-9433b9-6746686dd5-5b2nd\" (UID: \"84658a03-47b0-4f32-8385-4d36f59cdd8b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-9433b9-6746686dd5-5b2nd"
Sep 11 01:19:52.622531 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:19:52.622508    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aad0fca5-b959-47f7-8a46-a8da24458e5b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-9433b9-6b64d44946-bt9h7\" (UID: \"aad0fca5-b959-47f7-8a46-a8da24458e5b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-9433b9-6b64d44946-bt9h7"
Sep 11 01:19:52.629669 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:19:52.629645    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkw5k\" (UniqueName: \"kubernetes.io/projected/aad0fca5-b959-47f7-8a46-a8da24458e5b-kube-api-access-kkw5k\") pod \"payload-pre-processing-e2e-shared-b-w3-9433b9-6b64d44946-bt9h7\" (UID: \"aad0fca5-b959-47f7-8a46-a8da24458e5b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-9433b9-6b64d44946-bt9h7"
Sep 11 01:19:52.629780 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:19:52.629711    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkqxl\" (UniqueName: \"kubernetes.io/projected/84658a03-47b0-4f32-8385-4d36f59cdd8b-kube-api-access-gkqxl\") pod \"payload-processing-e2e-shared-b-w3-9433b9-6746686dd5-5b2nd\" (UID: \"84658a03-47b0-4f32-8385-4d36f59cdd8b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-9433b9-6746686dd5-5b2nd"
Sep 11 01:19:52.749298 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:19:52.749207    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-9433b9-6b64d44946-bt9h7"
Sep 11 01:19:52.779288 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:19:52.779255    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-9433b9-6746686dd5-5b2nd"
Sep 11 01:19:53.028040 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:19:53.028016    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-9433b9-6b64d44946-bt9h7"]
Sep 11 01:19:53.030138 ip-10-0-129-222 kubenswrapper[2569]: W0911 01:19:53.030107    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaad0fca5_b959_47f7_8a46_a8da24458e5b.slice/crio-43f412cef83b5675855da539f1a3955caba11ed3c5a50853004a9c210a84f5f2 WatchSource:0}: Error finding container 43f412cef83b5675855da539f1a3955caba11ed3c5a50853004a9c210a84f5f2: Status 404 returned error can't find the container with id 43f412cef83b5675855da539f1a3955caba11ed3c5a50853004a9c210a84f5f2
Sep 11 01:19:53.088545 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:19:53.088518    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-9433b9-6746686dd5-5b2nd"]
Sep 11 01:19:53.089903 ip-10-0-129-222 kubenswrapper[2569]: W0911 01:19:53.089821    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84658a03_47b0_4f32_8385_4d36f59cdd8b.slice/crio-fb4061f87fda557eb0bec2b26dc444fe7ba2e3607b2a481e9e2f8f227d14211d WatchSource:0}: Error finding container fb4061f87fda557eb0bec2b26dc444fe7ba2e3607b2a481e9e2f8f227d14211d: Status 404 returned error can't find the container with id fb4061f87fda557eb0bec2b26dc444fe7ba2e3607b2a481e9e2f8f227d14211d
Sep 11 01:19:53.101655 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:19:53.101629    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-9433b9-6746686dd5-5b2nd" event={"ID":"84658a03-47b0-4f32-8385-4d36f59cdd8b","Type":"ContainerStarted","Data":"fb4061f87fda557eb0bec2b26dc444fe7ba2e3607b2a481e9e2f8f227d14211d"}
Sep 11 01:19:53.102946 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:19:53.102917    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-9433b9-6b64d44946-bt9h7" event={"ID":"aad0fca5-b959-47f7-8a46-a8da24458e5b","Type":"ContainerStarted","Data":"43f412cef83b5675855da539f1a3955caba11ed3c5a50853004a9c210a84f5f2"}
Sep 11 01:19:53.426518 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:19:53.426483    2569 patch_prober.go:28] interesting pod/e2e-shared-a-w3-2482ac-openshift-default-75b5d5c9b8-zfr9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body=
Sep 11 01:19:53.426999 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:19:53.426545    2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-2482ac-openshift-default-75b5d5c9b8-zfr9t" podUID="7a92f15b-fd83-4f83-be73-8e5249d0686a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused"
Sep 11 01:19:54.110579 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:19:54.110542    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-9433b9-6b64d44946-bt9h7" event={"ID":"aad0fca5-b959-47f7-8a46-a8da24458e5b","Type":"ContainerStarted","Data":"b8cd8c93b2bc99229b9609d91754d265caa91371990e60d1248e73e4ad2faf97"}
Sep 11 01:19:54.110782 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:19:54.110748    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-9433b9-6b64d44946-bt9h7"
Sep 11 01:19:54.112303 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:19:54.112278    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-9433b9-6746686dd5-5b2nd" event={"ID":"84658a03-47b0-4f32-8385-4d36f59cdd8b","Type":"ContainerStarted","Data":"89b101a5339ff779368dc4d8ed3ccf1dee06a55070d3cc0014535b90e6cbe93f"}
Sep 11 01:19:54.112501 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:19:54.112480    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-9433b9-6746686dd5-5b2nd"
Sep 11 01:19:54.127319 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:19:54.127275    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-9433b9-6b64d44946-bt9h7" podStartSLOduration=2.127260981 podStartE2EDuration="2.127260981s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 01:19:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 01:19:54.126764482 +0000 UTC m=+1328.778138732" watchObservedRunningTime="2026-09-11 01:19:54.127260981 +0000 UTC m=+1328.778635229"
Sep 11 01:19:54.143801 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:19:54.143745    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-9433b9-6746686dd5-5b2nd" podStartSLOduration=2.143725915 podStartE2EDuration="2.143725915s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 01:19:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 01:19:54.142462029 +0000 UTC m=+1328.793836282" watchObservedRunningTime="2026-09-11 01:19:54.143725915 +0000 UTC m=+1328.795100175"
Sep 11 01:19:54.430741 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:19:54.430663    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-2482ac-openshift-default-75b5d5c9b8-zfr9t"
Sep 11 01:19:54.431671 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:19:54.431650    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-2482ac-openshift-default-75b5d5c9b8-zfr9t"
Sep 11 01:19:58.303627 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:19:58.303592    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kmx5x"]
Sep 11 01:20:05.120675 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:20:05.120616    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-9433b9-6746686dd5-5b2nd"
Sep 11 01:20:05.121053 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:20:05.120718    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-9433b9-6b64d44946-bt9h7"
Sep 11 01:20:14.214198 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:20:14.214123    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-194eb64c-t7m45"]
Sep 11 01:20:14.219056 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:20:14.219038    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-194eb64c-t7m45"
Sep 11 01:20:14.221702 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:20:14.221672    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pppkt\""
Sep 11 01:20:14.222041 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:20:14.221955    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 01:20:14.228764 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:20:14.228734    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-194eb64c-t7m45"]
Sep 11 01:20:14.345610 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:20:14.345577    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1cd8195f-e9f2-43d2-8e30-ac46e9b5afbb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-194eb64c-t7m45\" (UID: \"1cd8195f-e9f2-43d2-8e30-ac46e9b5afbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-194eb64c-t7m45"
Sep 11 01:20:14.446180 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:20:14.446145    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1cd8195f-e9f2-43d2-8e30-ac46e9b5afbb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-194eb64c-t7m45\" (UID: \"1cd8195f-e9f2-43d2-8e30-ac46e9b5afbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-194eb64c-t7m45"
Sep 11 01:20:14.446773 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:20:14.446752    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1cd8195f-e9f2-43d2-8e30-ac46e9b5afbb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-194eb64c-t7m45\" (UID: \"1cd8195f-e9f2-43d2-8e30-ac46e9b5afbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-194eb64c-t7m45"
Sep 11 01:20:14.543126 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:20:14.543052    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-194eb64c-t7m45"
Sep 11 01:20:14.812686 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:20:14.812657    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-194eb64c-t7m45"]
Sep 11 01:20:14.814125 ip-10-0-129-222 kubenswrapper[2569]: W0911 01:20:14.814100    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1cd8195f_e9f2_43d2_8e30_ac46e9b5afbb.slice/crio-d30e11b2be66730ccadb965202275f6289b1c205469a9d100001769b0450f5d9 WatchSource:0}: Error finding container d30e11b2be66730ccadb965202275f6289b1c205469a9d100001769b0450f5d9: Status 404 returned error can't find the container with id d30e11b2be66730ccadb965202275f6289b1c205469a9d100001769b0450f5d9
Sep 11 01:20:15.237083 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:20:15.237052    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-194eb64c-t7m45" event={"ID":"1cd8195f-e9f2-43d2-8e30-ac46e9b5afbb","Type":"ContainerStarted","Data":"f2c62342ad8252c6a21fd124a72ed103d3e77bc8396084717fd0bc663cbf5cfa"}
Sep 11 01:20:15.237488 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:20:15.237097    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-194eb64c-t7m45" event={"ID":"1cd8195f-e9f2-43d2-8e30-ac46e9b5afbb","Type":"ContainerStarted","Data":"d30e11b2be66730ccadb965202275f6289b1c205469a9d100001769b0450f5d9"}
Sep 11 01:20:15.252322 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:20:15.252279    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-194eb64c-t7m45" podStartSLOduration=1.252260993 podStartE2EDuration="1.252260993s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 01:20:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 01:20:15.250106994 +0000 UTC m=+1349.901481245" watchObservedRunningTime="2026-09-11 01:20:15.252260993 +0000 UTC m=+1349.903635244"
Sep 11 01:20:16.244108 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:20:16.244071    2569 generic.go:356] "Generic (PLEG): container finished" podID="1cd8195f-e9f2-43d2-8e30-ac46e9b5afbb" containerID="f2c62342ad8252c6a21fd124a72ed103d3e77bc8396084717fd0bc663cbf5cfa" exitCode=0
Sep 11 01:20:16.244522 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:20:16.244153    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-194eb64c-t7m45" event={"ID":"1cd8195f-e9f2-43d2-8e30-ac46e9b5afbb","Type":"ContainerDied","Data":"f2c62342ad8252c6a21fd124a72ed103d3e77bc8396084717fd0bc663cbf5cfa"}
Sep 11 01:20:17.389469 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:20:17.389443    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-194eb64c-t7m45"
Sep 11 01:20:17.474811 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:20:17.474779    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1cd8195f-e9f2-43d2-8e30-ac46e9b5afbb-maas-api-service-ca\") pod \"1cd8195f-e9f2-43d2-8e30-ac46e9b5afbb\" (UID: \"1cd8195f-e9f2-43d2-8e30-ac46e9b5afbb\") "
Sep 11 01:20:17.475206 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:20:17.475186    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1cd8195f-e9f2-43d2-8e30-ac46e9b5afbb-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1cd8195f-e9f2-43d2-8e30-ac46e9b5afbb" (UID: "1cd8195f-e9f2-43d2-8e30-ac46e9b5afbb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 01:20:17.576584 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:20:17.576555    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1cd8195f-e9f2-43d2-8e30-ac46e9b5afbb-maas-api-service-ca\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:20:18.256657 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:20:18.256579    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-194eb64c-t7m45"
Sep 11 01:20:18.256657 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:20:18.256616    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-194eb64c-t7m45" event={"ID":"1cd8195f-e9f2-43d2-8e30-ac46e9b5afbb","Type":"ContainerDied","Data":"d30e11b2be66730ccadb965202275f6289b1c205469a9d100001769b0450f5d9"}
Sep 11 01:20:18.256826 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:20:18.256664    2569 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d30e11b2be66730ccadb965202275f6289b1c205469a9d100001769b0450f5d9"
Sep 11 01:20:22.904674 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:20:22.904641    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kmx5x"]
Sep 11 01:20:24.561054 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:20:24.561015    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-194eb64c-7f5bc6c478-fhdj6"]
Sep 11 01:20:24.561412 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:20:24.561285    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-194eb64c-7f5bc6c478-fhdj6" podUID="3bccd1bc-710f-45ce-a974-985963822797" containerName="payload-pre-processing" containerID="cri-o://9d050b8a352d601182b0fb391d8406a08a69282d85e5ac345160d33653f87c1c" gracePeriod=30
Sep 11 01:20:24.718225 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:20:24.718193    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-194eb64c-t7m45"]
Sep 11 01:20:24.720487 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:20:24.720465    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-194eb64c-t7m45"]
Sep 11 01:20:24.828174 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:20:24.828148    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-194eb64c-7f5bc6c478-fhdj6"
Sep 11 01:20:24.946060 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:20:24.946017    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3bccd1bc-710f-45ce-a974-985963822797-plugins-config-volume\") pod \"3bccd1bc-710f-45ce-a974-985963822797\" (UID: \"3bccd1bc-710f-45ce-a974-985963822797\") "
Sep 11 01:20:24.946263 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:20:24.946090    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6qnc8\" (UniqueName: \"kubernetes.io/projected/3bccd1bc-710f-45ce-a974-985963822797-kube-api-access-6qnc8\") pod \"3bccd1bc-710f-45ce-a974-985963822797\" (UID: \"3bccd1bc-710f-45ce-a974-985963822797\") "
Sep 11 01:20:24.946452 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:20:24.946426    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3bccd1bc-710f-45ce-a974-985963822797-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3bccd1bc-710f-45ce-a974-985963822797" (UID: "3bccd1bc-710f-45ce-a974-985963822797"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 01:20:24.948175 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:20:24.948147    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bccd1bc-710f-45ce-a974-985963822797-kube-api-access-6qnc8" (OuterVolumeSpecName: "kube-api-access-6qnc8") pod "3bccd1bc-710f-45ce-a974-985963822797" (UID: "3bccd1bc-710f-45ce-a974-985963822797"). InnerVolumeSpecName "kube-api-access-6qnc8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 01:20:25.047093 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:20:25.047042    2569 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3bccd1bc-710f-45ce-a974-985963822797-plugins-config-volume\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:20:25.047093 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:20:25.047082    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6qnc8\" (UniqueName: \"kubernetes.io/projected/3bccd1bc-710f-45ce-a974-985963822797-kube-api-access-6qnc8\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:20:25.299684 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:20:25.299595    2569 generic.go:356] "Generic (PLEG): container finished" podID="3bccd1bc-710f-45ce-a974-985963822797" containerID="9d050b8a352d601182b0fb391d8406a08a69282d85e5ac345160d33653f87c1c" exitCode=0
Sep 11 01:20:25.299684 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:20:25.299639    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-194eb64c-7f5bc6c478-fhdj6"
Sep 11 01:20:25.299864 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:20:25.299680    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-194eb64c-7f5bc6c478-fhdj6" event={"ID":"3bccd1bc-710f-45ce-a974-985963822797","Type":"ContainerDied","Data":"9d050b8a352d601182b0fb391d8406a08a69282d85e5ac345160d33653f87c1c"}
Sep 11 01:20:25.299864 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:20:25.299727    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-194eb64c-7f5bc6c478-fhdj6" event={"ID":"3bccd1bc-710f-45ce-a974-985963822797","Type":"ContainerDied","Data":"5b93f30edb2a8a46e17eeb309d31ce506cf850edd31518186abc3fe41c1e8a40"}
Sep 11 01:20:25.299864 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:20:25.299748    2569 scope.go:117] "RemoveContainer" containerID="9d050b8a352d601182b0fb391d8406a08a69282d85e5ac345160d33653f87c1c"
Sep 11 01:20:25.309384 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:20:25.309366    2569 scope.go:117] "RemoveContainer" containerID="9d050b8a352d601182b0fb391d8406a08a69282d85e5ac345160d33653f87c1c"
Sep 11 01:20:25.309691 ip-10-0-129-222 kubenswrapper[2569]: E0911 01:20:25.309671    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d050b8a352d601182b0fb391d8406a08a69282d85e5ac345160d33653f87c1c\": container with ID starting with 9d050b8a352d601182b0fb391d8406a08a69282d85e5ac345160d33653f87c1c not found: ID does not exist" containerID="9d050b8a352d601182b0fb391d8406a08a69282d85e5ac345160d33653f87c1c"
Sep 11 01:20:25.309768 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:20:25.309702    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d050b8a352d601182b0fb391d8406a08a69282d85e5ac345160d33653f87c1c"} err="failed to get container status \"9d050b8a352d601182b0fb391d8406a08a69282d85e5ac345160d33653f87c1c\": rpc error: code = NotFound desc = could not find container \"9d050b8a352d601182b0fb391d8406a08a69282d85e5ac345160d33653f87c1c\": container with ID starting with 9d050b8a352d601182b0fb391d8406a08a69282d85e5ac345160d33653f87c1c not found: ID does not exist"
Sep 11 01:20:25.322496 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:20:25.322465    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-194eb64c-7f5bc6c478-fhdj6"]
Sep 11 01:20:25.326762 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:20:25.326740    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-194eb64c-7f5bc6c478-fhdj6"]
Sep 11 01:20:25.873015 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:20:25.872988    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1cd8195f-e9f2-43d2-8e30-ac46e9b5afbb" path="/var/lib/kubelet/pods/1cd8195f-e9f2-43d2-8e30-ac46e9b5afbb/volumes"
Sep 11 01:20:25.873373 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:20:25.873346    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3bccd1bc-710f-45ce-a974-985963822797" path="/var/lib/kubelet/pods/3bccd1bc-710f-45ce-a974-985963822797/volumes"
Sep 11 01:20:31.077740 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:20:31.077705    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-194eb64c-openshift-default-779f4656cb-lcpcg"]
Sep 11 01:20:31.078187 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:20:31.077920    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-194eb64c-openshift-default-779f4656cb-lcpcg" podUID="41517eb2-ec7f-488c-86b9-fc70f16bcd3d" containerName="istio-proxy" containerID="cri-o://68d0baa36cb739257ed1f28827c5737ac1b884a9f72ea4ff247da6b54b0eed1e" gracePeriod=30
Sep 11 01:20:36.348056 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:20:36.347998    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-194eb64c-openshift-default-779f4656cb-lcpcg"
Sep 11 01:20:36.364671 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:20:36.364646    2569 generic.go:356] "Generic (PLEG): container finished" podID="41517eb2-ec7f-488c-86b9-fc70f16bcd3d" containerID="68d0baa36cb739257ed1f28827c5737ac1b884a9f72ea4ff247da6b54b0eed1e" exitCode=0
Sep 11 01:20:36.364797 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:20:36.364686    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-194eb64c-openshift-default-779f4656cb-lcpcg"
Sep 11 01:20:36.364797 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:20:36.364732    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-194eb64c-openshift-default-779f4656cb-lcpcg" event={"ID":"41517eb2-ec7f-488c-86b9-fc70f16bcd3d","Type":"ContainerDied","Data":"68d0baa36cb739257ed1f28827c5737ac1b884a9f72ea4ff247da6b54b0eed1e"}
Sep 11 01:20:36.364797 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:20:36.364783    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-194eb64c-openshift-default-779f4656cb-lcpcg" event={"ID":"41517eb2-ec7f-488c-86b9-fc70f16bcd3d","Type":"ContainerDied","Data":"cb914d91a4fbf94561520490fa962226e737913ab9270d170953942a6bbbe698"}
Sep 11 01:20:36.364797 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:20:36.364798    2569 scope.go:117] "RemoveContainer" containerID="68d0baa36cb739257ed1f28827c5737ac1b884a9f72ea4ff247da6b54b0eed1e"
Sep 11 01:20:36.376725 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:20:36.376704    2569 scope.go:117] "RemoveContainer" containerID="03a963847114368a0ce55ad6877cdc12878a8ebaf8b16218389dfd17006b4e3b"
Sep 11 01:20:36.385231 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:20:36.385213    2569 scope.go:117] "RemoveContainer" containerID="68d0baa36cb739257ed1f28827c5737ac1b884a9f72ea4ff247da6b54b0eed1e"
Sep 11 01:20:36.385549 ip-10-0-129-222 kubenswrapper[2569]: E0911 01:20:36.385530    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68d0baa36cb739257ed1f28827c5737ac1b884a9f72ea4ff247da6b54b0eed1e\": container with ID starting with 68d0baa36cb739257ed1f28827c5737ac1b884a9f72ea4ff247da6b54b0eed1e not found: ID does not exist" containerID="68d0baa36cb739257ed1f28827c5737ac1b884a9f72ea4ff247da6b54b0eed1e"
Sep 11 01:20:36.385593 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:20:36.385556    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68d0baa36cb739257ed1f28827c5737ac1b884a9f72ea4ff247da6b54b0eed1e"} err="failed to get container status \"68d0baa36cb739257ed1f28827c5737ac1b884a9f72ea4ff247da6b54b0eed1e\": rpc error: code = NotFound desc = could not find container \"68d0baa36cb739257ed1f28827c5737ac1b884a9f72ea4ff247da6b54b0eed1e\": container with ID starting with 68d0baa36cb739257ed1f28827c5737ac1b884a9f72ea4ff247da6b54b0eed1e not found: ID does not exist"
Sep 11 01:20:36.385593 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:20:36.385574    2569 scope.go:117] "RemoveContainer" containerID="03a963847114368a0ce55ad6877cdc12878a8ebaf8b16218389dfd17006b4e3b"
Sep 11 01:20:36.385809 ip-10-0-129-222 kubenswrapper[2569]: E0911 01:20:36.385794    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03a963847114368a0ce55ad6877cdc12878a8ebaf8b16218389dfd17006b4e3b\": container with ID starting with 03a963847114368a0ce55ad6877cdc12878a8ebaf8b16218389dfd17006b4e3b not found: ID does not exist" containerID="03a963847114368a0ce55ad6877cdc12878a8ebaf8b16218389dfd17006b4e3b"
Sep 11 01:20:36.385851 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:20:36.385811    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03a963847114368a0ce55ad6877cdc12878a8ebaf8b16218389dfd17006b4e3b"} err="failed to get container status \"03a963847114368a0ce55ad6877cdc12878a8ebaf8b16218389dfd17006b4e3b\": rpc error: code = NotFound desc = could not find container \"03a963847114368a0ce55ad6877cdc12878a8ebaf8b16218389dfd17006b4e3b\": container with ID starting with 03a963847114368a0ce55ad6877cdc12878a8ebaf8b16218389dfd17006b4e3b not found: ID does not exist"
Sep 11 01:20:36.460988 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:20:36.460963    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41517eb2-ec7f-488c-86b9-fc70f16bcd3d-istio-data\") pod \"41517eb2-ec7f-488c-86b9-fc70f16bcd3d\" (UID: \"41517eb2-ec7f-488c-86b9-fc70f16bcd3d\") "
Sep 11 01:20:36.460988 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:20:36.460991    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41517eb2-ec7f-488c-86b9-fc70f16bcd3d-credential-socket\") pod \"41517eb2-ec7f-488c-86b9-fc70f16bcd3d\" (UID: \"41517eb2-ec7f-488c-86b9-fc70f16bcd3d\") "
Sep 11 01:20:36.461178 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:20:36.461032    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41517eb2-ec7f-488c-86b9-fc70f16bcd3d-istio-token\") pod \"41517eb2-ec7f-488c-86b9-fc70f16bcd3d\" (UID: \"41517eb2-ec7f-488c-86b9-fc70f16bcd3d\") "
Sep 11 01:20:36.461178 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:20:36.461107    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41517eb2-ec7f-488c-86b9-fc70f16bcd3d-workload-certs\") pod \"41517eb2-ec7f-488c-86b9-fc70f16bcd3d\" (UID: \"41517eb2-ec7f-488c-86b9-fc70f16bcd3d\") "
Sep 11 01:20:36.461178 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:20:36.461135    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5kw7f\" (UniqueName: \"kubernetes.io/projected/41517eb2-ec7f-488c-86b9-fc70f16bcd3d-kube-api-access-5kw7f\") pod \"41517eb2-ec7f-488c-86b9-fc70f16bcd3d\" (UID: \"41517eb2-ec7f-488c-86b9-fc70f16bcd3d\") "
Sep 11 01:20:36.461342 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:20:36.461193    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41517eb2-ec7f-488c-86b9-fc70f16bcd3d-istiod-ca-cert\") pod \"41517eb2-ec7f-488c-86b9-fc70f16bcd3d\" (UID: \"41517eb2-ec7f-488c-86b9-fc70f16bcd3d\") "
Sep 11 01:20:36.461342 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:20:36.461221    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41517eb2-ec7f-488c-86b9-fc70f16bcd3d-istio-data" (OuterVolumeSpecName: "istio-data") pod "41517eb2-ec7f-488c-86b9-fc70f16bcd3d" (UID: "41517eb2-ec7f-488c-86b9-fc70f16bcd3d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 01:20:36.461342 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:20:36.461263    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41517eb2-ec7f-488c-86b9-fc70f16bcd3d-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "41517eb2-ec7f-488c-86b9-fc70f16bcd3d" (UID: "41517eb2-ec7f-488c-86b9-fc70f16bcd3d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 01:20:36.461342 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:20:36.461278    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41517eb2-ec7f-488c-86b9-fc70f16bcd3d-istio-podinfo\") pod \"41517eb2-ec7f-488c-86b9-fc70f16bcd3d\" (UID: \"41517eb2-ec7f-488c-86b9-fc70f16bcd3d\") "
Sep 11 01:20:36.461342 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:20:36.461318    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41517eb2-ec7f-488c-86b9-fc70f16bcd3d-workload-socket\") pod \"41517eb2-ec7f-488c-86b9-fc70f16bcd3d\" (UID: \"41517eb2-ec7f-488c-86b9-fc70f16bcd3d\") "
Sep 11 01:20:36.461605 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:20:36.461354    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41517eb2-ec7f-488c-86b9-fc70f16bcd3d-istio-envoy\") pod \"41517eb2-ec7f-488c-86b9-fc70f16bcd3d\" (UID: \"41517eb2-ec7f-488c-86b9-fc70f16bcd3d\") "
Sep 11 01:20:36.461605 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:20:36.461383    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41517eb2-ec7f-488c-86b9-fc70f16bcd3d-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "41517eb2-ec7f-488c-86b9-fc70f16bcd3d" (UID: "41517eb2-ec7f-488c-86b9-fc70f16bcd3d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 01:20:36.461605 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:20:36.461591    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41517eb2-ec7f-488c-86b9-fc70f16bcd3d-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "41517eb2-ec7f-488c-86b9-fc70f16bcd3d" (UID: "41517eb2-ec7f-488c-86b9-fc70f16bcd3d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 01:20:36.461782 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:20:36.461605    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41517eb2-ec7f-488c-86b9-fc70f16bcd3d-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "41517eb2-ec7f-488c-86b9-fc70f16bcd3d" (UID: "41517eb2-ec7f-488c-86b9-fc70f16bcd3d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 01:20:36.461782 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:20:36.461743    2569 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41517eb2-ec7f-488c-86b9-fc70f16bcd3d-workload-certs\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:20:36.461782 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:20:36.461756    2569 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41517eb2-ec7f-488c-86b9-fc70f16bcd3d-istiod-ca-cert\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:20:36.461782 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:20:36.461764    2569 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41517eb2-ec7f-488c-86b9-fc70f16bcd3d-workload-socket\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:20:36.461782 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:20:36.461773    2569 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41517eb2-ec7f-488c-86b9-fc70f16bcd3d-istio-data\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:20:36.462075 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:20:36.461785    2569 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41517eb2-ec7f-488c-86b9-fc70f16bcd3d-credential-socket\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:20:36.463464 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:20:36.463437    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/41517eb2-ec7f-488c-86b9-fc70f16bcd3d-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "41517eb2-ec7f-488c-86b9-fc70f16bcd3d" (UID: "41517eb2-ec7f-488c-86b9-fc70f16bcd3d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 01:20:36.463464 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:20:36.463446    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41517eb2-ec7f-488c-86b9-fc70f16bcd3d-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "41517eb2-ec7f-488c-86b9-fc70f16bcd3d" (UID: "41517eb2-ec7f-488c-86b9-fc70f16bcd3d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 01:20:36.463592 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:20:36.463438    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41517eb2-ec7f-488c-86b9-fc70f16bcd3d-istio-token" (OuterVolumeSpecName: "istio-token") pod "41517eb2-ec7f-488c-86b9-fc70f16bcd3d" (UID: "41517eb2-ec7f-488c-86b9-fc70f16bcd3d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 01:20:36.463592 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:20:36.463450    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41517eb2-ec7f-488c-86b9-fc70f16bcd3d-kube-api-access-5kw7f" (OuterVolumeSpecName: "kube-api-access-5kw7f") pod "41517eb2-ec7f-488c-86b9-fc70f16bcd3d" (UID: "41517eb2-ec7f-488c-86b9-fc70f16bcd3d"). InnerVolumeSpecName "kube-api-access-5kw7f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 01:20:36.563098 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:20:36.563067    2569 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41517eb2-ec7f-488c-86b9-fc70f16bcd3d-istio-token\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:20:36.563098 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:20:36.563095    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5kw7f\" (UniqueName: \"kubernetes.io/projected/41517eb2-ec7f-488c-86b9-fc70f16bcd3d-kube-api-access-5kw7f\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:20:36.563098 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:20:36.563105    2569 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41517eb2-ec7f-488c-86b9-fc70f16bcd3d-istio-podinfo\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:20:36.563305 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:20:36.563114    2569 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41517eb2-ec7f-488c-86b9-fc70f16bcd3d-istio-envoy\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:20:36.686563 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:20:36.686535    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-194eb64c-openshift-default-779f4656cb-lcpcg"]
Sep 11 01:20:36.692513 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:20:36.692489    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-194eb64c-openshift-default-779f4656cb-lcpcg"]
Sep 11 01:20:37.871361 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:20:37.871326    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="41517eb2-ec7f-488c-86b9-fc70f16bcd3d" path="/var/lib/kubelet/pods/41517eb2-ec7f-488c-86b9-fc70f16bcd3d/volumes"
Sep 11 01:20:46.110319 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:20:46.110286    2569 scope.go:117] "RemoveContainer" containerID="b822c221c1c92f53c142996033cba8f57ecf90ef6f60a2c02c83208dff3eb11e"
Sep 11 01:20:55.351167 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:20:55.351130    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kmx5x"]
Sep 11 01:21:17.698504 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:21:17.698469    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kmx5x"]
Sep 11 01:21:25.816542 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:21:25.816510    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-f1f91d-openshift-default-7b546d6c86-kmhkm"]
Sep 11 01:21:25.817021 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:21:25.817006    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3bccd1bc-710f-45ce-a974-985963822797" containerName="payload-pre-processing"
Sep 11 01:21:25.817080 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:21:25.817023    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="3bccd1bc-710f-45ce-a974-985963822797" containerName="payload-pre-processing"
Sep 11 01:21:25.817080 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:21:25.817036    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41517eb2-ec7f-488c-86b9-fc70f16bcd3d" containerName="istio-proxy"
Sep 11 01:21:25.817080 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:21:25.817043    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="41517eb2-ec7f-488c-86b9-fc70f16bcd3d" containerName="istio-proxy"
Sep 11 01:21:25.817080 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:21:25.817069    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1cd8195f-e9f2-43d2-8e30-ac46e9b5afbb" containerName="revoke-keys"
Sep 11 01:21:25.817080 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:21:25.817076    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="1cd8195f-e9f2-43d2-8e30-ac46e9b5afbb" containerName="revoke-keys"
Sep 11 01:21:25.817259 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:21:25.817087    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41517eb2-ec7f-488c-86b9-fc70f16bcd3d" containerName="istio-proxy"
Sep 11 01:21:25.817259 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:21:25.817093    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="41517eb2-ec7f-488c-86b9-fc70f16bcd3d" containerName="istio-proxy"
Sep 11 01:21:25.817259 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:21:25.817156    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="41517eb2-ec7f-488c-86b9-fc70f16bcd3d" containerName="istio-proxy"
Sep 11 01:21:25.817259 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:21:25.817166    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="41517eb2-ec7f-488c-86b9-fc70f16bcd3d" containerName="istio-proxy"
Sep 11 01:21:25.817259 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:21:25.817172    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="1cd8195f-e9f2-43d2-8e30-ac46e9b5afbb" containerName="revoke-keys"
Sep 11 01:21:25.817259 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:21:25.817182    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="3bccd1bc-710f-45ce-a974-985963822797" containerName="payload-pre-processing"
Sep 11 01:21:25.820552 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:21:25.820534    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f1f91d-openshift-default-7b546d6c86-kmhkm"
Sep 11 01:21:25.822913 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:21:25.822856    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-f1f91d-openshift-default-dockercfg-r2g5f\""
Sep 11 01:21:25.830706 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:21:25.830685    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-f1f91d-openshift-default-7b546d6c86-kmhkm"]
Sep 11 01:21:25.918463 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:21:25.918426    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5176249-c100-42e1-ab5b-9bbfcee0c6f2-istio-data\") pod \"e2e-api-a-f1f91d-openshift-default-7b546d6c86-kmhkm\" (UID: \"e5176249-c100-42e1-ab5b-9bbfcee0c6f2\") " pod="openshift-ingress/e2e-api-a-f1f91d-openshift-default-7b546d6c86-kmhkm"
Sep 11 01:21:25.918669 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:21:25.918557    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5176249-c100-42e1-ab5b-9bbfcee0c6f2-istio-envoy\") pod \"e2e-api-a-f1f91d-openshift-default-7b546d6c86-kmhkm\" (UID: \"e5176249-c100-42e1-ab5b-9bbfcee0c6f2\") " pod="openshift-ingress/e2e-api-a-f1f91d-openshift-default-7b546d6c86-kmhkm"
Sep 11 01:21:25.918669 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:21:25.918625    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5176249-c100-42e1-ab5b-9bbfcee0c6f2-credential-socket\") pod \"e2e-api-a-f1f91d-openshift-default-7b546d6c86-kmhkm\" (UID: \"e5176249-c100-42e1-ab5b-9bbfcee0c6f2\") " pod="openshift-ingress/e2e-api-a-f1f91d-openshift-default-7b546d6c86-kmhkm"
Sep 11 01:21:25.918777 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:21:25.918755    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5176249-c100-42e1-ab5b-9bbfcee0c6f2-workload-socket\") pod \"e2e-api-a-f1f91d-openshift-default-7b546d6c86-kmhkm\" (UID: \"e5176249-c100-42e1-ab5b-9bbfcee0c6f2\") " pod="openshift-ingress/e2e-api-a-f1f91d-openshift-default-7b546d6c86-kmhkm"
Sep 11 01:21:25.918842 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:21:25.918826    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-898lr\" (UniqueName: \"kubernetes.io/projected/e5176249-c100-42e1-ab5b-9bbfcee0c6f2-kube-api-access-898lr\") pod \"e2e-api-a-f1f91d-openshift-default-7b546d6c86-kmhkm\" (UID: \"e5176249-c100-42e1-ab5b-9bbfcee0c6f2\") " pod="openshift-ingress/e2e-api-a-f1f91d-openshift-default-7b546d6c86-kmhkm"
Sep 11 01:21:25.918915 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:21:25.918861    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5176249-c100-42e1-ab5b-9bbfcee0c6f2-workload-certs\") pod \"e2e-api-a-f1f91d-openshift-default-7b546d6c86-kmhkm\" (UID: \"e5176249-c100-42e1-ab5b-9bbfcee0c6f2\") " pod="openshift-ingress/e2e-api-a-f1f91d-openshift-default-7b546d6c86-kmhkm"
Sep 11 01:21:25.918991 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:21:25.918967    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5176249-c100-42e1-ab5b-9bbfcee0c6f2-istiod-ca-cert\") pod \"e2e-api-a-f1f91d-openshift-default-7b546d6c86-kmhkm\" (UID: \"e5176249-c100-42e1-ab5b-9bbfcee0c6f2\") " pod="openshift-ingress/e2e-api-a-f1f91d-openshift-default-7b546d6c86-kmhkm"
Sep 11 01:21:25.919042 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:21:25.919027    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5176249-c100-42e1-ab5b-9bbfcee0c6f2-istio-podinfo\") pod \"e2e-api-a-f1f91d-openshift-default-7b546d6c86-kmhkm\" (UID: \"e5176249-c100-42e1-ab5b-9bbfcee0c6f2\") " pod="openshift-ingress/e2e-api-a-f1f91d-openshift-default-7b546d6c86-kmhkm"
Sep 11 01:21:25.919101 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:21:25.919086    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5176249-c100-42e1-ab5b-9bbfcee0c6f2-istio-token\") pod \"e2e-api-a-f1f91d-openshift-default-7b546d6c86-kmhkm\" (UID: \"e5176249-c100-42e1-ab5b-9bbfcee0c6f2\") " pod="openshift-ingress/e2e-api-a-f1f91d-openshift-default-7b546d6c86-kmhkm"
Sep 11 01:21:26.019962 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:21:26.019930    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5176249-c100-42e1-ab5b-9bbfcee0c6f2-workload-socket\") pod \"e2e-api-a-f1f91d-openshift-default-7b546d6c86-kmhkm\" (UID: \"e5176249-c100-42e1-ab5b-9bbfcee0c6f2\") " pod="openshift-ingress/e2e-api-a-f1f91d-openshift-default-7b546d6c86-kmhkm"
Sep 11 01:21:26.020167 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:21:26.019977    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-898lr\" (UniqueName: \"kubernetes.io/projected/e5176249-c100-42e1-ab5b-9bbfcee0c6f2-kube-api-access-898lr\") pod \"e2e-api-a-f1f91d-openshift-default-7b546d6c86-kmhkm\" (UID: \"e5176249-c100-42e1-ab5b-9bbfcee0c6f2\") " pod="openshift-ingress/e2e-api-a-f1f91d-openshift-default-7b546d6c86-kmhkm"
Sep 11 01:21:26.020167 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:21:26.019998    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5176249-c100-42e1-ab5b-9bbfcee0c6f2-workload-certs\") pod \"e2e-api-a-f1f91d-openshift-default-7b546d6c86-kmhkm\" (UID: \"e5176249-c100-42e1-ab5b-9bbfcee0c6f2\") " pod="openshift-ingress/e2e-api-a-f1f91d-openshift-default-7b546d6c86-kmhkm"
Sep 11 01:21:26.020167 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:21:26.020036    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5176249-c100-42e1-ab5b-9bbfcee0c6f2-istiod-ca-cert\") pod \"e2e-api-a-f1f91d-openshift-default-7b546d6c86-kmhkm\" (UID: \"e5176249-c100-42e1-ab5b-9bbfcee0c6f2\") " pod="openshift-ingress/e2e-api-a-f1f91d-openshift-default-7b546d6c86-kmhkm"
Sep 11 01:21:26.020167 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:21:26.020074    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5176249-c100-42e1-ab5b-9bbfcee0c6f2-istio-podinfo\") pod \"e2e-api-a-f1f91d-openshift-default-7b546d6c86-kmhkm\" (UID: \"e5176249-c100-42e1-ab5b-9bbfcee0c6f2\") " pod="openshift-ingress/e2e-api-a-f1f91d-openshift-default-7b546d6c86-kmhkm"
Sep 11 01:21:26.020167 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:21:26.020114    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5176249-c100-42e1-ab5b-9bbfcee0c6f2-istio-token\") pod \"e2e-api-a-f1f91d-openshift-default-7b546d6c86-kmhkm\" (UID: \"e5176249-c100-42e1-ab5b-9bbfcee0c6f2\") " pod="openshift-ingress/e2e-api-a-f1f91d-openshift-default-7b546d6c86-kmhkm"
Sep 11 01:21:26.020456 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:21:26.020195    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5176249-c100-42e1-ab5b-9bbfcee0c6f2-istio-data\") pod \"e2e-api-a-f1f91d-openshift-default-7b546d6c86-kmhkm\" (UID: \"e5176249-c100-42e1-ab5b-9bbfcee0c6f2\") " pod="openshift-ingress/e2e-api-a-f1f91d-openshift-default-7b546d6c86-kmhkm"
Sep 11 01:21:26.020456 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:21:26.020279    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5176249-c100-42e1-ab5b-9bbfcee0c6f2-istio-envoy\") pod \"e2e-api-a-f1f91d-openshift-default-7b546d6c86-kmhkm\" (UID: \"e5176249-c100-42e1-ab5b-9bbfcee0c6f2\") " pod="openshift-ingress/e2e-api-a-f1f91d-openshift-default-7b546d6c86-kmhkm"
Sep 11 01:21:26.020456 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:21:26.020345    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5176249-c100-42e1-ab5b-9bbfcee0c6f2-credential-socket\") pod \"e2e-api-a-f1f91d-openshift-default-7b546d6c86-kmhkm\" (UID: \"e5176249-c100-42e1-ab5b-9bbfcee0c6f2\") " pod="openshift-ingress/e2e-api-a-f1f91d-openshift-default-7b546d6c86-kmhkm"
Sep 11 01:21:26.020456 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:21:26.020388    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5176249-c100-42e1-ab5b-9bbfcee0c6f2-workload-socket\") pod \"e2e-api-a-f1f91d-openshift-default-7b546d6c86-kmhkm\" (UID: \"e5176249-c100-42e1-ab5b-9bbfcee0c6f2\") " pod="openshift-ingress/e2e-api-a-f1f91d-openshift-default-7b546d6c86-kmhkm"
Sep 11 01:21:26.020671 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:21:26.020643    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5176249-c100-42e1-ab5b-9bbfcee0c6f2-istio-data\") pod \"e2e-api-a-f1f91d-openshift-default-7b546d6c86-kmhkm\" (UID: \"e5176249-c100-42e1-ab5b-9bbfcee0c6f2\") " pod="openshift-ingress/e2e-api-a-f1f91d-openshift-default-7b546d6c86-kmhkm"
Sep 11 01:21:26.020727 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:21:26.020687    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5176249-c100-42e1-ab5b-9bbfcee0c6f2-workload-certs\") pod \"e2e-api-a-f1f91d-openshift-default-7b546d6c86-kmhkm\" (UID: \"e5176249-c100-42e1-ab5b-9bbfcee0c6f2\") " pod="openshift-ingress/e2e-api-a-f1f91d-openshift-default-7b546d6c86-kmhkm"
Sep 11 01:21:26.020776 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:21:26.020738    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5176249-c100-42e1-ab5b-9bbfcee0c6f2-istiod-ca-cert\") pod \"e2e-api-a-f1f91d-openshift-default-7b546d6c86-kmhkm\" (UID: \"e5176249-c100-42e1-ab5b-9bbfcee0c6f2\") " pod="openshift-ingress/e2e-api-a-f1f91d-openshift-default-7b546d6c86-kmhkm"
Sep 11 01:21:26.020776 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:21:26.020756    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5176249-c100-42e1-ab5b-9bbfcee0c6f2-credential-socket\") pod \"e2e-api-a-f1f91d-openshift-default-7b546d6c86-kmhkm\" (UID: \"e5176249-c100-42e1-ab5b-9bbfcee0c6f2\") " pod="openshift-ingress/e2e-api-a-f1f91d-openshift-default-7b546d6c86-kmhkm"
Sep 11 01:21:26.022628 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:21:26.022603    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5176249-c100-42e1-ab5b-9bbfcee0c6f2-istio-envoy\") pod \"e2e-api-a-f1f91d-openshift-default-7b546d6c86-kmhkm\" (UID: \"e5176249-c100-42e1-ab5b-9bbfcee0c6f2\") " pod="openshift-ingress/e2e-api-a-f1f91d-openshift-default-7b546d6c86-kmhkm"
Sep 11 01:21:26.022734 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:21:26.022612    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5176249-c100-42e1-ab5b-9bbfcee0c6f2-istio-podinfo\") pod \"e2e-api-a-f1f91d-openshift-default-7b546d6c86-kmhkm\" (UID: \"e5176249-c100-42e1-ab5b-9bbfcee0c6f2\") " pod="openshift-ingress/e2e-api-a-f1f91d-openshift-default-7b546d6c86-kmhkm"
Sep 11 01:21:26.028252 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:21:26.028226    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5176249-c100-42e1-ab5b-9bbfcee0c6f2-istio-token\") pod \"e2e-api-a-f1f91d-openshift-default-7b546d6c86-kmhkm\" (UID: \"e5176249-c100-42e1-ab5b-9bbfcee0c6f2\") " pod="openshift-ingress/e2e-api-a-f1f91d-openshift-default-7b546d6c86-kmhkm"
Sep 11 01:21:26.028806 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:21:26.028776    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-898lr\" (UniqueName: \"kubernetes.io/projected/e5176249-c100-42e1-ab5b-9bbfcee0c6f2-kube-api-access-898lr\") pod \"e2e-api-a-f1f91d-openshift-default-7b546d6c86-kmhkm\" (UID: \"e5176249-c100-42e1-ab5b-9bbfcee0c6f2\") " pod="openshift-ingress/e2e-api-a-f1f91d-openshift-default-7b546d6c86-kmhkm"
Sep 11 01:21:26.133254 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:21:26.133177    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f1f91d-openshift-default-7b546d6c86-kmhkm"
Sep 11 01:21:26.409122 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:21:26.409095    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-f1f91d-openshift-default-7b546d6c86-kmhkm"]
Sep 11 01:21:26.410642 ip-10-0-129-222 kubenswrapper[2569]: W0911 01:21:26.410616    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5176249_c100_42e1_ab5b_9bbfcee0c6f2.slice/crio-c0fe0b3f6ee2d65fa02d44125e39000fdbcd91715c9d660c58a9a8e804921de9 WatchSource:0}: Error finding container c0fe0b3f6ee2d65fa02d44125e39000fdbcd91715c9d660c58a9a8e804921de9: Status 404 returned error can't find the container with id c0fe0b3f6ee2d65fa02d44125e39000fdbcd91715c9d660c58a9a8e804921de9
Sep 11 01:21:26.412992 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:21:26.412954    2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 01:21:26.413098 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:21:26.413017    2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 01:21:26.413098 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:21:26.413044    2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 01:21:26.652613 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:21:26.652571    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f1f91d-openshift-default-7b546d6c86-kmhkm" event={"ID":"e5176249-c100-42e1-ab5b-9bbfcee0c6f2","Type":"ContainerStarted","Data":"913e23fc27b5aa5b74a9b741e0ba16d1553df645e85f1e21e00541765550b054"}
Sep 11 01:21:26.652804 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:21:26.652632    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f1f91d-openshift-default-7b546d6c86-kmhkm" event={"ID":"e5176249-c100-42e1-ab5b-9bbfcee0c6f2","Type":"ContainerStarted","Data":"c0fe0b3f6ee2d65fa02d44125e39000fdbcd91715c9d660c58a9a8e804921de9"}
Sep 11 01:21:26.672178 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:21:26.672058    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-f1f91d-openshift-default-7b546d6c86-kmhkm" podStartSLOduration=1.672037636 podStartE2EDuration="1.672037636s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 01:21:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 01:21:26.671160193 +0000 UTC m=+1421.322534441" watchObservedRunningTime="2026-09-11 01:21:26.672037636 +0000 UTC m=+1421.323411886"
Sep 11 01:21:27.133836 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:21:27.133802    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-f1f91d-openshift-default-7b546d6c86-kmhkm"
Sep 11 01:21:27.138651 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:21:27.138621    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-f1f91d-openshift-default-7b546d6c86-kmhkm"
Sep 11 01:21:27.658023 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:21:27.657992    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-f1f91d-openshift-default-7b546d6c86-kmhkm"
Sep 11 01:21:27.658956 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:21:27.658935    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-f1f91d-openshift-default-7b546d6c86-kmhkm"
Sep 11 01:21:42.099683 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:21:42.099593    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kmx5x"]
Sep 11 01:21:46.141034 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:21:46.140990    2569 scope.go:117] "RemoveContainer" containerID="6204b0f50e449b004329d33c38d705dd9ff1cd601d600909c69048abb9591ad6"
Sep 11 01:21:46.153529 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:21:46.153497    2569 scope.go:117] "RemoveContainer" containerID="4925d9c3f1ad2ce0d5bd2d901c6f1627fb6f8b4b18a51e0c2ecfb09f17eb66a3"
Sep 11 01:21:46.166806 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:21:46.166781    2569 scope.go:117] "RemoveContainer" containerID="e9202fc50c541f08db29853d495e0f0055f5302d687e9cff66d02912a15191b8"
Sep 11 01:21:52.807325 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:21:52.807286    2569 generic.go:356] "Generic (PLEG): container finished" podID="e5176249-c100-42e1-ab5b-9bbfcee0c6f2" containerID="913e23fc27b5aa5b74a9b741e0ba16d1553df645e85f1e21e00541765550b054" exitCode=0
Sep 11 01:21:52.807732 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:21:52.807356    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f1f91d-openshift-default-7b546d6c86-kmhkm" event={"ID":"e5176249-c100-42e1-ab5b-9bbfcee0c6f2","Type":"ContainerDied","Data":"913e23fc27b5aa5b74a9b741e0ba16d1553df645e85f1e21e00541765550b054"}
Sep 11 01:21:52.807732 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:21:52.807722    2569 scope.go:117] "RemoveContainer" containerID="913e23fc27b5aa5b74a9b741e0ba16d1553df645e85f1e21e00541765550b054"
Sep 11 01:21:52.808866 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:21:52.808833    2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 01:21:52.808995 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:21:52.808944    2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 01:21:52.809064 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:21:52.808992    2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 01:21:53.815101 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:21:53.815068    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f1f91d-openshift-default-7b546d6c86-kmhkm" event={"ID":"e5176249-c100-42e1-ab5b-9bbfcee0c6f2","Type":"ContainerStarted","Data":"ac6c1ba27f04c8b5dbdebd4678596f1a93af2126f42dcf9eac42e6c0893eaf18"}
Sep 11 01:21:54.133652 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:21:54.133571    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-f1f91d-openshift-default-7b546d6c86-kmhkm"
Sep 11 01:21:54.134741 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:21:54.134713    2569 patch_prober.go:28] interesting pod/e2e-api-a-f1f91d-openshift-default-7b546d6c86-kmhkm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body=
Sep 11 01:21:54.134906 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:21:54.134770    2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f1f91d-openshift-default-7b546d6c86-kmhkm" podUID="e5176249-c100-42e1-ab5b-9bbfcee0c6f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused"
Sep 11 01:21:54.399115 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:21:54.399038    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-816605-openshift-default-76dd9c445d-259dd"]
Sep 11 01:21:54.403108 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:21:54.403088    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-816605-openshift-default-76dd9c445d-259dd"
Sep 11 01:21:54.405374 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:21:54.405351    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-816605-openshift-default-dockercfg-w5nsg\""
Sep 11 01:21:54.414015 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:21:54.413994    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-816605-openshift-default-76dd9c445d-259dd"]
Sep 11 01:21:54.508886 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:21:54.508833    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0da348e1-2882-4e40-9563-976e64575a49-istio-podinfo\") pod \"e2e-api-b-816605-openshift-default-76dd9c445d-259dd\" (UID: \"0da348e1-2882-4e40-9563-976e64575a49\") " pod="openshift-ingress/e2e-api-b-816605-openshift-default-76dd9c445d-259dd"
Sep 11 01:21:54.509087 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:21:54.508890    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0da348e1-2882-4e40-9563-976e64575a49-workload-socket\") pod \"e2e-api-b-816605-openshift-default-76dd9c445d-259dd\" (UID: \"0da348e1-2882-4e40-9563-976e64575a49\") " pod="openshift-ingress/e2e-api-b-816605-openshift-default-76dd9c445d-259dd"
Sep 11 01:21:54.509087 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:21:54.508922    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0da348e1-2882-4e40-9563-976e64575a49-workload-certs\") pod \"e2e-api-b-816605-openshift-default-76dd9c445d-259dd\" (UID: \"0da348e1-2882-4e40-9563-976e64575a49\") " pod="openshift-ingress/e2e-api-b-816605-openshift-default-76dd9c445d-259dd"
Sep 11 01:21:54.509087 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:21:54.508938    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0da348e1-2882-4e40-9563-976e64575a49-istio-data\") pod \"e2e-api-b-816605-openshift-default-76dd9c445d-259dd\" (UID: \"0da348e1-2882-4e40-9563-976e64575a49\") " pod="openshift-ingress/e2e-api-b-816605-openshift-default-76dd9c445d-259dd"
Sep 11 01:21:54.509087 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:21:54.508995    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0da348e1-2882-4e40-9563-976e64575a49-istiod-ca-cert\") pod \"e2e-api-b-816605-openshift-default-76dd9c445d-259dd\" (UID: \"0da348e1-2882-4e40-9563-976e64575a49\") " pod="openshift-ingress/e2e-api-b-816605-openshift-default-76dd9c445d-259dd"
Sep 11 01:21:54.509087 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:21:54.509027    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0da348e1-2882-4e40-9563-976e64575a49-credential-socket\") pod \"e2e-api-b-816605-openshift-default-76dd9c445d-259dd\" (UID: \"0da348e1-2882-4e40-9563-976e64575a49\") " pod="openshift-ingress/e2e-api-b-816605-openshift-default-76dd9c445d-259dd"
Sep 11 01:21:54.509385 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:21:54.509104    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0da348e1-2882-4e40-9563-976e64575a49-istio-envoy\") pod \"e2e-api-b-816605-openshift-default-76dd9c445d-259dd\" (UID: \"0da348e1-2882-4e40-9563-976e64575a49\") " pod="openshift-ingress/e2e-api-b-816605-openshift-default-76dd9c445d-259dd"
Sep 11 01:21:54.509385 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:21:54.509160    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0da348e1-2882-4e40-9563-976e64575a49-istio-token\") pod \"e2e-api-b-816605-openshift-default-76dd9c445d-259dd\" (UID: \"0da348e1-2882-4e40-9563-976e64575a49\") " pod="openshift-ingress/e2e-api-b-816605-openshift-default-76dd9c445d-259dd"
Sep 11 01:21:54.509385 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:21:54.509195    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zh94h\" (UniqueName: \"kubernetes.io/projected/0da348e1-2882-4e40-9563-976e64575a49-kube-api-access-zh94h\") pod \"e2e-api-b-816605-openshift-default-76dd9c445d-259dd\" (UID: \"0da348e1-2882-4e40-9563-976e64575a49\") " pod="openshift-ingress/e2e-api-b-816605-openshift-default-76dd9c445d-259dd"
Sep 11 01:21:54.610618 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:21:54.610589    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0da348e1-2882-4e40-9563-976e64575a49-istio-token\") pod \"e2e-api-b-816605-openshift-default-76dd9c445d-259dd\" (UID: \"0da348e1-2882-4e40-9563-976e64575a49\") " pod="openshift-ingress/e2e-api-b-816605-openshift-default-76dd9c445d-259dd"
Sep 11 01:21:54.610798 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:21:54.610624    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zh94h\" (UniqueName: \"kubernetes.io/projected/0da348e1-2882-4e40-9563-976e64575a49-kube-api-access-zh94h\") pod \"e2e-api-b-816605-openshift-default-76dd9c445d-259dd\" (UID: \"0da348e1-2882-4e40-9563-976e64575a49\") " pod="openshift-ingress/e2e-api-b-816605-openshift-default-76dd9c445d-259dd"
Sep 11 01:21:54.610798 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:21:54.610695    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0da348e1-2882-4e40-9563-976e64575a49-istio-podinfo\") pod \"e2e-api-b-816605-openshift-default-76dd9c445d-259dd\" (UID: \"0da348e1-2882-4e40-9563-976e64575a49\") " pod="openshift-ingress/e2e-api-b-816605-openshift-default-76dd9c445d-259dd"
Sep 11 01:21:54.610798 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:21:54.610717    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0da348e1-2882-4e40-9563-976e64575a49-workload-socket\") pod \"e2e-api-b-816605-openshift-default-76dd9c445d-259dd\" (UID: \"0da348e1-2882-4e40-9563-976e64575a49\") " pod="openshift-ingress/e2e-api-b-816605-openshift-default-76dd9c445d-259dd"
Sep 11 01:21:54.610798 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:21:54.610738    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0da348e1-2882-4e40-9563-976e64575a49-workload-certs\") pod \"e2e-api-b-816605-openshift-default-76dd9c445d-259dd\" (UID: \"0da348e1-2882-4e40-9563-976e64575a49\") " pod="openshift-ingress/e2e-api-b-816605-openshift-default-76dd9c445d-259dd"
Sep 11 01:21:54.610798 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:21:54.610762    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0da348e1-2882-4e40-9563-976e64575a49-istio-data\") pod \"e2e-api-b-816605-openshift-default-76dd9c445d-259dd\" (UID: \"0da348e1-2882-4e40-9563-976e64575a49\") " pod="openshift-ingress/e2e-api-b-816605-openshift-default-76dd9c445d-259dd"
Sep 11 01:21:54.611100 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:21:54.610803    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0da348e1-2882-4e40-9563-976e64575a49-istiod-ca-cert\") pod \"e2e-api-b-816605-openshift-default-76dd9c445d-259dd\" (UID: \"0da348e1-2882-4e40-9563-976e64575a49\") " pod="openshift-ingress/e2e-api-b-816605-openshift-default-76dd9c445d-259dd"
Sep 11 01:21:54.611100 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:21:54.610850    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0da348e1-2882-4e40-9563-976e64575a49-credential-socket\") pod \"e2e-api-b-816605-openshift-default-76dd9c445d-259dd\" (UID: \"0da348e1-2882-4e40-9563-976e64575a49\") " pod="openshift-ingress/e2e-api-b-816605-openshift-default-76dd9c445d-259dd"
Sep 11 01:21:54.611100 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:21:54.610924    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0da348e1-2882-4e40-9563-976e64575a49-istio-envoy\") pod \"e2e-api-b-816605-openshift-default-76dd9c445d-259dd\" (UID: \"0da348e1-2882-4e40-9563-976e64575a49\") " pod="openshift-ingress/e2e-api-b-816605-openshift-default-76dd9c445d-259dd"
Sep 11 01:21:54.611264 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:21:54.611190    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0da348e1-2882-4e40-9563-976e64575a49-workload-socket\") pod \"e2e-api-b-816605-openshift-default-76dd9c445d-259dd\" (UID: \"0da348e1-2882-4e40-9563-976e64575a49\") " pod="openshift-ingress/e2e-api-b-816605-openshift-default-76dd9c445d-259dd"
Sep 11 01:21:54.611264 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:21:54.611203    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0da348e1-2882-4e40-9563-976e64575a49-istio-data\") pod \"e2e-api-b-816605-openshift-default-76dd9c445d-259dd\" (UID: \"0da348e1-2882-4e40-9563-976e64575a49\") " pod="openshift-ingress/e2e-api-b-816605-openshift-default-76dd9c445d-259dd"
Sep 11 01:21:54.611367 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:21:54.611292    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0da348e1-2882-4e40-9563-976e64575a49-workload-certs\") pod \"e2e-api-b-816605-openshift-default-76dd9c445d-259dd\" (UID: \"0da348e1-2882-4e40-9563-976e64575a49\") " pod="openshift-ingress/e2e-api-b-816605-openshift-default-76dd9c445d-259dd"
Sep 11 01:21:54.611548 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:21:54.611526    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0da348e1-2882-4e40-9563-976e64575a49-credential-socket\") pod \"e2e-api-b-816605-openshift-default-76dd9c445d-259dd\" (UID: \"0da348e1-2882-4e40-9563-976e64575a49\") " pod="openshift-ingress/e2e-api-b-816605-openshift-default-76dd9c445d-259dd"
Sep 11 01:21:54.611625 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:21:54.611601    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0da348e1-2882-4e40-9563-976e64575a49-istiod-ca-cert\") pod \"e2e-api-b-816605-openshift-default-76dd9c445d-259dd\" (UID: \"0da348e1-2882-4e40-9563-976e64575a49\") " pod="openshift-ingress/e2e-api-b-816605-openshift-default-76dd9c445d-259dd"
Sep 11 01:21:54.613296 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:21:54.613275    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0da348e1-2882-4e40-9563-976e64575a49-istio-podinfo\") pod \"e2e-api-b-816605-openshift-default-76dd9c445d-259dd\" (UID: \"0da348e1-2882-4e40-9563-976e64575a49\") " pod="openshift-ingress/e2e-api-b-816605-openshift-default-76dd9c445d-259dd"
Sep 11 01:21:54.613296 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:21:54.613271    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0da348e1-2882-4e40-9563-976e64575a49-istio-envoy\") pod \"e2e-api-b-816605-openshift-default-76dd9c445d-259dd\" (UID: \"0da348e1-2882-4e40-9563-976e64575a49\") " pod="openshift-ingress/e2e-api-b-816605-openshift-default-76dd9c445d-259dd"
Sep 11 01:21:54.620188 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:21:54.620158    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0da348e1-2882-4e40-9563-976e64575a49-istio-token\") pod \"e2e-api-b-816605-openshift-default-76dd9c445d-259dd\" (UID: \"0da348e1-2882-4e40-9563-976e64575a49\") " pod="openshift-ingress/e2e-api-b-816605-openshift-default-76dd9c445d-259dd"
Sep 11 01:21:54.620347 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:21:54.620289    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zh94h\" (UniqueName: \"kubernetes.io/projected/0da348e1-2882-4e40-9563-976e64575a49-kube-api-access-zh94h\") pod \"e2e-api-b-816605-openshift-default-76dd9c445d-259dd\" (UID: \"0da348e1-2882-4e40-9563-976e64575a49\") " pod="openshift-ingress/e2e-api-b-816605-openshift-default-76dd9c445d-259dd"
Sep 11 01:21:54.715840 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:21:54.715754    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-816605-openshift-default-76dd9c445d-259dd"
Sep 11 01:21:55.009186 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:21:55.009143    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-816605-openshift-default-76dd9c445d-259dd"]
Sep 11 01:21:55.010819 ip-10-0-129-222 kubenswrapper[2569]: W0911 01:21:55.010766    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0da348e1_2882_4e40_9563_976e64575a49.slice/crio-0151aa500f35802892c62271c071323033c6354c5cc53cc08ecfb9a6ce341236 WatchSource:0}: Error finding container 0151aa500f35802892c62271c071323033c6354c5cc53cc08ecfb9a6ce341236: Status 404 returned error can't find the container with id 0151aa500f35802892c62271c071323033c6354c5cc53cc08ecfb9a6ce341236
Sep 11 01:21:55.013341 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:21:55.013302    2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 01:21:55.013430 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:21:55.013372    2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 01:21:55.013430 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:21:55.013399    2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 01:21:55.134578 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:21:55.134544    2569 patch_prober.go:28] interesting pod/e2e-api-a-f1f91d-openshift-default-7b546d6c86-kmhkm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body=
Sep 11 01:21:55.134729 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:21:55.134607    2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f1f91d-openshift-default-7b546d6c86-kmhkm" podUID="e5176249-c100-42e1-ab5b-9bbfcee0c6f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused"
Sep 11 01:21:55.828711 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:21:55.828674    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-816605-openshift-default-76dd9c445d-259dd" event={"ID":"0da348e1-2882-4e40-9563-976e64575a49","Type":"ContainerStarted","Data":"52aadfc98510fd798c22935b9223fdbee083f17c0928cc816c1b8fd808d62c64"}
Sep 11 01:21:55.828931 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:21:55.828729    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-816605-openshift-default-76dd9c445d-259dd" event={"ID":"0da348e1-2882-4e40-9563-976e64575a49","Type":"ContainerStarted","Data":"0151aa500f35802892c62271c071323033c6354c5cc53cc08ecfb9a6ce341236"}
Sep 11 01:21:55.849127 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:21:55.849063    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-816605-openshift-default-76dd9c445d-259dd" podStartSLOduration=1.849042574 podStartE2EDuration="1.849042574s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 01:21:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 01:21:55.847274433 +0000 UTC m=+1450.498648682" watchObservedRunningTime="2026-09-11 01:21:55.849042574 +0000 UTC m=+1450.500416824"
Sep 11 01:21:56.134333 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:21:56.134244    2569 patch_prober.go:28] interesting pod/e2e-api-a-f1f91d-openshift-default-7b546d6c86-kmhkm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body=
Sep 11 01:21:56.134333 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:21:56.134305    2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f1f91d-openshift-default-7b546d6c86-kmhkm" podUID="e5176249-c100-42e1-ab5b-9bbfcee0c6f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused"
Sep 11 01:21:56.716477 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:21:56.716446    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-816605-openshift-default-76dd9c445d-259dd"
Sep 11 01:21:56.721462 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:21:56.721438    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-816605-openshift-default-76dd9c445d-259dd"
Sep 11 01:21:56.834372 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:21:56.834339    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-816605-openshift-default-76dd9c445d-259dd"
Sep 11 01:21:56.835828 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:21:56.835795    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-816605-openshift-default-76dd9c445d-259dd"
Sep 11 01:21:57.133472 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:21:57.133435    2569 patch_prober.go:28] interesting pod/e2e-api-a-f1f91d-openshift-default-7b546d6c86-kmhkm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body=
Sep 11 01:21:57.133760 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:21:57.133496    2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f1f91d-openshift-default-7b546d6c86-kmhkm" podUID="e5176249-c100-42e1-ab5b-9bbfcee0c6f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused"
Sep 11 01:21:57.658470 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:21:57.658434    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-f1f91d-openshift-default-7b546d6c86-kmhkm"
Sep 11 01:21:58.133534 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:21:58.133503    2569 patch_prober.go:28] interesting pod/e2e-api-a-f1f91d-openshift-default-7b546d6c86-kmhkm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body=
Sep 11 01:21:58.133715 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:21:58.133557    2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f1f91d-openshift-default-7b546d6c86-kmhkm" podUID="e5176249-c100-42e1-ab5b-9bbfcee0c6f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused"
Sep 11 01:21:59.138506 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:21:59.138479    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-f1f91d-openshift-default-7b546d6c86-kmhkm"
Sep 11 01:21:59.139564 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:21:59.139544    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-f1f91d-openshift-default-7b546d6c86-kmhkm"
Sep 11 01:22:03.713051 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:22:03.713015    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kmx5x"]
Sep 11 01:22:10.386231 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:22:10.386199    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-651569-zjl6r"]
Sep 11 01:22:10.391385 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:22:10.391367    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-651569-zjl6r"
Sep 11 01:22:10.394526 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:22:10.394503    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pppkt\""
Sep 11 01:22:10.394655 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:22:10.394534    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 01:22:10.408167 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:22:10.405639    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-651569-zjl6r"]
Sep 11 01:22:10.571828 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:22:10.571787    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc8413af-3deb-488c-998e-3bc0e76a1ef4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-651569-zjl6r\" (UID: \"cc8413af-3deb-488c-998e-3bc0e76a1ef4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-651569-zjl6r"
Sep 11 01:22:10.673412 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:22:10.673320    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc8413af-3deb-488c-998e-3bc0e76a1ef4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-651569-zjl6r\" (UID: \"cc8413af-3deb-488c-998e-3bc0e76a1ef4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-651569-zjl6r"
Sep 11 01:22:10.673956 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:22:10.673934    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc8413af-3deb-488c-998e-3bc0e76a1ef4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-651569-zjl6r\" (UID: \"cc8413af-3deb-488c-998e-3bc0e76a1ef4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-651569-zjl6r"
Sep 11 01:22:10.712190 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:22:10.712152    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-651569-zjl6r"
Sep 11 01:22:10.992331 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:22:10.992301    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-651569-zjl6r"]
Sep 11 01:22:10.993849 ip-10-0-129-222 kubenswrapper[2569]: W0911 01:22:10.993814    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc8413af_3deb_488c_998e_3bc0e76a1ef4.slice/crio-925e05884aac2105953d70f218b96e8fb105939e6e44603bc6ba8786dcf2e250 WatchSource:0}: Error finding container 925e05884aac2105953d70f218b96e8fb105939e6e44603bc6ba8786dcf2e250: Status 404 returned error can't find the container with id 925e05884aac2105953d70f218b96e8fb105939e6e44603bc6ba8786dcf2e250
Sep 11 01:22:11.917340 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:22:11.917303    2569 generic.go:356] "Generic (PLEG): container finished" podID="cc8413af-3deb-488c-998e-3bc0e76a1ef4" containerID="b1858b088e8a588a7543185cbcba3bbe69cd7809e4e10d72dd461634c690854b" exitCode=0
Sep 11 01:22:11.917696 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:22:11.917379    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-651569-zjl6r" event={"ID":"cc8413af-3deb-488c-998e-3bc0e76a1ef4","Type":"ContainerDied","Data":"b1858b088e8a588a7543185cbcba3bbe69cd7809e4e10d72dd461634c690854b"}
Sep 11 01:22:11.917696 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:22:11.917431    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-651569-zjl6r" event={"ID":"cc8413af-3deb-488c-998e-3bc0e76a1ef4","Type":"ContainerStarted","Data":"925e05884aac2105953d70f218b96e8fb105939e6e44603bc6ba8786dcf2e250"}
Sep 11 01:22:13.064470 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:22:13.064446    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-651569-zjl6r"
Sep 11 01:22:13.196397 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:22:13.196317    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc8413af-3deb-488c-998e-3bc0e76a1ef4-maas-api-service-ca\") pod \"cc8413af-3deb-488c-998e-3bc0e76a1ef4\" (UID: \"cc8413af-3deb-488c-998e-3bc0e76a1ef4\") "
Sep 11 01:22:13.196708 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:22:13.196678    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc8413af-3deb-488c-998e-3bc0e76a1ef4-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "cc8413af-3deb-488c-998e-3bc0e76a1ef4" (UID: "cc8413af-3deb-488c-998e-3bc0e76a1ef4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 01:22:13.297406 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:22:13.297377    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc8413af-3deb-488c-998e-3bc0e76a1ef4-maas-api-service-ca\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:22:13.933196 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:22:13.933167    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-651569-zjl6r"
Sep 11 01:22:13.933346 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:22:13.933204    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-651569-zjl6r" event={"ID":"cc8413af-3deb-488c-998e-3bc0e76a1ef4","Type":"ContainerDied","Data":"925e05884aac2105953d70f218b96e8fb105939e6e44603bc6ba8786dcf2e250"}
Sep 11 01:22:13.933346 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:22:13.933250    2569 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="925e05884aac2105953d70f218b96e8fb105939e6e44603bc6ba8786dcf2e250"
Sep 11 01:22:25.685475 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:22:25.685438    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f1f91d-rfr4n"]
Sep 11 01:22:25.685952 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:22:25.685938    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc8413af-3deb-488c-998e-3bc0e76a1ef4" containerName="revoke-keys"
Sep 11 01:22:25.686013 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:22:25.685954    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc8413af-3deb-488c-998e-3bc0e76a1ef4" containerName="revoke-keys"
Sep 11 01:22:25.686057 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:22:25.686048    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="cc8413af-3deb-488c-998e-3bc0e76a1ef4" containerName="revoke-keys"
Sep 11 01:22:25.689414 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:22:25.689397    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f1f91d-rfr4n"
Sep 11 01:22:25.691552 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:22:25.691531    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 01:22:25.691669 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:22:25.691571    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pppkt\""
Sep 11 01:22:25.695967 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:22:25.695945    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f1f91d-rfr4n"]
Sep 11 01:22:25.714545 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:22:25.714516    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4ec425c-5ebb-4b22-aa58-fcc11a270947-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-f1f91d-rfr4n\" (UID: \"d4ec425c-5ebb-4b22-aa58-fcc11a270947\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f1f91d-rfr4n"
Sep 11 01:22:25.815244 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:22:25.815207    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4ec425c-5ebb-4b22-aa58-fcc11a270947-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-f1f91d-rfr4n\" (UID: \"d4ec425c-5ebb-4b22-aa58-fcc11a270947\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f1f91d-rfr4n"
Sep 11 01:22:25.815901 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:22:25.815846    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4ec425c-5ebb-4b22-aa58-fcc11a270947-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-f1f91d-rfr4n\" (UID: \"d4ec425c-5ebb-4b22-aa58-fcc11a270947\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f1f91d-rfr4n"
Sep 11 01:22:26.001689 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:22:26.001597    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f1f91d-rfr4n"
Sep 11 01:22:26.292396 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:22:26.292366    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f1f91d-rfr4n"]
Sep 11 01:22:26.293837 ip-10-0-129-222 kubenswrapper[2569]: W0911 01:22:26.293811    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4ec425c_5ebb_4b22_aa58_fcc11a270947.slice/crio-0a20eea5316259dfa7184ecb26270de773561cc00db62f0aec57a1ecc8f7ac09 WatchSource:0}: Error finding container 0a20eea5316259dfa7184ecb26270de773561cc00db62f0aec57a1ecc8f7ac09: Status 404 returned error can't find the container with id 0a20eea5316259dfa7184ecb26270de773561cc00db62f0aec57a1ecc8f7ac09
Sep 11 01:22:26.774904 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:22:26.774859    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-651569-zjl6r"]
Sep 11 01:22:26.780748 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:22:26.780722    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-651569-zjl6r"]
Sep 11 01:22:27.015248 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:22:27.015156    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f1f91d-rfr4n" event={"ID":"d4ec425c-5ebb-4b22-aa58-fcc11a270947","Type":"ContainerStarted","Data":"bb9e1fb47946d56bb2262ee7692d2b74c4e2d85e0cb4616a032354142ce82ed9"}
Sep 11 01:22:27.015248 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:22:27.015216    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f1f91d-rfr4n" event={"ID":"d4ec425c-5ebb-4b22-aa58-fcc11a270947","Type":"ContainerStarted","Data":"0a20eea5316259dfa7184ecb26270de773561cc00db62f0aec57a1ecc8f7ac09"}
Sep 11 01:22:27.031053 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:22:27.031001    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f1f91d-rfr4n" podStartSLOduration=2.0309837 podStartE2EDuration="2.0309837s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 01:22:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 01:22:27.029105653 +0000 UTC m=+1481.680479902" watchObservedRunningTime="2026-09-11 01:22:27.0309837 +0000 UTC m=+1481.682357951"
Sep 11 01:22:27.873706 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:22:27.873671    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cc8413af-3deb-488c-998e-3bc0e76a1ef4" path="/var/lib/kubelet/pods/cc8413af-3deb-488c-998e-3bc0e76a1ef4/volumes"
Sep 11 01:22:28.022627 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:22:28.022593    2569 generic.go:356] "Generic (PLEG): container finished" podID="d4ec425c-5ebb-4b22-aa58-fcc11a270947" containerID="bb9e1fb47946d56bb2262ee7692d2b74c4e2d85e0cb4616a032354142ce82ed9" exitCode=0
Sep 11 01:22:28.022796 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:22:28.022654    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f1f91d-rfr4n" event={"ID":"d4ec425c-5ebb-4b22-aa58-fcc11a270947","Type":"ContainerDied","Data":"bb9e1fb47946d56bb2262ee7692d2b74c4e2d85e0cb4616a032354142ce82ed9"}
Sep 11 01:22:29.221115 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:22:29.221093    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f1f91d-rfr4n"
Sep 11 01:22:29.266285 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:22:29.266253    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4ec425c-5ebb-4b22-aa58-fcc11a270947-maas-api-service-ca\") pod \"d4ec425c-5ebb-4b22-aa58-fcc11a270947\" (UID: \"d4ec425c-5ebb-4b22-aa58-fcc11a270947\") "
Sep 11 01:22:29.266657 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:22:29.266631    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4ec425c-5ebb-4b22-aa58-fcc11a270947-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d4ec425c-5ebb-4b22-aa58-fcc11a270947" (UID: "d4ec425c-5ebb-4b22-aa58-fcc11a270947"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 01:22:29.367113 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:22:29.367079    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4ec425c-5ebb-4b22-aa58-fcc11a270947-maas-api-service-ca\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:22:30.049568 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:22:30.049539    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f1f91d-rfr4n"
Sep 11 01:22:30.049847 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:22:30.049538    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f1f91d-rfr4n" event={"ID":"d4ec425c-5ebb-4b22-aa58-fcc11a270947","Type":"ContainerDied","Data":"0a20eea5316259dfa7184ecb26270de773561cc00db62f0aec57a1ecc8f7ac09"}
Sep 11 01:22:30.049967 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:22:30.049925    2569 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0a20eea5316259dfa7184ecb26270de773561cc00db62f0aec57a1ecc8f7ac09"
Sep 11 01:22:34.078084 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:22:34.078005    2569 generic.go:356] "Generic (PLEG): container finished" podID="0da348e1-2882-4e40-9563-976e64575a49" containerID="52aadfc98510fd798c22935b9223fdbee083f17c0928cc816c1b8fd808d62c64" exitCode=0
Sep 11 01:22:34.078557 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:22:34.078078    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-816605-openshift-default-76dd9c445d-259dd" event={"ID":"0da348e1-2882-4e40-9563-976e64575a49","Type":"ContainerDied","Data":"52aadfc98510fd798c22935b9223fdbee083f17c0928cc816c1b8fd808d62c64"}
Sep 11 01:22:34.078557 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:22:34.078484    2569 scope.go:117] "RemoveContainer" containerID="52aadfc98510fd798c22935b9223fdbee083f17c0928cc816c1b8fd808d62c64"
Sep 11 01:22:34.079640 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:22:34.079609    2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 01:22:34.079721 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:22:34.079689    2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 01:22:34.079779 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:22:34.079721    2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 01:22:35.086463 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:22:35.086425    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-816605-openshift-default-76dd9c445d-259dd" event={"ID":"0da348e1-2882-4e40-9563-976e64575a49","Type":"ContainerStarted","Data":"eb067e6fd84177eea7b59e4aae222556fa4ce5b17722682860a6148b70d2ac4b"}
Sep 11 01:22:35.716309 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:22:35.716274    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-816605-openshift-default-76dd9c445d-259dd"
Sep 11 01:22:35.716518 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:22:35.716486    2569 patch_prober.go:28] interesting pod/e2e-api-b-816605-openshift-default-76dd9c445d-259dd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body=
Sep 11 01:22:35.716602 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:22:35.716552    2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-816605-openshift-default-76dd9c445d-259dd" podUID="0da348e1-2882-4e40-9563-976e64575a49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused"
Sep 11 01:22:36.717007 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:22:36.716973    2569 patch_prober.go:28] interesting pod/e2e-api-b-816605-openshift-default-76dd9c445d-259dd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body=
Sep 11 01:22:36.717378 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:22:36.717040    2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-816605-openshift-default-76dd9c445d-259dd" podUID="0da348e1-2882-4e40-9563-976e64575a49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused"
Sep 11 01:22:37.717158 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:22:37.717126    2569 patch_prober.go:28] interesting pod/e2e-api-b-816605-openshift-default-76dd9c445d-259dd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body=
Sep 11 01:22:37.717557 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:22:37.717204    2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-816605-openshift-default-76dd9c445d-259dd" podUID="0da348e1-2882-4e40-9563-976e64575a49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused"
Sep 11 01:22:38.716202 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:22:38.716164    2569 patch_prober.go:28] interesting pod/e2e-api-b-816605-openshift-default-76dd9c445d-259dd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body=
Sep 11 01:22:38.716381 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:22:38.716226    2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-816605-openshift-default-76dd9c445d-259dd" podUID="0da348e1-2882-4e40-9563-976e64575a49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused"
Sep 11 01:22:39.716860 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:22:39.716820    2569 patch_prober.go:28] interesting pod/e2e-api-b-816605-openshift-default-76dd9c445d-259dd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body=
Sep 11 01:22:39.717254 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:22:39.716899    2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-816605-openshift-default-76dd9c445d-259dd" podUID="0da348e1-2882-4e40-9563-976e64575a49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused"
Sep 11 01:22:40.720562 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:22:40.720533    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-816605-openshift-default-76dd9c445d-259dd"
Sep 11 01:22:40.720996 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:22:40.720818    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-816605-openshift-default-76dd9c445d-259dd"
Sep 11 01:22:40.721471 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:22:40.721454    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-816605-openshift-default-76dd9c445d-259dd"
Sep 11 01:22:41.648979 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:22:41.648938    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f1f91d-rfr4n"]
Sep 11 01:22:41.651416 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:22:41.651386    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f1f91d-rfr4n"]
Sep 11 01:22:41.871348 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:22:41.871317    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d4ec425c-5ebb-4b22-aa58-fcc11a270947" path="/var/lib/kubelet/pods/d4ec425c-5ebb-4b22-aa58-fcc11a270947/volumes"
Sep 11 01:22:41.901735 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:22:41.901650    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kmx5x"]
Sep 11 01:22:46.194200 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:22:46.194166    2569 scope.go:117] "RemoveContainer" containerID="43267619a78852ec4d2b1be2887a4f0b31c44c2012ca8ea24d83485b8e99a194"
Sep 11 01:22:47.944474 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:22:47.944438    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-f1f91d-openshift-default-7b546d6c86-kmhkm"]
Sep 11 01:22:47.944947 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:22:47.944692    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-f1f91d-openshift-default-7b546d6c86-kmhkm" podUID="e5176249-c100-42e1-ab5b-9bbfcee0c6f2" containerName="istio-proxy" containerID="cri-o://ac6c1ba27f04c8b5dbdebd4678596f1a93af2126f42dcf9eac42e6c0893eaf18" gracePeriod=30
Sep 11 01:22:48.314379 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:22:48.314349    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-816605-kzt2n"]
Sep 11 01:22:48.314835 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:22:48.314822    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4ec425c-5ebb-4b22-aa58-fcc11a270947" containerName="revoke-keys"
Sep 11 01:22:48.314915 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:22:48.314837    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4ec425c-5ebb-4b22-aa58-fcc11a270947" containerName="revoke-keys"
Sep 11 01:22:48.314915 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:22:48.314910    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="d4ec425c-5ebb-4b22-aa58-fcc11a270947" containerName="revoke-keys"
Sep 11 01:22:48.319570 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:22:48.319552    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-816605-kzt2n"
Sep 11 01:22:48.322159 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:22:48.322130    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 01:22:48.322294 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:22:48.322133    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pppkt\""
Sep 11 01:22:48.325415 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:22:48.325390    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-816605-kzt2n"]
Sep 11 01:22:48.366373 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:22:48.366344    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3bf3b8b5-eeb4-452c-a4c3-94e247cd6ee6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-816605-kzt2n\" (UID: \"3bf3b8b5-eeb4-452c-a4c3-94e247cd6ee6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-816605-kzt2n"
Sep 11 01:22:48.467543 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:22:48.467504    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3bf3b8b5-eeb4-452c-a4c3-94e247cd6ee6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-816605-kzt2n\" (UID: \"3bf3b8b5-eeb4-452c-a4c3-94e247cd6ee6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-816605-kzt2n"
Sep 11 01:22:48.468121 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:22:48.468101    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3bf3b8b5-eeb4-452c-a4c3-94e247cd6ee6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-816605-kzt2n\" (UID: \"3bf3b8b5-eeb4-452c-a4c3-94e247cd6ee6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-816605-kzt2n"
Sep 11 01:22:48.632823 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:22:48.632736    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-816605-kzt2n"
Sep 11 01:22:48.903903 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:22:48.903863    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-816605-kzt2n"]
Sep 11 01:22:48.905717 ip-10-0-129-222 kubenswrapper[2569]: W0911 01:22:48.905690    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bf3b8b5_eeb4_452c_a4c3_94e247cd6ee6.slice/crio-0625bd34ec5251df5afb673398ecb31770b628289c1fd3e00083b61592bf2d1b WatchSource:0}: Error finding container 0625bd34ec5251df5afb673398ecb31770b628289c1fd3e00083b61592bf2d1b: Status 404 returned error can't find the container with id 0625bd34ec5251df5afb673398ecb31770b628289c1fd3e00083b61592bf2d1b
Sep 11 01:22:49.169350 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:22:49.169268    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-816605-kzt2n" event={"ID":"3bf3b8b5-eeb4-452c-a4c3-94e247cd6ee6","Type":"ContainerStarted","Data":"0625bd34ec5251df5afb673398ecb31770b628289c1fd3e00083b61592bf2d1b"}
Sep 11 01:22:50.176966 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:22:50.176936    2569 generic.go:356] "Generic (PLEG): container finished" podID="3bf3b8b5-eeb4-452c-a4c3-94e247cd6ee6" containerID="23f416bfa55bc84c428697280404e9679a90a75b2b99e4a5d1c2dabcc6d11810" exitCode=0
Sep 11 01:22:50.177367 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:22:50.177035    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-816605-kzt2n" event={"ID":"3bf3b8b5-eeb4-452c-a4c3-94e247cd6ee6","Type":"ContainerDied","Data":"23f416bfa55bc84c428697280404e9679a90a75b2b99e4a5d1c2dabcc6d11810"}
Sep 11 01:22:51.321428 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:22:51.321405    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-816605-kzt2n"
Sep 11 01:22:51.396271 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:22:51.396244    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3bf3b8b5-eeb4-452c-a4c3-94e247cd6ee6-maas-api-service-ca\") pod \"3bf3b8b5-eeb4-452c-a4c3-94e247cd6ee6\" (UID: \"3bf3b8b5-eeb4-452c-a4c3-94e247cd6ee6\") "
Sep 11 01:22:51.396577 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:22:51.396555    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3bf3b8b5-eeb4-452c-a4c3-94e247cd6ee6-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3bf3b8b5-eeb4-452c-a4c3-94e247cd6ee6" (UID: "3bf3b8b5-eeb4-452c-a4c3-94e247cd6ee6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 01:22:51.498046 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:22:51.497980    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3bf3b8b5-eeb4-452c-a4c3-94e247cd6ee6-maas-api-service-ca\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:22:52.189794 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:22:52.189715    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-816605-kzt2n"
Sep 11 01:22:52.189794 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:22:52.189733    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-816605-kzt2n" event={"ID":"3bf3b8b5-eeb4-452c-a4c3-94e247cd6ee6","Type":"ContainerDied","Data":"0625bd34ec5251df5afb673398ecb31770b628289c1fd3e00083b61592bf2d1b"}
Sep 11 01:22:52.189794 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:22:52.189782    2569 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0625bd34ec5251df5afb673398ecb31770b628289c1fd3e00083b61592bf2d1b"
Sep 11 01:22:53.197200 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:22:53.197159    2569 generic.go:356] "Generic (PLEG): container finished" podID="e5176249-c100-42e1-ab5b-9bbfcee0c6f2" containerID="ac6c1ba27f04c8b5dbdebd4678596f1a93af2126f42dcf9eac42e6c0893eaf18" exitCode=0
Sep 11 01:22:53.197567 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:22:53.197190    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f1f91d-openshift-default-7b546d6c86-kmhkm" event={"ID":"e5176249-c100-42e1-ab5b-9bbfcee0c6f2","Type":"ContainerDied","Data":"ac6c1ba27f04c8b5dbdebd4678596f1a93af2126f42dcf9eac42e6c0893eaf18"}
Sep 11 01:22:53.197567 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:22:53.197264    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f1f91d-openshift-default-7b546d6c86-kmhkm" event={"ID":"e5176249-c100-42e1-ab5b-9bbfcee0c6f2","Type":"ContainerDied","Data":"c0fe0b3f6ee2d65fa02d44125e39000fdbcd91715c9d660c58a9a8e804921de9"}
Sep 11 01:22:53.197567 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:22:53.197287    2569 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c0fe0b3f6ee2d65fa02d44125e39000fdbcd91715c9d660c58a9a8e804921de9"
Sep 11 01:22:53.197567 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:22:53.197301    2569 scope.go:117] "RemoveContainer" containerID="913e23fc27b5aa5b74a9b741e0ba16d1553df645e85f1e21e00541765550b054"
Sep 11 01:22:53.210463 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:22:53.210438    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f1f91d-openshift-default-7b546d6c86-kmhkm"
Sep 11 01:22:53.314822 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:22:53.314792    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5176249-c100-42e1-ab5b-9bbfcee0c6f2-workload-certs\") pod \"e5176249-c100-42e1-ab5b-9bbfcee0c6f2\" (UID: \"e5176249-c100-42e1-ab5b-9bbfcee0c6f2\") "
Sep 11 01:22:53.315034 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:22:53.314863    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-898lr\" (UniqueName: \"kubernetes.io/projected/e5176249-c100-42e1-ab5b-9bbfcee0c6f2-kube-api-access-898lr\") pod \"e5176249-c100-42e1-ab5b-9bbfcee0c6f2\" (UID: \"e5176249-c100-42e1-ab5b-9bbfcee0c6f2\") "
Sep 11 01:22:53.315034 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:22:53.314915    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5176249-c100-42e1-ab5b-9bbfcee0c6f2-workload-socket\") pod \"e5176249-c100-42e1-ab5b-9bbfcee0c6f2\" (UID: \"e5176249-c100-42e1-ab5b-9bbfcee0c6f2\") "
Sep 11 01:22:53.315034 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:22:53.314940    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5176249-c100-42e1-ab5b-9bbfcee0c6f2-istiod-ca-cert\") pod \"e5176249-c100-42e1-ab5b-9bbfcee0c6f2\" (UID: \"e5176249-c100-42e1-ab5b-9bbfcee0c6f2\") "
Sep 11 01:22:53.315203 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:22:53.315049    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5176249-c100-42e1-ab5b-9bbfcee0c6f2-istio-data\") pod \"e5176249-c100-42e1-ab5b-9bbfcee0c6f2\" (UID: \"e5176249-c100-42e1-ab5b-9bbfcee0c6f2\") "
Sep 11 01:22:53.315203 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:22:53.315081    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5176249-c100-42e1-ab5b-9bbfcee0c6f2-credential-socket\") pod \"e5176249-c100-42e1-ab5b-9bbfcee0c6f2\" (UID: \"e5176249-c100-42e1-ab5b-9bbfcee0c6f2\") "
Sep 11 01:22:53.315203 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:22:53.315109    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5176249-c100-42e1-ab5b-9bbfcee0c6f2-istio-envoy\") pod \"e5176249-c100-42e1-ab5b-9bbfcee0c6f2\" (UID: \"e5176249-c100-42e1-ab5b-9bbfcee0c6f2\") "
Sep 11 01:22:53.315203 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:22:53.315142    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5176249-c100-42e1-ab5b-9bbfcee0c6f2-istio-token\") pod \"e5176249-c100-42e1-ab5b-9bbfcee0c6f2\" (UID: \"e5176249-c100-42e1-ab5b-9bbfcee0c6f2\") "
Sep 11 01:22:53.315203 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:22:53.315173    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5176249-c100-42e1-ab5b-9bbfcee0c6f2-istio-podinfo\") pod \"e5176249-c100-42e1-ab5b-9bbfcee0c6f2\" (UID: \"e5176249-c100-42e1-ab5b-9bbfcee0c6f2\") "
Sep 11 01:22:53.315483 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:22:53.315208    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5176249-c100-42e1-ab5b-9bbfcee0c6f2-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e5176249-c100-42e1-ab5b-9bbfcee0c6f2" (UID: "e5176249-c100-42e1-ab5b-9bbfcee0c6f2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 01:22:53.315483 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:22:53.315139    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5176249-c100-42e1-ab5b-9bbfcee0c6f2-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e5176249-c100-42e1-ab5b-9bbfcee0c6f2" (UID: "e5176249-c100-42e1-ab5b-9bbfcee0c6f2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 01:22:53.315483 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:22:53.315361    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5176249-c100-42e1-ab5b-9bbfcee0c6f2-istio-data" (OuterVolumeSpecName: "istio-data") pod "e5176249-c100-42e1-ab5b-9bbfcee0c6f2" (UID: "e5176249-c100-42e1-ab5b-9bbfcee0c6f2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 01:22:53.315483 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:22:53.315384    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5176249-c100-42e1-ab5b-9bbfcee0c6f2-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e5176249-c100-42e1-ab5b-9bbfcee0c6f2" (UID: "e5176249-c100-42e1-ab5b-9bbfcee0c6f2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 01:22:53.315483 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:22:53.315462    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5176249-c100-42e1-ab5b-9bbfcee0c6f2-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e5176249-c100-42e1-ab5b-9bbfcee0c6f2" (UID: "e5176249-c100-42e1-ab5b-9bbfcee0c6f2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 01:22:53.315723 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:22:53.315563    2569 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5176249-c100-42e1-ab5b-9bbfcee0c6f2-workload-socket\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:22:53.315723 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:22:53.315583    2569 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5176249-c100-42e1-ab5b-9bbfcee0c6f2-istiod-ca-cert\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:22:53.315723 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:22:53.315598    2569 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5176249-c100-42e1-ab5b-9bbfcee0c6f2-istio-data\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:22:53.315723 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:22:53.315610    2569 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5176249-c100-42e1-ab5b-9bbfcee0c6f2-credential-socket\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:22:53.315723 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:22:53.315623    2569 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5176249-c100-42e1-ab5b-9bbfcee0c6f2-workload-certs\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:22:53.317345 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:22:53.317325    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e5176249-c100-42e1-ab5b-9bbfcee0c6f2-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e5176249-c100-42e1-ab5b-9bbfcee0c6f2" (UID: "e5176249-c100-42e1-ab5b-9bbfcee0c6f2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 01:22:53.317422 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:22:53.317322    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5176249-c100-42e1-ab5b-9bbfcee0c6f2-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e5176249-c100-42e1-ab5b-9bbfcee0c6f2" (UID: "e5176249-c100-42e1-ab5b-9bbfcee0c6f2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 01:22:53.317493 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:22:53.317477    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5176249-c100-42e1-ab5b-9bbfcee0c6f2-kube-api-access-898lr" (OuterVolumeSpecName: "kube-api-access-898lr") pod "e5176249-c100-42e1-ab5b-9bbfcee0c6f2" (UID: "e5176249-c100-42e1-ab5b-9bbfcee0c6f2"). InnerVolumeSpecName "kube-api-access-898lr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 01:22:53.317632 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:22:53.317610    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5176249-c100-42e1-ab5b-9bbfcee0c6f2-istio-token" (OuterVolumeSpecName: "istio-token") pod "e5176249-c100-42e1-ab5b-9bbfcee0c6f2" (UID: "e5176249-c100-42e1-ab5b-9bbfcee0c6f2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 01:22:53.416255 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:22:53.416223    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-898lr\" (UniqueName: \"kubernetes.io/projected/e5176249-c100-42e1-ab5b-9bbfcee0c6f2-kube-api-access-898lr\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:22:53.416255 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:22:53.416251    2569 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5176249-c100-42e1-ab5b-9bbfcee0c6f2-istio-envoy\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:22:53.416408 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:22:53.416265    2569 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5176249-c100-42e1-ab5b-9bbfcee0c6f2-istio-token\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:22:53.416408 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:22:53.416277    2569 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5176249-c100-42e1-ab5b-9bbfcee0c6f2-istio-podinfo\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:22:54.204305 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:22:54.204224    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f1f91d-openshift-default-7b546d6c86-kmhkm"
Sep 11 01:22:54.235155 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:22:54.235119    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-f1f91d-openshift-default-7b546d6c86-kmhkm"]
Sep 11 01:22:54.240725 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:22:54.240690    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-f1f91d-openshift-default-7b546d6c86-kmhkm"]
Sep 11 01:22:55.871155 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:22:55.871125    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e5176249-c100-42e1-ab5b-9bbfcee0c6f2" path="/var/lib/kubelet/pods/e5176249-c100-42e1-ab5b-9bbfcee0c6f2/volumes"
Sep 11 01:23:04.062654 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:04.062622    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-816605-kzt2n"]
Sep 11 01:23:04.064799 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:04.064764    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-816605-kzt2n"]
Sep 11 01:23:05.871435 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:05.871401    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3bf3b8b5-eeb4-452c-a4c3-94e247cd6ee6" path="/var/lib/kubelet/pods/3bf3b8b5-eeb4-452c-a4c3-94e247cd6ee6/volumes"
Sep 11 01:23:07.596464 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:07.596435    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kmx5x"]
Sep 11 01:23:10.591590 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:10.591539    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-816605-openshift-default-76dd9c445d-259dd"]
Sep 11 01:23:10.592059 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:10.591771    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-816605-openshift-default-76dd9c445d-259dd" podUID="0da348e1-2882-4e40-9563-976e64575a49" containerName="istio-proxy" containerID="cri-o://eb067e6fd84177eea7b59e4aae222556fa4ce5b17722682860a6148b70d2ac4b" gracePeriod=30
Sep 11 01:23:10.723317 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:10.723271    2569 patch_prober.go:28] interesting pod/e2e-api-b-816605-openshift-default-76dd9c445d-259dd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused
Sep 11 01:23:10.723486 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:10.723348    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-816605-openshift-default-76dd9c445d-259dd" podUID="0da348e1-2882-4e40-9563-976e64575a49" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 11 01:23:11.041070 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:11.040987    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-799002-qnrdc"]
Sep 11 01:23:11.041437 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:11.041424    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e5176249-c100-42e1-ab5b-9bbfcee0c6f2" containerName="istio-proxy"
Sep 11 01:23:11.041479 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:11.041438    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5176249-c100-42e1-ab5b-9bbfcee0c6f2" containerName="istio-proxy"
Sep 11 01:23:11.041479 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:11.041451    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e5176249-c100-42e1-ab5b-9bbfcee0c6f2" containerName="istio-proxy"
Sep 11 01:23:11.041479 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:11.041456    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5176249-c100-42e1-ab5b-9bbfcee0c6f2" containerName="istio-proxy"
Sep 11 01:23:11.041479 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:11.041470    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3bf3b8b5-eeb4-452c-a4c3-94e247cd6ee6" containerName="revoke-keys"
Sep 11 01:23:11.041479 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:11.041476    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="3bf3b8b5-eeb4-452c-a4c3-94e247cd6ee6" containerName="revoke-keys"
Sep 11 01:23:11.041636 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:11.041546    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="e5176249-c100-42e1-ab5b-9bbfcee0c6f2" containerName="istio-proxy"
Sep 11 01:23:11.041636 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:11.041555    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="3bf3b8b5-eeb4-452c-a4c3-94e247cd6ee6" containerName="revoke-keys"
Sep 11 01:23:11.046459 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:11.046441    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-799002-qnrdc"
Sep 11 01:23:11.049100 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:11.049075    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 01:23:11.049100 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:11.049099    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pppkt\""
Sep 11 01:23:11.054765 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:11.054742    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-799002-qnrdc"]
Sep 11 01:23:11.193326 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:11.193291    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9841099-bb5e-462e-9fa0-e36f75c8fe18-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-799002-qnrdc\" (UID: \"c9841099-bb5e-462e-9fa0-e36f75c8fe18\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-799002-qnrdc"
Sep 11 01:23:11.294137 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:11.294048    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9841099-bb5e-462e-9fa0-e36f75c8fe18-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-799002-qnrdc\" (UID: \"c9841099-bb5e-462e-9fa0-e36f75c8fe18\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-799002-qnrdc"
Sep 11 01:23:11.294659 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:11.294640    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9841099-bb5e-462e-9fa0-e36f75c8fe18-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-799002-qnrdc\" (UID: \"c9841099-bb5e-462e-9fa0-e36f75c8fe18\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-799002-qnrdc"
Sep 11 01:23:11.358243 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:11.358202    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-799002-qnrdc"
Sep 11 01:23:11.626782 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:11.626756    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-799002-qnrdc"]
Sep 11 01:23:11.628813 ip-10-0-129-222 kubenswrapper[2569]: W0911 01:23:11.628787    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9841099_bb5e_462e_9fa0_e36f75c8fe18.slice/crio-d7c91cf333a277f249ca039d70795275677503b01b01dabf090291e4a92c15f5 WatchSource:0}: Error finding container d7c91cf333a277f249ca039d70795275677503b01b01dabf090291e4a92c15f5: Status 404 returned error can't find the container with id d7c91cf333a277f249ca039d70795275677503b01b01dabf090291e4a92c15f5
Sep 11 01:23:12.309603 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:12.309574    2569 generic.go:356] "Generic (PLEG): container finished" podID="c9841099-bb5e-462e-9fa0-e36f75c8fe18" containerID="937bcf80eb54a03918e50155af6984cf7157263bade7cea72647a9b5d6366e6a" exitCode=0
Sep 11 01:23:12.309733 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:12.309627    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-799002-qnrdc" event={"ID":"c9841099-bb5e-462e-9fa0-e36f75c8fe18","Type":"ContainerDied","Data":"937bcf80eb54a03918e50155af6984cf7157263bade7cea72647a9b5d6366e6a"}
Sep 11 01:23:12.309733 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:12.309666    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-799002-qnrdc" event={"ID":"c9841099-bb5e-462e-9fa0-e36f75c8fe18","Type":"ContainerStarted","Data":"d7c91cf333a277f249ca039d70795275677503b01b01dabf090291e4a92c15f5"}
Sep 11 01:23:13.445939 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:13.445915    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-799002-qnrdc"
Sep 11 01:23:13.616193 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:13.616114    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9841099-bb5e-462e-9fa0-e36f75c8fe18-maas-api-service-ca\") pod \"c9841099-bb5e-462e-9fa0-e36f75c8fe18\" (UID: \"c9841099-bb5e-462e-9fa0-e36f75c8fe18\") "
Sep 11 01:23:13.616460 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:13.616438    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9841099-bb5e-462e-9fa0-e36f75c8fe18-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c9841099-bb5e-462e-9fa0-e36f75c8fe18" (UID: "c9841099-bb5e-462e-9fa0-e36f75c8fe18"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 01:23:13.717315 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:13.717285    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9841099-bb5e-462e-9fa0-e36f75c8fe18-maas-api-service-ca\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:23:14.322204 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:14.322173    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-799002-qnrdc"
Sep 11 01:23:14.322377 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:14.322210    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-799002-qnrdc" event={"ID":"c9841099-bb5e-462e-9fa0-e36f75c8fe18","Type":"ContainerDied","Data":"d7c91cf333a277f249ca039d70795275677503b01b01dabf090291e4a92c15f5"}
Sep 11 01:23:14.322377 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:14.322260    2569 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d7c91cf333a277f249ca039d70795275677503b01b01dabf090291e4a92c15f5"
Sep 11 01:23:15.851086 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:15.851060    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-816605-openshift-default-76dd9c445d-259dd"
Sep 11 01:23:15.937764 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:15.937692    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0da348e1-2882-4e40-9563-976e64575a49-istio-data\") pod \"0da348e1-2882-4e40-9563-976e64575a49\" (UID: \"0da348e1-2882-4e40-9563-976e64575a49\") "
Sep 11 01:23:15.937764 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:15.937726    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0da348e1-2882-4e40-9563-976e64575a49-workload-certs\") pod \"0da348e1-2882-4e40-9563-976e64575a49\" (UID: \"0da348e1-2882-4e40-9563-976e64575a49\") "
Sep 11 01:23:15.937764 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:15.937760    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0da348e1-2882-4e40-9563-976e64575a49-istio-podinfo\") pod \"0da348e1-2882-4e40-9563-976e64575a49\" (UID: \"0da348e1-2882-4e40-9563-976e64575a49\") "
Sep 11 01:23:15.938038 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:15.937811    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0da348e1-2882-4e40-9563-976e64575a49-istiod-ca-cert\") pod \"0da348e1-2882-4e40-9563-976e64575a49\" (UID: \"0da348e1-2882-4e40-9563-976e64575a49\") "
Sep 11 01:23:15.938038 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:15.937897    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0da348e1-2882-4e40-9563-976e64575a49-credential-socket\") pod \"0da348e1-2882-4e40-9563-976e64575a49\" (UID: \"0da348e1-2882-4e40-9563-976e64575a49\") "
Sep 11 01:23:15.938038 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:15.937934    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0da348e1-2882-4e40-9563-976e64575a49-istio-token\") pod \"0da348e1-2882-4e40-9563-976e64575a49\" (UID: \"0da348e1-2882-4e40-9563-976e64575a49\") "
Sep 11 01:23:15.938038 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:15.937995    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zh94h\" (UniqueName: \"kubernetes.io/projected/0da348e1-2882-4e40-9563-976e64575a49-kube-api-access-zh94h\") pod \"0da348e1-2882-4e40-9563-976e64575a49\" (UID: \"0da348e1-2882-4e40-9563-976e64575a49\") "
Sep 11 01:23:15.938038 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:15.938023    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0da348e1-2882-4e40-9563-976e64575a49-istio-data" (OuterVolumeSpecName: "istio-data") pod "0da348e1-2882-4e40-9563-976e64575a49" (UID: "0da348e1-2882-4e40-9563-976e64575a49"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 01:23:15.938299 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:15.938041    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0da348e1-2882-4e40-9563-976e64575a49-workload-socket\") pod \"0da348e1-2882-4e40-9563-976e64575a49\" (UID: \"0da348e1-2882-4e40-9563-976e64575a49\") "
Sep 11 01:23:15.938299 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:15.938041    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0da348e1-2882-4e40-9563-976e64575a49-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "0da348e1-2882-4e40-9563-976e64575a49" (UID: "0da348e1-2882-4e40-9563-976e64575a49"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 01:23:15.938299 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:15.938071    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0da348e1-2882-4e40-9563-976e64575a49-istio-envoy\") pod \"0da348e1-2882-4e40-9563-976e64575a49\" (UID: \"0da348e1-2882-4e40-9563-976e64575a49\") "
Sep 11 01:23:15.938299 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:15.938241    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0da348e1-2882-4e40-9563-976e64575a49-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "0da348e1-2882-4e40-9563-976e64575a49" (UID: "0da348e1-2882-4e40-9563-976e64575a49"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 01:23:15.938532 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:15.938246    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0da348e1-2882-4e40-9563-976e64575a49-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "0da348e1-2882-4e40-9563-976e64575a49" (UID: "0da348e1-2882-4e40-9563-976e64575a49"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 01:23:15.938532 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:15.938461    2569 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0da348e1-2882-4e40-9563-976e64575a49-workload-socket\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:23:15.938532 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:15.938512    2569 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0da348e1-2882-4e40-9563-976e64575a49-istio-data\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:23:15.938532 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:15.938527    2569 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0da348e1-2882-4e40-9563-976e64575a49-workload-certs\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:23:15.938741 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:15.938540    2569 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0da348e1-2882-4e40-9563-976e64575a49-credential-socket\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:23:15.938741 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:15.938626    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0da348e1-2882-4e40-9563-976e64575a49-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "0da348e1-2882-4e40-9563-976e64575a49" (UID: "0da348e1-2882-4e40-9563-976e64575a49"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 01:23:15.940164 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:15.940138    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0da348e1-2882-4e40-9563-976e64575a49-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "0da348e1-2882-4e40-9563-976e64575a49" (UID: "0da348e1-2882-4e40-9563-976e64575a49"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 01:23:15.940274 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:15.940197    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0da348e1-2882-4e40-9563-976e64575a49-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "0da348e1-2882-4e40-9563-976e64575a49" (UID: "0da348e1-2882-4e40-9563-976e64575a49"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 01:23:15.940614 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:15.940595    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0da348e1-2882-4e40-9563-976e64575a49-istio-token" (OuterVolumeSpecName: "istio-token") pod "0da348e1-2882-4e40-9563-976e64575a49" (UID: "0da348e1-2882-4e40-9563-976e64575a49"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 01:23:15.940864 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:15.940849    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0da348e1-2882-4e40-9563-976e64575a49-kube-api-access-zh94h" (OuterVolumeSpecName: "kube-api-access-zh94h") pod "0da348e1-2882-4e40-9563-976e64575a49" (UID: "0da348e1-2882-4e40-9563-976e64575a49"). InnerVolumeSpecName "kube-api-access-zh94h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 01:23:16.039968 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:16.039940    2569 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0da348e1-2882-4e40-9563-976e64575a49-istio-podinfo\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:23:16.039968 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:16.039965    2569 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0da348e1-2882-4e40-9563-976e64575a49-istiod-ca-cert\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:23:16.039968 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:16.039973    2569 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0da348e1-2882-4e40-9563-976e64575a49-istio-token\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:23:16.040175 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:16.039981    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zh94h\" (UniqueName: \"kubernetes.io/projected/0da348e1-2882-4e40-9563-976e64575a49-kube-api-access-zh94h\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:23:16.040175 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:16.039990    2569 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0da348e1-2882-4e40-9563-976e64575a49-istio-envoy\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:23:16.334693 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:16.334662    2569 generic.go:356] "Generic (PLEG): container finished" podID="0da348e1-2882-4e40-9563-976e64575a49" containerID="eb067e6fd84177eea7b59e4aae222556fa4ce5b17722682860a6148b70d2ac4b" exitCode=0
Sep 11 01:23:16.334911 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:16.334705    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-816605-openshift-default-76dd9c445d-259dd"
Sep 11 01:23:16.334911 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:16.334752    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-816605-openshift-default-76dd9c445d-259dd" event={"ID":"0da348e1-2882-4e40-9563-976e64575a49","Type":"ContainerDied","Data":"eb067e6fd84177eea7b59e4aae222556fa4ce5b17722682860a6148b70d2ac4b"}
Sep 11 01:23:16.334911 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:16.334811    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-816605-openshift-default-76dd9c445d-259dd" event={"ID":"0da348e1-2882-4e40-9563-976e64575a49","Type":"ContainerDied","Data":"0151aa500f35802892c62271c071323033c6354c5cc53cc08ecfb9a6ce341236"}
Sep 11 01:23:16.334911 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:16.334839    2569 scope.go:117] "RemoveContainer" containerID="eb067e6fd84177eea7b59e4aae222556fa4ce5b17722682860a6148b70d2ac4b"
Sep 11 01:23:16.345163 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:16.345142    2569 scope.go:117] "RemoveContainer" containerID="52aadfc98510fd798c22935b9223fdbee083f17c0928cc816c1b8fd808d62c64"
Sep 11 01:23:16.353852 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:16.353791    2569 scope.go:117] "RemoveContainer" containerID="eb067e6fd84177eea7b59e4aae222556fa4ce5b17722682860a6148b70d2ac4b"
Sep 11 01:23:16.354512 ip-10-0-129-222 kubenswrapper[2569]: E0911 01:23:16.354488    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb067e6fd84177eea7b59e4aae222556fa4ce5b17722682860a6148b70d2ac4b\": container with ID starting with eb067e6fd84177eea7b59e4aae222556fa4ce5b17722682860a6148b70d2ac4b not found: ID does not exist" containerID="eb067e6fd84177eea7b59e4aae222556fa4ce5b17722682860a6148b70d2ac4b"
Sep 11 01:23:16.354677 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:16.354647    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb067e6fd84177eea7b59e4aae222556fa4ce5b17722682860a6148b70d2ac4b"} err="failed to get container status \"eb067e6fd84177eea7b59e4aae222556fa4ce5b17722682860a6148b70d2ac4b\": rpc error: code = NotFound desc = could not find container \"eb067e6fd84177eea7b59e4aae222556fa4ce5b17722682860a6148b70d2ac4b\": container with ID starting with eb067e6fd84177eea7b59e4aae222556fa4ce5b17722682860a6148b70d2ac4b not found: ID does not exist"
Sep 11 01:23:16.354677 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:16.354677    2569 scope.go:117] "RemoveContainer" containerID="52aadfc98510fd798c22935b9223fdbee083f17c0928cc816c1b8fd808d62c64"
Sep 11 01:23:16.355013 ip-10-0-129-222 kubenswrapper[2569]: E0911 01:23:16.354993    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52aadfc98510fd798c22935b9223fdbee083f17c0928cc816c1b8fd808d62c64\": container with ID starting with 52aadfc98510fd798c22935b9223fdbee083f17c0928cc816c1b8fd808d62c64 not found: ID does not exist" containerID="52aadfc98510fd798c22935b9223fdbee083f17c0928cc816c1b8fd808d62c64"
Sep 11 01:23:16.355108 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:16.355021    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52aadfc98510fd798c22935b9223fdbee083f17c0928cc816c1b8fd808d62c64"} err="failed to get container status \"52aadfc98510fd798c22935b9223fdbee083f17c0928cc816c1b8fd808d62c64\": rpc error: code = NotFound desc = could not find container \"52aadfc98510fd798c22935b9223fdbee083f17c0928cc816c1b8fd808d62c64\": container with ID starting with 52aadfc98510fd798c22935b9223fdbee083f17c0928cc816c1b8fd808d62c64 not found: ID does not exist"
Sep 11 01:23:16.356841 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:16.356823    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-816605-openshift-default-76dd9c445d-259dd"]
Sep 11 01:23:16.361124 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:16.361104    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-816605-openshift-default-76dd9c445d-259dd"]
Sep 11 01:23:17.871921 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:17.871888    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0da348e1-2882-4e40-9563-976e64575a49" path="/var/lib/kubelet/pods/0da348e1-2882-4e40-9563-976e64575a49/volumes"
Sep 11 01:23:21.483089 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:21.483054    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-799002-d567ff65f-z74b7"]
Sep 11 01:23:21.483650 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:21.483296    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-799002-d567ff65f-z74b7" podUID="05bdec86-c9eb-44c0-9c76-7e44625a96c7" containerName="payload-processing" containerID="cri-o://a8f41554e6925ced1178920113a5b106830b363870f2a5153d1f6b19174acaa1" gracePeriod=30
Sep 11 01:23:21.494711 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:21.494680    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-799002-b8d69fd78-5chn7"]
Sep 11 01:23:21.495157 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:21.494978    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-799002-b8d69fd78-5chn7" podUID="a465f3ba-d922-48fc-aea7-76be3e6e8c16" containerName="payload-pre-processing" containerID="cri-o://dd108f389896446d217922a4bb39ff98bd647e7caf07ec753b03429994414797" gracePeriod=30
Sep 11 01:23:21.542619 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:21.542585    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-799002-b8d69fd78-5chn7" podUID="a465f3ba-d922-48fc-aea7-76be3e6e8c16" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.132.0.67:9004: connect: connection refused"
Sep 11 01:23:21.542749 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:21.542585    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-799002-d567ff65f-z74b7" podUID="05bdec86-c9eb-44c0-9c76-7e44625a96c7" containerName="payload-processing" probeResult="failure" output="dial tcp 10.132.0.68:9004: connect: connection refused"
Sep 11 01:23:21.650789 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:21.650752    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-799002-qnrdc"]
Sep 11 01:23:21.653349 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:21.653321    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-799002-qnrdc"]
Sep 11 01:23:21.770485 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:21.770461    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-799002-d567ff65f-z74b7"
Sep 11 01:23:21.773598 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:21.773580    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-799002-b8d69fd78-5chn7"
Sep 11 01:23:21.871511 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:21.871481    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c9841099-bb5e-462e-9fa0-e36f75c8fe18" path="/var/lib/kubelet/pods/c9841099-bb5e-462e-9fa0-e36f75c8fe18/volumes"
Sep 11 01:23:21.897011 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:21.896986    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05bdec86-c9eb-44c0-9c76-7e44625a96c7-plugins-config-volume\") pod \"05bdec86-c9eb-44c0-9c76-7e44625a96c7\" (UID: \"05bdec86-c9eb-44c0-9c76-7e44625a96c7\") "
Sep 11 01:23:21.897150 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:21.897033    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tmj4t\" (UniqueName: \"kubernetes.io/projected/05bdec86-c9eb-44c0-9c76-7e44625a96c7-kube-api-access-tmj4t\") pod \"05bdec86-c9eb-44c0-9c76-7e44625a96c7\" (UID: \"05bdec86-c9eb-44c0-9c76-7e44625a96c7\") "
Sep 11 01:23:21.897150 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:21.897125    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a465f3ba-d922-48fc-aea7-76be3e6e8c16-plugins-config-volume\") pod \"a465f3ba-d922-48fc-aea7-76be3e6e8c16\" (UID: \"a465f3ba-d922-48fc-aea7-76be3e6e8c16\") "
Sep 11 01:23:21.897264 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:21.897243    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-67zgk\" (UniqueName: \"kubernetes.io/projected/a465f3ba-d922-48fc-aea7-76be3e6e8c16-kube-api-access-67zgk\") pod \"a465f3ba-d922-48fc-aea7-76be3e6e8c16\" (UID: \"a465f3ba-d922-48fc-aea7-76be3e6e8c16\") "
Sep 11 01:23:21.897381 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:21.897356    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05bdec86-c9eb-44c0-9c76-7e44625a96c7-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "05bdec86-c9eb-44c0-9c76-7e44625a96c7" (UID: "05bdec86-c9eb-44c0-9c76-7e44625a96c7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 01:23:21.897431 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:21.897393    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a465f3ba-d922-48fc-aea7-76be3e6e8c16-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a465f3ba-d922-48fc-aea7-76be3e6e8c16" (UID: "a465f3ba-d922-48fc-aea7-76be3e6e8c16"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 01:23:21.897629 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:21.897609    2569 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05bdec86-c9eb-44c0-9c76-7e44625a96c7-plugins-config-volume\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:23:21.897684 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:21.897631    2569 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a465f3ba-d922-48fc-aea7-76be3e6e8c16-plugins-config-volume\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:23:21.899291 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:21.899271    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05bdec86-c9eb-44c0-9c76-7e44625a96c7-kube-api-access-tmj4t" (OuterVolumeSpecName: "kube-api-access-tmj4t") pod "05bdec86-c9eb-44c0-9c76-7e44625a96c7" (UID: "05bdec86-c9eb-44c0-9c76-7e44625a96c7"). InnerVolumeSpecName "kube-api-access-tmj4t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 01:23:21.899400 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:21.899296    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a465f3ba-d922-48fc-aea7-76be3e6e8c16-kube-api-access-67zgk" (OuterVolumeSpecName: "kube-api-access-67zgk") pod "a465f3ba-d922-48fc-aea7-76be3e6e8c16" (UID: "a465f3ba-d922-48fc-aea7-76be3e6e8c16"). InnerVolumeSpecName "kube-api-access-67zgk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 01:23:21.998707 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:21.998646    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tmj4t\" (UniqueName: \"kubernetes.io/projected/05bdec86-c9eb-44c0-9c76-7e44625a96c7-kube-api-access-tmj4t\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:23:21.998707 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:21.998670    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-67zgk\" (UniqueName: \"kubernetes.io/projected/a465f3ba-d922-48fc-aea7-76be3e6e8c16-kube-api-access-67zgk\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:23:22.375301 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:22.375267    2569 generic.go:356] "Generic (PLEG): container finished" podID="05bdec86-c9eb-44c0-9c76-7e44625a96c7" containerID="a8f41554e6925ced1178920113a5b106830b363870f2a5153d1f6b19174acaa1" exitCode=0
Sep 11 01:23:22.375502 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:22.375310    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-799002-d567ff65f-z74b7"
Sep 11 01:23:22.375502 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:22.375364    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-799002-d567ff65f-z74b7" event={"ID":"05bdec86-c9eb-44c0-9c76-7e44625a96c7","Type":"ContainerDied","Data":"a8f41554e6925ced1178920113a5b106830b363870f2a5153d1f6b19174acaa1"}
Sep 11 01:23:22.375502 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:22.375413    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-799002-d567ff65f-z74b7" event={"ID":"05bdec86-c9eb-44c0-9c76-7e44625a96c7","Type":"ContainerDied","Data":"063369b6ae684f1cc2d85801d40875a1391241dfadbf662fcbeddd123661a434"}
Sep 11 01:23:22.375502 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:22.375441    2569 scope.go:117] "RemoveContainer" containerID="a8f41554e6925ced1178920113a5b106830b363870f2a5153d1f6b19174acaa1"
Sep 11 01:23:22.376824 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:22.376804    2569 generic.go:356] "Generic (PLEG): container finished" podID="a465f3ba-d922-48fc-aea7-76be3e6e8c16" containerID="dd108f389896446d217922a4bb39ff98bd647e7caf07ec753b03429994414797" exitCode=0
Sep 11 01:23:22.376963 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:22.376829    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-799002-b8d69fd78-5chn7" event={"ID":"a465f3ba-d922-48fc-aea7-76be3e6e8c16","Type":"ContainerDied","Data":"dd108f389896446d217922a4bb39ff98bd647e7caf07ec753b03429994414797"}
Sep 11 01:23:22.376963 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:22.376851    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-799002-b8d69fd78-5chn7"
Sep 11 01:23:22.376963 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:22.376899    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-799002-b8d69fd78-5chn7" event={"ID":"a465f3ba-d922-48fc-aea7-76be3e6e8c16","Type":"ContainerDied","Data":"2dcbb1d5ef59981a12c410db8b7c37f9a46c61cf1dc7dcac2248c19e1f4a809b"}
Sep 11 01:23:22.384844 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:22.384825    2569 scope.go:117] "RemoveContainer" containerID="a8f41554e6925ced1178920113a5b106830b363870f2a5153d1f6b19174acaa1"
Sep 11 01:23:22.385159 ip-10-0-129-222 kubenswrapper[2569]: E0911 01:23:22.385136    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8f41554e6925ced1178920113a5b106830b363870f2a5153d1f6b19174acaa1\": container with ID starting with a8f41554e6925ced1178920113a5b106830b363870f2a5153d1f6b19174acaa1 not found: ID does not exist" containerID="a8f41554e6925ced1178920113a5b106830b363870f2a5153d1f6b19174acaa1"
Sep 11 01:23:22.385256 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:22.385163    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8f41554e6925ced1178920113a5b106830b363870f2a5153d1f6b19174acaa1"} err="failed to get container status \"a8f41554e6925ced1178920113a5b106830b363870f2a5153d1f6b19174acaa1\": rpc error: code = NotFound desc = could not find container \"a8f41554e6925ced1178920113a5b106830b363870f2a5153d1f6b19174acaa1\": container with ID starting with a8f41554e6925ced1178920113a5b106830b363870f2a5153d1f6b19174acaa1 not found: ID does not exist"
Sep 11 01:23:22.385256 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:22.385178    2569 scope.go:117] "RemoveContainer" containerID="dd108f389896446d217922a4bb39ff98bd647e7caf07ec753b03429994414797"
Sep 11 01:23:22.393364 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:22.393348    2569 scope.go:117] "RemoveContainer" containerID="dd108f389896446d217922a4bb39ff98bd647e7caf07ec753b03429994414797"
Sep 11 01:23:22.393625 ip-10-0-129-222 kubenswrapper[2569]: E0911 01:23:22.393606    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd108f389896446d217922a4bb39ff98bd647e7caf07ec753b03429994414797\": container with ID starting with dd108f389896446d217922a4bb39ff98bd647e7caf07ec753b03429994414797 not found: ID does not exist" containerID="dd108f389896446d217922a4bb39ff98bd647e7caf07ec753b03429994414797"
Sep 11 01:23:22.393677 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:22.393632    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd108f389896446d217922a4bb39ff98bd647e7caf07ec753b03429994414797"} err="failed to get container status \"dd108f389896446d217922a4bb39ff98bd647e7caf07ec753b03429994414797\": rpc error: code = NotFound desc = could not find container \"dd108f389896446d217922a4bb39ff98bd647e7caf07ec753b03429994414797\": container with ID starting with dd108f389896446d217922a4bb39ff98bd647e7caf07ec753b03429994414797 not found: ID does not exist"
Sep 11 01:23:22.401029 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:22.400993    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-799002-d567ff65f-z74b7"]
Sep 11 01:23:22.403145 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:22.403126    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-799002-d567ff65f-z74b7"]
Sep 11 01:23:22.414758 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:22.414737    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-799002-b8d69fd78-5chn7"]
Sep 11 01:23:22.418841 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:22.418821    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-799002-b8d69fd78-5chn7"]
Sep 11 01:23:23.875153 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:23.872561    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="05bdec86-c9eb-44c0-9c76-7e44625a96c7" path="/var/lib/kubelet/pods/05bdec86-c9eb-44c0-9c76-7e44625a96c7/volumes"
Sep 11 01:23:23.875153 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:23.873195    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a465f3ba-d922-48fc-aea7-76be3e6e8c16" path="/var/lib/kubelet/pods/a465f3ba-d922-48fc-aea7-76be3e6e8c16/volumes"
Sep 11 01:23:23.895551 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:23.895519    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kmx5x"]
Sep 11 01:23:28.095830 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:28.095797    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-799002-openshift-default-69d7698d8d-gfrd4"]
Sep 11 01:23:28.096271 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:28.096028    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-799002-openshift-default-69d7698d8d-gfrd4" podUID="64a075bf-600d-4fd2-8ec1-a74c1f4cedd5" containerName="istio-proxy" containerID="cri-o://199dc80f63c9e8f7fee9b62d64eaec4f1104c63c77a2df2d6277226135a76a30" gracePeriod=30
Sep 11 01:23:28.449049 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:28.448970    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-71ba6d-jr26t"]
Sep 11 01:23:28.449430 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:28.449417    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0da348e1-2882-4e40-9563-976e64575a49" containerName="istio-proxy"
Sep 11 01:23:28.449480 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:28.449432    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="0da348e1-2882-4e40-9563-976e64575a49" containerName="istio-proxy"
Sep 11 01:23:28.449480 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:28.449448    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c9841099-bb5e-462e-9fa0-e36f75c8fe18" containerName="revoke-keys"
Sep 11 01:23:28.449480 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:28.449455    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="c9841099-bb5e-462e-9fa0-e36f75c8fe18" containerName="revoke-keys"
Sep 11 01:23:28.449480 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:28.449466    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="05bdec86-c9eb-44c0-9c76-7e44625a96c7" containerName="payload-processing"
Sep 11 01:23:28.449480 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:28.449473    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="05bdec86-c9eb-44c0-9c76-7e44625a96c7" containerName="payload-processing"
Sep 11 01:23:28.449640 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:28.449483    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0da348e1-2882-4e40-9563-976e64575a49" containerName="istio-proxy"
Sep 11 01:23:28.449640 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:28.449490    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="0da348e1-2882-4e40-9563-976e64575a49" containerName="istio-proxy"
Sep 11 01:23:28.449640 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:28.449498    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a465f3ba-d922-48fc-aea7-76be3e6e8c16" containerName="payload-pre-processing"
Sep 11 01:23:28.449640 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:28.449503    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="a465f3ba-d922-48fc-aea7-76be3e6e8c16" containerName="payload-pre-processing"
Sep 11 01:23:28.449640 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:28.449588    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="e5176249-c100-42e1-ab5b-9bbfcee0c6f2" containerName="istio-proxy"
Sep 11 01:23:28.449640 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:28.449597    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="0da348e1-2882-4e40-9563-976e64575a49" containerName="istio-proxy"
Sep 11 01:23:28.449640 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:28.449606    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="a465f3ba-d922-48fc-aea7-76be3e6e8c16" containerName="payload-pre-processing"
Sep 11 01:23:28.449640 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:28.449612    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="c9841099-bb5e-462e-9fa0-e36f75c8fe18" containerName="revoke-keys"
Sep 11 01:23:28.449640 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:28.449620    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="05bdec86-c9eb-44c0-9c76-7e44625a96c7" containerName="payload-processing"
Sep 11 01:23:28.454095 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:28.454079    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-71ba6d-jr26t"
Sep 11 01:23:28.456639 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:28.456612    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 01:23:28.457168 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:28.457147    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pppkt\""
Sep 11 01:23:28.459408 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:28.459366    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-71ba6d-jr26t"]
Sep 11 01:23:28.551049 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:28.551011    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/647f5789-8cb0-411a-996a-f439f7240739-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-71ba6d-jr26t\" (UID: \"647f5789-8cb0-411a-996a-f439f7240739\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-71ba6d-jr26t"
Sep 11 01:23:28.651946 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:28.651918    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/647f5789-8cb0-411a-996a-f439f7240739-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-71ba6d-jr26t\" (UID: \"647f5789-8cb0-411a-996a-f439f7240739\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-71ba6d-jr26t"
Sep 11 01:23:28.652543 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:28.652524    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/647f5789-8cb0-411a-996a-f439f7240739-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-71ba6d-jr26t\" (UID: \"647f5789-8cb0-411a-996a-f439f7240739\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-71ba6d-jr26t"
Sep 11 01:23:28.766689 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:28.766605    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-71ba6d-jr26t"
Sep 11 01:23:29.041167 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:29.041008    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-71ba6d-jr26t"]
Sep 11 01:23:29.419390 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:29.419354    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-71ba6d-jr26t" event={"ID":"647f5789-8cb0-411a-996a-f439f7240739","Type":"ContainerStarted","Data":"48fa0736957818490546a31c3afb14eb24de6985ab9de9ff92255f405f15cf00"}
Sep 11 01:23:29.419735 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:29.419417    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-71ba6d-jr26t" event={"ID":"647f5789-8cb0-411a-996a-f439f7240739","Type":"ContainerStarted","Data":"5edcc06216ff93147e76df1b1adefbc4039b8240757e80ceacf561101fc4ee2f"}
Sep 11 01:23:29.435038 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:29.434993    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-71ba6d-jr26t" podStartSLOduration=1.434978962 podStartE2EDuration="1.434978962s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 01:23:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 01:23:29.434936013 +0000 UTC m=+1544.086310269" watchObservedRunningTime="2026-09-11 01:23:29.434978962 +0000 UTC m=+1544.086353211"
Sep 11 01:23:30.425969 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:30.425934    2569 generic.go:356] "Generic (PLEG): container finished" podID="647f5789-8cb0-411a-996a-f439f7240739" containerID="48fa0736957818490546a31c3afb14eb24de6985ab9de9ff92255f405f15cf00" exitCode=0
Sep 11 01:23:30.426371 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:30.426014    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-71ba6d-jr26t" event={"ID":"647f5789-8cb0-411a-996a-f439f7240739","Type":"ContainerDied","Data":"48fa0736957818490546a31c3afb14eb24de6985ab9de9ff92255f405f15cf00"}
Sep 11 01:23:31.565132 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:31.565109    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-71ba6d-jr26t"
Sep 11 01:23:31.578762 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:31.578738    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/647f5789-8cb0-411a-996a-f439f7240739-maas-api-service-ca\") pod \"647f5789-8cb0-411a-996a-f439f7240739\" (UID: \"647f5789-8cb0-411a-996a-f439f7240739\") "
Sep 11 01:23:31.579373 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:31.579267    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/647f5789-8cb0-411a-996a-f439f7240739-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "647f5789-8cb0-411a-996a-f439f7240739" (UID: "647f5789-8cb0-411a-996a-f439f7240739"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 01:23:31.680133 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:31.680097    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/647f5789-8cb0-411a-996a-f439f7240739-maas-api-service-ca\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:23:32.440017 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:32.439978    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-71ba6d-jr26t" event={"ID":"647f5789-8cb0-411a-996a-f439f7240739","Type":"ContainerDied","Data":"5edcc06216ff93147e76df1b1adefbc4039b8240757e80ceacf561101fc4ee2f"}
Sep 11 01:23:32.440222 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:32.440031    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-71ba6d-jr26t"
Sep 11 01:23:32.440222 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:32.440054    2569 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5edcc06216ff93147e76df1b1adefbc4039b8240757e80ceacf561101fc4ee2f"
Sep 11 01:23:33.357229 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:33.357204    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-799002-openshift-default-69d7698d8d-gfrd4"
Sep 11 01:23:33.399775 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:33.399705    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dsbjq\" (UniqueName: \"kubernetes.io/projected/64a075bf-600d-4fd2-8ec1-a74c1f4cedd5-kube-api-access-dsbjq\") pod \"64a075bf-600d-4fd2-8ec1-a74c1f4cedd5\" (UID: \"64a075bf-600d-4fd2-8ec1-a74c1f4cedd5\") "
Sep 11 01:23:33.399775 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:33.399758    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64a075bf-600d-4fd2-8ec1-a74c1f4cedd5-workload-certs\") pod \"64a075bf-600d-4fd2-8ec1-a74c1f4cedd5\" (UID: \"64a075bf-600d-4fd2-8ec1-a74c1f4cedd5\") "
Sep 11 01:23:33.399987 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:33.399787    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64a075bf-600d-4fd2-8ec1-a74c1f4cedd5-istio-token\") pod \"64a075bf-600d-4fd2-8ec1-a74c1f4cedd5\" (UID: \"64a075bf-600d-4fd2-8ec1-a74c1f4cedd5\") "
Sep 11 01:23:33.399987 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:33.399819    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64a075bf-600d-4fd2-8ec1-a74c1f4cedd5-istio-data\") pod \"64a075bf-600d-4fd2-8ec1-a74c1f4cedd5\" (UID: \"64a075bf-600d-4fd2-8ec1-a74c1f4cedd5\") "
Sep 11 01:23:33.399987 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:33.399842    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64a075bf-600d-4fd2-8ec1-a74c1f4cedd5-istio-envoy\") pod \"64a075bf-600d-4fd2-8ec1-a74c1f4cedd5\" (UID: \"64a075bf-600d-4fd2-8ec1-a74c1f4cedd5\") "
Sep 11 01:23:33.399987 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:33.399862    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64a075bf-600d-4fd2-8ec1-a74c1f4cedd5-istiod-ca-cert\") pod \"64a075bf-600d-4fd2-8ec1-a74c1f4cedd5\" (UID: \"64a075bf-600d-4fd2-8ec1-a74c1f4cedd5\") "
Sep 11 01:23:33.399987 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:33.399901    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64a075bf-600d-4fd2-8ec1-a74c1f4cedd5-workload-socket\") pod \"64a075bf-600d-4fd2-8ec1-a74c1f4cedd5\" (UID: \"64a075bf-600d-4fd2-8ec1-a74c1f4cedd5\") "
Sep 11 01:23:33.399987 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:33.399953    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64a075bf-600d-4fd2-8ec1-a74c1f4cedd5-istio-podinfo\") pod \"64a075bf-600d-4fd2-8ec1-a74c1f4cedd5\" (UID: \"64a075bf-600d-4fd2-8ec1-a74c1f4cedd5\") "
Sep 11 01:23:33.399987 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:33.399987    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64a075bf-600d-4fd2-8ec1-a74c1f4cedd5-credential-socket\") pod \"64a075bf-600d-4fd2-8ec1-a74c1f4cedd5\" (UID: \"64a075bf-600d-4fd2-8ec1-a74c1f4cedd5\") "
Sep 11 01:23:33.400252 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:33.400119    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64a075bf-600d-4fd2-8ec1-a74c1f4cedd5-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "64a075bf-600d-4fd2-8ec1-a74c1f4cedd5" (UID: "64a075bf-600d-4fd2-8ec1-a74c1f4cedd5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 01:23:33.400412 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:33.400344    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64a075bf-600d-4fd2-8ec1-a74c1f4cedd5-istio-data" (OuterVolumeSpecName: "istio-data") pod "64a075bf-600d-4fd2-8ec1-a74c1f4cedd5" (UID: "64a075bf-600d-4fd2-8ec1-a74c1f4cedd5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 01:23:33.400412 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:33.400360    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64a075bf-600d-4fd2-8ec1-a74c1f4cedd5-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "64a075bf-600d-4fd2-8ec1-a74c1f4cedd5" (UID: "64a075bf-600d-4fd2-8ec1-a74c1f4cedd5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 01:23:33.400412 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:33.400369    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64a075bf-600d-4fd2-8ec1-a74c1f4cedd5-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "64a075bf-600d-4fd2-8ec1-a74c1f4cedd5" (UID: "64a075bf-600d-4fd2-8ec1-a74c1f4cedd5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 01:23:33.400610 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:33.400470    2569 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64a075bf-600d-4fd2-8ec1-a74c1f4cedd5-workload-certs\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:23:33.400610 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:33.400492    2569 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64a075bf-600d-4fd2-8ec1-a74c1f4cedd5-istio-data\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:23:33.400610 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:33.400508    2569 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64a075bf-600d-4fd2-8ec1-a74c1f4cedd5-credential-socket\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:23:33.400772 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:33.400636    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64a075bf-600d-4fd2-8ec1-a74c1f4cedd5-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "64a075bf-600d-4fd2-8ec1-a74c1f4cedd5" (UID: "64a075bf-600d-4fd2-8ec1-a74c1f4cedd5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 01:23:33.402110 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:33.402088    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64a075bf-600d-4fd2-8ec1-a74c1f4cedd5-kube-api-access-dsbjq" (OuterVolumeSpecName: "kube-api-access-dsbjq") pod "64a075bf-600d-4fd2-8ec1-a74c1f4cedd5" (UID: "64a075bf-600d-4fd2-8ec1-a74c1f4cedd5"). InnerVolumeSpecName "kube-api-access-dsbjq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 01:23:33.402222 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:33.402206    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64a075bf-600d-4fd2-8ec1-a74c1f4cedd5-istio-token" (OuterVolumeSpecName: "istio-token") pod "64a075bf-600d-4fd2-8ec1-a74c1f4cedd5" (UID: "64a075bf-600d-4fd2-8ec1-a74c1f4cedd5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 01:23:33.402330 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:33.402309    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/64a075bf-600d-4fd2-8ec1-a74c1f4cedd5-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "64a075bf-600d-4fd2-8ec1-a74c1f4cedd5" (UID: "64a075bf-600d-4fd2-8ec1-a74c1f4cedd5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 01:23:33.402753 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:33.402725    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64a075bf-600d-4fd2-8ec1-a74c1f4cedd5-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "64a075bf-600d-4fd2-8ec1-a74c1f4cedd5" (UID: "64a075bf-600d-4fd2-8ec1-a74c1f4cedd5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 01:23:33.454861 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:33.454830    2569 generic.go:356] "Generic (PLEG): container finished" podID="64a075bf-600d-4fd2-8ec1-a74c1f4cedd5" containerID="199dc80f63c9e8f7fee9b62d64eaec4f1104c63c77a2df2d6277226135a76a30" exitCode=0
Sep 11 01:23:33.455029 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:33.454922    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-799002-openshift-default-69d7698d8d-gfrd4"
Sep 11 01:23:33.455029 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:33.454918    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-799002-openshift-default-69d7698d8d-gfrd4" event={"ID":"64a075bf-600d-4fd2-8ec1-a74c1f4cedd5","Type":"ContainerDied","Data":"199dc80f63c9e8f7fee9b62d64eaec4f1104c63c77a2df2d6277226135a76a30"}
Sep 11 01:23:33.455145 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:33.455065    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-799002-openshift-default-69d7698d8d-gfrd4" event={"ID":"64a075bf-600d-4fd2-8ec1-a74c1f4cedd5","Type":"ContainerDied","Data":"ed838955582c82d0c044329134a1604789260c05efc2a782f06371792312b390"}
Sep 11 01:23:33.455145 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:33.455088    2569 scope.go:117] "RemoveContainer" containerID="199dc80f63c9e8f7fee9b62d64eaec4f1104c63c77a2df2d6277226135a76a30"
Sep 11 01:23:33.467659 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:33.467641    2569 scope.go:117] "RemoveContainer" containerID="a12e5d3dec99af4d34aa5ee0ba2517c060ab0e133b424f3a23c27a531486b7fd"
Sep 11 01:23:33.476183 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:33.476165    2569 scope.go:117] "RemoveContainer" containerID="199dc80f63c9e8f7fee9b62d64eaec4f1104c63c77a2df2d6277226135a76a30"
Sep 11 01:23:33.476483 ip-10-0-129-222 kubenswrapper[2569]: E0911 01:23:33.476458    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"199dc80f63c9e8f7fee9b62d64eaec4f1104c63c77a2df2d6277226135a76a30\": container with ID starting with 199dc80f63c9e8f7fee9b62d64eaec4f1104c63c77a2df2d6277226135a76a30 not found: ID does not exist" containerID="199dc80f63c9e8f7fee9b62d64eaec4f1104c63c77a2df2d6277226135a76a30"
Sep 11 01:23:33.476583 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:33.476486    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"199dc80f63c9e8f7fee9b62d64eaec4f1104c63c77a2df2d6277226135a76a30"} err="failed to get container status \"199dc80f63c9e8f7fee9b62d64eaec4f1104c63c77a2df2d6277226135a76a30\": rpc error: code = NotFound desc = could not find container \"199dc80f63c9e8f7fee9b62d64eaec4f1104c63c77a2df2d6277226135a76a30\": container with ID starting with 199dc80f63c9e8f7fee9b62d64eaec4f1104c63c77a2df2d6277226135a76a30 not found: ID does not exist"
Sep 11 01:23:33.476583 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:33.476504    2569 scope.go:117] "RemoveContainer" containerID="a12e5d3dec99af4d34aa5ee0ba2517c060ab0e133b424f3a23c27a531486b7fd"
Sep 11 01:23:33.476764 ip-10-0-129-222 kubenswrapper[2569]: E0911 01:23:33.476740    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a12e5d3dec99af4d34aa5ee0ba2517c060ab0e133b424f3a23c27a531486b7fd\": container with ID starting with a12e5d3dec99af4d34aa5ee0ba2517c060ab0e133b424f3a23c27a531486b7fd not found: ID does not exist" containerID="a12e5d3dec99af4d34aa5ee0ba2517c060ab0e133b424f3a23c27a531486b7fd"
Sep 11 01:23:33.476823 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:33.476774    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a12e5d3dec99af4d34aa5ee0ba2517c060ab0e133b424f3a23c27a531486b7fd"} err="failed to get container status \"a12e5d3dec99af4d34aa5ee0ba2517c060ab0e133b424f3a23c27a531486b7fd\": rpc error: code = NotFound desc = could not find container \"a12e5d3dec99af4d34aa5ee0ba2517c060ab0e133b424f3a23c27a531486b7fd\": container with ID starting with a12e5d3dec99af4d34aa5ee0ba2517c060ab0e133b424f3a23c27a531486b7fd not found: ID does not exist"
Sep 11 01:23:33.477898 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:33.477861    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-799002-openshift-default-69d7698d8d-gfrd4"]
Sep 11 01:23:33.481465 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:33.481442    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-799002-openshift-default-69d7698d8d-gfrd4"]
Sep 11 01:23:33.502009 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:33.501987    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dsbjq\" (UniqueName: \"kubernetes.io/projected/64a075bf-600d-4fd2-8ec1-a74c1f4cedd5-kube-api-access-dsbjq\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:23:33.502009 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:33.502008    2569 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64a075bf-600d-4fd2-8ec1-a74c1f4cedd5-istio-token\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:23:33.502121 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:33.502018    2569 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64a075bf-600d-4fd2-8ec1-a74c1f4cedd5-istio-envoy\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:23:33.502121 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:33.502026    2569 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64a075bf-600d-4fd2-8ec1-a74c1f4cedd5-istiod-ca-cert\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:23:33.502121 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:33.502033    2569 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64a075bf-600d-4fd2-8ec1-a74c1f4cedd5-workload-socket\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:23:33.502121 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:33.502041    2569 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64a075bf-600d-4fd2-8ec1-a74c1f4cedd5-istio-podinfo\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:23:33.872591 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:33.872562    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="64a075bf-600d-4fd2-8ec1-a74c1f4cedd5" path="/var/lib/kubelet/pods/64a075bf-600d-4fd2-8ec1-a74c1f4cedd5/volumes"
Sep 11 01:23:39.123598 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:39.123567    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-71ba6d-jr26t"]
Sep 11 01:23:39.127631 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:39.127605    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-71ba6d-jr26t"]
Sep 11 01:23:39.870802 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:39.870768    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="647f5789-8cb0-411a-996a-f439f7240739" path="/var/lib/kubelet/pods/647f5789-8cb0-411a-996a-f439f7240739/volumes"
Sep 11 01:23:39.996382 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:39.996351    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kmx5x"]
Sep 11 01:23:45.673802 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:45.673764    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-71ba6d-openshift-default-765f5dc878-82nr2"]
Sep 11 01:23:45.674213 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:45.674115    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-71ba6d-openshift-default-765f5dc878-82nr2" podUID="a9896000-8b68-4e4a-afd6-b3dd99e275b6" containerName="istio-proxy" containerID="cri-o://82a222ba16c60d65879eccc766539947eaf362e3b5e513b38720cfb54d71a53a" gracePeriod=30
Sep 11 01:23:50.934742 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:50.934715    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-71ba6d-openshift-default-765f5dc878-82nr2"
Sep 11 01:23:51.070310 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:51.070277    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9896000-8b68-4e4a-afd6-b3dd99e275b6-istiod-ca-cert\") pod \"a9896000-8b68-4e4a-afd6-b3dd99e275b6\" (UID: \"a9896000-8b68-4e4a-afd6-b3dd99e275b6\") "
Sep 11 01:23:51.070474 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:51.070325    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9896000-8b68-4e4a-afd6-b3dd99e275b6-workload-certs\") pod \"a9896000-8b68-4e4a-afd6-b3dd99e275b6\" (UID: \"a9896000-8b68-4e4a-afd6-b3dd99e275b6\") "
Sep 11 01:23:51.070474 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:51.070376    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9896000-8b68-4e4a-afd6-b3dd99e275b6-istio-podinfo\") pod \"a9896000-8b68-4e4a-afd6-b3dd99e275b6\" (UID: \"a9896000-8b68-4e4a-afd6-b3dd99e275b6\") "
Sep 11 01:23:51.070474 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:51.070434    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9896000-8b68-4e4a-afd6-b3dd99e275b6-istio-token\") pod \"a9896000-8b68-4e4a-afd6-b3dd99e275b6\" (UID: \"a9896000-8b68-4e4a-afd6-b3dd99e275b6\") "
Sep 11 01:23:51.070474 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:51.070467    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9896000-8b68-4e4a-afd6-b3dd99e275b6-workload-socket\") pod \"a9896000-8b68-4e4a-afd6-b3dd99e275b6\" (UID: \"a9896000-8b68-4e4a-afd6-b3dd99e275b6\") "
Sep 11 01:23:51.070699 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:51.070503    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9896000-8b68-4e4a-afd6-b3dd99e275b6-istio-data\") pod \"a9896000-8b68-4e4a-afd6-b3dd99e275b6\" (UID: \"a9896000-8b68-4e4a-afd6-b3dd99e275b6\") "
Sep 11 01:23:51.070699 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:51.070525    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9896000-8b68-4e4a-afd6-b3dd99e275b6-istio-envoy\") pod \"a9896000-8b68-4e4a-afd6-b3dd99e275b6\" (UID: \"a9896000-8b68-4e4a-afd6-b3dd99e275b6\") "
Sep 11 01:23:51.070699 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:51.070604    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4vsc7\" (UniqueName: \"kubernetes.io/projected/a9896000-8b68-4e4a-afd6-b3dd99e275b6-kube-api-access-4vsc7\") pod \"a9896000-8b68-4e4a-afd6-b3dd99e275b6\" (UID: \"a9896000-8b68-4e4a-afd6-b3dd99e275b6\") "
Sep 11 01:23:51.070699 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:51.070643    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9896000-8b68-4e4a-afd6-b3dd99e275b6-credential-socket\") pod \"a9896000-8b68-4e4a-afd6-b3dd99e275b6\" (UID: \"a9896000-8b68-4e4a-afd6-b3dd99e275b6\") "
Sep 11 01:23:51.070699 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:51.070651    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9896000-8b68-4e4a-afd6-b3dd99e275b6-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "a9896000-8b68-4e4a-afd6-b3dd99e275b6" (UID: "a9896000-8b68-4e4a-afd6-b3dd99e275b6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 01:23:51.070973 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:51.070695    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9896000-8b68-4e4a-afd6-b3dd99e275b6-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "a9896000-8b68-4e4a-afd6-b3dd99e275b6" (UID: "a9896000-8b68-4e4a-afd6-b3dd99e275b6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 01:23:51.070973 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:51.070793    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9896000-8b68-4e4a-afd6-b3dd99e275b6-istio-data" (OuterVolumeSpecName: "istio-data") pod "a9896000-8b68-4e4a-afd6-b3dd99e275b6" (UID: "a9896000-8b68-4e4a-afd6-b3dd99e275b6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 01:23:51.070973 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:51.070804    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9896000-8b68-4e4a-afd6-b3dd99e275b6-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "a9896000-8b68-4e4a-afd6-b3dd99e275b6" (UID: "a9896000-8b68-4e4a-afd6-b3dd99e275b6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 01:23:51.071116 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:51.071044    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9896000-8b68-4e4a-afd6-b3dd99e275b6-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "a9896000-8b68-4e4a-afd6-b3dd99e275b6" (UID: "a9896000-8b68-4e4a-afd6-b3dd99e275b6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 01:23:51.071116 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:51.071087    2569 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9896000-8b68-4e4a-afd6-b3dd99e275b6-istiod-ca-cert\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:23:51.071116 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:51.071108    2569 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9896000-8b68-4e4a-afd6-b3dd99e275b6-workload-certs\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:23:51.071236 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:51.071120    2569 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9896000-8b68-4e4a-afd6-b3dd99e275b6-workload-socket\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:23:51.071236 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:51.071134    2569 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9896000-8b68-4e4a-afd6-b3dd99e275b6-istio-data\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:23:51.072604 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:51.072581    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a9896000-8b68-4e4a-afd6-b3dd99e275b6-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "a9896000-8b68-4e4a-afd6-b3dd99e275b6" (UID: "a9896000-8b68-4e4a-afd6-b3dd99e275b6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 01:23:51.072740 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:51.072719    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9896000-8b68-4e4a-afd6-b3dd99e275b6-kube-api-access-4vsc7" (OuterVolumeSpecName: "kube-api-access-4vsc7") pod "a9896000-8b68-4e4a-afd6-b3dd99e275b6" (UID: "a9896000-8b68-4e4a-afd6-b3dd99e275b6"). InnerVolumeSpecName "kube-api-access-4vsc7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 01:23:51.073031 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:51.073012    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9896000-8b68-4e4a-afd6-b3dd99e275b6-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "a9896000-8b68-4e4a-afd6-b3dd99e275b6" (UID: "a9896000-8b68-4e4a-afd6-b3dd99e275b6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 01:23:51.073031 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:51.073022    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9896000-8b68-4e4a-afd6-b3dd99e275b6-istio-token" (OuterVolumeSpecName: "istio-token") pod "a9896000-8b68-4e4a-afd6-b3dd99e275b6" (UID: "a9896000-8b68-4e4a-afd6-b3dd99e275b6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 01:23:51.172032 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:51.171995    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4vsc7\" (UniqueName: \"kubernetes.io/projected/a9896000-8b68-4e4a-afd6-b3dd99e275b6-kube-api-access-4vsc7\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:23:51.172032 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:51.172026    2569 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9896000-8b68-4e4a-afd6-b3dd99e275b6-credential-socket\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:23:51.172032 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:51.172037    2569 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9896000-8b68-4e4a-afd6-b3dd99e275b6-istio-podinfo\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:23:51.172253 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:51.172045    2569 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9896000-8b68-4e4a-afd6-b3dd99e275b6-istio-token\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:23:51.172253 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:51.172053    2569 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9896000-8b68-4e4a-afd6-b3dd99e275b6-istio-envoy\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:23:51.558922 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:51.558860    2569 generic.go:356] "Generic (PLEG): container finished" podID="a9896000-8b68-4e4a-afd6-b3dd99e275b6" containerID="82a222ba16c60d65879eccc766539947eaf362e3b5e513b38720cfb54d71a53a" exitCode=0
Sep 11 01:23:51.559075 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:51.558934    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-71ba6d-openshift-default-765f5dc878-82nr2"
Sep 11 01:23:51.559075 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:51.558951    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-71ba6d-openshift-default-765f5dc878-82nr2" event={"ID":"a9896000-8b68-4e4a-afd6-b3dd99e275b6","Type":"ContainerDied","Data":"82a222ba16c60d65879eccc766539947eaf362e3b5e513b38720cfb54d71a53a"}
Sep 11 01:23:51.559075 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:51.559039    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-71ba6d-openshift-default-765f5dc878-82nr2" event={"ID":"a9896000-8b68-4e4a-afd6-b3dd99e275b6","Type":"ContainerDied","Data":"0f1f4c29bb4053e330adae29340e4e5ccea9374ea04975a3158b4e51073b72bd"}
Sep 11 01:23:51.559075 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:51.559063    2569 scope.go:117] "RemoveContainer" containerID="82a222ba16c60d65879eccc766539947eaf362e3b5e513b38720cfb54d71a53a"
Sep 11 01:23:51.568341 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:51.568326    2569 scope.go:117] "RemoveContainer" containerID="738a9435af486af08772608e7259fe29e4cb1fa841172dd3a81aef04897191cc"
Sep 11 01:23:51.576474 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:51.576459    2569 scope.go:117] "RemoveContainer" containerID="82a222ba16c60d65879eccc766539947eaf362e3b5e513b38720cfb54d71a53a"
Sep 11 01:23:51.576736 ip-10-0-129-222 kubenswrapper[2569]: E0911 01:23:51.576714    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82a222ba16c60d65879eccc766539947eaf362e3b5e513b38720cfb54d71a53a\": container with ID starting with 82a222ba16c60d65879eccc766539947eaf362e3b5e513b38720cfb54d71a53a not found: ID does not exist" containerID="82a222ba16c60d65879eccc766539947eaf362e3b5e513b38720cfb54d71a53a"
Sep 11 01:23:51.576775 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:51.576745    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82a222ba16c60d65879eccc766539947eaf362e3b5e513b38720cfb54d71a53a"} err="failed to get container status \"82a222ba16c60d65879eccc766539947eaf362e3b5e513b38720cfb54d71a53a\": rpc error: code = NotFound desc = could not find container \"82a222ba16c60d65879eccc766539947eaf362e3b5e513b38720cfb54d71a53a\": container with ID starting with 82a222ba16c60d65879eccc766539947eaf362e3b5e513b38720cfb54d71a53a not found: ID does not exist"
Sep 11 01:23:51.576775 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:51.576760    2569 scope.go:117] "RemoveContainer" containerID="738a9435af486af08772608e7259fe29e4cb1fa841172dd3a81aef04897191cc"
Sep 11 01:23:51.577034 ip-10-0-129-222 kubenswrapper[2569]: E0911 01:23:51.577019    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"738a9435af486af08772608e7259fe29e4cb1fa841172dd3a81aef04897191cc\": container with ID starting with 738a9435af486af08772608e7259fe29e4cb1fa841172dd3a81aef04897191cc not found: ID does not exist" containerID="738a9435af486af08772608e7259fe29e4cb1fa841172dd3a81aef04897191cc"
Sep 11 01:23:51.577082 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:51.577042    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"738a9435af486af08772608e7259fe29e4cb1fa841172dd3a81aef04897191cc"} err="failed to get container status \"738a9435af486af08772608e7259fe29e4cb1fa841172dd3a81aef04897191cc\": rpc error: code = NotFound desc = could not find container \"738a9435af486af08772608e7259fe29e4cb1fa841172dd3a81aef04897191cc\": container with ID starting with 738a9435af486af08772608e7259fe29e4cb1fa841172dd3a81aef04897191cc not found: ID does not exist"
Sep 11 01:23:51.579654 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:51.579633    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-71ba6d-openshift-default-765f5dc878-82nr2"]
Sep 11 01:23:51.583631 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:51.583611    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-71ba6d-openshift-default-765f5dc878-82nr2"]
Sep 11 01:23:51.870998 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:51.870925    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a9896000-8b68-4e4a-afd6-b3dd99e275b6" path="/var/lib/kubelet/pods/a9896000-8b68-4e4a-afd6-b3dd99e275b6/volumes"
Sep 11 01:23:53.392089 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:23:53.392057    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kmx5x"]
Sep 11 01:24:06.091762 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:24:06.091727    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kmx5x"]
Sep 11 01:24:19.605269 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:24:19.605235    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kmx5x"]
Sep 11 01:24:29.096357 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:24:29.096316    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kmx5x"]
Sep 11 01:24:46.199665 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:24:46.199588    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kmx5x"]
Sep 11 01:25:07.391162 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:25:07.391130    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kmx5x"]
Sep 11 01:25:24.004372 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:25:24.004337    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kmx5x"]
Sep 11 01:25:39.801341 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:25:39.801303    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kmx5x"]
Sep 11 01:25:50.504564 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:25:50.504531    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kmx5x"]
Sep 11 01:25:56.213118 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:25:56.213078    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kmx5x"]
Sep 11 01:26:27.997212 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:26:27.997175    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kmx5x"]
Sep 11 01:26:44.491664 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:26:44.491625    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kmx5x"]
Sep 11 01:26:46.265656 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:26:46.265616    2569 scope.go:117] "RemoveContainer" containerID="f2c62342ad8252c6a21fd124a72ed103d3e77bc8396084717fd0bc663cbf5cfa"
Sep 11 01:26:59.294456 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:26:59.294424    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kmx5x"]
Sep 11 01:27:09.802758 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:27:09.802723    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kmx5x"]
Sep 11 01:27:20.194860 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:27:20.194823    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kmx5x"]
Sep 11 01:27:41.274582 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:27:41.274507    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-bae8d3-7ddfd8cc4-tmb2s"]
Sep 11 01:27:41.277015 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:27:41.274933    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a9896000-8b68-4e4a-afd6-b3dd99e275b6" containerName="istio-proxy"
Sep 11 01:27:41.277015 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:27:41.274945    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="a9896000-8b68-4e4a-afd6-b3dd99e275b6" containerName="istio-proxy"
Sep 11 01:27:41.277015 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:27:41.274961    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="64a075bf-600d-4fd2-8ec1-a74c1f4cedd5" containerName="istio-proxy"
Sep 11 01:27:41.277015 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:27:41.274966    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="64a075bf-600d-4fd2-8ec1-a74c1f4cedd5" containerName="istio-proxy"
Sep 11 01:27:41.277015 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:27:41.274980    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="64a075bf-600d-4fd2-8ec1-a74c1f4cedd5" containerName="istio-proxy"
Sep 11 01:27:41.277015 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:27:41.274986    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="64a075bf-600d-4fd2-8ec1-a74c1f4cedd5" containerName="istio-proxy"
Sep 11 01:27:41.277015 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:27:41.274999    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="647f5789-8cb0-411a-996a-f439f7240739" containerName="revoke-keys"
Sep 11 01:27:41.277015 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:27:41.275004    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="647f5789-8cb0-411a-996a-f439f7240739" containerName="revoke-keys"
Sep 11 01:27:41.277015 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:27:41.275059    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="64a075bf-600d-4fd2-8ec1-a74c1f4cedd5" containerName="istio-proxy"
Sep 11 01:27:41.277015 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:27:41.275066    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="647f5789-8cb0-411a-996a-f439f7240739" containerName="revoke-keys"
Sep 11 01:27:41.277015 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:27:41.275071    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="a9896000-8b68-4e4a-afd6-b3dd99e275b6" containerName="istio-proxy"
Sep 11 01:27:41.277015 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:27:41.275077    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="0da348e1-2882-4e40-9563-976e64575a49" containerName="istio-proxy"
Sep 11 01:27:41.277015 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:27:41.275083    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="64a075bf-600d-4fd2-8ec1-a74c1f4cedd5" containerName="istio-proxy"
Sep 11 01:27:41.277015 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:27:41.275090    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="a9896000-8b68-4e4a-afd6-b3dd99e275b6" containerName="istio-proxy"
Sep 11 01:27:41.278036 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:27:41.278021    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-bae8d3-7ddfd8cc4-tmb2s"
Sep 11 01:27:41.280441 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:27:41.280418    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-bae8d3\""
Sep 11 01:27:41.288529 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:27:41.288506    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-bae8d3-7ddfd8cc4-tmb2s"]
Sep 11 01:27:41.299407 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:27:41.299376    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-bae8d3-5db4c6fdd5-ldnjc"]
Sep 11 01:27:41.299840 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:27:41.299822    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a9896000-8b68-4e4a-afd6-b3dd99e275b6" containerName="istio-proxy"
Sep 11 01:27:41.299948 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:27:41.299843    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="a9896000-8b68-4e4a-afd6-b3dd99e275b6" containerName="istio-proxy"
Sep 11 01:27:41.303314 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:27:41.303295    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-bae8d3-5db4c6fdd5-ldnjc"
Sep 11 01:27:41.305533 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:27:41.305514    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-bae8d3-dockercfg-qgzzv\""
Sep 11 01:27:41.313833 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:27:41.313804    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-bae8d3-5db4c6fdd5-ldnjc"]
Sep 11 01:27:41.438407 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:27:41.438376    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e746168-41d8-41d8-a35d-0730f7f40ed7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-bae8d3-7ddfd8cc4-tmb2s\" (UID: \"8e746168-41d8-41d8-a35d-0730f7f40ed7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-bae8d3-7ddfd8cc4-tmb2s"
Sep 11 01:27:41.438565 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:27:41.438458    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4507e865-3c97-479b-a029-f9bee7af7d36-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-bae8d3-5db4c6fdd5-ldnjc\" (UID: \"4507e865-3c97-479b-a029-f9bee7af7d36\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-bae8d3-5db4c6fdd5-ldnjc"
Sep 11 01:27:41.438565 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:27:41.438493    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pppqv\" (UniqueName: \"kubernetes.io/projected/8e746168-41d8-41d8-a35d-0730f7f40ed7-kube-api-access-pppqv\") pod \"payload-pre-processing-e2e-ipp-b-bae8d3-7ddfd8cc4-tmb2s\" (UID: \"8e746168-41d8-41d8-a35d-0730f7f40ed7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-bae8d3-7ddfd8cc4-tmb2s"
Sep 11 01:27:41.438565 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:27:41.438515    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7wgqb\" (UniqueName: \"kubernetes.io/projected/4507e865-3c97-479b-a029-f9bee7af7d36-kube-api-access-7wgqb\") pod \"payload-processing-e2e-ipp-b-bae8d3-5db4c6fdd5-ldnjc\" (UID: \"4507e865-3c97-479b-a029-f9bee7af7d36\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-bae8d3-5db4c6fdd5-ldnjc"
Sep 11 01:27:41.539757 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:27:41.539653    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4507e865-3c97-479b-a029-f9bee7af7d36-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-bae8d3-5db4c6fdd5-ldnjc\" (UID: \"4507e865-3c97-479b-a029-f9bee7af7d36\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-bae8d3-5db4c6fdd5-ldnjc"
Sep 11 01:27:41.539757 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:27:41.539743    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pppqv\" (UniqueName: \"kubernetes.io/projected/8e746168-41d8-41d8-a35d-0730f7f40ed7-kube-api-access-pppqv\") pod \"payload-pre-processing-e2e-ipp-b-bae8d3-7ddfd8cc4-tmb2s\" (UID: \"8e746168-41d8-41d8-a35d-0730f7f40ed7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-bae8d3-7ddfd8cc4-tmb2s"
Sep 11 01:27:41.539975 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:27:41.539780    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wgqb\" (UniqueName: \"kubernetes.io/projected/4507e865-3c97-479b-a029-f9bee7af7d36-kube-api-access-7wgqb\") pod \"payload-processing-e2e-ipp-b-bae8d3-5db4c6fdd5-ldnjc\" (UID: \"4507e865-3c97-479b-a029-f9bee7af7d36\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-bae8d3-5db4c6fdd5-ldnjc"
Sep 11 01:27:41.539975 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:27:41.539840    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e746168-41d8-41d8-a35d-0730f7f40ed7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-bae8d3-7ddfd8cc4-tmb2s\" (UID: \"8e746168-41d8-41d8-a35d-0730f7f40ed7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-bae8d3-7ddfd8cc4-tmb2s"
Sep 11 01:27:41.540413 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:27:41.540393    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4507e865-3c97-479b-a029-f9bee7af7d36-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-bae8d3-5db4c6fdd5-ldnjc\" (UID: \"4507e865-3c97-479b-a029-f9bee7af7d36\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-bae8d3-5db4c6fdd5-ldnjc"
Sep 11 01:27:41.540709 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:27:41.540688    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e746168-41d8-41d8-a35d-0730f7f40ed7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-bae8d3-7ddfd8cc4-tmb2s\" (UID: \"8e746168-41d8-41d8-a35d-0730f7f40ed7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-bae8d3-7ddfd8cc4-tmb2s"
Sep 11 01:27:41.548989 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:27:41.548968    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pppqv\" (UniqueName: \"kubernetes.io/projected/8e746168-41d8-41d8-a35d-0730f7f40ed7-kube-api-access-pppqv\") pod \"payload-pre-processing-e2e-ipp-b-bae8d3-7ddfd8cc4-tmb2s\" (UID: \"8e746168-41d8-41d8-a35d-0730f7f40ed7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-bae8d3-7ddfd8cc4-tmb2s"
Sep 11 01:27:41.549127 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:27:41.549028    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wgqb\" (UniqueName: \"kubernetes.io/projected/4507e865-3c97-479b-a029-f9bee7af7d36-kube-api-access-7wgqb\") pod \"payload-processing-e2e-ipp-b-bae8d3-5db4c6fdd5-ldnjc\" (UID: \"4507e865-3c97-479b-a029-f9bee7af7d36\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-bae8d3-5db4c6fdd5-ldnjc"
Sep 11 01:27:41.588574 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:27:41.588548    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-bae8d3-7ddfd8cc4-tmb2s"
Sep 11 01:27:41.613725 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:27:41.613681    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-bae8d3-5db4c6fdd5-ldnjc"
Sep 11 01:27:41.864658 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:27:41.864634    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-bae8d3-7ddfd8cc4-tmb2s"]
Sep 11 01:27:41.867117 ip-10-0-129-222 kubenswrapper[2569]: W0911 01:27:41.867083    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e746168_41d8_41d8_a35d_0730f7f40ed7.slice/crio-a8e87ca09820e7007aba63d52ac2556fa9db556028ed70599ba4714a05d88b53 WatchSource:0}: Error finding container a8e87ca09820e7007aba63d52ac2556fa9db556028ed70599ba4714a05d88b53: Status 404 returned error can't find the container with id a8e87ca09820e7007aba63d52ac2556fa9db556028ed70599ba4714a05d88b53
Sep 11 01:27:41.901530 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:27:41.901472    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-bae8d3-5db4c6fdd5-ldnjc"]
Sep 11 01:27:41.910076 ip-10-0-129-222 kubenswrapper[2569]: W0911 01:27:41.910050    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4507e865_3c97_479b_a029_f9bee7af7d36.slice/crio-05ba74fd088ea7596e59c9fb4119177555ff4c8d76e1f525eab6d524a6d21459 WatchSource:0}: Error finding container 05ba74fd088ea7596e59c9fb4119177555ff4c8d76e1f525eab6d524a6d21459: Status 404 returned error can't find the container with id 05ba74fd088ea7596e59c9fb4119177555ff4c8d76e1f525eab6d524a6d21459
Sep 11 01:27:42.802907 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:27:42.802848    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-bae8d3-5db4c6fdd5-ldnjc" event={"ID":"4507e865-3c97-479b-a029-f9bee7af7d36","Type":"ContainerStarted","Data":"01d0b916991102eaa52972981f57424ee816853bedb32b9bde60cc9218a5dc2c"}
Sep 11 01:27:42.803331 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:27:42.802923    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-bae8d3-5db4c6fdd5-ldnjc" event={"ID":"4507e865-3c97-479b-a029-f9bee7af7d36","Type":"ContainerStarted","Data":"05ba74fd088ea7596e59c9fb4119177555ff4c8d76e1f525eab6d524a6d21459"}
Sep 11 01:27:42.803331 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:27:42.802949    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-bae8d3-5db4c6fdd5-ldnjc"
Sep 11 01:27:42.804435 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:27:42.804416    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-bae8d3-7ddfd8cc4-tmb2s" event={"ID":"8e746168-41d8-41d8-a35d-0730f7f40ed7","Type":"ContainerStarted","Data":"692ae21d4de354f40cf20852e0527b9a89fe411f5c88a676de4f6e94a4913d33"}
Sep 11 01:27:42.804486 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:27:42.804456    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-bae8d3-7ddfd8cc4-tmb2s" event={"ID":"8e746168-41d8-41d8-a35d-0730f7f40ed7","Type":"ContainerStarted","Data":"a8e87ca09820e7007aba63d52ac2556fa9db556028ed70599ba4714a05d88b53"}
Sep 11 01:27:42.804561 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:27:42.804547    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-bae8d3-7ddfd8cc4-tmb2s"
Sep 11 01:27:42.820379 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:27:42.820329    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-bae8d3-5db4c6fdd5-ldnjc" podStartSLOduration=1.8203172429999999 podStartE2EDuration="1.820317243s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 01:27:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 01:27:42.817732172 +0000 UTC m=+1797.469106432" watchObservedRunningTime="2026-09-11 01:27:42.820317243 +0000 UTC m=+1797.471691531"
Sep 11 01:27:42.831612 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:27:42.831573    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-bae8d3-7ddfd8cc4-tmb2s" podStartSLOduration=1.831558228 podStartE2EDuration="1.831558228s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 01:27:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 01:27:42.83115976 +0000 UTC m=+1797.482534007" watchObservedRunningTime="2026-09-11 01:27:42.831558228 +0000 UTC m=+1797.482932474"
Sep 11 01:27:46.805051 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:27:46.805015    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kmx5x"]
Sep 11 01:27:53.812616 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:27:53.812581    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-bae8d3-5db4c6fdd5-ldnjc"
Sep 11 01:27:53.813005 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:27:53.812761    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-bae8d3-7ddfd8cc4-tmb2s"
Sep 11 01:28:09.685801 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:28:09.685772    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-bae8d3-5db4c6fdd5-ldnjc_4507e865-3c97-479b-a029-f9bee7af7d36/payload-processing/0.log"
Sep 11 01:28:23.101575 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:28:23.101498    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kmx5x"]
Sep 11 01:28:46.293795 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:28:46.293756    2569 scope.go:117] "RemoveContainer" containerID="ac6c1ba27f04c8b5dbdebd4678596f1a93af2126f42dcf9eac42e6c0893eaf18"
Sep 11 01:28:46.303042 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:28:46.303016    2569 scope.go:117] "RemoveContainer" containerID="b1858b088e8a588a7543185cbcba3bbe69cd7809e4e10d72dd461634c690854b"
Sep 11 01:28:46.311185 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:28:46.311163    2569 scope.go:117] "RemoveContainer" containerID="bb9e1fb47946d56bb2262ee7692d2b74c4e2d85e0cb4616a032354142ce82ed9"
Sep 11 01:28:49.500084 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:28:49.500048    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kmx5x"]
Sep 11 01:28:56.055299 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:28:56.055271    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-bae8d3-5db4c6fdd5-ldnjc_4507e865-3c97-479b-a029-f9bee7af7d36/payload-processing/0.log"
Sep 11 01:28:56.609204 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:28:56.609165    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-40e6bb-openshift-default-579dfcb59d-h7rck"]
Sep 11 01:28:56.613534 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:28:56.613510    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-40e6bb-openshift-default-579dfcb59d-h7rck"
Sep 11 01:28:56.615861 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:28:56.615841    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-cleanup-40e6bb-openshift-default-dockercfg-gl5dx\""
Sep 11 01:28:56.622754 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:28:56.622733    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-40e6bb-openshift-default-579dfcb59d-h7rck"]
Sep 11 01:28:56.740353 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:28:56.740321    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c57f95d5-daa7-4dee-a41b-4291e69c2b70-workload-certs\") pod \"e2e-ipp-cleanup-40e6bb-openshift-default-579dfcb59d-h7rck\" (UID: \"c57f95d5-daa7-4dee-a41b-4291e69c2b70\") " pod="openshift-ingress/e2e-ipp-cleanup-40e6bb-openshift-default-579dfcb59d-h7rck"
Sep 11 01:28:56.740498 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:28:56.740362    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c57f95d5-daa7-4dee-a41b-4291e69c2b70-workload-socket\") pod \"e2e-ipp-cleanup-40e6bb-openshift-default-579dfcb59d-h7rck\" (UID: \"c57f95d5-daa7-4dee-a41b-4291e69c2b70\") " pod="openshift-ingress/e2e-ipp-cleanup-40e6bb-openshift-default-579dfcb59d-h7rck"
Sep 11 01:28:56.740498 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:28:56.740384    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c57f95d5-daa7-4dee-a41b-4291e69c2b70-credential-socket\") pod \"e2e-ipp-cleanup-40e6bb-openshift-default-579dfcb59d-h7rck\" (UID: \"c57f95d5-daa7-4dee-a41b-4291e69c2b70\") " pod="openshift-ingress/e2e-ipp-cleanup-40e6bb-openshift-default-579dfcb59d-h7rck"
Sep 11 01:28:56.740498 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:28:56.740454    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdqfx\" (UniqueName: \"kubernetes.io/projected/c57f95d5-daa7-4dee-a41b-4291e69c2b70-kube-api-access-fdqfx\") pod \"e2e-ipp-cleanup-40e6bb-openshift-default-579dfcb59d-h7rck\" (UID: \"c57f95d5-daa7-4dee-a41b-4291e69c2b70\") " pod="openshift-ingress/e2e-ipp-cleanup-40e6bb-openshift-default-579dfcb59d-h7rck"
Sep 11 01:28:56.740498 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:28:56.740484    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c57f95d5-daa7-4dee-a41b-4291e69c2b70-istio-envoy\") pod \"e2e-ipp-cleanup-40e6bb-openshift-default-579dfcb59d-h7rck\" (UID: \"c57f95d5-daa7-4dee-a41b-4291e69c2b70\") " pod="openshift-ingress/e2e-ipp-cleanup-40e6bb-openshift-default-579dfcb59d-h7rck"
Sep 11 01:28:56.740654 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:28:56.740507    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c57f95d5-daa7-4dee-a41b-4291e69c2b70-istio-data\") pod \"e2e-ipp-cleanup-40e6bb-openshift-default-579dfcb59d-h7rck\" (UID: \"c57f95d5-daa7-4dee-a41b-4291e69c2b70\") " pod="openshift-ingress/e2e-ipp-cleanup-40e6bb-openshift-default-579dfcb59d-h7rck"
Sep 11 01:28:56.740654 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:28:56.740551    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c57f95d5-daa7-4dee-a41b-4291e69c2b70-istio-token\") pod \"e2e-ipp-cleanup-40e6bb-openshift-default-579dfcb59d-h7rck\" (UID: \"c57f95d5-daa7-4dee-a41b-4291e69c2b70\") " pod="openshift-ingress/e2e-ipp-cleanup-40e6bb-openshift-default-579dfcb59d-h7rck"
Sep 11 01:28:56.740654 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:28:56.740611    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c57f95d5-daa7-4dee-a41b-4291e69c2b70-istio-podinfo\") pod \"e2e-ipp-cleanup-40e6bb-openshift-default-579dfcb59d-h7rck\" (UID: \"c57f95d5-daa7-4dee-a41b-4291e69c2b70\") " pod="openshift-ingress/e2e-ipp-cleanup-40e6bb-openshift-default-579dfcb59d-h7rck"
Sep 11 01:28:56.740755 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:28:56.740657    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c57f95d5-daa7-4dee-a41b-4291e69c2b70-istiod-ca-cert\") pod \"e2e-ipp-cleanup-40e6bb-openshift-default-579dfcb59d-h7rck\" (UID: \"c57f95d5-daa7-4dee-a41b-4291e69c2b70\") " pod="openshift-ingress/e2e-ipp-cleanup-40e6bb-openshift-default-579dfcb59d-h7rck"
Sep 11 01:28:56.842044 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:28:56.842009    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c57f95d5-daa7-4dee-a41b-4291e69c2b70-istio-token\") pod \"e2e-ipp-cleanup-40e6bb-openshift-default-579dfcb59d-h7rck\" (UID: \"c57f95d5-daa7-4dee-a41b-4291e69c2b70\") " pod="openshift-ingress/e2e-ipp-cleanup-40e6bb-openshift-default-579dfcb59d-h7rck"
Sep 11 01:28:56.842230 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:28:56.842057    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c57f95d5-daa7-4dee-a41b-4291e69c2b70-istio-podinfo\") pod \"e2e-ipp-cleanup-40e6bb-openshift-default-579dfcb59d-h7rck\" (UID: \"c57f95d5-daa7-4dee-a41b-4291e69c2b70\") " pod="openshift-ingress/e2e-ipp-cleanup-40e6bb-openshift-default-579dfcb59d-h7rck"
Sep 11 01:28:56.842230 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:28:56.842091    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c57f95d5-daa7-4dee-a41b-4291e69c2b70-istiod-ca-cert\") pod \"e2e-ipp-cleanup-40e6bb-openshift-default-579dfcb59d-h7rck\" (UID: \"c57f95d5-daa7-4dee-a41b-4291e69c2b70\") " pod="openshift-ingress/e2e-ipp-cleanup-40e6bb-openshift-default-579dfcb59d-h7rck"
Sep 11 01:28:56.842230 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:28:56.842129    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c57f95d5-daa7-4dee-a41b-4291e69c2b70-workload-certs\") pod \"e2e-ipp-cleanup-40e6bb-openshift-default-579dfcb59d-h7rck\" (UID: \"c57f95d5-daa7-4dee-a41b-4291e69c2b70\") " pod="openshift-ingress/e2e-ipp-cleanup-40e6bb-openshift-default-579dfcb59d-h7rck"
Sep 11 01:28:56.842230 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:28:56.842162    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c57f95d5-daa7-4dee-a41b-4291e69c2b70-workload-socket\") pod \"e2e-ipp-cleanup-40e6bb-openshift-default-579dfcb59d-h7rck\" (UID: \"c57f95d5-daa7-4dee-a41b-4291e69c2b70\") " pod="openshift-ingress/e2e-ipp-cleanup-40e6bb-openshift-default-579dfcb59d-h7rck"
Sep 11 01:28:56.842230 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:28:56.842188    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c57f95d5-daa7-4dee-a41b-4291e69c2b70-credential-socket\") pod \"e2e-ipp-cleanup-40e6bb-openshift-default-579dfcb59d-h7rck\" (UID: \"c57f95d5-daa7-4dee-a41b-4291e69c2b70\") " pod="openshift-ingress/e2e-ipp-cleanup-40e6bb-openshift-default-579dfcb59d-h7rck"
Sep 11 01:28:56.842507 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:28:56.842233    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdqfx\" (UniqueName: \"kubernetes.io/projected/c57f95d5-daa7-4dee-a41b-4291e69c2b70-kube-api-access-fdqfx\") pod \"e2e-ipp-cleanup-40e6bb-openshift-default-579dfcb59d-h7rck\" (UID: \"c57f95d5-daa7-4dee-a41b-4291e69c2b70\") " pod="openshift-ingress/e2e-ipp-cleanup-40e6bb-openshift-default-579dfcb59d-h7rck"
Sep 11 01:28:56.842507 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:28:56.842294    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c57f95d5-daa7-4dee-a41b-4291e69c2b70-istio-envoy\") pod \"e2e-ipp-cleanup-40e6bb-openshift-default-579dfcb59d-h7rck\" (UID: \"c57f95d5-daa7-4dee-a41b-4291e69c2b70\") " pod="openshift-ingress/e2e-ipp-cleanup-40e6bb-openshift-default-579dfcb59d-h7rck"
Sep 11 01:28:56.842507 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:28:56.842333    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c57f95d5-daa7-4dee-a41b-4291e69c2b70-istio-data\") pod \"e2e-ipp-cleanup-40e6bb-openshift-default-579dfcb59d-h7rck\" (UID: \"c57f95d5-daa7-4dee-a41b-4291e69c2b70\") " pod="openshift-ingress/e2e-ipp-cleanup-40e6bb-openshift-default-579dfcb59d-h7rck"
Sep 11 01:28:56.842853 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:28:56.842799    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c57f95d5-daa7-4dee-a41b-4291e69c2b70-credential-socket\") pod \"e2e-ipp-cleanup-40e6bb-openshift-default-579dfcb59d-h7rck\" (UID: \"c57f95d5-daa7-4dee-a41b-4291e69c2b70\") " pod="openshift-ingress/e2e-ipp-cleanup-40e6bb-openshift-default-579dfcb59d-h7rck"
Sep 11 01:28:56.842853 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:28:56.842832    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c57f95d5-daa7-4dee-a41b-4291e69c2b70-workload-socket\") pod \"e2e-ipp-cleanup-40e6bb-openshift-default-579dfcb59d-h7rck\" (UID: \"c57f95d5-daa7-4dee-a41b-4291e69c2b70\") " pod="openshift-ingress/e2e-ipp-cleanup-40e6bb-openshift-default-579dfcb59d-h7rck"
Sep 11 01:28:56.843064 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:28:56.842975    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c57f95d5-daa7-4dee-a41b-4291e69c2b70-workload-certs\") pod \"e2e-ipp-cleanup-40e6bb-openshift-default-579dfcb59d-h7rck\" (UID: \"c57f95d5-daa7-4dee-a41b-4291e69c2b70\") " pod="openshift-ingress/e2e-ipp-cleanup-40e6bb-openshift-default-579dfcb59d-h7rck"
Sep 11 01:28:56.843111 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:28:56.843054    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c57f95d5-daa7-4dee-a41b-4291e69c2b70-istiod-ca-cert\") pod \"e2e-ipp-cleanup-40e6bb-openshift-default-579dfcb59d-h7rck\" (UID: \"c57f95d5-daa7-4dee-a41b-4291e69c2b70\") " pod="openshift-ingress/e2e-ipp-cleanup-40e6bb-openshift-default-579dfcb59d-h7rck"
Sep 11 01:28:56.843228 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:28:56.843208    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c57f95d5-daa7-4dee-a41b-4291e69c2b70-istio-data\") pod \"e2e-ipp-cleanup-40e6bb-openshift-default-579dfcb59d-h7rck\" (UID: \"c57f95d5-daa7-4dee-a41b-4291e69c2b70\") " pod="openshift-ingress/e2e-ipp-cleanup-40e6bb-openshift-default-579dfcb59d-h7rck"
Sep 11 01:28:56.844662 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:28:56.844641    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c57f95d5-daa7-4dee-a41b-4291e69c2b70-istio-envoy\") pod \"e2e-ipp-cleanup-40e6bb-openshift-default-579dfcb59d-h7rck\" (UID: \"c57f95d5-daa7-4dee-a41b-4291e69c2b70\") " pod="openshift-ingress/e2e-ipp-cleanup-40e6bb-openshift-default-579dfcb59d-h7rck"
Sep 11 01:28:56.845012 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:28:56.844993    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c57f95d5-daa7-4dee-a41b-4291e69c2b70-istio-podinfo\") pod \"e2e-ipp-cleanup-40e6bb-openshift-default-579dfcb59d-h7rck\" (UID: \"c57f95d5-daa7-4dee-a41b-4291e69c2b70\") " pod="openshift-ingress/e2e-ipp-cleanup-40e6bb-openshift-default-579dfcb59d-h7rck"
Sep 11 01:28:56.849458 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:28:56.849432    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c57f95d5-daa7-4dee-a41b-4291e69c2b70-istio-token\") pod \"e2e-ipp-cleanup-40e6bb-openshift-default-579dfcb59d-h7rck\" (UID: \"c57f95d5-daa7-4dee-a41b-4291e69c2b70\") " pod="openshift-ingress/e2e-ipp-cleanup-40e6bb-openshift-default-579dfcb59d-h7rck"
Sep 11 01:28:56.849626 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:28:56.849604    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdqfx\" (UniqueName: \"kubernetes.io/projected/c57f95d5-daa7-4dee-a41b-4291e69c2b70-kube-api-access-fdqfx\") pod \"e2e-ipp-cleanup-40e6bb-openshift-default-579dfcb59d-h7rck\" (UID: \"c57f95d5-daa7-4dee-a41b-4291e69c2b70\") " pod="openshift-ingress/e2e-ipp-cleanup-40e6bb-openshift-default-579dfcb59d-h7rck"
Sep 11 01:28:56.925283 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:28:56.925214    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-40e6bb-openshift-default-579dfcb59d-h7rck"
Sep 11 01:28:57.208440 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:28:57.208357    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-40e6bb-openshift-default-579dfcb59d-h7rck"]
Sep 11 01:28:57.211628 ip-10-0-129-222 kubenswrapper[2569]: W0911 01:28:57.211602    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc57f95d5_daa7_4dee_a41b_4291e69c2b70.slice/crio-fd88059b47c977b401da2448c860c06fbbf26616222320cd29275fdf01999b11 WatchSource:0}: Error finding container fd88059b47c977b401da2448c860c06fbbf26616222320cd29275fdf01999b11: Status 404 returned error can't find the container with id fd88059b47c977b401da2448c860c06fbbf26616222320cd29275fdf01999b11
Sep 11 01:28:57.214278 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:28:57.214245    2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 01:28:57.214359 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:28:57.214316    2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 01:28:57.214359 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:28:57.214348    2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 01:28:58.209319 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:28:58.209281    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-40e6bb-openshift-default-579dfcb59d-h7rck" event={"ID":"c57f95d5-daa7-4dee-a41b-4291e69c2b70","Type":"ContainerStarted","Data":"79e98b1d4ffcf27e32b53fd90627ac1c254dd44fafea2556c178e84f2c00c27a"}
Sep 11 01:28:58.209780 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:28:58.209350    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-40e6bb-openshift-default-579dfcb59d-h7rck" event={"ID":"c57f95d5-daa7-4dee-a41b-4291e69c2b70","Type":"ContainerStarted","Data":"fd88059b47c977b401da2448c860c06fbbf26616222320cd29275fdf01999b11"}
Sep 11 01:28:58.231355 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:28:58.231307    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-cleanup-40e6bb-openshift-default-579dfcb59d-h7rck" podStartSLOduration=2.231291452 podStartE2EDuration="2.231291452s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 01:28:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 01:28:58.229757896 +0000 UTC m=+1872.881132173" watchObservedRunningTime="2026-09-11 01:28:58.231291452 +0000 UTC m=+1872.882665700"
Sep 11 01:28:58.925476 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:28:58.925439    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-40e6bb-openshift-default-579dfcb59d-h7rck"
Sep 11 01:28:58.930720 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:28:58.930691    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-40e6bb-openshift-default-579dfcb59d-h7rck"
Sep 11 01:28:59.214592 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:28:59.214503    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-40e6bb-openshift-default-579dfcb59d-h7rck"
Sep 11 01:28:59.215441 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:28:59.215419    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-40e6bb-openshift-default-579dfcb59d-h7rck"
Sep 11 01:29:08.998866 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:29:08.998783    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kmx5x"]
Sep 11 01:29:19.327630 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:29:19.327596    2569 generic.go:356] "Generic (PLEG): container finished" podID="c57f95d5-daa7-4dee-a41b-4291e69c2b70" containerID="79e98b1d4ffcf27e32b53fd90627ac1c254dd44fafea2556c178e84f2c00c27a" exitCode=0
Sep 11 01:29:19.328049 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:29:19.327667    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-40e6bb-openshift-default-579dfcb59d-h7rck" event={"ID":"c57f95d5-daa7-4dee-a41b-4291e69c2b70","Type":"ContainerDied","Data":"79e98b1d4ffcf27e32b53fd90627ac1c254dd44fafea2556c178e84f2c00c27a"}
Sep 11 01:29:19.328109 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:29:19.328094    2569 scope.go:117] "RemoveContainer" containerID="79e98b1d4ffcf27e32b53fd90627ac1c254dd44fafea2556c178e84f2c00c27a"
Sep 11 01:29:19.329224 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:29:19.329196    2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 01:29:19.329300 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:29:19.329262    2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 01:29:19.329300 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:29:19.329293    2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 01:29:20.336038 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:29:20.335983    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-40e6bb-openshift-default-579dfcb59d-h7rck" event={"ID":"c57f95d5-daa7-4dee-a41b-4291e69c2b70","Type":"ContainerStarted","Data":"b00b7d8636801d6dfd47c607a565b9a54fbc20bc1070be89d63e9d1a5936b7a0"}
Sep 11 01:29:20.925853 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:29:20.925815    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-40e6bb-openshift-default-579dfcb59d-h7rck"
Sep 11 01:29:20.926093 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:29:20.926069    2569 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-40e6bb-openshift-default-579dfcb59d-h7rck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body=
Sep 11 01:29:20.926179 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:29:20.926139    2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-40e6bb-openshift-default-579dfcb59d-h7rck" podUID="c57f95d5-daa7-4dee-a41b-4291e69c2b70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused"
Sep 11 01:29:21.926318 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:29:21.926277    2569 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-40e6bb-openshift-default-579dfcb59d-h7rck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body=
Sep 11 01:29:21.926794 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:29:21.926356    2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-40e6bb-openshift-default-579dfcb59d-h7rck" podUID="c57f95d5-daa7-4dee-a41b-4291e69c2b70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused"
Sep 11 01:29:22.925590 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:29:22.925553    2569 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-40e6bb-openshift-default-579dfcb59d-h7rck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body=
Sep 11 01:29:22.925760 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:29:22.925622    2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-40e6bb-openshift-default-579dfcb59d-h7rck" podUID="c57f95d5-daa7-4dee-a41b-4291e69c2b70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused"
Sep 11 01:29:23.926512 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:29:23.926478    2569 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-40e6bb-openshift-default-579dfcb59d-h7rck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body=
Sep 11 01:29:23.926999 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:29:23.926549    2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-40e6bb-openshift-default-579dfcb59d-h7rck" podUID="c57f95d5-daa7-4dee-a41b-4291e69c2b70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused"
Sep 11 01:29:24.926053 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:29:24.926014    2569 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-40e6bb-openshift-default-579dfcb59d-h7rck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body=
Sep 11 01:29:24.926232 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:29:24.926066    2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-40e6bb-openshift-default-579dfcb59d-h7rck" podUID="c57f95d5-daa7-4dee-a41b-4291e69c2b70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused"
Sep 11 01:29:25.843804 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:29:25.843769    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-40e6bb-82fsl"]
Sep 11 01:29:25.847366 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:29:25.847350    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-40e6bb-82fsl"
Sep 11 01:29:25.849590 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:29:25.849570    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pppkt\""
Sep 11 01:29:25.849726 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:29:25.849572    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 01:29:25.854453 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:29:25.854429    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-40e6bb-82fsl"]
Sep 11 01:29:25.927849 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:29:25.927814    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ca6aeae-68ff-4088-a65d-7117d87f4535-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-40e6bb-82fsl\" (UID: \"2ca6aeae-68ff-4088-a65d-7117d87f4535\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-40e6bb-82fsl"
Sep 11 01:29:25.930121 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:29:25.930099    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-40e6bb-openshift-default-579dfcb59d-h7rck"
Sep 11 01:29:25.930384 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:29:25.930364    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-40e6bb-openshift-default-579dfcb59d-h7rck"
Sep 11 01:29:25.931125 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:29:25.931109    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-40e6bb-openshift-default-579dfcb59d-h7rck"
Sep 11 01:29:26.028981 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:29:26.028928    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ca6aeae-68ff-4088-a65d-7117d87f4535-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-40e6bb-82fsl\" (UID: \"2ca6aeae-68ff-4088-a65d-7117d87f4535\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-40e6bb-82fsl"
Sep 11 01:29:26.029642 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:29:26.029616    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ca6aeae-68ff-4088-a65d-7117d87f4535-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-40e6bb-82fsl\" (UID: \"2ca6aeae-68ff-4088-a65d-7117d87f4535\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-40e6bb-82fsl"
Sep 11 01:29:26.160368 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:29:26.160285    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-40e6bb-82fsl"
Sep 11 01:29:26.430552 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:29:26.430525    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-40e6bb-82fsl"]
Sep 11 01:29:26.432084 ip-10-0-129-222 kubenswrapper[2569]: W0911 01:29:26.432053    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ca6aeae_68ff_4088_a65d_7117d87f4535.slice/crio-fed5f26847792408dd10181c38600d95ada4e18d4e99eae1e6d86ce243dea31d WatchSource:0}: Error finding container fed5f26847792408dd10181c38600d95ada4e18d4e99eae1e6d86ce243dea31d: Status 404 returned error can't find the container with id fed5f26847792408dd10181c38600d95ada4e18d4e99eae1e6d86ce243dea31d
Sep 11 01:29:27.385689 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:29:27.385655    2569 generic.go:356] "Generic (PLEG): container finished" podID="2ca6aeae-68ff-4088-a65d-7117d87f4535" containerID="127a8f1ee29162213a9a9b85ed5b246e2d730e3e704023abb538615c58914996" exitCode=0
Sep 11 01:29:27.386103 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:29:27.385735    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-40e6bb-82fsl" event={"ID":"2ca6aeae-68ff-4088-a65d-7117d87f4535","Type":"ContainerDied","Data":"127a8f1ee29162213a9a9b85ed5b246e2d730e3e704023abb538615c58914996"}
Sep 11 01:29:27.386103 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:29:27.385784    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-40e6bb-82fsl" event={"ID":"2ca6aeae-68ff-4088-a65d-7117d87f4535","Type":"ContainerStarted","Data":"fed5f26847792408dd10181c38600d95ada4e18d4e99eae1e6d86ce243dea31d"}
Sep 11 01:29:28.530884 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:29:28.530847    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-40e6bb-82fsl"
Sep 11 01:29:28.654688 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:29:28.654604    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ca6aeae-68ff-4088-a65d-7117d87f4535-maas-api-service-ca\") pod \"2ca6aeae-68ff-4088-a65d-7117d87f4535\" (UID: \"2ca6aeae-68ff-4088-a65d-7117d87f4535\") "
Sep 11 01:29:28.654976 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:29:28.654952    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ca6aeae-68ff-4088-a65d-7117d87f4535-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2ca6aeae-68ff-4088-a65d-7117d87f4535" (UID: "2ca6aeae-68ff-4088-a65d-7117d87f4535"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 01:29:28.756207 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:29:28.756174    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ca6aeae-68ff-4088-a65d-7117d87f4535-maas-api-service-ca\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:29:29.400956 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:29:29.400926    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-40e6bb-82fsl"
Sep 11 01:29:29.400956 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:29:29.400945    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-40e6bb-82fsl" event={"ID":"2ca6aeae-68ff-4088-a65d-7117d87f4535","Type":"ContainerDied","Data":"fed5f26847792408dd10181c38600d95ada4e18d4e99eae1e6d86ce243dea31d"}
Sep 11 01:29:29.401158 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:29:29.400994    2569 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fed5f26847792408dd10181c38600d95ada4e18d4e99eae1e6d86ce243dea31d"
Sep 11 01:29:32.206012 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:29:32.205976    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-40e6bb-82fsl"]
Sep 11 01:29:32.208982 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:29:32.208956    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-40e6bb-82fsl"]
Sep 11 01:29:33.871642 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:29:33.871605    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2ca6aeae-68ff-4088-a65d-7117d87f4535" path="/var/lib/kubelet/pods/2ca6aeae-68ff-4088-a65d-7117d87f4535/volumes"
Sep 11 01:29:37.099415 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:29:37.099380    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kmx5x"]
Sep 11 01:29:38.500168 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:29:38.500138    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-40e6bb-openshift-default-579dfcb59d-h7rck"]
Sep 11 01:29:38.500576 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:29:38.500374    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-cleanup-40e6bb-openshift-default-579dfcb59d-h7rck" podUID="c57f95d5-daa7-4dee-a41b-4291e69c2b70" containerName="istio-proxy" containerID="cri-o://b00b7d8636801d6dfd47c607a565b9a54fbc20bc1070be89d63e9d1a5936b7a0" gracePeriod=30
Sep 11 01:29:40.932947 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:29:40.932909    2569 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-40e6bb-openshift-default-579dfcb59d-h7rck container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 11 01:29:40.933328 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:29:40.932981    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-cleanup-40e6bb-openshift-default-579dfcb59d-h7rck" podUID="c57f95d5-daa7-4dee-a41b-4291e69c2b70" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 11 01:29:42.276674 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:29:42.276643    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-976998-9cbq4"]
Sep 11 01:29:42.277286 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:29:42.277266    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ca6aeae-68ff-4088-a65d-7117d87f4535" containerName="revoke-keys"
Sep 11 01:29:42.277376 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:29:42.277288    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ca6aeae-68ff-4088-a65d-7117d87f4535" containerName="revoke-keys"
Sep 11 01:29:42.277456 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:29:42.277443    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="2ca6aeae-68ff-4088-a65d-7117d87f4535" containerName="revoke-keys"
Sep 11 01:29:42.280986 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:29:42.280962    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-976998-9cbq4"
Sep 11 01:29:42.283221 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:29:42.283196    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pppkt\""
Sep 11 01:29:42.283454 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:29:42.283437    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 01:29:42.287835 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:29:42.287816    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-976998-9cbq4"]
Sep 11 01:29:42.388066 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:29:42.388034    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/09a00a3a-7a49-453d-8c42-d0399f2738c3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-976998-9cbq4\" (UID: \"09a00a3a-7a49-453d-8c42-d0399f2738c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-976998-9cbq4"
Sep 11 01:29:42.489280 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:29:42.489247    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/09a00a3a-7a49-453d-8c42-d0399f2738c3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-976998-9cbq4\" (UID: \"09a00a3a-7a49-453d-8c42-d0399f2738c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-976998-9cbq4"
Sep 11 01:29:42.489848 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:29:42.489828    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/09a00a3a-7a49-453d-8c42-d0399f2738c3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-976998-9cbq4\" (UID: \"09a00a3a-7a49-453d-8c42-d0399f2738c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-976998-9cbq4"
Sep 11 01:29:42.593159 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:29:42.593117    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-976998-9cbq4"
Sep 11 01:29:42.863632 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:29:42.863556    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-976998-9cbq4"]
Sep 11 01:29:42.866999 ip-10-0-129-222 kubenswrapper[2569]: W0911 01:29:42.866971    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09a00a3a_7a49_453d_8c42_d0399f2738c3.slice/crio-5d738240e059c27e4e69c42dfa3827d3bc688c15829662fd782c657aa47e232b WatchSource:0}: Error finding container 5d738240e059c27e4e69c42dfa3827d3bc688c15829662fd782c657aa47e232b: Status 404 returned error can't find the container with id 5d738240e059c27e4e69c42dfa3827d3bc688c15829662fd782c657aa47e232b
Sep 11 01:29:43.485071 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:29:43.484980    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-976998-9cbq4" event={"ID":"09a00a3a-7a49-453d-8c42-d0399f2738c3","Type":"ContainerStarted","Data":"bfdd67a1253629a90e5fc103f5e53c23a1eb5f46cfd402cdef1604365800a8ca"}
Sep 11 01:29:43.485415 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:29:43.485106    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-976998-9cbq4" event={"ID":"09a00a3a-7a49-453d-8c42-d0399f2738c3","Type":"ContainerStarted","Data":"5d738240e059c27e4e69c42dfa3827d3bc688c15829662fd782c657aa47e232b"}
Sep 11 01:29:43.501118 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:29:43.501071    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-976998-9cbq4" podStartSLOduration=1.5010588440000001 podStartE2EDuration="1.501058844s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 01:29:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 01:29:43.499049774 +0000 UTC m=+1918.150424024" watchObservedRunningTime="2026-09-11 01:29:43.501058844 +0000 UTC m=+1918.152433093"
Sep 11 01:29:43.765305 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:29:43.765275    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-40e6bb-openshift-default-579dfcb59d-h7rck"
Sep 11 01:29:43.900504 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:29:43.900475    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c57f95d5-daa7-4dee-a41b-4291e69c2b70-credential-socket\") pod \"c57f95d5-daa7-4dee-a41b-4291e69c2b70\" (UID: \"c57f95d5-daa7-4dee-a41b-4291e69c2b70\") "
Sep 11 01:29:43.900679 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:29:43.900540    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c57f95d5-daa7-4dee-a41b-4291e69c2b70-workload-certs\") pod \"c57f95d5-daa7-4dee-a41b-4291e69c2b70\" (UID: \"c57f95d5-daa7-4dee-a41b-4291e69c2b70\") "
Sep 11 01:29:43.900679 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:29:43.900559    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c57f95d5-daa7-4dee-a41b-4291e69c2b70-istio-podinfo\") pod \"c57f95d5-daa7-4dee-a41b-4291e69c2b70\" (UID: \"c57f95d5-daa7-4dee-a41b-4291e69c2b70\") "
Sep 11 01:29:43.900679 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:29:43.900617    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fdqfx\" (UniqueName: \"kubernetes.io/projected/c57f95d5-daa7-4dee-a41b-4291e69c2b70-kube-api-access-fdqfx\") pod \"c57f95d5-daa7-4dee-a41b-4291e69c2b70\" (UID: \"c57f95d5-daa7-4dee-a41b-4291e69c2b70\") "
Sep 11 01:29:43.900679 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:29:43.900653    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c57f95d5-daa7-4dee-a41b-4291e69c2b70-workload-socket\") pod \"c57f95d5-daa7-4dee-a41b-4291e69c2b70\" (UID: \"c57f95d5-daa7-4dee-a41b-4291e69c2b70\") "
Sep 11 01:29:43.900679 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:29:43.900676    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c57f95d5-daa7-4dee-a41b-4291e69c2b70-istiod-ca-cert\") pod \"c57f95d5-daa7-4dee-a41b-4291e69c2b70\" (UID: \"c57f95d5-daa7-4dee-a41b-4291e69c2b70\") "
Sep 11 01:29:43.900986 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:29:43.900692    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c57f95d5-daa7-4dee-a41b-4291e69c2b70-istio-data\") pod \"c57f95d5-daa7-4dee-a41b-4291e69c2b70\" (UID: \"c57f95d5-daa7-4dee-a41b-4291e69c2b70\") "
Sep 11 01:29:43.900986 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:29:43.900721    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c57f95d5-daa7-4dee-a41b-4291e69c2b70-istio-envoy\") pod \"c57f95d5-daa7-4dee-a41b-4291e69c2b70\" (UID: \"c57f95d5-daa7-4dee-a41b-4291e69c2b70\") "
Sep 11 01:29:43.900986 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:29:43.900740    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c57f95d5-daa7-4dee-a41b-4291e69c2b70-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "c57f95d5-daa7-4dee-a41b-4291e69c2b70" (UID: "c57f95d5-daa7-4dee-a41b-4291e69c2b70"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 01:29:43.900986 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:29:43.900753    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c57f95d5-daa7-4dee-a41b-4291e69c2b70-istio-token\") pod \"c57f95d5-daa7-4dee-a41b-4291e69c2b70\" (UID: \"c57f95d5-daa7-4dee-a41b-4291e69c2b70\") "
Sep 11 01:29:43.900986 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:29:43.900814    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c57f95d5-daa7-4dee-a41b-4291e69c2b70-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "c57f95d5-daa7-4dee-a41b-4291e69c2b70" (UID: "c57f95d5-daa7-4dee-a41b-4291e69c2b70"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 01:29:43.901228 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:29:43.900993    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c57f95d5-daa7-4dee-a41b-4291e69c2b70-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "c57f95d5-daa7-4dee-a41b-4291e69c2b70" (UID: "c57f95d5-daa7-4dee-a41b-4291e69c2b70"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 01:29:43.901228 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:29:43.901004    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c57f95d5-daa7-4dee-a41b-4291e69c2b70-istio-data" (OuterVolumeSpecName: "istio-data") pod "c57f95d5-daa7-4dee-a41b-4291e69c2b70" (UID: "c57f95d5-daa7-4dee-a41b-4291e69c2b70"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 01:29:43.901228 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:29:43.901081    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c57f95d5-daa7-4dee-a41b-4291e69c2b70-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "c57f95d5-daa7-4dee-a41b-4291e69c2b70" (UID: "c57f95d5-daa7-4dee-a41b-4291e69c2b70"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 01:29:43.901228 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:29:43.901194    2569 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c57f95d5-daa7-4dee-a41b-4291e69c2b70-workload-socket\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:29:43.901228 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:29:43.901213    2569 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c57f95d5-daa7-4dee-a41b-4291e69c2b70-istiod-ca-cert\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:29:43.901228 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:29:43.901226    2569 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c57f95d5-daa7-4dee-a41b-4291e69c2b70-istio-data\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:29:43.901462 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:29:43.901238    2569 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c57f95d5-daa7-4dee-a41b-4291e69c2b70-credential-socket\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:29:43.901462 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:29:43.901251    2569 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c57f95d5-daa7-4dee-a41b-4291e69c2b70-workload-certs\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:29:43.903072 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:29:43.903037    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c57f95d5-daa7-4dee-a41b-4291e69c2b70-kube-api-access-fdqfx" (OuterVolumeSpecName: "kube-api-access-fdqfx") pod "c57f95d5-daa7-4dee-a41b-4291e69c2b70" (UID: "c57f95d5-daa7-4dee-a41b-4291e69c2b70"). InnerVolumeSpecName "kube-api-access-fdqfx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 01:29:43.903175 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:29:43.903111    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c57f95d5-daa7-4dee-a41b-4291e69c2b70-istio-token" (OuterVolumeSpecName: "istio-token") pod "c57f95d5-daa7-4dee-a41b-4291e69c2b70" (UID: "c57f95d5-daa7-4dee-a41b-4291e69c2b70"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 01:29:43.903307 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:29:43.903287    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c57f95d5-daa7-4dee-a41b-4291e69c2b70-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "c57f95d5-daa7-4dee-a41b-4291e69c2b70" (UID: "c57f95d5-daa7-4dee-a41b-4291e69c2b70"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 01:29:43.903400 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:29:43.903316    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c57f95d5-daa7-4dee-a41b-4291e69c2b70-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "c57f95d5-daa7-4dee-a41b-4291e69c2b70" (UID: "c57f95d5-daa7-4dee-a41b-4291e69c2b70"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 01:29:44.002489 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:29:44.002409    2569 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c57f95d5-daa7-4dee-a41b-4291e69c2b70-istio-podinfo\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:29:44.002620 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:29:44.002516    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fdqfx\" (UniqueName: \"kubernetes.io/projected/c57f95d5-daa7-4dee-a41b-4291e69c2b70-kube-api-access-fdqfx\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:29:44.002620 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:29:44.002542    2569 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c57f95d5-daa7-4dee-a41b-4291e69c2b70-istio-envoy\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:29:44.002620 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:29:44.002552    2569 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c57f95d5-daa7-4dee-a41b-4291e69c2b70-istio-token\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:29:44.492573 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:29:44.492536    2569 generic.go:356] "Generic (PLEG): container finished" podID="c57f95d5-daa7-4dee-a41b-4291e69c2b70" containerID="b00b7d8636801d6dfd47c607a565b9a54fbc20bc1070be89d63e9d1a5936b7a0" exitCode=0
Sep 11 01:29:44.493036 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:29:44.492588    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-40e6bb-openshift-default-579dfcb59d-h7rck"
Sep 11 01:29:44.493036 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:29:44.492619    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-40e6bb-openshift-default-579dfcb59d-h7rck" event={"ID":"c57f95d5-daa7-4dee-a41b-4291e69c2b70","Type":"ContainerDied","Data":"b00b7d8636801d6dfd47c607a565b9a54fbc20bc1070be89d63e9d1a5936b7a0"}
Sep 11 01:29:44.493036 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:29:44.492681    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-40e6bb-openshift-default-579dfcb59d-h7rck" event={"ID":"c57f95d5-daa7-4dee-a41b-4291e69c2b70","Type":"ContainerDied","Data":"fd88059b47c977b401da2448c860c06fbbf26616222320cd29275fdf01999b11"}
Sep 11 01:29:44.493036 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:29:44.492714    2569 scope.go:117] "RemoveContainer" containerID="b00b7d8636801d6dfd47c607a565b9a54fbc20bc1070be89d63e9d1a5936b7a0"
Sep 11 01:29:44.494487 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:29:44.494465    2569 generic.go:356] "Generic (PLEG): container finished" podID="09a00a3a-7a49-453d-8c42-d0399f2738c3" containerID="bfdd67a1253629a90e5fc103f5e53c23a1eb5f46cfd402cdef1604365800a8ca" exitCode=0
Sep 11 01:29:44.494605 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:29:44.494552    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-976998-9cbq4" event={"ID":"09a00a3a-7a49-453d-8c42-d0399f2738c3","Type":"ContainerDied","Data":"bfdd67a1253629a90e5fc103f5e53c23a1eb5f46cfd402cdef1604365800a8ca"}
Sep 11 01:29:44.503366 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:29:44.503338    2569 scope.go:117] "RemoveContainer" containerID="79e98b1d4ffcf27e32b53fd90627ac1c254dd44fafea2556c178e84f2c00c27a"
Sep 11 01:29:44.512179 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:29:44.512158    2569 scope.go:117] "RemoveContainer" containerID="b00b7d8636801d6dfd47c607a565b9a54fbc20bc1070be89d63e9d1a5936b7a0"
Sep 11 01:29:44.512456 ip-10-0-129-222 kubenswrapper[2569]: E0911 01:29:44.512436    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b00b7d8636801d6dfd47c607a565b9a54fbc20bc1070be89d63e9d1a5936b7a0\": container with ID starting with b00b7d8636801d6dfd47c607a565b9a54fbc20bc1070be89d63e9d1a5936b7a0 not found: ID does not exist" containerID="b00b7d8636801d6dfd47c607a565b9a54fbc20bc1070be89d63e9d1a5936b7a0"
Sep 11 01:29:44.512509 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:29:44.512464    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b00b7d8636801d6dfd47c607a565b9a54fbc20bc1070be89d63e9d1a5936b7a0"} err="failed to get container status \"b00b7d8636801d6dfd47c607a565b9a54fbc20bc1070be89d63e9d1a5936b7a0\": rpc error: code = NotFound desc = could not find container \"b00b7d8636801d6dfd47c607a565b9a54fbc20bc1070be89d63e9d1a5936b7a0\": container with ID starting with b00b7d8636801d6dfd47c607a565b9a54fbc20bc1070be89d63e9d1a5936b7a0 not found: ID does not exist"
Sep 11 01:29:44.512509 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:29:44.512481    2569 scope.go:117] "RemoveContainer" containerID="79e98b1d4ffcf27e32b53fd90627ac1c254dd44fafea2556c178e84f2c00c27a"
Sep 11 01:29:44.512762 ip-10-0-129-222 kubenswrapper[2569]: E0911 01:29:44.512744    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79e98b1d4ffcf27e32b53fd90627ac1c254dd44fafea2556c178e84f2c00c27a\": container with ID starting with 79e98b1d4ffcf27e32b53fd90627ac1c254dd44fafea2556c178e84f2c00c27a not found: ID does not exist" containerID="79e98b1d4ffcf27e32b53fd90627ac1c254dd44fafea2556c178e84f2c00c27a"
Sep 11 01:29:44.512819 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:29:44.512768    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79e98b1d4ffcf27e32b53fd90627ac1c254dd44fafea2556c178e84f2c00c27a"} err="failed to get container status \"79e98b1d4ffcf27e32b53fd90627ac1c254dd44fafea2556c178e84f2c00c27a\": rpc error: code = NotFound desc = could not find container \"79e98b1d4ffcf27e32b53fd90627ac1c254dd44fafea2556c178e84f2c00c27a\": container with ID starting with 79e98b1d4ffcf27e32b53fd90627ac1c254dd44fafea2556c178e84f2c00c27a not found: ID does not exist"
Sep 11 01:29:44.527450 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:29:44.527426    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-40e6bb-openshift-default-579dfcb59d-h7rck"]
Sep 11 01:29:44.532832 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:29:44.532813    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-40e6bb-openshift-default-579dfcb59d-h7rck"]
Sep 11 01:29:45.643302 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:29:45.643269    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-976998-9cbq4"
Sep 11 01:29:45.819832 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:29:45.819804    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/09a00a3a-7a49-453d-8c42-d0399f2738c3-maas-api-service-ca\") pod \"09a00a3a-7a49-453d-8c42-d0399f2738c3\" (UID: \"09a00a3a-7a49-453d-8c42-d0399f2738c3\") "
Sep 11 01:29:45.820200 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:29:45.820177    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09a00a3a-7a49-453d-8c42-d0399f2738c3-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "09a00a3a-7a49-453d-8c42-d0399f2738c3" (UID: "09a00a3a-7a49-453d-8c42-d0399f2738c3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 01:29:45.872162 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:29:45.872138    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c57f95d5-daa7-4dee-a41b-4291e69c2b70" path="/var/lib/kubelet/pods/c57f95d5-daa7-4dee-a41b-4291e69c2b70/volumes"
Sep 11 01:29:45.920711 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:29:45.920673    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/09a00a3a-7a49-453d-8c42-d0399f2738c3-maas-api-service-ca\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:29:46.340806 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:29:46.340777    2569 scope.go:117] "RemoveContainer" containerID="937bcf80eb54a03918e50155af6984cf7157263bade7cea72647a9b5d6366e6a"
Sep 11 01:29:46.349307 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:29:46.349286    2569 scope.go:117] "RemoveContainer" containerID="48fa0736957818490546a31c3afb14eb24de6985ab9de9ff92255f405f15cf00"
Sep 11 01:29:46.356853 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:29:46.356824    2569 scope.go:117] "RemoveContainer" containerID="23f416bfa55bc84c428697280404e9679a90a75b2b99e4a5d1c2dabcc6d11810"
Sep 11 01:29:46.507354 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:29:46.507327    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-976998-9cbq4"
Sep 11 01:29:46.507469 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:29:46.507371    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-976998-9cbq4" event={"ID":"09a00a3a-7a49-453d-8c42-d0399f2738c3","Type":"ContainerDied","Data":"5d738240e059c27e4e69c42dfa3827d3bc688c15829662fd782c657aa47e232b"}
Sep 11 01:29:46.507469 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:29:46.507412    2569 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5d738240e059c27e4e69c42dfa3827d3bc688c15829662fd782c657aa47e232b"
Sep 11 01:29:55.793286 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:29:55.793251    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kmx5x"]
Sep 11 01:29:58.206372 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:29:58.206329    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-976998-9cbq4"]
Sep 11 01:29:58.208957 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:29:58.208934    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-976998-9cbq4"]
Sep 11 01:29:59.870552 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:29:59.870509    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="09a00a3a-7a49-453d-8c42-d0399f2738c3" path="/var/lib/kubelet/pods/09a00a3a-7a49-453d-8c42-d0399f2738c3/volumes"
Sep 11 01:30:00.145298 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:30:00.145220    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2482ac-29818170-kxbbl"]
Sep 11 01:30:00.145653 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:30:00.145641    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c57f95d5-daa7-4dee-a41b-4291e69c2b70" containerName="istio-proxy"
Sep 11 01:30:00.145701 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:30:00.145655    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="c57f95d5-daa7-4dee-a41b-4291e69c2b70" containerName="istio-proxy"
Sep 11 01:30:00.145701 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:30:00.145671    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09a00a3a-7a49-453d-8c42-d0399f2738c3" containerName="revoke-keys"
Sep 11 01:30:00.145701 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:30:00.145677    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="09a00a3a-7a49-453d-8c42-d0399f2738c3" containerName="revoke-keys"
Sep 11 01:30:00.145812 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:30:00.145741    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="c57f95d5-daa7-4dee-a41b-4291e69c2b70" containerName="istio-proxy"
Sep 11 01:30:00.145812 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:30:00.145749    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="09a00a3a-7a49-453d-8c42-d0399f2738c3" containerName="revoke-keys"
Sep 11 01:30:00.149344 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:30:00.149314    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2482ac-29818170-kxbbl"
Sep 11 01:30:00.152630 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:30:00.152586    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-9433b9-29818170-j7lkd"]
Sep 11 01:30:00.152903 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:30:00.152884    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pppkt\""
Sep 11 01:30:00.153146 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:30:00.153127    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c57f95d5-daa7-4dee-a41b-4291e69c2b70" containerName="istio-proxy"
Sep 11 01:30:00.153206 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:30:00.153148    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="c57f95d5-daa7-4dee-a41b-4291e69c2b70" containerName="istio-proxy"
Sep 11 01:30:00.153269 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:30:00.153255    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="c57f95d5-daa7-4dee-a41b-4291e69c2b70" containerName="istio-proxy"
Sep 11 01:30:00.156689 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:30:00.156669    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-9433b9-29818170-j7lkd"
Sep 11 01:30:00.160658 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:30:00.160633    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2482ac-29818170-kxbbl"]
Sep 11 01:30:00.161666 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:30:00.161646    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2482ac-29818170-kxbbl"
Sep 11 01:30:00.175287 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:30:00.175265    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-9433b9-29818170-j7lkd"
Sep 11 01:30:00.184138 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:30:00.184100    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-9433b9-29818170-j7lkd"]
Sep 11 01:30:00.445917 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:30:00.445890    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2482ac-29818170-kxbbl"]
Sep 11 01:30:00.447965 ip-10-0-129-222 kubenswrapper[2569]: W0911 01:30:00.447934    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode20b24c2_11fb_4ed5_a710_04cd39193d19.slice/crio-1870c75beb795b2847984c78f6e242cf4167f3526fdabc9cbf4305555e70111f WatchSource:0}: Error finding container 1870c75beb795b2847984c78f6e242cf4167f3526fdabc9cbf4305555e70111f: Status 404 returned error can't find the container with id 1870c75beb795b2847984c78f6e242cf4167f3526fdabc9cbf4305555e70111f
Sep 11 01:30:00.459711 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:30:00.459691    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-9433b9-29818170-j7lkd"]
Sep 11 01:30:00.462190 ip-10-0-129-222 kubenswrapper[2569]: W0911 01:30:00.462163    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f509aae_0bd1_4431_8f38_47b5b4340d00.slice/crio-ed9fcf180b04f454070297afcc496aece8df11ed17a392708b251ca34cb43fb4 WatchSource:0}: Error finding container ed9fcf180b04f454070297afcc496aece8df11ed17a392708b251ca34cb43fb4: Status 404 returned error can't find the container with id ed9fcf180b04f454070297afcc496aece8df11ed17a392708b251ca34cb43fb4
Sep 11 01:30:00.581740 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:30:00.581706    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2482ac-29818170-kxbbl" event={"ID":"e20b24c2-11fb-4ed5-a710-04cd39193d19","Type":"ContainerStarted","Data":"1870c75beb795b2847984c78f6e242cf4167f3526fdabc9cbf4305555e70111f"}
Sep 11 01:30:00.583069 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:30:00.583041    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-9433b9-29818170-j7lkd" event={"ID":"6f509aae-0bd1-4431-8f38-47b5b4340d00","Type":"ContainerStarted","Data":"ed9fcf180b04f454070297afcc496aece8df11ed17a392708b251ca34cb43fb4"}
Sep 11 01:30:01.589746 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:30:01.589709    2569 generic.go:356] "Generic (PLEG): container finished" podID="6f509aae-0bd1-4431-8f38-47b5b4340d00" containerID="e51eb99b1fd324483eb2975cb72f54935dd5cda49a192b76ece21885c671e8f7" exitCode=0
Sep 11 01:30:01.590156 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:30:01.589799    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-9433b9-29818170-j7lkd" event={"ID":"6f509aae-0bd1-4431-8f38-47b5b4340d00","Type":"ContainerDied","Data":"e51eb99b1fd324483eb2975cb72f54935dd5cda49a192b76ece21885c671e8f7"}
Sep 11 01:30:01.591579 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:30:01.591556    2569 generic.go:356] "Generic (PLEG): container finished" podID="e20b24c2-11fb-4ed5-a710-04cd39193d19" containerID="cb450458b14970c49e5e1e9f56e536398441861f29f172d32b69bb0a68553e30" exitCode=0
Sep 11 01:30:01.591696 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:30:01.591602    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2482ac-29818170-kxbbl" event={"ID":"e20b24c2-11fb-4ed5-a710-04cd39193d19","Type":"ContainerDied","Data":"cb450458b14970c49e5e1e9f56e536398441861f29f172d32b69bb0a68553e30"}
Sep 11 01:30:02.756026 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:30:02.755999    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2482ac-29818170-kxbbl"
Sep 11 01:30:02.758989 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:30:02.758972    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-9433b9-29818170-j7lkd"
Sep 11 01:30:03.603130 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:30:03.603099    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2482ac-29818170-kxbbl"
Sep 11 01:30:03.603306 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:30:03.603135    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2482ac-29818170-kxbbl" event={"ID":"e20b24c2-11fb-4ed5-a710-04cd39193d19","Type":"ContainerDied","Data":"1870c75beb795b2847984c78f6e242cf4167f3526fdabc9cbf4305555e70111f"}
Sep 11 01:30:03.603306 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:30:03.603199    2569 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1870c75beb795b2847984c78f6e242cf4167f3526fdabc9cbf4305555e70111f"
Sep 11 01:30:03.604435 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:30:03.604416    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-9433b9-29818170-j7lkd"
Sep 11 01:30:03.604526 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:30:03.604450    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-9433b9-29818170-j7lkd" event={"ID":"6f509aae-0bd1-4431-8f38-47b5b4340d00","Type":"ContainerDied","Data":"ed9fcf180b04f454070297afcc496aece8df11ed17a392708b251ca34cb43fb4"}
Sep 11 01:30:03.604526 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:30:03.604489    2569 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ed9fcf180b04f454070297afcc496aece8df11ed17a392708b251ca34cb43fb4"
Sep 11 01:30:08.002619 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:30:08.002589    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kmx5x"]
Sep 11 01:30:46.950784 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:30:46.950703    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-bae8d3-xv67p"]
Sep 11 01:30:46.951296 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:30:46.951179    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f509aae-0bd1-4431-8f38-47b5b4340d00" containerName="cleanup"
Sep 11 01:30:46.951296 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:30:46.951193    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f509aae-0bd1-4431-8f38-47b5b4340d00" containerName="cleanup"
Sep 11 01:30:46.951296 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:30:46.951211    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e20b24c2-11fb-4ed5-a710-04cd39193d19" containerName="cleanup"
Sep 11 01:30:46.951296 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:30:46.951216    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="e20b24c2-11fb-4ed5-a710-04cd39193d19" containerName="cleanup"
Sep 11 01:30:46.951296 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:30:46.951288    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="e20b24c2-11fb-4ed5-a710-04cd39193d19" containerName="cleanup"
Sep 11 01:30:46.951296 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:30:46.951299    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="6f509aae-0bd1-4431-8f38-47b5b4340d00" containerName="cleanup"
Sep 11 01:30:46.954651 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:30:46.954636    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-bae8d3-xv67p"
Sep 11 01:30:46.956792 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:30:46.956767    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 01:30:46.956920 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:30:46.956901    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pppkt\""
Sep 11 01:30:46.960644 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:30:46.960619    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-bae8d3-xv67p"]
Sep 11 01:30:46.965574 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:30:46.965545    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ee924d9-4129-4a56-aa47-3995c323b320-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-bae8d3-xv67p\" (UID: \"2ee924d9-4129-4a56-aa47-3995c323b320\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-bae8d3-xv67p"
Sep 11 01:30:47.066657 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:30:47.066613    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ee924d9-4129-4a56-aa47-3995c323b320-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-bae8d3-xv67p\" (UID: \"2ee924d9-4129-4a56-aa47-3995c323b320\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-bae8d3-xv67p"
Sep 11 01:30:47.067283 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:30:47.067263    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ee924d9-4129-4a56-aa47-3995c323b320-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-bae8d3-xv67p\" (UID: \"2ee924d9-4129-4a56-aa47-3995c323b320\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-bae8d3-xv67p"
Sep 11 01:30:47.267539 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:30:47.267459    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-bae8d3-xv67p"
Sep 11 01:30:47.567388 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:30:47.567353    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-bae8d3-xv67p"]
Sep 11 01:30:47.568711 ip-10-0-129-222 kubenswrapper[2569]: W0911 01:30:47.568682    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ee924d9_4129_4a56_aa47_3995c323b320.slice/crio-e577a30eaee1e72442cc5823159f053bd8651501face67cc101db85bb4cf0682 WatchSource:0}: Error finding container e577a30eaee1e72442cc5823159f053bd8651501face67cc101db85bb4cf0682: Status 404 returned error can't find the container with id e577a30eaee1e72442cc5823159f053bd8651501face67cc101db85bb4cf0682
Sep 11 01:30:47.843321 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:30:47.843290    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-bae8d3-xv67p" event={"ID":"2ee924d9-4129-4a56-aa47-3995c323b320","Type":"ContainerStarted","Data":"e577a30eaee1e72442cc5823159f053bd8651501face67cc101db85bb4cf0682"}
Sep 11 01:30:48.850590 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:30:48.850556    2569 generic.go:356] "Generic (PLEG): container finished" podID="2ee924d9-4129-4a56-aa47-3995c323b320" containerID="40d2eae1410bf211234285c8e3632a9b01671d25cb4ecc023c86e2e8f97f2dd5" exitCode=0
Sep 11 01:30:48.851008 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:30:48.850623    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-bae8d3-xv67p" event={"ID":"2ee924d9-4129-4a56-aa47-3995c323b320","Type":"ContainerDied","Data":"40d2eae1410bf211234285c8e3632a9b01671d25cb4ecc023c86e2e8f97f2dd5"}
Sep 11 01:30:49.992272 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:30:49.992248    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-bae8d3-xv67p"
Sep 11 01:30:50.093959 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:30:50.093924    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ee924d9-4129-4a56-aa47-3995c323b320-maas-api-service-ca\") pod \"2ee924d9-4129-4a56-aa47-3995c323b320\" (UID: \"2ee924d9-4129-4a56-aa47-3995c323b320\") "
Sep 11 01:30:50.094267 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:30:50.094244    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ee924d9-4129-4a56-aa47-3995c323b320-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2ee924d9-4129-4a56-aa47-3995c323b320" (UID: "2ee924d9-4129-4a56-aa47-3995c323b320"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 01:30:50.194971 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:30:50.194902    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ee924d9-4129-4a56-aa47-3995c323b320-maas-api-service-ca\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:30:50.862493 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:30:50.862464    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-bae8d3-xv67p"
Sep 11 01:30:50.862661 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:30:50.862470    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-bae8d3-xv67p" event={"ID":"2ee924d9-4129-4a56-aa47-3995c323b320","Type":"ContainerDied","Data":"e577a30eaee1e72442cc5823159f053bd8651501face67cc101db85bb4cf0682"}
Sep 11 01:30:50.862661 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:30:50.862590    2569 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e577a30eaee1e72442cc5823159f053bd8651501face67cc101db85bb4cf0682"
Sep 11 01:30:57.384736 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:30:57.384698    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-bae8d3-5db4c6fdd5-ldnjc"]
Sep 11 01:30:57.385222 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:30:57.384974    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-bae8d3-5db4c6fdd5-ldnjc" podUID="4507e865-3c97-479b-a029-f9bee7af7d36" containerName="payload-processing" containerID="cri-o://01d0b916991102eaa52972981f57424ee816853bedb32b9bde60cc9218a5dc2c" gracePeriod=30
Sep 11 01:30:57.392701 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:30:57.392548    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-bae8d3-7ddfd8cc4-tmb2s"]
Sep 11 01:30:57.392804 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:30:57.392772    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-bae8d3-7ddfd8cc4-tmb2s" podUID="8e746168-41d8-41d8-a35d-0730f7f40ed7" containerName="payload-pre-processing" containerID="cri-o://692ae21d4de354f40cf20852e0527b9a89fe411f5c88a676de4f6e94a4913d33" gracePeriod=30
Sep 11 01:30:57.573965 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:30:57.573933    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-bae8d3-xv67p"]
Sep 11 01:30:57.576948 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:30:57.576926    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-bae8d3-xv67p"]
Sep 11 01:30:57.672763 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:30:57.672739    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-bae8d3-5db4c6fdd5-ldnjc"
Sep 11 01:30:57.673716 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:30:57.673697    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-bae8d3-7ddfd8cc4-tmb2s"
Sep 11 01:30:57.760307 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:30:57.760277    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pppqv\" (UniqueName: \"kubernetes.io/projected/8e746168-41d8-41d8-a35d-0730f7f40ed7-kube-api-access-pppqv\") pod \"8e746168-41d8-41d8-a35d-0730f7f40ed7\" (UID: \"8e746168-41d8-41d8-a35d-0730f7f40ed7\") "
Sep 11 01:30:57.760464 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:30:57.760327    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e746168-41d8-41d8-a35d-0730f7f40ed7-plugins-config-volume\") pod \"8e746168-41d8-41d8-a35d-0730f7f40ed7\" (UID: \"8e746168-41d8-41d8-a35d-0730f7f40ed7\") "
Sep 11 01:30:57.760511 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:30:57.760472    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7wgqb\" (UniqueName: \"kubernetes.io/projected/4507e865-3c97-479b-a029-f9bee7af7d36-kube-api-access-7wgqb\") pod \"4507e865-3c97-479b-a029-f9bee7af7d36\" (UID: \"4507e865-3c97-479b-a029-f9bee7af7d36\") "
Sep 11 01:30:57.760552 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:30:57.760537    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4507e865-3c97-479b-a029-f9bee7af7d36-plugins-config-volume\") pod \"4507e865-3c97-479b-a029-f9bee7af7d36\" (UID: \"4507e865-3c97-479b-a029-f9bee7af7d36\") "
Sep 11 01:30:57.760725 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:30:57.760702    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e746168-41d8-41d8-a35d-0730f7f40ed7-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8e746168-41d8-41d8-a35d-0730f7f40ed7" (UID: "8e746168-41d8-41d8-a35d-0730f7f40ed7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 01:30:57.761000 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:30:57.760968    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4507e865-3c97-479b-a029-f9bee7af7d36-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4507e865-3c97-479b-a029-f9bee7af7d36" (UID: "4507e865-3c97-479b-a029-f9bee7af7d36"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 01:30:57.761000 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:30:57.760985    2569 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e746168-41d8-41d8-a35d-0730f7f40ed7-plugins-config-volume\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:30:57.762471 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:30:57.762446    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e746168-41d8-41d8-a35d-0730f7f40ed7-kube-api-access-pppqv" (OuterVolumeSpecName: "kube-api-access-pppqv") pod "8e746168-41d8-41d8-a35d-0730f7f40ed7" (UID: "8e746168-41d8-41d8-a35d-0730f7f40ed7"). InnerVolumeSpecName "kube-api-access-pppqv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 01:30:57.762576 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:30:57.762506    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4507e865-3c97-479b-a029-f9bee7af7d36-kube-api-access-7wgqb" (OuterVolumeSpecName: "kube-api-access-7wgqb") pod "4507e865-3c97-479b-a029-f9bee7af7d36" (UID: "4507e865-3c97-479b-a029-f9bee7af7d36"). InnerVolumeSpecName "kube-api-access-7wgqb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 01:30:57.861657 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:30:57.861628    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7wgqb\" (UniqueName: \"kubernetes.io/projected/4507e865-3c97-479b-a029-f9bee7af7d36-kube-api-access-7wgqb\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:30:57.861657 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:30:57.861652    2569 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4507e865-3c97-479b-a029-f9bee7af7d36-plugins-config-volume\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:30:57.861824 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:30:57.861663    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pppqv\" (UniqueName: \"kubernetes.io/projected/8e746168-41d8-41d8-a35d-0730f7f40ed7-kube-api-access-pppqv\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:30:57.878492 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:30:57.878462    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2ee924d9-4129-4a56-aa47-3995c323b320" path="/var/lib/kubelet/pods/2ee924d9-4129-4a56-aa47-3995c323b320/volumes"
Sep 11 01:30:57.903747 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:30:57.903685    2569 generic.go:356] "Generic (PLEG): container finished" podID="8e746168-41d8-41d8-a35d-0730f7f40ed7" containerID="692ae21d4de354f40cf20852e0527b9a89fe411f5c88a676de4f6e94a4913d33" exitCode=0
Sep 11 01:30:57.903747 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:30:57.903736    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-bae8d3-7ddfd8cc4-tmb2s"
Sep 11 01:30:57.903935 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:30:57.903768    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-bae8d3-7ddfd8cc4-tmb2s" event={"ID":"8e746168-41d8-41d8-a35d-0730f7f40ed7","Type":"ContainerDied","Data":"692ae21d4de354f40cf20852e0527b9a89fe411f5c88a676de4f6e94a4913d33"}
Sep 11 01:30:57.903935 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:30:57.903812    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-bae8d3-7ddfd8cc4-tmb2s" event={"ID":"8e746168-41d8-41d8-a35d-0730f7f40ed7","Type":"ContainerDied","Data":"a8e87ca09820e7007aba63d52ac2556fa9db556028ed70599ba4714a05d88b53"}
Sep 11 01:30:57.903935 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:30:57.903835    2569 scope.go:117] "RemoveContainer" containerID="692ae21d4de354f40cf20852e0527b9a89fe411f5c88a676de4f6e94a4913d33"
Sep 11 01:30:57.905274 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:30:57.905253    2569 generic.go:356] "Generic (PLEG): container finished" podID="4507e865-3c97-479b-a029-f9bee7af7d36" containerID="01d0b916991102eaa52972981f57424ee816853bedb32b9bde60cc9218a5dc2c" exitCode=0
Sep 11 01:30:57.905366 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:30:57.905308    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-bae8d3-5db4c6fdd5-ldnjc"
Sep 11 01:30:57.905366 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:30:57.905318    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-bae8d3-5db4c6fdd5-ldnjc" event={"ID":"4507e865-3c97-479b-a029-f9bee7af7d36","Type":"ContainerDied","Data":"01d0b916991102eaa52972981f57424ee816853bedb32b9bde60cc9218a5dc2c"}
Sep 11 01:30:57.905366 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:30:57.905346    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-bae8d3-5db4c6fdd5-ldnjc" event={"ID":"4507e865-3c97-479b-a029-f9bee7af7d36","Type":"ContainerDied","Data":"05ba74fd088ea7596e59c9fb4119177555ff4c8d76e1f525eab6d524a6d21459"}
Sep 11 01:30:57.913312 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:30:57.913295    2569 scope.go:117] "RemoveContainer" containerID="692ae21d4de354f40cf20852e0527b9a89fe411f5c88a676de4f6e94a4913d33"
Sep 11 01:30:57.913580 ip-10-0-129-222 kubenswrapper[2569]: E0911 01:30:57.913562    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"692ae21d4de354f40cf20852e0527b9a89fe411f5c88a676de4f6e94a4913d33\": container with ID starting with 692ae21d4de354f40cf20852e0527b9a89fe411f5c88a676de4f6e94a4913d33 not found: ID does not exist" containerID="692ae21d4de354f40cf20852e0527b9a89fe411f5c88a676de4f6e94a4913d33"
Sep 11 01:30:57.913644 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:30:57.913593    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"692ae21d4de354f40cf20852e0527b9a89fe411f5c88a676de4f6e94a4913d33"} err="failed to get container status \"692ae21d4de354f40cf20852e0527b9a89fe411f5c88a676de4f6e94a4913d33\": rpc error: code = NotFound desc = could not find container \"692ae21d4de354f40cf20852e0527b9a89fe411f5c88a676de4f6e94a4913d33\": container with ID starting with 692ae21d4de354f40cf20852e0527b9a89fe411f5c88a676de4f6e94a4913d33 not found: ID does not exist"
Sep 11 01:30:57.913644 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:30:57.913609    2569 scope.go:117] "RemoveContainer" containerID="01d0b916991102eaa52972981f57424ee816853bedb32b9bde60cc9218a5dc2c"
Sep 11 01:30:57.922051 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:30:57.922031    2569 scope.go:117] "RemoveContainer" containerID="01d0b916991102eaa52972981f57424ee816853bedb32b9bde60cc9218a5dc2c"
Sep 11 01:30:57.922352 ip-10-0-129-222 kubenswrapper[2569]: E0911 01:30:57.922330    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01d0b916991102eaa52972981f57424ee816853bedb32b9bde60cc9218a5dc2c\": container with ID starting with 01d0b916991102eaa52972981f57424ee816853bedb32b9bde60cc9218a5dc2c not found: ID does not exist" containerID="01d0b916991102eaa52972981f57424ee816853bedb32b9bde60cc9218a5dc2c"
Sep 11 01:30:57.922432 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:30:57.922358    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01d0b916991102eaa52972981f57424ee816853bedb32b9bde60cc9218a5dc2c"} err="failed to get container status \"01d0b916991102eaa52972981f57424ee816853bedb32b9bde60cc9218a5dc2c\": rpc error: code = NotFound desc = could not find container \"01d0b916991102eaa52972981f57424ee816853bedb32b9bde60cc9218a5dc2c\": container with ID starting with 01d0b916991102eaa52972981f57424ee816853bedb32b9bde60cc9218a5dc2c not found: ID does not exist"
Sep 11 01:30:57.923532 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:30:57.923508    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-bae8d3-7ddfd8cc4-tmb2s"]
Sep 11 01:30:57.928734 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:30:57.928711    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-bae8d3-7ddfd8cc4-tmb2s"]
Sep 11 01:30:57.945432 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:30:57.945411    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-bae8d3-5db4c6fdd5-ldnjc"]
Sep 11 01:30:57.947255 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:30:57.947232    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-bae8d3-5db4c6fdd5-ldnjc"]
Sep 11 01:30:59.871436 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:30:59.871400    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4507e865-3c97-479b-a029-f9bee7af7d36" path="/var/lib/kubelet/pods/4507e865-3c97-479b-a029-f9bee7af7d36/volumes"
Sep 11 01:30:59.871944 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:30:59.871781    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8e746168-41d8-41d8-a35d-0730f7f40ed7" path="/var/lib/kubelet/pods/8e746168-41d8-41d8-a35d-0730f7f40ed7/volumes"
Sep 11 01:31:00.798861 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:31:00.798830    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kmx5x"]
Sep 11 01:31:04.439573 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:31:04.439535    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-2482ac-d4sc9"]
Sep 11 01:31:04.439998 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:31:04.439978    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4507e865-3c97-479b-a029-f9bee7af7d36" containerName="payload-processing"
Sep 11 01:31:04.439998 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:31:04.439991    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="4507e865-3c97-479b-a029-f9bee7af7d36" containerName="payload-processing"
Sep 11 01:31:04.440104 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:31:04.440008    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e746168-41d8-41d8-a35d-0730f7f40ed7" containerName="payload-pre-processing"
Sep 11 01:31:04.440104 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:31:04.440014    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e746168-41d8-41d8-a35d-0730f7f40ed7" containerName="payload-pre-processing"
Sep 11 01:31:04.440104 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:31:04.440021    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ee924d9-4129-4a56-aa47-3995c323b320" containerName="revoke-keys"
Sep 11 01:31:04.440104 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:31:04.440027    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ee924d9-4129-4a56-aa47-3995c323b320" containerName="revoke-keys"
Sep 11 01:31:04.440104 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:31:04.440103    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="4507e865-3c97-479b-a029-f9bee7af7d36" containerName="payload-processing"
Sep 11 01:31:04.440278 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:31:04.440115    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e746168-41d8-41d8-a35d-0730f7f40ed7" containerName="payload-pre-processing"
Sep 11 01:31:04.440278 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:31:04.440123    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="2ee924d9-4129-4a56-aa47-3995c323b320" containerName="revoke-keys"
Sep 11 01:31:04.443065 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:31:04.443048    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-2482ac-d4sc9"
Sep 11 01:31:04.445390 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:31:04.445370    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pppkt\""
Sep 11 01:31:04.445496 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:31:04.445370    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 01:31:04.450286 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:31:04.450259    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-2482ac-d4sc9"]
Sep 11 01:31:04.521968 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:31:04.521937    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e556216-1967-484c-b5b1-04ed93b90042-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-2482ac-d4sc9\" (UID: \"4e556216-1967-484c-b5b1-04ed93b90042\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-2482ac-d4sc9"
Sep 11 01:31:04.622920 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:31:04.622859    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e556216-1967-484c-b5b1-04ed93b90042-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-2482ac-d4sc9\" (UID: \"4e556216-1967-484c-b5b1-04ed93b90042\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-2482ac-d4sc9"
Sep 11 01:31:04.623504 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:31:04.623485    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e556216-1967-484c-b5b1-04ed93b90042-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-2482ac-d4sc9\" (UID: \"4e556216-1967-484c-b5b1-04ed93b90042\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-2482ac-d4sc9"
Sep 11 01:31:04.756026 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:31:04.755959    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-2482ac-d4sc9"
Sep 11 01:31:05.029131 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:31:05.029103    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-2482ac-d4sc9"]
Sep 11 01:31:05.032144 ip-10-0-129-222 kubenswrapper[2569]: W0911 01:31:05.032116    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e556216_1967_484c_b5b1_04ed93b90042.slice/crio-f5173f8892ff29976529c5c0627bb2f95312520542c93467391968fa115f41ef WatchSource:0}: Error finding container f5173f8892ff29976529c5c0627bb2f95312520542c93467391968fa115f41ef: Status 404 returned error can't find the container with id f5173f8892ff29976529c5c0627bb2f95312520542c93467391968fa115f41ef
Sep 11 01:31:05.950441 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:31:05.950408    2569 generic.go:356] "Generic (PLEG): container finished" podID="4e556216-1967-484c-b5b1-04ed93b90042" containerID="a0ad6be02545621afb8832ce51f6e766c9860a129d4bf76a9601bc4ccf6df35d" exitCode=0
Sep 11 01:31:05.950808 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:31:05.950459    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-2482ac-d4sc9" event={"ID":"4e556216-1967-484c-b5b1-04ed93b90042","Type":"ContainerDied","Data":"a0ad6be02545621afb8832ce51f6e766c9860a129d4bf76a9601bc4ccf6df35d"}
Sep 11 01:31:05.950808 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:31:05.950501    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-2482ac-d4sc9" event={"ID":"4e556216-1967-484c-b5b1-04ed93b90042","Type":"ContainerStarted","Data":"f5173f8892ff29976529c5c0627bb2f95312520542c93467391968fa115f41ef"}
Sep 11 01:31:07.085610 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:31:07.085586    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-2482ac-d4sc9"
Sep 11 01:31:07.142236 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:31:07.142205    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e556216-1967-484c-b5b1-04ed93b90042-maas-api-service-ca\") pod \"4e556216-1967-484c-b5b1-04ed93b90042\" (UID: \"4e556216-1967-484c-b5b1-04ed93b90042\") "
Sep 11 01:31:07.142645 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:31:07.142615    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e556216-1967-484c-b5b1-04ed93b90042-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "4e556216-1967-484c-b5b1-04ed93b90042" (UID: "4e556216-1967-484c-b5b1-04ed93b90042"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 01:31:07.243800 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:31:07.243733    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e556216-1967-484c-b5b1-04ed93b90042-maas-api-service-ca\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:31:07.961351 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:31:07.961327    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-2482ac-d4sc9"
Sep 11 01:31:07.961493 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:31:07.961368    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-2482ac-d4sc9" event={"ID":"4e556216-1967-484c-b5b1-04ed93b90042","Type":"ContainerDied","Data":"f5173f8892ff29976529c5c0627bb2f95312520542c93467391968fa115f41ef"}
Sep 11 01:31:07.961493 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:31:07.961416    2569 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f5173f8892ff29976529c5c0627bb2f95312520542c93467391968fa115f41ef"
Sep 11 01:31:15.035862 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:31:15.035819    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2482ac-29818170-kxbbl"]
Sep 11 01:31:15.038037 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:31:15.038007    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2482ac-29818170-kxbbl"]
Sep 11 01:31:15.062104 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:31:15.062074    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-2482ac-6c4ddccb95-xs7tc"]
Sep 11 01:31:15.062409 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:31:15.062354    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2482ac-6c4ddccb95-xs7tc" podUID="ebb5ee7b-b3c8-45e8-a361-c2bdf98ccd20" containerName="payload-processing" containerID="cri-o://490e0c9833c52b9ade373352235c2c9828b989b7d1619580e247281e8f7c0501" gracePeriod=30
Sep 11 01:31:15.066707 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:31:15.066680    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2482ac-75f7b999d6-bjvlg"]
Sep 11 01:31:15.067141 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:31:15.067090    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2482ac-75f7b999d6-bjvlg" podUID="7b5dbde6-6b81-444e-822e-0ee063ef4065" containerName="payload-pre-processing" containerID="cri-o://ddd62c8ba49c95c613a2ef0b52588beb9555bd22b59ab9e62cb3f5abb4e28ca8" gracePeriod=30
Sep 11 01:31:15.209741 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:31:15.209703    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-2482ac-d4sc9"]
Sep 11 01:31:15.212257 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:31:15.212231    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-2482ac-d4sc9"]
Sep 11 01:31:15.353167 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:31:15.353139    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2482ac-75f7b999d6-bjvlg"
Sep 11 01:31:15.356176 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:31:15.356156    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2482ac-6c4ddccb95-xs7tc"
Sep 11 01:31:15.425490 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:31:15.425463    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-znwjf\" (UniqueName: \"kubernetes.io/projected/7b5dbde6-6b81-444e-822e-0ee063ef4065-kube-api-access-znwjf\") pod \"7b5dbde6-6b81-444e-822e-0ee063ef4065\" (UID: \"7b5dbde6-6b81-444e-822e-0ee063ef4065\") "
Sep 11 01:31:15.425676 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:31:15.425526    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ebb5ee7b-b3c8-45e8-a361-c2bdf98ccd20-plugins-config-volume\") pod \"ebb5ee7b-b3c8-45e8-a361-c2bdf98ccd20\" (UID: \"ebb5ee7b-b3c8-45e8-a361-c2bdf98ccd20\") "
Sep 11 01:31:15.425676 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:31:15.425565    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zzcwp\" (UniqueName: \"kubernetes.io/projected/ebb5ee7b-b3c8-45e8-a361-c2bdf98ccd20-kube-api-access-zzcwp\") pod \"ebb5ee7b-b3c8-45e8-a361-c2bdf98ccd20\" (UID: \"ebb5ee7b-b3c8-45e8-a361-c2bdf98ccd20\") "
Sep 11 01:31:15.425676 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:31:15.425624    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b5dbde6-6b81-444e-822e-0ee063ef4065-plugins-config-volume\") pod \"7b5dbde6-6b81-444e-822e-0ee063ef4065\" (UID: \"7b5dbde6-6b81-444e-822e-0ee063ef4065\") "
Sep 11 01:31:15.426046 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:31:15.426014    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ebb5ee7b-b3c8-45e8-a361-c2bdf98ccd20-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ebb5ee7b-b3c8-45e8-a361-c2bdf98ccd20" (UID: "ebb5ee7b-b3c8-45e8-a361-c2bdf98ccd20"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 01:31:15.426173 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:31:15.426056    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b5dbde6-6b81-444e-822e-0ee063ef4065-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7b5dbde6-6b81-444e-822e-0ee063ef4065" (UID: "7b5dbde6-6b81-444e-822e-0ee063ef4065"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 01:31:15.427736 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:31:15.427709    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b5dbde6-6b81-444e-822e-0ee063ef4065-kube-api-access-znwjf" (OuterVolumeSpecName: "kube-api-access-znwjf") pod "7b5dbde6-6b81-444e-822e-0ee063ef4065" (UID: "7b5dbde6-6b81-444e-822e-0ee063ef4065"). InnerVolumeSpecName "kube-api-access-znwjf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 01:31:15.427840 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:31:15.427744    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebb5ee7b-b3c8-45e8-a361-c2bdf98ccd20-kube-api-access-zzcwp" (OuterVolumeSpecName: "kube-api-access-zzcwp") pod "ebb5ee7b-b3c8-45e8-a361-c2bdf98ccd20" (UID: "ebb5ee7b-b3c8-45e8-a361-c2bdf98ccd20"). InnerVolumeSpecName "kube-api-access-zzcwp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 01:31:15.526644 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:31:15.526618    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-znwjf\" (UniqueName: \"kubernetes.io/projected/7b5dbde6-6b81-444e-822e-0ee063ef4065-kube-api-access-znwjf\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:31:15.526644 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:31:15.526643    2569 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ebb5ee7b-b3c8-45e8-a361-c2bdf98ccd20-plugins-config-volume\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:31:15.526820 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:31:15.526652    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zzcwp\" (UniqueName: \"kubernetes.io/projected/ebb5ee7b-b3c8-45e8-a361-c2bdf98ccd20-kube-api-access-zzcwp\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:31:15.526820 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:31:15.526661    2569 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b5dbde6-6b81-444e-822e-0ee063ef4065-plugins-config-volume\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:31:15.870540 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:31:15.870507    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4e556216-1967-484c-b5b1-04ed93b90042" path="/var/lib/kubelet/pods/4e556216-1967-484c-b5b1-04ed93b90042/volumes"
Sep 11 01:31:15.870811 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:31:15.870798    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e20b24c2-11fb-4ed5-a710-04cd39193d19" path="/var/lib/kubelet/pods/e20b24c2-11fb-4ed5-a710-04cd39193d19/volumes"
Sep 11 01:31:16.012543 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:31:16.012514    2569 generic.go:356] "Generic (PLEG): container finished" podID="7b5dbde6-6b81-444e-822e-0ee063ef4065" containerID="ddd62c8ba49c95c613a2ef0b52588beb9555bd22b59ab9e62cb3f5abb4e28ca8" exitCode=0
Sep 11 01:31:16.012683 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:31:16.012559    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2482ac-75f7b999d6-bjvlg"
Sep 11 01:31:16.012683 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:31:16.012597    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2482ac-75f7b999d6-bjvlg" event={"ID":"7b5dbde6-6b81-444e-822e-0ee063ef4065","Type":"ContainerDied","Data":"ddd62c8ba49c95c613a2ef0b52588beb9555bd22b59ab9e62cb3f5abb4e28ca8"}
Sep 11 01:31:16.012683 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:31:16.012647    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2482ac-75f7b999d6-bjvlg" event={"ID":"7b5dbde6-6b81-444e-822e-0ee063ef4065","Type":"ContainerDied","Data":"631b9da27dd4aa8dda424b464a3319e0138da1e0a8b0ed891fb54845a53b17c6"}
Sep 11 01:31:16.012683 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:31:16.012668    2569 scope.go:117] "RemoveContainer" containerID="ddd62c8ba49c95c613a2ef0b52588beb9555bd22b59ab9e62cb3f5abb4e28ca8"
Sep 11 01:31:16.014105 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:31:16.014067    2569 generic.go:356] "Generic (PLEG): container finished" podID="ebb5ee7b-b3c8-45e8-a361-c2bdf98ccd20" containerID="490e0c9833c52b9ade373352235c2c9828b989b7d1619580e247281e8f7c0501" exitCode=0
Sep 11 01:31:16.014226 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:31:16.014117    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2482ac-6c4ddccb95-xs7tc"
Sep 11 01:31:16.014226 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:31:16.014122    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2482ac-6c4ddccb95-xs7tc" event={"ID":"ebb5ee7b-b3c8-45e8-a361-c2bdf98ccd20","Type":"ContainerDied","Data":"490e0c9833c52b9ade373352235c2c9828b989b7d1619580e247281e8f7c0501"}
Sep 11 01:31:16.014226 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:31:16.014223    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2482ac-6c4ddccb95-xs7tc" event={"ID":"ebb5ee7b-b3c8-45e8-a361-c2bdf98ccd20","Type":"ContainerDied","Data":"ef93a51ba96963f5ab01a90ef40a04a259eaadd1b48baa2ba057be3b340fb2e1"}
Sep 11 01:31:16.021543 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:31:16.021528    2569 scope.go:117] "RemoveContainer" containerID="ddd62c8ba49c95c613a2ef0b52588beb9555bd22b59ab9e62cb3f5abb4e28ca8"
Sep 11 01:31:16.021788 ip-10-0-129-222 kubenswrapper[2569]: E0911 01:31:16.021772    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ddd62c8ba49c95c613a2ef0b52588beb9555bd22b59ab9e62cb3f5abb4e28ca8\": container with ID starting with ddd62c8ba49c95c613a2ef0b52588beb9555bd22b59ab9e62cb3f5abb4e28ca8 not found: ID does not exist" containerID="ddd62c8ba49c95c613a2ef0b52588beb9555bd22b59ab9e62cb3f5abb4e28ca8"
Sep 11 01:31:16.021841 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:31:16.021794    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ddd62c8ba49c95c613a2ef0b52588beb9555bd22b59ab9e62cb3f5abb4e28ca8"} err="failed to get container status \"ddd62c8ba49c95c613a2ef0b52588beb9555bd22b59ab9e62cb3f5abb4e28ca8\": rpc error: code = NotFound desc = could not find container \"ddd62c8ba49c95c613a2ef0b52588beb9555bd22b59ab9e62cb3f5abb4e28ca8\": container with ID starting with ddd62c8ba49c95c613a2ef0b52588beb9555bd22b59ab9e62cb3f5abb4e28ca8 not found: ID does not exist"
Sep 11 01:31:16.021841 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:31:16.021809    2569 scope.go:117] "RemoveContainer" containerID="490e0c9833c52b9ade373352235c2c9828b989b7d1619580e247281e8f7c0501"
Sep 11 01:31:16.030082 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:31:16.030060    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-2482ac-6c4ddccb95-xs7tc"]
Sep 11 01:31:16.030196 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:31:16.030176    2569 scope.go:117] "RemoveContainer" containerID="490e0c9833c52b9ade373352235c2c9828b989b7d1619580e247281e8f7c0501"
Sep 11 01:31:16.030714 ip-10-0-129-222 kubenswrapper[2569]: E0911 01:31:16.030686    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"490e0c9833c52b9ade373352235c2c9828b989b7d1619580e247281e8f7c0501\": container with ID starting with 490e0c9833c52b9ade373352235c2c9828b989b7d1619580e247281e8f7c0501 not found: ID does not exist" containerID="490e0c9833c52b9ade373352235c2c9828b989b7d1619580e247281e8f7c0501"
Sep 11 01:31:16.030834 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:31:16.030722    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"490e0c9833c52b9ade373352235c2c9828b989b7d1619580e247281e8f7c0501"} err="failed to get container status \"490e0c9833c52b9ade373352235c2c9828b989b7d1619580e247281e8f7c0501\": rpc error: code = NotFound desc = could not find container \"490e0c9833c52b9ade373352235c2c9828b989b7d1619580e247281e8f7c0501\": container with ID starting with 490e0c9833c52b9ade373352235c2c9828b989b7d1619580e247281e8f7c0501 not found: ID does not exist"
Sep 11 01:31:16.032638 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:31:16.032619    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-2482ac-6c4ddccb95-xs7tc"]
Sep 11 01:31:16.041856 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:31:16.041836    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2482ac-75f7b999d6-bjvlg"]
Sep 11 01:31:16.048582 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:31:16.048562    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2482ac-75f7b999d6-bjvlg"]
Sep 11 01:31:17.808821 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:31:17.808783    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kmx5x"]
Sep 11 01:31:17.873572 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:31:17.873536    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7b5dbde6-6b81-444e-822e-0ee063ef4065" path="/var/lib/kubelet/pods/7b5dbde6-6b81-444e-822e-0ee063ef4065/volumes"
Sep 11 01:31:17.874097 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:31:17.874071    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ebb5ee7b-b3c8-45e8-a361-c2bdf98ccd20" path="/var/lib/kubelet/pods/ebb5ee7b-b3c8-45e8-a361-c2bdf98ccd20/volumes"
Sep 11 01:31:21.713124 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:31:21.713090    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-2482ac-openshift-default-75b5d5c9b8-zfr9t"]
Sep 11 01:31:21.713654 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:31:21.713301    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-2482ac-openshift-default-75b5d5c9b8-zfr9t" podUID="7a92f15b-fd83-4f83-be73-8e5249d0686a" containerName="istio-proxy" containerID="cri-o://b6c252e9e6eb12d104f6189d25627a1b7c4fef8634f53140657c802031fdd77a" gracePeriod=30
Sep 11 01:31:22.083560 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:31:22.083529    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-9433b9-ggr5z"]
Sep 11 01:31:22.083969 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:31:22.083955    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ebb5ee7b-b3c8-45e8-a361-c2bdf98ccd20" containerName="payload-processing"
Sep 11 01:31:22.084021 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:31:22.083970    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="ebb5ee7b-b3c8-45e8-a361-c2bdf98ccd20" containerName="payload-processing"
Sep 11 01:31:22.084021 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:31:22.083984    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e556216-1967-484c-b5b1-04ed93b90042" containerName="revoke-keys"
Sep 11 01:31:22.084021 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:31:22.083991    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e556216-1967-484c-b5b1-04ed93b90042" containerName="revoke-keys"
Sep 11 01:31:22.084021 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:31:22.083999    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b5dbde6-6b81-444e-822e-0ee063ef4065" containerName="payload-pre-processing"
Sep 11 01:31:22.084021 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:31:22.084005    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b5dbde6-6b81-444e-822e-0ee063ef4065" containerName="payload-pre-processing"
Sep 11 01:31:22.084204 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:31:22.084059    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="7b5dbde6-6b81-444e-822e-0ee063ef4065" containerName="payload-pre-processing"
Sep 11 01:31:22.084204 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:31:22.084066    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="ebb5ee7b-b3c8-45e8-a361-c2bdf98ccd20" containerName="payload-processing"
Sep 11 01:31:22.084204 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:31:22.084075    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e556216-1967-484c-b5b1-04ed93b90042" containerName="revoke-keys"
Sep 11 01:31:22.087049 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:31:22.087026    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-9433b9-ggr5z"
Sep 11 01:31:22.089566 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:31:22.089545    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pppkt\""
Sep 11 01:31:22.089684 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:31:22.089550    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 01:31:22.096481 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:31:22.096458    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-9433b9-ggr5z"]
Sep 11 01:31:22.199933 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:31:22.199905    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/68d224f4-99c9-4553-b21d-07b92209a86a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-9433b9-ggr5z\" (UID: \"68d224f4-99c9-4553-b21d-07b92209a86a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-9433b9-ggr5z"
Sep 11 01:31:22.300634 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:31:22.300602    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/68d224f4-99c9-4553-b21d-07b92209a86a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-9433b9-ggr5z\" (UID: \"68d224f4-99c9-4553-b21d-07b92209a86a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-9433b9-ggr5z"
Sep 11 01:31:22.301283 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:31:22.301260    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/68d224f4-99c9-4553-b21d-07b92209a86a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-9433b9-ggr5z\" (UID: \"68d224f4-99c9-4553-b21d-07b92209a86a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-9433b9-ggr5z"
Sep 11 01:31:22.399165 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:31:22.399098    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-9433b9-ggr5z"
Sep 11 01:31:22.669423 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:31:22.669393    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-9433b9-ggr5z"]
Sep 11 01:31:22.671559 ip-10-0-129-222 kubenswrapper[2569]: W0911 01:31:22.671530    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68d224f4_99c9_4553_b21d_07b92209a86a.slice/crio-c3849f76d5c852beac576d3bd843cde873dde445e97cb540b42ee6c1a44964ad WatchSource:0}: Error finding container c3849f76d5c852beac576d3bd843cde873dde445e97cb540b42ee6c1a44964ad: Status 404 returned error can't find the container with id c3849f76d5c852beac576d3bd843cde873dde445e97cb540b42ee6c1a44964ad
Sep 11 01:31:23.060202 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:31:23.060159    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-9433b9-ggr5z" event={"ID":"68d224f4-99c9-4553-b21d-07b92209a86a","Type":"ContainerStarted","Data":"9c1ac101c65799f255ad234e0d2f7ca8c75cdd243100c21fe007f1925de4443c"}
Sep 11 01:31:23.060594 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:31:23.060222    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-9433b9-ggr5z" event={"ID":"68d224f4-99c9-4553-b21d-07b92209a86a","Type":"ContainerStarted","Data":"c3849f76d5c852beac576d3bd843cde873dde445e97cb540b42ee6c1a44964ad"}
Sep 11 01:31:23.074974 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:31:23.074931    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-9433b9-ggr5z" podStartSLOduration=1.07491835 podStartE2EDuration="1.07491835s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 01:31:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 01:31:23.073600752 +0000 UTC m=+2017.724975003" watchObservedRunningTime="2026-09-11 01:31:23.07491835 +0000 UTC m=+2017.726292599"
Sep 11 01:31:24.067034 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:31:24.066994    2569 generic.go:356] "Generic (PLEG): container finished" podID="68d224f4-99c9-4553-b21d-07b92209a86a" containerID="9c1ac101c65799f255ad234e0d2f7ca8c75cdd243100c21fe007f1925de4443c" exitCode=0
Sep 11 01:31:24.067399 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:31:24.067080    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-9433b9-ggr5z" event={"ID":"68d224f4-99c9-4553-b21d-07b92209a86a","Type":"ContainerDied","Data":"9c1ac101c65799f255ad234e0d2f7ca8c75cdd243100c21fe007f1925de4443c"}
Sep 11 01:31:24.433133 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:31:24.433039    2569 patch_prober.go:28] interesting pod/e2e-shared-a-w3-2482ac-openshift-default-75b5d5c9b8-zfr9t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 11 01:31:24.433133 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:31:24.433102    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-2482ac-openshift-default-75b5d5c9b8-zfr9t" podUID="7a92f15b-fd83-4f83-be73-8e5249d0686a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 11 01:31:25.201865 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:31:25.201841    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-9433b9-ggr5z"
Sep 11 01:31:25.325943 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:31:25.325847    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/68d224f4-99c9-4553-b21d-07b92209a86a-maas-api-service-ca\") pod \"68d224f4-99c9-4553-b21d-07b92209a86a\" (UID: \"68d224f4-99c9-4553-b21d-07b92209a86a\") "
Sep 11 01:31:25.326197 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:31:25.326172    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68d224f4-99c9-4553-b21d-07b92209a86a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "68d224f4-99c9-4553-b21d-07b92209a86a" (UID: "68d224f4-99c9-4553-b21d-07b92209a86a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 01:31:25.426821 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:31:25.426791    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/68d224f4-99c9-4553-b21d-07b92209a86a-maas-api-service-ca\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:31:26.079395 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:31:26.079357    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-9433b9-ggr5z" event={"ID":"68d224f4-99c9-4553-b21d-07b92209a86a","Type":"ContainerDied","Data":"c3849f76d5c852beac576d3bd843cde873dde445e97cb540b42ee6c1a44964ad"}
Sep 11 01:31:26.079395 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:31:26.079390    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-9433b9-ggr5z"
Sep 11 01:31:26.079603 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:31:26.079417    2569 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c3849f76d5c852beac576d3bd843cde873dde445e97cb540b42ee6c1a44964ad"
Sep 11 01:31:26.967947 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:31:26.967922    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-2482ac-openshift-default-75b5d5c9b8-zfr9t"
Sep 11 01:31:27.086729 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:31:27.086697    2569 generic.go:356] "Generic (PLEG): container finished" podID="7a92f15b-fd83-4f83-be73-8e5249d0686a" containerID="b6c252e9e6eb12d104f6189d25627a1b7c4fef8634f53140657c802031fdd77a" exitCode=0
Sep 11 01:31:27.086911 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:31:27.086738    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-2482ac-openshift-default-75b5d5c9b8-zfr9t" event={"ID":"7a92f15b-fd83-4f83-be73-8e5249d0686a","Type":"ContainerDied","Data":"b6c252e9e6eb12d104f6189d25627a1b7c4fef8634f53140657c802031fdd77a"}
Sep 11 01:31:27.086911 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:31:27.086794    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-2482ac-openshift-default-75b5d5c9b8-zfr9t" event={"ID":"7a92f15b-fd83-4f83-be73-8e5249d0686a","Type":"ContainerDied","Data":"754e10168915541382b90102b415e453f74411d1c13391e225dfb078ac06b286"}
Sep 11 01:31:27.086911 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:31:27.086815    2569 scope.go:117] "RemoveContainer" containerID="b6c252e9e6eb12d104f6189d25627a1b7c4fef8634f53140657c802031fdd77a"
Sep 11 01:31:27.086911 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:31:27.086747    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-2482ac-openshift-default-75b5d5c9b8-zfr9t"
Sep 11 01:31:27.096222 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:31:27.096204    2569 scope.go:117] "RemoveContainer" containerID="3c76a24d83b28546610a57f01eeca3cc81db1bcb2d63887e4236e00b175380a5"
Sep 11 01:31:27.104433 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:31:27.104413    2569 scope.go:117] "RemoveContainer" containerID="b6c252e9e6eb12d104f6189d25627a1b7c4fef8634f53140657c802031fdd77a"
Sep 11 01:31:27.104709 ip-10-0-129-222 kubenswrapper[2569]: E0911 01:31:27.104688    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6c252e9e6eb12d104f6189d25627a1b7c4fef8634f53140657c802031fdd77a\": container with ID starting with b6c252e9e6eb12d104f6189d25627a1b7c4fef8634f53140657c802031fdd77a not found: ID does not exist" containerID="b6c252e9e6eb12d104f6189d25627a1b7c4fef8634f53140657c802031fdd77a"
Sep 11 01:31:27.104764 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:31:27.104716    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6c252e9e6eb12d104f6189d25627a1b7c4fef8634f53140657c802031fdd77a"} err="failed to get container status \"b6c252e9e6eb12d104f6189d25627a1b7c4fef8634f53140657c802031fdd77a\": rpc error: code = NotFound desc = could not find container \"b6c252e9e6eb12d104f6189d25627a1b7c4fef8634f53140657c802031fdd77a\": container with ID starting with b6c252e9e6eb12d104f6189d25627a1b7c4fef8634f53140657c802031fdd77a not found: ID does not exist"
Sep 11 01:31:27.104764 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:31:27.104734    2569 scope.go:117] "RemoveContainer" containerID="3c76a24d83b28546610a57f01eeca3cc81db1bcb2d63887e4236e00b175380a5"
Sep 11 01:31:27.105024 ip-10-0-129-222 kubenswrapper[2569]: E0911 01:31:27.105002    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c76a24d83b28546610a57f01eeca3cc81db1bcb2d63887e4236e00b175380a5\": container with ID starting with 3c76a24d83b28546610a57f01eeca3cc81db1bcb2d63887e4236e00b175380a5 not found: ID does not exist" containerID="3c76a24d83b28546610a57f01eeca3cc81db1bcb2d63887e4236e00b175380a5"
Sep 11 01:31:27.105075 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:31:27.105031    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c76a24d83b28546610a57f01eeca3cc81db1bcb2d63887e4236e00b175380a5"} err="failed to get container status \"3c76a24d83b28546610a57f01eeca3cc81db1bcb2d63887e4236e00b175380a5\": rpc error: code = NotFound desc = could not find container \"3c76a24d83b28546610a57f01eeca3cc81db1bcb2d63887e4236e00b175380a5\": container with ID starting with 3c76a24d83b28546610a57f01eeca3cc81db1bcb2d63887e4236e00b175380a5 not found: ID does not exist"
Sep 11 01:31:27.142302 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:31:27.142270    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a92f15b-fd83-4f83-be73-8e5249d0686a-istio-token\") pod \"7a92f15b-fd83-4f83-be73-8e5249d0686a\" (UID: \"7a92f15b-fd83-4f83-be73-8e5249d0686a\") "
Sep 11 01:31:27.142416 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:31:27.142343    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a92f15b-fd83-4f83-be73-8e5249d0686a-credential-socket\") pod \"7a92f15b-fd83-4f83-be73-8e5249d0686a\" (UID: \"7a92f15b-fd83-4f83-be73-8e5249d0686a\") "
Sep 11 01:31:27.142416 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:31:27.142403    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a92f15b-fd83-4f83-be73-8e5249d0686a-istio-data\") pod \"7a92f15b-fd83-4f83-be73-8e5249d0686a\" (UID: \"7a92f15b-fd83-4f83-be73-8e5249d0686a\") "
Sep 11 01:31:27.142529 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:31:27.142437    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a92f15b-fd83-4f83-be73-8e5249d0686a-istiod-ca-cert\") pod \"7a92f15b-fd83-4f83-be73-8e5249d0686a\" (UID: \"7a92f15b-fd83-4f83-be73-8e5249d0686a\") "
Sep 11 01:31:27.142529 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:31:27.142469    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s4wxl\" (UniqueName: \"kubernetes.io/projected/7a92f15b-fd83-4f83-be73-8e5249d0686a-kube-api-access-s4wxl\") pod \"7a92f15b-fd83-4f83-be73-8e5249d0686a\" (UID: \"7a92f15b-fd83-4f83-be73-8e5249d0686a\") "
Sep 11 01:31:27.142529 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:31:27.142495    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a92f15b-fd83-4f83-be73-8e5249d0686a-istio-envoy\") pod \"7a92f15b-fd83-4f83-be73-8e5249d0686a\" (UID: \"7a92f15b-fd83-4f83-be73-8e5249d0686a\") "
Sep 11 01:31:27.142529 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:31:27.142521    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a92f15b-fd83-4f83-be73-8e5249d0686a-istio-podinfo\") pod \"7a92f15b-fd83-4f83-be73-8e5249d0686a\" (UID: \"7a92f15b-fd83-4f83-be73-8e5249d0686a\") "
Sep 11 01:31:27.142731 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:31:27.142595    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a92f15b-fd83-4f83-be73-8e5249d0686a-workload-socket\") pod \"7a92f15b-fd83-4f83-be73-8e5249d0686a\" (UID: \"7a92f15b-fd83-4f83-be73-8e5249d0686a\") "
Sep 11 01:31:27.142731 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:31:27.142629    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a92f15b-fd83-4f83-be73-8e5249d0686a-workload-certs\") pod \"7a92f15b-fd83-4f83-be73-8e5249d0686a\" (UID: \"7a92f15b-fd83-4f83-be73-8e5249d0686a\") "
Sep 11 01:31:27.143015 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:31:27.142627    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a92f15b-fd83-4f83-be73-8e5249d0686a-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "7a92f15b-fd83-4f83-be73-8e5249d0686a" (UID: "7a92f15b-fd83-4f83-be73-8e5249d0686a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 01:31:27.143149 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:31:27.142909    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a92f15b-fd83-4f83-be73-8e5249d0686a-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "7a92f15b-fd83-4f83-be73-8e5249d0686a" (UID: "7a92f15b-fd83-4f83-be73-8e5249d0686a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 01:31:27.143149 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:31:27.142806    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a92f15b-fd83-4f83-be73-8e5249d0686a-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "7a92f15b-fd83-4f83-be73-8e5249d0686a" (UID: "7a92f15b-fd83-4f83-be73-8e5249d0686a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 01:31:27.143149 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:31:27.142963    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a92f15b-fd83-4f83-be73-8e5249d0686a-istio-data" (OuterVolumeSpecName: "istio-data") pod "7a92f15b-fd83-4f83-be73-8e5249d0686a" (UID: "7a92f15b-fd83-4f83-be73-8e5249d0686a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 01:31:27.143149 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:31:27.143076    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a92f15b-fd83-4f83-be73-8e5249d0686a-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "7a92f15b-fd83-4f83-be73-8e5249d0686a" (UID: "7a92f15b-fd83-4f83-be73-8e5249d0686a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 01:31:27.144583 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:31:27.144552    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a92f15b-fd83-4f83-be73-8e5249d0686a-kube-api-access-s4wxl" (OuterVolumeSpecName: "kube-api-access-s4wxl") pod "7a92f15b-fd83-4f83-be73-8e5249d0686a" (UID: "7a92f15b-fd83-4f83-be73-8e5249d0686a"). InnerVolumeSpecName "kube-api-access-s4wxl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 01:31:27.144671 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:31:27.144589    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a92f15b-fd83-4f83-be73-8e5249d0686a-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "7a92f15b-fd83-4f83-be73-8e5249d0686a" (UID: "7a92f15b-fd83-4f83-be73-8e5249d0686a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 01:31:27.144808 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:31:27.144792    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a92f15b-fd83-4f83-be73-8e5249d0686a-istio-token" (OuterVolumeSpecName: "istio-token") pod "7a92f15b-fd83-4f83-be73-8e5249d0686a" (UID: "7a92f15b-fd83-4f83-be73-8e5249d0686a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 01:31:27.144893 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:31:27.144831    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7a92f15b-fd83-4f83-be73-8e5249d0686a-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "7a92f15b-fd83-4f83-be73-8e5249d0686a" (UID: "7a92f15b-fd83-4f83-be73-8e5249d0686a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 01:31:27.244299 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:31:27.244270    2569 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a92f15b-fd83-4f83-be73-8e5249d0686a-credential-socket\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:31:27.244299 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:31:27.244295    2569 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a92f15b-fd83-4f83-be73-8e5249d0686a-istio-data\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:31:27.244299 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:31:27.244304    2569 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a92f15b-fd83-4f83-be73-8e5249d0686a-istiod-ca-cert\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:31:27.244481 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:31:27.244312    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s4wxl\" (UniqueName: \"kubernetes.io/projected/7a92f15b-fd83-4f83-be73-8e5249d0686a-kube-api-access-s4wxl\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:31:27.244481 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:31:27.244322    2569 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a92f15b-fd83-4f83-be73-8e5249d0686a-istio-envoy\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:31:27.244481 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:31:27.244330    2569 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a92f15b-fd83-4f83-be73-8e5249d0686a-istio-podinfo\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:31:27.244481 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:31:27.244338    2569 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a92f15b-fd83-4f83-be73-8e5249d0686a-workload-socket\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:31:27.244481 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:31:27.244347    2569 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a92f15b-fd83-4f83-be73-8e5249d0686a-workload-certs\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:31:27.244481 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:31:27.244356    2569 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a92f15b-fd83-4f83-be73-8e5249d0686a-istio-token\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:31:27.408575 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:31:27.408545    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-2482ac-openshift-default-75b5d5c9b8-zfr9t"]
Sep 11 01:31:27.411975 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:31:27.411948    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-2482ac-openshift-default-75b5d5c9b8-zfr9t"]
Sep 11 01:31:27.870720 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:31:27.870686    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7a92f15b-fd83-4f83-be73-8e5249d0686a" path="/var/lib/kubelet/pods/7a92f15b-fd83-4f83-be73-8e5249d0686a/volumes"
Sep 11 01:31:32.525610 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:31:32.525575    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-9433b9-29818170-j7lkd"]
Sep 11 01:31:32.527631 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:31:32.527606    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-9433b9-29818170-j7lkd"]
Sep 11 01:31:32.552625 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:31:32.552597    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-9433b9-6746686dd5-5b2nd"]
Sep 11 01:31:32.552919 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:31:32.552890    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-9433b9-6746686dd5-5b2nd" podUID="84658a03-47b0-4f32-8385-4d36f59cdd8b" containerName="payload-processing" containerID="cri-o://89b101a5339ff779368dc4d8ed3ccf1dee06a55070d3cc0014535b90e6cbe93f" gracePeriod=30
Sep 11 01:31:32.561952 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:31:32.561924    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-9433b9-6b64d44946-bt9h7"]
Sep 11 01:31:32.562177 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:31:32.562146    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-9433b9-6b64d44946-bt9h7" podUID="aad0fca5-b959-47f7-8a46-a8da24458e5b" containerName="payload-pre-processing" containerID="cri-o://b8cd8c93b2bc99229b9609d91754d265caa91371990e60d1248e73e4ad2faf97" gracePeriod=30
Sep 11 01:31:32.701750 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:31:32.701717    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-9433b9-ggr5z"]
Sep 11 01:31:32.704591 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:31:32.704565    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-9433b9-ggr5z"]
Sep 11 01:31:32.809144 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:31:32.809119    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-9433b9-6746686dd5-5b2nd"
Sep 11 01:31:32.838740 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:31:32.838710    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-9433b9-6b64d44946-bt9h7"
Sep 11 01:31:32.989727 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:31:32.989693    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aad0fca5-b959-47f7-8a46-a8da24458e5b-plugins-config-volume\") pod \"aad0fca5-b959-47f7-8a46-a8da24458e5b\" (UID: \"aad0fca5-b959-47f7-8a46-a8da24458e5b\") "
Sep 11 01:31:32.989933 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:31:32.989736    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84658a03-47b0-4f32-8385-4d36f59cdd8b-plugins-config-volume\") pod \"84658a03-47b0-4f32-8385-4d36f59cdd8b\" (UID: \"84658a03-47b0-4f32-8385-4d36f59cdd8b\") "
Sep 11 01:31:32.989933 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:31:32.989897    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gkqxl\" (UniqueName: \"kubernetes.io/projected/84658a03-47b0-4f32-8385-4d36f59cdd8b-kube-api-access-gkqxl\") pod \"84658a03-47b0-4f32-8385-4d36f59cdd8b\" (UID: \"84658a03-47b0-4f32-8385-4d36f59cdd8b\") "
Sep 11 01:31:32.990068 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:31:32.989954    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kkw5k\" (UniqueName: \"kubernetes.io/projected/aad0fca5-b959-47f7-8a46-a8da24458e5b-kube-api-access-kkw5k\") pod \"aad0fca5-b959-47f7-8a46-a8da24458e5b\" (UID: \"aad0fca5-b959-47f7-8a46-a8da24458e5b\") "
Sep 11 01:31:32.990200 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:31:32.990173    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aad0fca5-b959-47f7-8a46-a8da24458e5b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "aad0fca5-b959-47f7-8a46-a8da24458e5b" (UID: "aad0fca5-b959-47f7-8a46-a8da24458e5b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 01:31:32.990326 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:31:32.990173    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84658a03-47b0-4f32-8385-4d36f59cdd8b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "84658a03-47b0-4f32-8385-4d36f59cdd8b" (UID: "84658a03-47b0-4f32-8385-4d36f59cdd8b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 01:31:32.990326 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:31:32.990271    2569 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aad0fca5-b959-47f7-8a46-a8da24458e5b-plugins-config-volume\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:31:32.990326 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:31:32.990286    2569 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84658a03-47b0-4f32-8385-4d36f59cdd8b-plugins-config-volume\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:31:32.992033 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:31:32.992012    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84658a03-47b0-4f32-8385-4d36f59cdd8b-kube-api-access-gkqxl" (OuterVolumeSpecName: "kube-api-access-gkqxl") pod "84658a03-47b0-4f32-8385-4d36f59cdd8b" (UID: "84658a03-47b0-4f32-8385-4d36f59cdd8b"). InnerVolumeSpecName "kube-api-access-gkqxl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 01:31:32.992094 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:31:32.992073    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aad0fca5-b959-47f7-8a46-a8da24458e5b-kube-api-access-kkw5k" (OuterVolumeSpecName: "kube-api-access-kkw5k") pod "aad0fca5-b959-47f7-8a46-a8da24458e5b" (UID: "aad0fca5-b959-47f7-8a46-a8da24458e5b"). InnerVolumeSpecName "kube-api-access-kkw5k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 01:31:33.091591 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:31:33.091567    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gkqxl\" (UniqueName: \"kubernetes.io/projected/84658a03-47b0-4f32-8385-4d36f59cdd8b-kube-api-access-gkqxl\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:31:33.091591 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:31:33.091592    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kkw5k\" (UniqueName: \"kubernetes.io/projected/aad0fca5-b959-47f7-8a46-a8da24458e5b-kube-api-access-kkw5k\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:31:33.123354 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:31:33.123327    2569 generic.go:356] "Generic (PLEG): container finished" podID="84658a03-47b0-4f32-8385-4d36f59cdd8b" containerID="89b101a5339ff779368dc4d8ed3ccf1dee06a55070d3cc0014535b90e6cbe93f" exitCode=0
Sep 11 01:31:33.123497 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:31:33.123371    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-9433b9-6746686dd5-5b2nd"
Sep 11 01:31:33.123497 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:31:33.123413    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-9433b9-6746686dd5-5b2nd" event={"ID":"84658a03-47b0-4f32-8385-4d36f59cdd8b","Type":"ContainerDied","Data":"89b101a5339ff779368dc4d8ed3ccf1dee06a55070d3cc0014535b90e6cbe93f"}
Sep 11 01:31:33.123497 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:31:33.123472    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-9433b9-6746686dd5-5b2nd" event={"ID":"84658a03-47b0-4f32-8385-4d36f59cdd8b","Type":"ContainerDied","Data":"fb4061f87fda557eb0bec2b26dc444fe7ba2e3607b2a481e9e2f8f227d14211d"}
Sep 11 01:31:33.123672 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:31:33.123501    2569 scope.go:117] "RemoveContainer" containerID="89b101a5339ff779368dc4d8ed3ccf1dee06a55070d3cc0014535b90e6cbe93f"
Sep 11 01:31:33.124910 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:31:33.124866    2569 generic.go:356] "Generic (PLEG): container finished" podID="aad0fca5-b959-47f7-8a46-a8da24458e5b" containerID="b8cd8c93b2bc99229b9609d91754d265caa91371990e60d1248e73e4ad2faf97" exitCode=0
Sep 11 01:31:33.125020 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:31:33.124937    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-9433b9-6b64d44946-bt9h7"
Sep 11 01:31:33.125020 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:31:33.124964    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-9433b9-6b64d44946-bt9h7" event={"ID":"aad0fca5-b959-47f7-8a46-a8da24458e5b","Type":"ContainerDied","Data":"b8cd8c93b2bc99229b9609d91754d265caa91371990e60d1248e73e4ad2faf97"}
Sep 11 01:31:33.125128 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:31:33.125020    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-9433b9-6b64d44946-bt9h7" event={"ID":"aad0fca5-b959-47f7-8a46-a8da24458e5b","Type":"ContainerDied","Data":"43f412cef83b5675855da539f1a3955caba11ed3c5a50853004a9c210a84f5f2"}
Sep 11 01:31:33.133183 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:31:33.133162    2569 scope.go:117] "RemoveContainer" containerID="89b101a5339ff779368dc4d8ed3ccf1dee06a55070d3cc0014535b90e6cbe93f"
Sep 11 01:31:33.133495 ip-10-0-129-222 kubenswrapper[2569]: E0911 01:31:33.133469    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89b101a5339ff779368dc4d8ed3ccf1dee06a55070d3cc0014535b90e6cbe93f\": container with ID starting with 89b101a5339ff779368dc4d8ed3ccf1dee06a55070d3cc0014535b90e6cbe93f not found: ID does not exist" containerID="89b101a5339ff779368dc4d8ed3ccf1dee06a55070d3cc0014535b90e6cbe93f"
Sep 11 01:31:33.133553 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:31:33.133508    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89b101a5339ff779368dc4d8ed3ccf1dee06a55070d3cc0014535b90e6cbe93f"} err="failed to get container status \"89b101a5339ff779368dc4d8ed3ccf1dee06a55070d3cc0014535b90e6cbe93f\": rpc error: code = NotFound desc = could not find container \"89b101a5339ff779368dc4d8ed3ccf1dee06a55070d3cc0014535b90e6cbe93f\": container with ID starting with 89b101a5339ff779368dc4d8ed3ccf1dee06a55070d3cc0014535b90e6cbe93f not found: ID does not exist"
Sep 11 01:31:33.133553 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:31:33.133530    2569 scope.go:117] "RemoveContainer" containerID="b8cd8c93b2bc99229b9609d91754d265caa91371990e60d1248e73e4ad2faf97"
Sep 11 01:31:33.141564 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:31:33.141546    2569 scope.go:117] "RemoveContainer" containerID="b8cd8c93b2bc99229b9609d91754d265caa91371990e60d1248e73e4ad2faf97"
Sep 11 01:31:33.141829 ip-10-0-129-222 kubenswrapper[2569]: E0911 01:31:33.141809    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8cd8c93b2bc99229b9609d91754d265caa91371990e60d1248e73e4ad2faf97\": container with ID starting with b8cd8c93b2bc99229b9609d91754d265caa91371990e60d1248e73e4ad2faf97 not found: ID does not exist" containerID="b8cd8c93b2bc99229b9609d91754d265caa91371990e60d1248e73e4ad2faf97"
Sep 11 01:31:33.141901 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:31:33.141836    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8cd8c93b2bc99229b9609d91754d265caa91371990e60d1248e73e4ad2faf97"} err="failed to get container status \"b8cd8c93b2bc99229b9609d91754d265caa91371990e60d1248e73e4ad2faf97\": rpc error: code = NotFound desc = could not find container \"b8cd8c93b2bc99229b9609d91754d265caa91371990e60d1248e73e4ad2faf97\": container with ID starting with b8cd8c93b2bc99229b9609d91754d265caa91371990e60d1248e73e4ad2faf97 not found: ID does not exist"
Sep 11 01:31:33.145229 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:31:33.145208    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-9433b9-6746686dd5-5b2nd"]
Sep 11 01:31:33.149165 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:31:33.149145    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-9433b9-6746686dd5-5b2nd"]
Sep 11 01:31:33.157848 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:31:33.157827    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-9433b9-6b64d44946-bt9h7"]
Sep 11 01:31:33.161480 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:31:33.161456    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-9433b9-6b64d44946-bt9h7"]
Sep 11 01:31:33.870544 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:31:33.870468    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="68d224f4-99c9-4553-b21d-07b92209a86a" path="/var/lib/kubelet/pods/68d224f4-99c9-4553-b21d-07b92209a86a/volumes"
Sep 11 01:31:33.870905 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:31:33.870771    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6f509aae-0bd1-4431-8f38-47b5b4340d00" path="/var/lib/kubelet/pods/6f509aae-0bd1-4431-8f38-47b5b4340d00/volumes"
Sep 11 01:31:33.871107 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:31:33.871092    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="84658a03-47b0-4f32-8385-4d36f59cdd8b" path="/var/lib/kubelet/pods/84658a03-47b0-4f32-8385-4d36f59cdd8b/volumes"
Sep 11 01:31:33.871438 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:31:33.871427    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aad0fca5-b959-47f7-8a46-a8da24458e5b" path="/var/lib/kubelet/pods/aad0fca5-b959-47f7-8a46-a8da24458e5b/volumes"
Sep 11 01:31:35.003811 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:31:35.003777    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kmx5x"]
Sep 11 01:31:44.103393 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:31:44.103361    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kmx5x"]
Sep 11 01:33:05.203854 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:33:05.203818    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kmx5x"]
Sep 11 01:33:31.301375 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:33:31.301341    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kmx5x"]
Sep 11 01:33:38.889419 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:33:38.889319    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kmx5x"]
Sep 11 01:33:46.091640 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:33:46.091607    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kmx5x"]
Sep 11 01:33:53.696052 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:33:53.696013    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kmx5x"]
Sep 11 01:34:07.603779 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:34:07.603747    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kmx5x"]
Sep 11 01:34:15.198934 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:34:15.198903    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kmx5x"]
Sep 11 01:35:42.894153 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:35:42.894118    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kmx5x"]
Sep 11 01:35:46.455835 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:35:46.455802    2569 scope.go:117] "RemoveContainer" containerID="bfdd67a1253629a90e5fc103f5e53c23a1eb5f46cfd402cdef1604365800a8ca"
Sep 11 01:35:46.464253 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:35:46.464232    2569 scope.go:117] "RemoveContainer" containerID="127a8f1ee29162213a9a9b85ed5b246e2d730e3e704023abb538615c58914996"
Sep 11 01:35:49.603509 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:35:49.603473    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kmx5x"]
Sep 11 01:35:58.696021 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:35:58.695987    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kmx5x"]
Sep 11 01:36:08.697881 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:36:08.697833    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-1626d3e3-kserve-cb8fdb767-qvfnl"]
Sep 11 01:36:08.698357 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:36:08.698244    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a92f15b-fd83-4f83-be73-8e5249d0686a" containerName="istio-proxy"
Sep 11 01:36:08.698357 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:36:08.698256    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a92f15b-fd83-4f83-be73-8e5249d0686a" containerName="istio-proxy"
Sep 11 01:36:08.698357 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:36:08.698266    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="84658a03-47b0-4f32-8385-4d36f59cdd8b" containerName="payload-processing"
Sep 11 01:36:08.698357 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:36:08.698276    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="84658a03-47b0-4f32-8385-4d36f59cdd8b" containerName="payload-processing"
Sep 11 01:36:08.698357 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:36:08.698293    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a92f15b-fd83-4f83-be73-8e5249d0686a" containerName="istio-proxy"
Sep 11 01:36:08.698357 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:36:08.698298    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a92f15b-fd83-4f83-be73-8e5249d0686a" containerName="istio-proxy"
Sep 11 01:36:08.698357 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:36:08.698308    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="68d224f4-99c9-4553-b21d-07b92209a86a" containerName="revoke-keys"
Sep 11 01:36:08.698357 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:36:08.698313    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="68d224f4-99c9-4553-b21d-07b92209a86a" containerName="revoke-keys"
Sep 11 01:36:08.698357 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:36:08.698322    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aad0fca5-b959-47f7-8a46-a8da24458e5b" containerName="payload-pre-processing"
Sep 11 01:36:08.698357 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:36:08.698327    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="aad0fca5-b959-47f7-8a46-a8da24458e5b" containerName="payload-pre-processing"
Sep 11 01:36:08.698734 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:36:08.698395    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="aad0fca5-b959-47f7-8a46-a8da24458e5b" containerName="payload-pre-processing"
Sep 11 01:36:08.698734 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:36:08.698407    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="7a92f15b-fd83-4f83-be73-8e5249d0686a" containerName="istio-proxy"
Sep 11 01:36:08.698734 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:36:08.698414    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="7a92f15b-fd83-4f83-be73-8e5249d0686a" containerName="istio-proxy"
Sep 11 01:36:08.698734 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:36:08.698424    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="68d224f4-99c9-4553-b21d-07b92209a86a" containerName="revoke-keys"
Sep 11 01:36:08.698734 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:36:08.698435    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="84658a03-47b0-4f32-8385-4d36f59cdd8b" containerName="payload-processing"
Sep 11 01:36:08.701621 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:36:08.701604    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-1626d3e3-kserve-cb8fdb767-qvfnl"
Sep 11 01:36:08.703738 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:36:08.703721    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-1626d3e3-kserve-self-signed-certs\""
Sep 11 01:36:08.711124 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:36:08.711096    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-1626d3e3-kserve-cb8fdb767-qvfnl"]
Sep 11 01:36:08.770673 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:36:08.770649    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8c20a2bc-a3e0-4ce9-b753-39b20c067a8b-dshm\") pod \"e2e-same-id-a-1626d3e3-kserve-cb8fdb767-qvfnl\" (UID: \"8c20a2bc-a3e0-4ce9-b753-39b20c067a8b\") " pod="llm/e2e-same-id-a-1626d3e3-kserve-cb8fdb767-qvfnl"
Sep 11 01:36:08.770815 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:36:08.770683    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8c20a2bc-a3e0-4ce9-b753-39b20c067a8b-model-cache\") pod \"e2e-same-id-a-1626d3e3-kserve-cb8fdb767-qvfnl\" (UID: \"8c20a2bc-a3e0-4ce9-b753-39b20c067a8b\") " pod="llm/e2e-same-id-a-1626d3e3-kserve-cb8fdb767-qvfnl"
Sep 11 01:36:08.770815 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:36:08.770708    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8c20a2bc-a3e0-4ce9-b753-39b20c067a8b-home\") pod \"e2e-same-id-a-1626d3e3-kserve-cb8fdb767-qvfnl\" (UID: \"8c20a2bc-a3e0-4ce9-b753-39b20c067a8b\") " pod="llm/e2e-same-id-a-1626d3e3-kserve-cb8fdb767-qvfnl"
Sep 11 01:36:08.770815 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:36:08.770751    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c20a2bc-a3e0-4ce9-b753-39b20c067a8b-tls-certs\") pod \"e2e-same-id-a-1626d3e3-kserve-cb8fdb767-qvfnl\" (UID: \"8c20a2bc-a3e0-4ce9-b753-39b20c067a8b\") " pod="llm/e2e-same-id-a-1626d3e3-kserve-cb8fdb767-qvfnl"
Sep 11 01:36:08.770815 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:36:08.770795    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c20a2bc-a3e0-4ce9-b753-39b20c067a8b-tmp-dir\") pod \"e2e-same-id-a-1626d3e3-kserve-cb8fdb767-qvfnl\" (UID: \"8c20a2bc-a3e0-4ce9-b753-39b20c067a8b\") " pod="llm/e2e-same-id-a-1626d3e3-kserve-cb8fdb767-qvfnl"
Sep 11 01:36:08.770977 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:36:08.770834    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfkqf\" (UniqueName: \"kubernetes.io/projected/8c20a2bc-a3e0-4ce9-b753-39b20c067a8b-kube-api-access-sfkqf\") pod \"e2e-same-id-a-1626d3e3-kserve-cb8fdb767-qvfnl\" (UID: \"8c20a2bc-a3e0-4ce9-b753-39b20c067a8b\") " pod="llm/e2e-same-id-a-1626d3e3-kserve-cb8fdb767-qvfnl"
Sep 11 01:36:08.871373 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:36:08.871347    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfkqf\" (UniqueName: \"kubernetes.io/projected/8c20a2bc-a3e0-4ce9-b753-39b20c067a8b-kube-api-access-sfkqf\") pod \"e2e-same-id-a-1626d3e3-kserve-cb8fdb767-qvfnl\" (UID: \"8c20a2bc-a3e0-4ce9-b753-39b20c067a8b\") " pod="llm/e2e-same-id-a-1626d3e3-kserve-cb8fdb767-qvfnl"
Sep 11 01:36:08.871527 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:36:08.871420    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8c20a2bc-a3e0-4ce9-b753-39b20c067a8b-dshm\") pod \"e2e-same-id-a-1626d3e3-kserve-cb8fdb767-qvfnl\" (UID: \"8c20a2bc-a3e0-4ce9-b753-39b20c067a8b\") " pod="llm/e2e-same-id-a-1626d3e3-kserve-cb8fdb767-qvfnl"
Sep 11 01:36:08.871527 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:36:08.871444    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8c20a2bc-a3e0-4ce9-b753-39b20c067a8b-model-cache\") pod \"e2e-same-id-a-1626d3e3-kserve-cb8fdb767-qvfnl\" (UID: \"8c20a2bc-a3e0-4ce9-b753-39b20c067a8b\") " pod="llm/e2e-same-id-a-1626d3e3-kserve-cb8fdb767-qvfnl"
Sep 11 01:36:08.871527 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:36:08.871462    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8c20a2bc-a3e0-4ce9-b753-39b20c067a8b-home\") pod \"e2e-same-id-a-1626d3e3-kserve-cb8fdb767-qvfnl\" (UID: \"8c20a2bc-a3e0-4ce9-b753-39b20c067a8b\") " pod="llm/e2e-same-id-a-1626d3e3-kserve-cb8fdb767-qvfnl"
Sep 11 01:36:08.871527 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:36:08.871482    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c20a2bc-a3e0-4ce9-b753-39b20c067a8b-tls-certs\") pod \"e2e-same-id-a-1626d3e3-kserve-cb8fdb767-qvfnl\" (UID: \"8c20a2bc-a3e0-4ce9-b753-39b20c067a8b\") " pod="llm/e2e-same-id-a-1626d3e3-kserve-cb8fdb767-qvfnl"
Sep 11 01:36:08.871527 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:36:08.871508    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c20a2bc-a3e0-4ce9-b753-39b20c067a8b-tmp-dir\") pod \"e2e-same-id-a-1626d3e3-kserve-cb8fdb767-qvfnl\" (UID: \"8c20a2bc-a3e0-4ce9-b753-39b20c067a8b\") " pod="llm/e2e-same-id-a-1626d3e3-kserve-cb8fdb767-qvfnl"
Sep 11 01:36:08.871993 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:36:08.871970    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8c20a2bc-a3e0-4ce9-b753-39b20c067a8b-model-cache\") pod \"e2e-same-id-a-1626d3e3-kserve-cb8fdb767-qvfnl\" (UID: \"8c20a2bc-a3e0-4ce9-b753-39b20c067a8b\") " pod="llm/e2e-same-id-a-1626d3e3-kserve-cb8fdb767-qvfnl"
Sep 11 01:36:08.872106 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:36:08.872021    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8c20a2bc-a3e0-4ce9-b753-39b20c067a8b-home\") pod \"e2e-same-id-a-1626d3e3-kserve-cb8fdb767-qvfnl\" (UID: \"8c20a2bc-a3e0-4ce9-b753-39b20c067a8b\") " pod="llm/e2e-same-id-a-1626d3e3-kserve-cb8fdb767-qvfnl"
Sep 11 01:36:08.872106 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:36:08.872066    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c20a2bc-a3e0-4ce9-b753-39b20c067a8b-tmp-dir\") pod \"e2e-same-id-a-1626d3e3-kserve-cb8fdb767-qvfnl\" (UID: \"8c20a2bc-a3e0-4ce9-b753-39b20c067a8b\") " pod="llm/e2e-same-id-a-1626d3e3-kserve-cb8fdb767-qvfnl"
Sep 11 01:36:08.874511 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:36:08.874479    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8c20a2bc-a3e0-4ce9-b753-39b20c067a8b-dshm\") pod \"e2e-same-id-a-1626d3e3-kserve-cb8fdb767-qvfnl\" (UID: \"8c20a2bc-a3e0-4ce9-b753-39b20c067a8b\") " pod="llm/e2e-same-id-a-1626d3e3-kserve-cb8fdb767-qvfnl"
Sep 11 01:36:08.874710 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:36:08.874692    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c20a2bc-a3e0-4ce9-b753-39b20c067a8b-tls-certs\") pod \"e2e-same-id-a-1626d3e3-kserve-cb8fdb767-qvfnl\" (UID: \"8c20a2bc-a3e0-4ce9-b753-39b20c067a8b\") " pod="llm/e2e-same-id-a-1626d3e3-kserve-cb8fdb767-qvfnl"
Sep 11 01:36:08.879066 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:36:08.879044    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfkqf\" (UniqueName: \"kubernetes.io/projected/8c20a2bc-a3e0-4ce9-b753-39b20c067a8b-kube-api-access-sfkqf\") pod \"e2e-same-id-a-1626d3e3-kserve-cb8fdb767-qvfnl\" (UID: \"8c20a2bc-a3e0-4ce9-b753-39b20c067a8b\") " pod="llm/e2e-same-id-a-1626d3e3-kserve-cb8fdb767-qvfnl"
Sep 11 01:36:09.011063 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:36:09.010991    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-1626d3e3-kserve-cb8fdb767-qvfnl"
Sep 11 01:36:09.278073 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:36:09.278048    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-1626d3e3-kserve-cb8fdb767-qvfnl"]
Sep 11 01:36:09.280199 ip-10-0-129-222 kubenswrapper[2569]: W0911 01:36:09.280171    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8c20a2bc_a3e0_4ce9_b753_39b20c067a8b.slice/crio-eb0d8a37a3294aa724de7e441885592417cdfe89aa00ef9c24e8326102f97e63 WatchSource:0}: Error finding container eb0d8a37a3294aa724de7e441885592417cdfe89aa00ef9c24e8326102f97e63: Status 404 returned error can't find the container with id eb0d8a37a3294aa724de7e441885592417cdfe89aa00ef9c24e8326102f97e63
Sep 11 01:36:09.538180 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:36:09.538103    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-1626d3e3-kserve-cb8fdb767-qvfnl" event={"ID":"8c20a2bc-a3e0-4ce9-b753-39b20c067a8b","Type":"ContainerStarted","Data":"6ea173b859be44735db63853cab42a74c39472ae07ce21352d9bc2a78155f2f2"}
Sep 11 01:36:09.538180 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:36:09.538143    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-1626d3e3-kserve-cb8fdb767-qvfnl" event={"ID":"8c20a2bc-a3e0-4ce9-b753-39b20c067a8b","Type":"ContainerStarted","Data":"eb0d8a37a3294aa724de7e441885592417cdfe89aa00ef9c24e8326102f97e63"}
Sep 11 01:36:09.556856 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:36:09.556806    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-1626d3e3-kserve-cb8fdb767-qvfnl" podStartSLOduration=1.5567883660000001 podStartE2EDuration="1.556788366s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 01:36:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 01:36:09.5544932 +0000 UTC m=+2304.205867448" watchObservedRunningTime="2026-09-11 01:36:09.556788366 +0000 UTC m=+2304.208162657"
Sep 11 01:36:11.400255 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:36:11.400220    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kmx5x"]
Sep 11 01:36:19.011544 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:36:19.011506    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-1626d3e3-kserve-cb8fdb767-qvfnl"
Sep 11 01:36:19.011544 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:36:19.011552    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-1626d3e3-kserve-cb8fdb767-qvfnl"
Sep 11 01:36:19.024135 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:36:19.024112    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-1626d3e3-kserve-cb8fdb767-qvfnl"
Sep 11 01:36:19.603389 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:36:19.603353    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-1626d3e3-kserve-cb8fdb767-qvfnl"
Sep 11 01:36:19.998858 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:36:19.998776    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kmx5x"]
Sep 11 01:36:42.411819 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:36:42.411739    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-1626d3e3-kserve-cb8fdb767-qvfnl"]
Sep 11 01:36:45.499722 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:36:45.499690    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kmx5x"]
Sep 11 01:36:46.488324 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:36:46.488292    2569 scope.go:117] "RemoveContainer" containerID="cb450458b14970c49e5e1e9f56e536398441861f29f172d32b69bb0a68553e30"
Sep 11 01:36:46.496439 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:36:46.496420    2569 scope.go:117] "RemoveContainer" containerID="e51eb99b1fd324483eb2975cb72f54935dd5cda49a192b76ece21885c671e8f7"
Sep 11 01:36:52.498842 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:36:52.498801    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kmx5x"]
Sep 11 01:36:57.432269 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:36:57.432226    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-1626d3e3-kserve-cb8fdb767-qvfnl" podUID="8c20a2bc-a3e0-4ce9-b753-39b20c067a8b" containerName="main" containerID="cri-o://6ea173b859be44735db63853cab42a74c39472ae07ce21352d9bc2a78155f2f2" gracePeriod=45
Sep 11 01:36:57.678278 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:36:57.678248    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-1626d3e3-kserve-cb8fdb767-qvfnl"
Sep 11 01:36:57.791054 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:36:57.790979    2569 generic.go:356] "Generic (PLEG): container finished" podID="8c20a2bc-a3e0-4ce9-b753-39b20c067a8b" containerID="6ea173b859be44735db63853cab42a74c39472ae07ce21352d9bc2a78155f2f2" exitCode=0
Sep 11 01:36:57.791054 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:36:57.791021    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-1626d3e3-kserve-cb8fdb767-qvfnl" event={"ID":"8c20a2bc-a3e0-4ce9-b753-39b20c067a8b","Type":"ContainerDied","Data":"6ea173b859be44735db63853cab42a74c39472ae07ce21352d9bc2a78155f2f2"}
Sep 11 01:36:57.791054 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:36:57.791022    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-1626d3e3-kserve-cb8fdb767-qvfnl"
Sep 11 01:36:57.791306 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:36:57.791057    2569 scope.go:117] "RemoveContainer" containerID="6ea173b859be44735db63853cab42a74c39472ae07ce21352d9bc2a78155f2f2"
Sep 11 01:36:57.791306 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:36:57.791048    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-1626d3e3-kserve-cb8fdb767-qvfnl" event={"ID":"8c20a2bc-a3e0-4ce9-b753-39b20c067a8b","Type":"ContainerDied","Data":"eb0d8a37a3294aa724de7e441885592417cdfe89aa00ef9c24e8326102f97e63"}
Sep 11 01:36:57.799665 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:36:57.799648    2569 scope.go:117] "RemoveContainer" containerID="6ea173b859be44735db63853cab42a74c39472ae07ce21352d9bc2a78155f2f2"
Sep 11 01:36:57.799975 ip-10-0-129-222 kubenswrapper[2569]: E0911 01:36:57.799947    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ea173b859be44735db63853cab42a74c39472ae07ce21352d9bc2a78155f2f2\": container with ID starting with 6ea173b859be44735db63853cab42a74c39472ae07ce21352d9bc2a78155f2f2 not found: ID does not exist" containerID="6ea173b859be44735db63853cab42a74c39472ae07ce21352d9bc2a78155f2f2"
Sep 11 01:36:57.800074 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:36:57.799983    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ea173b859be44735db63853cab42a74c39472ae07ce21352d9bc2a78155f2f2"} err="failed to get container status \"6ea173b859be44735db63853cab42a74c39472ae07ce21352d9bc2a78155f2f2\": rpc error: code = NotFound desc = could not find container \"6ea173b859be44735db63853cab42a74c39472ae07ce21352d9bc2a78155f2f2\": container with ID starting with 6ea173b859be44735db63853cab42a74c39472ae07ce21352d9bc2a78155f2f2 not found: ID does not exist"
Sep 11 01:36:57.807980 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:36:57.807959    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sfkqf\" (UniqueName: \"kubernetes.io/projected/8c20a2bc-a3e0-4ce9-b753-39b20c067a8b-kube-api-access-sfkqf\") pod \"8c20a2bc-a3e0-4ce9-b753-39b20c067a8b\" (UID: \"8c20a2bc-a3e0-4ce9-b753-39b20c067a8b\") "
Sep 11 01:36:57.808054 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:36:57.808017    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8c20a2bc-a3e0-4ce9-b753-39b20c067a8b-dshm\") pod \"8c20a2bc-a3e0-4ce9-b753-39b20c067a8b\" (UID: \"8c20a2bc-a3e0-4ce9-b753-39b20c067a8b\") "
Sep 11 01:36:57.808054 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:36:57.808045    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c20a2bc-a3e0-4ce9-b753-39b20c067a8b-tls-certs\") pod \"8c20a2bc-a3e0-4ce9-b753-39b20c067a8b\" (UID: \"8c20a2bc-a3e0-4ce9-b753-39b20c067a8b\") "
Sep 11 01:36:57.808136 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:36:57.808070    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8c20a2bc-a3e0-4ce9-b753-39b20c067a8b-model-cache\") pod \"8c20a2bc-a3e0-4ce9-b753-39b20c067a8b\" (UID: \"8c20a2bc-a3e0-4ce9-b753-39b20c067a8b\") "
Sep 11 01:36:57.808136 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:36:57.808114    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8c20a2bc-a3e0-4ce9-b753-39b20c067a8b-home\") pod \"8c20a2bc-a3e0-4ce9-b753-39b20c067a8b\" (UID: \"8c20a2bc-a3e0-4ce9-b753-39b20c067a8b\") "
Sep 11 01:36:57.808243 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:36:57.808164    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c20a2bc-a3e0-4ce9-b753-39b20c067a8b-tmp-dir\") pod \"8c20a2bc-a3e0-4ce9-b753-39b20c067a8b\" (UID: \"8c20a2bc-a3e0-4ce9-b753-39b20c067a8b\") "
Sep 11 01:36:57.808450 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:36:57.808351    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c20a2bc-a3e0-4ce9-b753-39b20c067a8b-model-cache" (OuterVolumeSpecName: "model-cache") pod "8c20a2bc-a3e0-4ce9-b753-39b20c067a8b" (UID: "8c20a2bc-a3e0-4ce9-b753-39b20c067a8b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 01:36:57.808450 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:36:57.808393    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c20a2bc-a3e0-4ce9-b753-39b20c067a8b-home" (OuterVolumeSpecName: "home") pod "8c20a2bc-a3e0-4ce9-b753-39b20c067a8b" (UID: "8c20a2bc-a3e0-4ce9-b753-39b20c067a8b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 01:36:57.808608 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:36:57.808470    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c20a2bc-a3e0-4ce9-b753-39b20c067a8b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8c20a2bc-a3e0-4ce9-b753-39b20c067a8b" (UID: "8c20a2bc-a3e0-4ce9-b753-39b20c067a8b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 01:36:57.808608 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:36:57.808550    2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8c20a2bc-a3e0-4ce9-b753-39b20c067a8b-model-cache\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:36:57.808608 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:36:57.808568    2569 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8c20a2bc-a3e0-4ce9-b753-39b20c067a8b-home\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:36:57.808608 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:36:57.808581    2569 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c20a2bc-a3e0-4ce9-b753-39b20c067a8b-tmp-dir\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:36:57.810217 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:36:57.810200    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c20a2bc-a3e0-4ce9-b753-39b20c067a8b-kube-api-access-sfkqf" (OuterVolumeSpecName: "kube-api-access-sfkqf") pod "8c20a2bc-a3e0-4ce9-b753-39b20c067a8b" (UID: "8c20a2bc-a3e0-4ce9-b753-39b20c067a8b"). InnerVolumeSpecName "kube-api-access-sfkqf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 01:36:57.810605 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:36:57.810584    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c20a2bc-a3e0-4ce9-b753-39b20c067a8b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8c20a2bc-a3e0-4ce9-b753-39b20c067a8b" (UID: "8c20a2bc-a3e0-4ce9-b753-39b20c067a8b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 01:36:57.810674 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:36:57.810598    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c20a2bc-a3e0-4ce9-b753-39b20c067a8b-dshm" (OuterVolumeSpecName: "dshm") pod "8c20a2bc-a3e0-4ce9-b753-39b20c067a8b" (UID: "8c20a2bc-a3e0-4ce9-b753-39b20c067a8b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 01:36:57.909043 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:36:57.909014    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sfkqf\" (UniqueName: \"kubernetes.io/projected/8c20a2bc-a3e0-4ce9-b753-39b20c067a8b-kube-api-access-sfkqf\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:36:57.909043 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:36:57.909042    2569 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8c20a2bc-a3e0-4ce9-b753-39b20c067a8b-dshm\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:36:57.909200 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:36:57.909056    2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c20a2bc-a3e0-4ce9-b753-39b20c067a8b-tls-certs\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Sep 11 01:36:58.107656 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:36:58.107625    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-1626d3e3-kserve-cb8fdb767-qvfnl"]
Sep 11 01:36:58.109325 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:36:58.109304    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-1626d3e3-kserve-cb8fdb767-qvfnl"]
Sep 11 01:36:59.812611 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:36:59.812578    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kmx5x"]
Sep 11 01:36:59.870161 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:36:59.870128    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8c20a2bc-a3e0-4ce9-b753-39b20c067a8b" path="/var/lib/kubelet/pods/8c20a2bc-a3e0-4ce9-b753-39b20c067a8b/volumes"
Sep 11 01:37:07.788615 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:37:07.788573    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kmx5x"]
Sep 11 01:37:14.590319 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:37:14.590282    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kmx5x"]
Sep 11 01:37:20.699117 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:37:20.699079    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kmx5x"]
Sep 11 01:37:27.791420 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:37:27.791384    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kmx5x"]
Sep 11 01:37:36.011308 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:37:36.011277    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7c49b45946-25wxw_823b7670-1630-48fc-9963-fb96437d04b0/manager/0.log"
Sep 11 01:37:36.017725 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:37:36.017698    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7c49b45946-25wxw_823b7670-1630-48fc-9963-fb96437d04b0/copy-manifests/0.log"
Sep 11 01:37:38.067734 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:37:38.067702    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-wmkfk_61cd974e-1ee7-4b63-aa87-1d32dc4b2952/manager/0.log"
Sep 11 01:37:38.176781 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:37:38.176751    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-gqczs_73dee526-b179-4d7e-8ef0-375ef65b4cee/kuadrant-console-plugin/0.log"
Sep 11 01:37:38.414740 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:37:38.414667    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-kmx5x_257b2c19-d395-40eb-8e08-8ba9560fd070/limitador/0.log"
Sep 11 01:37:39.129233 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:37:39.129204    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5cc5ff74fb-8j6n9_b406e46a-719d-42d9-80b6-5706557ec9f8/kube-auth-proxy/0.log"
Sep 11 01:37:39.388579 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:37:39.388505    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-2t8jt_b8f2fea7-8e9c-45dd-9291-6e261d2e69db/istio-proxy/1.log"
Sep 11 01:37:40.352749 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:37:40.352716    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-bcc9ff8d8-ld7t4_a7f9b621-8d87-49d1-a254-c7c8a7238f8c/main/0.log"
Sep 11 01:37:40.687179 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:37:40.687091    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-pjbd2_5b0a0cbc-ae41-4060-8fc9-3b07ada1dd1f/main/0.log"
Sep 11 01:37:40.801281 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:37:40.801252    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-f8s9l_4b2a1717-67cc-41f4-b396-836fdf0e8036/main/0.log"
Sep 11 01:37:46.213948 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:37:46.213908    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-lvb79/must-gather-c7rfr"]
Sep 11 01:37:46.214538 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:37:46.214521    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c20a2bc-a3e0-4ce9-b753-39b20c067a8b" containerName="main"
Sep 11 01:37:46.214620 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:37:46.214544    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c20a2bc-a3e0-4ce9-b753-39b20c067a8b" containerName="main"
Sep 11 01:37:46.214680 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:37:46.214660    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="8c20a2bc-a3e0-4ce9-b753-39b20c067a8b" containerName="main"
Sep 11 01:37:46.218075 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:37:46.218052    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-lvb79/must-gather-c7rfr"
Sep 11 01:37:46.220249 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:37:46.220225    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-lvb79\"/\"default-dockercfg-s54hm\""
Sep 11 01:37:46.220388 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:37:46.220369    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-lvb79\"/\"openshift-service-ca.crt\""
Sep 11 01:37:46.221040 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:37:46.221024    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-lvb79\"/\"kube-root-ca.crt\""
Sep 11 01:37:46.226541 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:37:46.226519    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-lvb79/must-gather-c7rfr"]
Sep 11 01:37:46.346885 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:37:46.346848    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/7ef81e73-f9f4-456d-a760-c8d2c10ea819-must-gather-output\") pod \"must-gather-c7rfr\" (UID: \"7ef81e73-f9f4-456d-a760-c8d2c10ea819\") " pod="openshift-must-gather-lvb79/must-gather-c7rfr"
Sep 11 01:37:46.347043 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:37:46.346925    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8t56d\" (UniqueName: \"kubernetes.io/projected/7ef81e73-f9f4-456d-a760-c8d2c10ea819-kube-api-access-8t56d\") pod \"must-gather-c7rfr\" (UID: \"7ef81e73-f9f4-456d-a760-c8d2c10ea819\") " pod="openshift-must-gather-lvb79/must-gather-c7rfr"
Sep 11 01:37:46.447494 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:37:46.447467    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/7ef81e73-f9f4-456d-a760-c8d2c10ea819-must-gather-output\") pod \"must-gather-c7rfr\" (UID: \"7ef81e73-f9f4-456d-a760-c8d2c10ea819\") " pod="openshift-must-gather-lvb79/must-gather-c7rfr"
Sep 11 01:37:46.447629 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:37:46.447516    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8t56d\" (UniqueName: \"kubernetes.io/projected/7ef81e73-f9f4-456d-a760-c8d2c10ea819-kube-api-access-8t56d\") pod \"must-gather-c7rfr\" (UID: \"7ef81e73-f9f4-456d-a760-c8d2c10ea819\") " pod="openshift-must-gather-lvb79/must-gather-c7rfr"
Sep 11 01:37:46.447775 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:37:46.447758    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/7ef81e73-f9f4-456d-a760-c8d2c10ea819-must-gather-output\") pod \"must-gather-c7rfr\" (UID: \"7ef81e73-f9f4-456d-a760-c8d2c10ea819\") " pod="openshift-must-gather-lvb79/must-gather-c7rfr"
Sep 11 01:37:46.455919 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:37:46.455890    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8t56d\" (UniqueName: \"kubernetes.io/projected/7ef81e73-f9f4-456d-a760-c8d2c10ea819-kube-api-access-8t56d\") pod \"must-gather-c7rfr\" (UID: \"7ef81e73-f9f4-456d-a760-c8d2c10ea819\") " pod="openshift-must-gather-lvb79/must-gather-c7rfr"
Sep 11 01:37:46.521058 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:37:46.520989    2569 scope.go:117] "RemoveContainer" containerID="40d2eae1410bf211234285c8e3632a9b01671d25cb4ecc023c86e2e8f97f2dd5"
Sep 11 01:37:46.528059 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:37:46.528043    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-lvb79/must-gather-c7rfr"
Sep 11 01:37:46.529787 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:37:46.529768    2569 scope.go:117] "RemoveContainer" containerID="9c1ac101c65799f255ad234e0d2f7ca8c75cdd243100c21fe007f1925de4443c"
Sep 11 01:37:46.539186 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:37:46.539166    2569 scope.go:117] "RemoveContainer" containerID="a0ad6be02545621afb8832ce51f6e766c9860a129d4bf76a9601bc4ccf6df35d"
Sep 11 01:37:46.810721 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:37:46.810692    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-lvb79/must-gather-c7rfr"]
Sep 11 01:37:46.813373 ip-10-0-129-222 kubenswrapper[2569]: W0911 01:37:46.813342    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7ef81e73_f9f4_456d_a760_c8d2c10ea819.slice/crio-c062eb09b235e202ae5894ae719457b8bcf461444208679bb9a7c96984479293 WatchSource:0}: Error finding container c062eb09b235e202ae5894ae719457b8bcf461444208679bb9a7c96984479293: Status 404 returned error can't find the container with id c062eb09b235e202ae5894ae719457b8bcf461444208679bb9a7c96984479293
Sep 11 01:37:46.815497 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:37:46.815478    2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 01:37:47.038364 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:37:47.038326    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-lvb79/must-gather-c7rfr" event={"ID":"7ef81e73-f9f4-456d-a760-c8d2c10ea819","Type":"ContainerStarted","Data":"c062eb09b235e202ae5894ae719457b8bcf461444208679bb9a7c96984479293"}
Sep 11 01:37:48.048909 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:37:48.048783    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-lvb79/must-gather-c7rfr" event={"ID":"7ef81e73-f9f4-456d-a760-c8d2c10ea819","Type":"ContainerStarted","Data":"061a6c0743bc341130a40d642036f8f1f812385742633d435d63c13cb1d8573e"}
Sep 11 01:37:48.048909 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:37:48.048848    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-lvb79/must-gather-c7rfr" event={"ID":"7ef81e73-f9f4-456d-a760-c8d2c10ea819","Type":"ContainerStarted","Data":"f7edd6a470a101a78afa690b92b97a81150b80d192a8b6e052431f2928897db6"}
Sep 11 01:37:48.067843 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:37:48.067780    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-lvb79/must-gather-c7rfr" podStartSLOduration=1.1685820439999999 podStartE2EDuration="2.067756954s" totalImagesPullingTime="899.17491ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 01:37:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 01:37:48.064799607 +0000 UTC m=+2402.716173856" watchObservedRunningTime="2026-09-11 01:37:48.067756954 +0000 UTC m=+2402.719131204"
Sep 11 01:37:49.419109 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:37:49.419074    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-qgmsz_d5d763bb-6142-409f-8155-7143a6d91dc6/global-pull-secret-syncer/0.log"
Sep 11 01:37:49.535977 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:37:49.535949    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-st6p9_bbc61763-126c-46a6-8607-9f1071004770/konnectivity-agent/0.log"
Sep 11 01:37:49.560578 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:37:49.560550    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-129-222.ec2.internal_bef6176fe69d54950289897ea3150444/haproxy/0.log"
Sep 11 01:37:54.192023 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:37:54.191846    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-wmkfk_61cd974e-1ee7-4b63-aa87-1d32dc4b2952/manager/0.log"
Sep 11 01:37:54.224213 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:37:54.224184    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-gqczs_73dee526-b179-4d7e-8ef0-375ef65b4cee/kuadrant-console-plugin/0.log"
Sep 11 01:37:54.287941 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:37:54.287918    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-kmx5x_257b2c19-d395-40eb-8e08-8ba9560fd070/limitador/0.log"
Sep 11 01:37:55.878851 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:37:55.878816    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_575eb6a2-71fc-481b-a5da-84ebbdef443f/alertmanager/0.log"
Sep 11 01:37:55.902999 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:37:55.902971    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_575eb6a2-71fc-481b-a5da-84ebbdef443f/config-reloader/0.log"
Sep 11 01:37:55.925452 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:37:55.925423    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_575eb6a2-71fc-481b-a5da-84ebbdef443f/kube-rbac-proxy-web/0.log"
Sep 11 01:37:55.951480 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:37:55.950622    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_575eb6a2-71fc-481b-a5da-84ebbdef443f/kube-rbac-proxy/0.log"
Sep 11 01:37:55.975421 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:37:55.975397    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_575eb6a2-71fc-481b-a5da-84ebbdef443f/kube-rbac-proxy-metric/0.log"
Sep 11 01:37:55.998209 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:37:55.998183    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_575eb6a2-71fc-481b-a5da-84ebbdef443f/prom-label-proxy/0.log"
Sep 11 01:37:56.020947 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:37:56.020916    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_575eb6a2-71fc-481b-a5da-84ebbdef443f/init-config-reloader/0.log"
Sep 11 01:37:56.106178 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:37:56.106144    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-v9pqc_f7a5dad1-f163-4920-b8a4-3ab9e37de477/kube-state-metrics/0.log"
Sep 11 01:37:56.123262 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:37:56.123230    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-v9pqc_f7a5dad1-f163-4920-b8a4-3ab9e37de477/kube-rbac-proxy-main/0.log"
Sep 11 01:37:56.142333 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:37:56.142254    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-v9pqc_f7a5dad1-f163-4920-b8a4-3ab9e37de477/kube-rbac-proxy-self/0.log"
Sep 11 01:37:56.411926 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:37:56.411834    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-pqgn8_78562e6b-b985-4098-a024-2d2b21e73fe3/node-exporter/0.log"
Sep 11 01:37:56.440311 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:37:56.440284    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-pqgn8_78562e6b-b985-4098-a024-2d2b21e73fe3/kube-rbac-proxy/0.log"
Sep 11 01:37:56.469095 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:37:56.468926    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-pqgn8_78562e6b-b985-4098-a024-2d2b21e73fe3/init-textfile/0.log"
Sep 11 01:37:56.496225 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:37:56.496200    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-zl6np_c9757f20-e580-4d80-aa5f-2f148ce4690d/kube-rbac-proxy-main/0.log"
Sep 11 01:37:56.516662 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:37:56.516608    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-zl6np_c9757f20-e580-4d80-aa5f-2f148ce4690d/kube-rbac-proxy-self/0.log"
Sep 11 01:37:56.546394 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:37:56.546365    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-zl6np_c9757f20-e580-4d80-aa5f-2f148ce4690d/openshift-state-metrics/0.log"
Sep 11 01:37:56.597065 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:37:56.597035    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_06d240fd-ec52-46cd-a90d-40cd7ad8c7d9/prometheus/0.log"
Sep 11 01:37:56.618336 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:37:56.618221    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_06d240fd-ec52-46cd-a90d-40cd7ad8c7d9/config-reloader/0.log"
Sep 11 01:37:56.639914 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:37:56.639858    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_06d240fd-ec52-46cd-a90d-40cd7ad8c7d9/thanos-sidecar/0.log"
Sep 11 01:37:56.667230 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:37:56.662231    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_06d240fd-ec52-46cd-a90d-40cd7ad8c7d9/kube-rbac-proxy-web/0.log"
Sep 11 01:37:56.690036 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:37:56.690011    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_06d240fd-ec52-46cd-a90d-40cd7ad8c7d9/kube-rbac-proxy/0.log"
Sep 11 01:37:56.710561 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:37:56.710532    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_06d240fd-ec52-46cd-a90d-40cd7ad8c7d9/kube-rbac-proxy-thanos/0.log"
Sep 11 01:37:56.732244 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:37:56.732214    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_06d240fd-ec52-46cd-a90d-40cd7ad8c7d9/init-config-reloader/0.log"
Sep 11 01:37:56.817293 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:37:56.817264    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-5864cd99d7-ctjfd_735598ab-461b-4145-824b-b197ee49967e/prometheus-operator-admission-webhook/0.log"
Sep 11 01:37:56.846632 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:37:56.846530    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6cd4587857-vvhjq_9c2b3f67-8ddc-4cce-994c-364bf13799fa/telemeter-client/0.log"
Sep 11 01:37:56.871110 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:37:56.871036    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6cd4587857-vvhjq_9c2b3f67-8ddc-4cce-994c-364bf13799fa/reload/0.log"
Sep 11 01:37:56.891892 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:37:56.891793    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6cd4587857-vvhjq_9c2b3f67-8ddc-4cce-994c-364bf13799fa/kube-rbac-proxy/0.log"
Sep 11 01:37:59.610837 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:37:59.610796    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-lvb79/perf-node-gather-daemonset-twgc7"]
Sep 11 01:37:59.618670 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:37:59.618644    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-lvb79/perf-node-gather-daemonset-twgc7"
Sep 11 01:37:59.624438 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:37:59.624405    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-lvb79/perf-node-gather-daemonset-twgc7"]
Sep 11 01:37:59.636148 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:37:59.636067    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-76f99584f4-j6gxg_722ada5b-0930-4b9b-8752-49be36c404b4/download-server/0.log"
Sep 11 01:37:59.695733 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:37:59.695562    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6m94\" (UniqueName: \"kubernetes.io/projected/9d5eaa0e-57a2-44f8-a521-af3d3ad7d490-kube-api-access-k6m94\") pod \"perf-node-gather-daemonset-twgc7\" (UID: \"9d5eaa0e-57a2-44f8-a521-af3d3ad7d490\") " pod="openshift-must-gather-lvb79/perf-node-gather-daemonset-twgc7"
Sep 11 01:37:59.695733 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:37:59.695629    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/9d5eaa0e-57a2-44f8-a521-af3d3ad7d490-proc\") pod \"perf-node-gather-daemonset-twgc7\" (UID: \"9d5eaa0e-57a2-44f8-a521-af3d3ad7d490\") " pod="openshift-must-gather-lvb79/perf-node-gather-daemonset-twgc7"
Sep 11 01:37:59.695733 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:37:59.695685    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9d5eaa0e-57a2-44f8-a521-af3d3ad7d490-sys\") pod \"perf-node-gather-daemonset-twgc7\" (UID: \"9d5eaa0e-57a2-44f8-a521-af3d3ad7d490\") " pod="openshift-must-gather-lvb79/perf-node-gather-daemonset-twgc7"
Sep 11 01:37:59.695733 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:37:59.695740    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/9d5eaa0e-57a2-44f8-a521-af3d3ad7d490-podres\") pod \"perf-node-gather-daemonset-twgc7\" (UID: \"9d5eaa0e-57a2-44f8-a521-af3d3ad7d490\") " pod="openshift-must-gather-lvb79/perf-node-gather-daemonset-twgc7"
Sep 11 01:37:59.696112 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:37:59.695788    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9d5eaa0e-57a2-44f8-a521-af3d3ad7d490-lib-modules\") pod \"perf-node-gather-daemonset-twgc7\" (UID: \"9d5eaa0e-57a2-44f8-a521-af3d3ad7d490\") " pod="openshift-must-gather-lvb79/perf-node-gather-daemonset-twgc7"
Sep 11 01:37:59.797089 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:37:59.797052    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6m94\" (UniqueName: \"kubernetes.io/projected/9d5eaa0e-57a2-44f8-a521-af3d3ad7d490-kube-api-access-k6m94\") pod \"perf-node-gather-daemonset-twgc7\" (UID: \"9d5eaa0e-57a2-44f8-a521-af3d3ad7d490\") " pod="openshift-must-gather-lvb79/perf-node-gather-daemonset-twgc7"
Sep 11 01:37:59.797286 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:37:59.797108    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/9d5eaa0e-57a2-44f8-a521-af3d3ad7d490-proc\") pod \"perf-node-gather-daemonset-twgc7\" (UID: \"9d5eaa0e-57a2-44f8-a521-af3d3ad7d490\") " pod="openshift-must-gather-lvb79/perf-node-gather-daemonset-twgc7"
Sep 11 01:37:59.797286 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:37:59.797159    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9d5eaa0e-57a2-44f8-a521-af3d3ad7d490-sys\") pod \"perf-node-gather-daemonset-twgc7\" (UID: \"9d5eaa0e-57a2-44f8-a521-af3d3ad7d490\") " pod="openshift-must-gather-lvb79/perf-node-gather-daemonset-twgc7"
Sep 11 01:37:59.797286 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:37:59.797212    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/9d5eaa0e-57a2-44f8-a521-af3d3ad7d490-podres\") pod \"perf-node-gather-daemonset-twgc7\" (UID: \"9d5eaa0e-57a2-44f8-a521-af3d3ad7d490\") " pod="openshift-must-gather-lvb79/perf-node-gather-daemonset-twgc7"
Sep 11 01:37:59.797286 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:37:59.797260    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9d5eaa0e-57a2-44f8-a521-af3d3ad7d490-lib-modules\") pod \"perf-node-gather-daemonset-twgc7\" (UID: \"9d5eaa0e-57a2-44f8-a521-af3d3ad7d490\") " pod="openshift-must-gather-lvb79/perf-node-gather-daemonset-twgc7"
Sep 11 01:37:59.797286 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:37:59.797268    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/9d5eaa0e-57a2-44f8-a521-af3d3ad7d490-proc\") pod \"perf-node-gather-daemonset-twgc7\" (UID: \"9d5eaa0e-57a2-44f8-a521-af3d3ad7d490\") " pod="openshift-must-gather-lvb79/perf-node-gather-daemonset-twgc7"
Sep 11 01:37:59.797529 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:37:59.797285    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9d5eaa0e-57a2-44f8-a521-af3d3ad7d490-sys\") pod \"perf-node-gather-daemonset-twgc7\" (UID: \"9d5eaa0e-57a2-44f8-a521-af3d3ad7d490\") " pod="openshift-must-gather-lvb79/perf-node-gather-daemonset-twgc7"
Sep 11 01:37:59.797529 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:37:59.797430    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/9d5eaa0e-57a2-44f8-a521-af3d3ad7d490-podres\") pod \"perf-node-gather-daemonset-twgc7\" (UID: \"9d5eaa0e-57a2-44f8-a521-af3d3ad7d490\") " pod="openshift-must-gather-lvb79/perf-node-gather-daemonset-twgc7"
Sep 11 01:37:59.797529 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:37:59.797461    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9d5eaa0e-57a2-44f8-a521-af3d3ad7d490-lib-modules\") pod \"perf-node-gather-daemonset-twgc7\" (UID: \"9d5eaa0e-57a2-44f8-a521-af3d3ad7d490\") " pod="openshift-must-gather-lvb79/perf-node-gather-daemonset-twgc7"
Sep 11 01:37:59.805380 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:37:59.805346    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6m94\" (UniqueName: \"kubernetes.io/projected/9d5eaa0e-57a2-44f8-a521-af3d3ad7d490-kube-api-access-k6m94\") pod \"perf-node-gather-daemonset-twgc7\" (UID: \"9d5eaa0e-57a2-44f8-a521-af3d3ad7d490\") " pod="openshift-must-gather-lvb79/perf-node-gather-daemonset-twgc7"
Sep 11 01:37:59.938547 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:37:59.938456    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-lvb79/perf-node-gather-daemonset-twgc7"
Sep 11 01:38:00.232447 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:38:00.231554    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-lvb79/perf-node-gather-daemonset-twgc7"]
Sep 11 01:38:00.235303 ip-10-0-129-222 kubenswrapper[2569]: W0911 01:38:00.235273    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod9d5eaa0e_57a2_44f8_a521_af3d3ad7d490.slice/crio-1f70f389e137d2157ec188ffc62ee900cf93826fb07475211df31dcbc9621628 WatchSource:0}: Error finding container 1f70f389e137d2157ec188ffc62ee900cf93826fb07475211df31dcbc9621628: Status 404 returned error can't find the container with id 1f70f389e137d2157ec188ffc62ee900cf93826fb07475211df31dcbc9621628
Sep 11 01:38:01.143595 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:38:01.143546    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-lvb79/perf-node-gather-daemonset-twgc7" event={"ID":"9d5eaa0e-57a2-44f8-a521-af3d3ad7d490","Type":"ContainerStarted","Data":"ccc5d6c5693d5bae2015c259d0fa074be732fbc4a2ebc1f8883114147a2f2b15"}
Sep 11 01:38:01.144074 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:38:01.143629    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-lvb79/perf-node-gather-daemonset-twgc7" event={"ID":"9d5eaa0e-57a2-44f8-a521-af3d3ad7d490","Type":"ContainerStarted","Data":"1f70f389e137d2157ec188ffc62ee900cf93826fb07475211df31dcbc9621628"}
Sep 11 01:38:01.144074 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:38:01.143763    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-lvb79/perf-node-gather-daemonset-twgc7"
Sep 11 01:38:01.159996 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:38:01.159945    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-lvb79/perf-node-gather-daemonset-twgc7" podStartSLOduration=2.15993039 podStartE2EDuration="2.15993039s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 01:37:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 01:38:01.157912093 +0000 UTC m=+2415.809286343" watchObservedRunningTime="2026-09-11 01:38:01.15993039 +0000 UTC m=+2415.811304638"
Sep 11 01:38:01.463417 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:38:01.463338    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-6q5zl_d10c25a9-1b18-43a7-94f6-ae86285efcec/dns/0.log"
Sep 11 01:38:01.483667 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:38:01.483643    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-6q5zl_d10c25a9-1b18-43a7-94f6-ae86285efcec/kube-rbac-proxy/0.log"
Sep 11 01:38:01.594977 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:38:01.594952    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-2dvjn_c7486f1f-9947-4405-9186-66dfa5621253/dns-node-resolver/0.log"
Sep 11 01:38:02.252096 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:38:02.252054    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-jrszh_304c1b0f-f5f9-4c4c-8041-ac384ad21516/node-ca/0.log"
Sep 11 01:38:03.468883 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:38:03.468837    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5cc5ff74fb-8j6n9_b406e46a-719d-42d9-80b6-5706557ec9f8/kube-auth-proxy/0.log"
Sep 11 01:38:03.518431 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:38:03.518405    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-2t8jt_b8f2fea7-8e9c-45dd-9291-6e261d2e69db/istio-proxy/0.log"
Sep 11 01:38:03.521641 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:38:03.521618    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-2t8jt_b8f2fea7-8e9c-45dd-9291-6e261d2e69db/istio-proxy/1.log"
Sep 11 01:38:04.498236 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:38:04.498204    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-c2ph7_f31a4ea6-4a42-40a1-bffa-7d59ce62a268/serve-healthcheck-canary/0.log"
Sep 11 01:38:05.040321 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:38:05.040284    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-bmdx6_ef9445dc-e523-489f-8112-04141c08da1d/kube-rbac-proxy/0.log"
Sep 11 01:38:05.058811 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:38:05.058779    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-bmdx6_ef9445dc-e523-489f-8112-04141c08da1d/exporter/0.log"
Sep 11 01:38:05.077570 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:38:05.077545    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-bmdx6_ef9445dc-e523-489f-8112-04141c08da1d/extractor/0.log"
Sep 11 01:38:07.160814 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:38:07.160779    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-lvb79/perf-node-gather-daemonset-twgc7"
Sep 11 01:38:07.660942 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:38:07.660890    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7c49b45946-25wxw_823b7670-1630-48fc-9963-fb96437d04b0/manager/0.log"
Sep 11 01:38:07.684623 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:38:07.684585    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7c49b45946-25wxw_823b7670-1630-48fc-9963-fb96437d04b0/copy-manifests/0.log"
Sep 11 01:38:09.235009 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:38:09.234979    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-6cff9449b4-vtxpl_91d835d3-5954-4a73-970e-3f2c924b3daa/manager/0.log"
Sep 11 01:38:10.200535 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:38:10.200505    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5cc5ff74fb-8j6n9_b406e46a-719d-42d9-80b6-5706557ec9f8/kube-auth-proxy/0.log"
Sep 11 01:38:10.232520 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:38:10.232494    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-2t8jt_b8f2fea7-8e9c-45dd-9291-6e261d2e69db/istio-proxy/0.log"
Sep 11 01:38:10.236039 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:38:10.236011    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-2t8jt_b8f2fea7-8e9c-45dd-9291-6e261d2e69db/istio-proxy/1.log"
Sep 11 01:38:12.134575 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:38:12.134549    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5cc5ff74fb-8j6n9_b406e46a-719d-42d9-80b6-5706557ec9f8/kube-auth-proxy/0.log"
Sep 11 01:38:12.169256 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:38:12.169225    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-2t8jt_b8f2fea7-8e9c-45dd-9291-6e261d2e69db/istio-proxy/0.log"
Sep 11 01:38:12.173372 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:38:12.173318    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-2t8jt_b8f2fea7-8e9c-45dd-9291-6e261d2e69db/istio-proxy/1.log"
Sep 11 01:38:14.304252 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:38:14.304219    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-8b4d697c5-bt7tj_aa2b1bf9-4803-4325-89bf-fdb1389d3ecb/migrator/0.log"
Sep 11 01:38:14.323764 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:38:14.323740    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-8b4d697c5-bt7tj_aa2b1bf9-4803-4325-89bf-fdb1389d3ecb/graceful-termination/0.log"
Sep 11 01:38:15.767159 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:38:15.767128    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-8vxxt_bda35fc9-c767-4b21-9509-994a67ffdeaf/kube-multus/0.log"
Sep 11 01:38:15.789727 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:38:15.789701    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8bg6q_799942b1-a6d8-4f5a-8d17-e7b2761b68b7/kube-multus-additional-cni-plugins/0.log"
Sep 11 01:38:15.809203 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:38:15.809179    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8bg6q_799942b1-a6d8-4f5a-8d17-e7b2761b68b7/egress-router-binary-copy/0.log"
Sep 11 01:38:15.828439 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:38:15.828414    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8bg6q_799942b1-a6d8-4f5a-8d17-e7b2761b68b7/cni-plugins/0.log"
Sep 11 01:38:15.847126 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:38:15.847102    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8bg6q_799942b1-a6d8-4f5a-8d17-e7b2761b68b7/bond-cni-plugin/0.log"
Sep 11 01:38:15.866919 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:38:15.866901    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8bg6q_799942b1-a6d8-4f5a-8d17-e7b2761b68b7/routeoverride-cni/0.log"
Sep 11 01:38:15.885492 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:38:15.885472    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8bg6q_799942b1-a6d8-4f5a-8d17-e7b2761b68b7/whereabouts-cni-bincopy/0.log"
Sep 11 01:38:15.904433 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:38:15.904415    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8bg6q_799942b1-a6d8-4f5a-8d17-e7b2761b68b7/whereabouts-cni/0.log"
Sep 11 01:38:16.339466 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:38:16.339439    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-r8n96_85c82c9f-52d0-49cd-afd9-b791628e1a50/network-metrics-daemon/0.log"
Sep 11 01:38:16.355797 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:38:16.355771    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-r8n96_85c82c9f-52d0-49cd-afd9-b791628e1a50/kube-rbac-proxy/0.log"
Sep 11 01:38:17.266472 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:38:17.266434    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qsqkd_37ee44c2-7e75-4918-bc66-636974fa9b94/ovn-controller/0.log"
Sep 11 01:38:17.282197 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:38:17.282159    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qsqkd_37ee44c2-7e75-4918-bc66-636974fa9b94/ovn-acl-logging/0.log"
Sep 11 01:38:17.304682 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:38:17.304659    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qsqkd_37ee44c2-7e75-4918-bc66-636974fa9b94/kube-rbac-proxy-node/0.log"
Sep 11 01:38:17.324806 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:38:17.324777    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qsqkd_37ee44c2-7e75-4918-bc66-636974fa9b94/kube-rbac-proxy-ovn-metrics/0.log"
Sep 11 01:38:17.349846 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:38:17.349825    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qsqkd_37ee44c2-7e75-4918-bc66-636974fa9b94/northd/0.log"
Sep 11 01:38:17.369142 ip-10-0-129-222 kubenswrapper[2569]: I0911 01:38:17.369117    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qsqkd_37ee44c2-7e75-4918-bc66-636974fa9b94/nbdb/0.log"