Sep 16 02:11:12.515037 ip-10-0-140-34 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 16 02:11:12.515052 ip-10-0-140-34 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 16 02:11:12.515061 ip-10-0-140-34 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 16 02:11:12.515382 ip-10-0-140-34 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 16 02:11:22.734188 ip-10-0-140-34 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 16 02:11:22.734203 ip-10-0-140-34 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot bf600a45af194e21938fec7bfff47b7c --
Sep 16 02:13:06.953035 ip-10-0-140-34 systemd[1]: Starting Kubernetes Kubelet...
Sep 16 02:13:07.388770 ip-10-0-140-34 kubenswrapper[2558]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 02:13:07.388770 ip-10-0-140-34 kubenswrapper[2558]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 16 02:13:07.388770 ip-10-0-140-34 kubenswrapper[2558]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 02:13:07.388770 ip-10-0-140-34 kubenswrapper[2558]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 16 02:13:07.388770 ip-10-0-140-34 kubenswrapper[2558]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 02:13:07.391557 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.391471    2558 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 16 02:13:07.395994 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.395979    2558 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 02:13:07.395994 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.395993    2558 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 02:13:07.396056 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.395997    2558 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 02:13:07.396056 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.396000    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 02:13:07.396056 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.396003    2558 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 02:13:07.396056 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.396007    2558 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 02:13:07.396056 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.396010    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 02:13:07.396056 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.396013    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 02:13:07.396056 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.396016    2558 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 02:13:07.396056 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.396019    2558 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 02:13:07.396056 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.396021    2558 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 02:13:07.396056 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.396029    2558 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 02:13:07.396056 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.396032    2558 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 02:13:07.396056 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.396035    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 02:13:07.396056 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.396037    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 02:13:07.396056 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.396040    2558 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 02:13:07.396056 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.396042    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 02:13:07.396056 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.396045    2558 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 02:13:07.396056 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.396048    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 02:13:07.396056 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.396050    2558 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 02:13:07.396056 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.396052    2558 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 02:13:07.396598 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.396055    2558 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 02:13:07.396598 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.396058    2558 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 02:13:07.396598 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.396060    2558 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 02:13:07.396598 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.396063    2558 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 02:13:07.396598 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.396066    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 02:13:07.396598 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.396069    2558 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 02:13:07.396598 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.396072    2558 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 02:13:07.396598 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.396074    2558 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 02:13:07.396598 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.396077    2558 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 02:13:07.396598 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.396080    2558 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 02:13:07.396598 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.396082    2558 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 02:13:07.396598 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.396085    2558 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 02:13:07.396598 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.396087    2558 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 02:13:07.396598 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.396090    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 02:13:07.396598 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.396092    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 02:13:07.396598 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.396095    2558 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 02:13:07.396598 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.396097    2558 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 02:13:07.396598 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.396100    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 02:13:07.396598 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.396102    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 02:13:07.396598 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.396105    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 02:13:07.397393 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.396107    2558 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 02:13:07.397393 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.396109    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 02:13:07.397393 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.396112    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 02:13:07.397393 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.396115    2558 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 02:13:07.397393 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.396117    2558 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 02:13:07.397393 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.396120    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 02:13:07.397393 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.396122    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 02:13:07.397393 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.396125    2558 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 02:13:07.397393 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.396127    2558 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 02:13:07.397393 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.396130    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 02:13:07.397393 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.396132    2558 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 02:13:07.397393 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.396134    2558 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 02:13:07.397393 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.396137    2558 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 02:13:07.397393 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.396141    2558 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 02:13:07.397393 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.396144    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 02:13:07.397393 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.396147    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 02:13:07.397393 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.396150    2558 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 02:13:07.397393 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.396154    2558 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 02:13:07.397393 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.396158    2558 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 02:13:07.397902 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.396160    2558 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 02:13:07.397902 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.396163    2558 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 02:13:07.397902 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.396166    2558 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 02:13:07.397902 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.396168    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 02:13:07.397902 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.396171    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 02:13:07.397902 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.396173    2558 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 02:13:07.397902 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.396176    2558 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 02:13:07.397902 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.396178    2558 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 02:13:07.397902 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.396181    2558 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 02:13:07.397902 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.396184    2558 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 02:13:07.397902 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.396189    2558 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 02:13:07.397902 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.396193    2558 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 02:13:07.397902 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.396196    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 02:13:07.397902 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.396199    2558 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 02:13:07.397902 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.396203    2558 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 02:13:07.397902 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.396206    2558 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 02:13:07.397902 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.396209    2558 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 02:13:07.397902 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.396212    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 02:13:07.397902 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.396215    2558 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 02:13:07.398385 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.396218    2558 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 02:13:07.398385 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.396221    2558 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 02:13:07.398385 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.396224    2558 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 02:13:07.398385 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.396228    2558 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 02:13:07.398385 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.396231    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 02:13:07.398385 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.396233    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 02:13:07.398385 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.396237    2558 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 02:13:07.398385 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.396240    2558 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 02:13:07.398385 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.396242    2558 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 02:13:07.398385 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.396245    2558 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 02:13:07.398385 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.396248    2558 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 02:13:07.398385 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.396250    2558 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 02:13:07.398385 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.396253    2558 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 02:13:07.398385 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.396256    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 02:13:07.398385 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.396258    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 02:13:07.398385 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.396261    2558 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 02:13:07.398385 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.396264    2558 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 02:13:07.398385 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.396266    2558 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 02:13:07.398385 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.396269    2558 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 02:13:07.398385 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.396271    2558 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 02:13:07.400263 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.400246    2558 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 02:13:07.400263 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.400263    2558 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 02:13:07.400333 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.400267    2558 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 02:13:07.400333 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.400270    2558 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 02:13:07.400333 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.400274    2558 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 02:13:07.400333 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.400277    2558 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 02:13:07.400333 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.400280    2558 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 02:13:07.400333 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.400282    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 02:13:07.400333 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.400285    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 02:13:07.400333 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.400288    2558 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 02:13:07.400333 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.400290    2558 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 02:13:07.400333 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.400293    2558 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 02:13:07.400333 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.400295    2558 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 02:13:07.400333 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.400298    2558 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 02:13:07.400333 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.400300    2558 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 02:13:07.400333 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.400304    2558 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 02:13:07.400333 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.400306    2558 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 02:13:07.400333 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.400309    2558 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 02:13:07.400333 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.400312    2558 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 02:13:07.400333 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.400315    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 02:13:07.400333 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.400318    2558 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 02:13:07.400333 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.400321    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 02:13:07.400844 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.400324    2558 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 02:13:07.400844 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.400326    2558 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 02:13:07.400844 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.400329    2558 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 02:13:07.400844 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.400331    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 02:13:07.400844 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.400334    2558 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 02:13:07.400844 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.400337    2558 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 02:13:07.400844 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.400340    2558 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 02:13:07.400844 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.400343    2558 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 02:13:07.400844 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.400346    2558 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 02:13:07.400844 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.400349    2558 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 02:13:07.400844 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.400351    2558 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 02:13:07.400844 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.400354    2558 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 02:13:07.400844 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.400357    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 02:13:07.400844 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.400359    2558 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 02:13:07.400844 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.400362    2558 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 02:13:07.400844 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.400364    2558 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 02:13:07.400844 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.400367    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 02:13:07.400844 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.400369    2558 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 02:13:07.400844 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.400372    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 02:13:07.400844 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.400374    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 02:13:07.401352 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.400377    2558 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 02:13:07.401352 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.400379    2558 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 02:13:07.401352 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.400382    2558 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 02:13:07.401352 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.400385    2558 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 02:13:07.401352 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.400387    2558 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 02:13:07.401352 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.400390    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 02:13:07.401352 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.400393    2558 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 02:13:07.401352 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.400395    2558 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 02:13:07.401352 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.400399    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 02:13:07.401352 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.400402    2558 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 02:13:07.401352 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.400404    2558 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 02:13:07.401352 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.400406    2558 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 02:13:07.401352 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.400409    2558 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 02:13:07.401352 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.400412    2558 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 02:13:07.401352 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.400415    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 02:13:07.401352 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.400418    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 02:13:07.401352 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.400420    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 02:13:07.401352 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.400423    2558 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 02:13:07.401352 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.400426    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 02:13:07.401352 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.400428    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 02:13:07.401862 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.400431    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 02:13:07.401862 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.400433    2558 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 02:13:07.401862 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.400436    2558 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 02:13:07.401862 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.400440    2558 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 02:13:07.401862 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.400444    2558 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 02:13:07.401862 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.400447    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 02:13:07.401862 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.400449    2558 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 02:13:07.401862 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.400452    2558 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 02:13:07.401862 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.400455    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 02:13:07.401862 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.400458    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 02:13:07.401862 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.400460    2558 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 02:13:07.401862 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.400463    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 02:13:07.401862 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.400465    2558 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 02:13:07.401862 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.400468    2558 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 02:13:07.401862 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.400470    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 02:13:07.401862 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.400473    2558 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 02:13:07.401862 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.400476    2558 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 02:13:07.401862 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.400480    2558 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 02:13:07.401862 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.400483    2558 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 02:13:07.402330 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.400486    2558 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 02:13:07.402330 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.400488    2558 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 02:13:07.402330 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.400492    2558 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 02:13:07.402330 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.400495    2558 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 02:13:07.402330 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.400499    2558 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 02:13:07.402330 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.400502    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 02:13:07.402330 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.400506    2558 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 02:13:07.402330 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.400509    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 02:13:07.402330 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.400512    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 02:13:07.402330 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.400514    2558 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 02:13:07.402330 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.400517    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 02:13:07.402330 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.400521    2558 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 02:13:07.402330 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.400523    2558 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 02:13:07.402330 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.400526    2558 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 02:13:07.402330 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.400528    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 02:13:07.402330 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.400531    2558 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 02:13:07.402330 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.400533    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 02:13:07.402330 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.400536    2558 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 02:13:07.402330 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.400966    2558 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 02:13:07.402812 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.400973    2558 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 02:13:07.402812 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.400976    2558 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 02:13:07.402812 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.400979    2558 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 02:13:07.402812 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.400982    2558 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 02:13:07.402812 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.400985    2558 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 02:13:07.402812 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.400988    2558 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 02:13:07.402812 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.400990    2558 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 02:13:07.402812 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.400994    2558 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 02:13:07.402812 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.400996    2558 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 02:13:07.402812 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.400999    2558 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 02:13:07.402812 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.401002    2558 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 02:13:07.402812 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.401004    2558 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 02:13:07.402812 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.401007    2558 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 02:13:07.402812 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.401009    2558 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 02:13:07.402812 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.401012    2558 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 02:13:07.402812 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.401015    2558 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 02:13:07.402812 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.401018    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 02:13:07.402812 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.401021    2558 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 02:13:07.402812 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.401024    2558 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 02:13:07.402812 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.401027    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 02:13:07.403326 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.401030    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 02:13:07.403326 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.401033    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 02:13:07.403326 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.401035    2558 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 02:13:07.403326 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.401038    2558 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 02:13:07.403326 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.401041    2558 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 02:13:07.403326 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.401043    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 02:13:07.403326 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.401046    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 02:13:07.403326 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.401048    2558 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 02:13:07.403326 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.401051    2558 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 02:13:07.403326 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.401054    2558 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 02:13:07.403326 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.401057    2558 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 02:13:07.403326 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.401059    2558 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 02:13:07.403326 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.401062    2558 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 02:13:07.403326 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.401064    2558 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 02:13:07.403326 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.401067    2558 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 02:13:07.403326 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.401071    2558 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 02:13:07.403326 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.401075    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 02:13:07.403326 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.401078    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 02:13:07.403326 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.401081    2558 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 02:13:07.403917 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.401084    2558 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 02:13:07.403917 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.401087    2558 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 02:13:07.403917 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.401089    2558 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 02:13:07.403917 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.401092    2558 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 02:13:07.403917 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.401095    2558 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 02:13:07.403917 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.401097    2558 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 02:13:07.403917 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.401100    2558 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 02:13:07.403917 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.401103    2558 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 02:13:07.403917 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.401106    2558 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 02:13:07.403917 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.401110    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 02:13:07.403917 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.401112    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 02:13:07.403917 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.401116    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 02:13:07.403917 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.401119    2558 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 02:13:07.403917 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.401121    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 02:13:07.403917 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.401124    2558 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 02:13:07.403917 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.401126    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 02:13:07.403917 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.401129    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 02:13:07.403917 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.401132    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 02:13:07.403917 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.401134    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 02:13:07.404394 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.401137    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 02:13:07.404394 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.401139    2558 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 02:13:07.404394 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.401141    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 02:13:07.404394 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.401144    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 02:13:07.404394 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.401147    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 02:13:07.404394 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.401149    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 02:13:07.404394 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.401152    2558 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 02:13:07.404394 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.401155    2558 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 02:13:07.404394 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.401157    2558 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 02:13:07.404394 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.401160    2558 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 02:13:07.404394 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.401162    2558 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 02:13:07.404394 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.401165    2558 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 02:13:07.404394 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.401168    2558 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 02:13:07.404394 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.401170    2558 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 02:13:07.404394 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.401173    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 02:13:07.404394 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.401177    2558 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 02:13:07.404394 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.401180    2558 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 02:13:07.404394 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.401183    2558 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 02:13:07.404394 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.401186    2558 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 02:13:07.404916 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.401188    2558 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 02:13:07.404916 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.401192    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 02:13:07.404916 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.401195    2558 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 02:13:07.404916 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.401199    2558 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 02:13:07.404916 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.401204    2558 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 02:13:07.404916 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.401207    2558 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 02:13:07.404916 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.401211    2558 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 02:13:07.404916 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.401214    2558 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 02:13:07.404916 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.401216    2558 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 02:13:07.404916 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.401218    2558 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 02:13:07.404916 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.401221    2558 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 02:13:07.404916 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.401224    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 02:13:07.404916 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.401226    2558 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 02:13:07.404916 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.401228    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 02:13:07.404916 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.401231    2558 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 02:13:07.404916 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.401234    2558 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 02:13:07.404916 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.401236    2558 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 02:13:07.404916 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.401239    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 02:13:07.404916 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.401241    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 02:13:07.405379 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.401244    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 02:13:07.405379 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.401246    2558 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 02:13:07.405379 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.401332    2558 flags.go:64] FLAG: --address="0.0.0.0"
Sep 16 02:13:07.405379 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.401366    2558 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 16 02:13:07.405379 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.401376    2558 flags.go:64] FLAG: --anonymous-auth="true"
Sep 16 02:13:07.405379 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.401380    2558 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 16 02:13:07.405379 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.401384    2558 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 16 02:13:07.405379 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.401388    2558 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 16 02:13:07.405379 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.401392    2558 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 16 02:13:07.405379 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.401396    2558 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 16 02:13:07.405379 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.401400    2558 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 16 02:13:07.405379 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.401403    2558 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 16 02:13:07.405379 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.401406    2558 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 16 02:13:07.405379 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.401409    2558 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 16 02:13:07.405379 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.401413    2558 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 16 02:13:07.405379 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.401416    2558 flags.go:64] FLAG: --cgroup-root=""
Sep 16 02:13:07.405379 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.401419    2558 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 16 02:13:07.405379 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.401422    2558 flags.go:64] FLAG: --client-ca-file=""
Sep 16 02:13:07.405379 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.401426    2558 flags.go:64] FLAG: --cloud-provider="external"
Sep 16 02:13:07.405379 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.401429    2558 flags.go:64] FLAG: --cluster-dns="[]"
Sep 16 02:13:07.405379 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.401433    2558 flags.go:64] FLAG: --cluster-domain=""
Sep 16 02:13:07.405379 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.401436    2558 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 16 02:13:07.405379 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.401439    2558 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 16 02:13:07.405949 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.401442    2558 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 16 02:13:07.405949 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.401446    2558 flags.go:64] FLAG: --container-log-max-files="5"
Sep 16 02:13:07.405949 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.401449    2558 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 16 02:13:07.405949 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.401452    2558 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 16 02:13:07.405949 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.401455    2558 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 16 02:13:07.405949 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.401459    2558 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 16 02:13:07.405949 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.401462    2558 flags.go:64] FLAG: --contention-profiling="false"
Sep 16 02:13:07.405949 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.401465    2558 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 16 02:13:07.405949 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.401468    2558 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 16 02:13:07.405949 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.401471    2558 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 16 02:13:07.405949 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.401474    2558 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 16 02:13:07.405949 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.401478    2558 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 16 02:13:07.405949 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.401481    2558 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 16 02:13:07.405949 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.401484    2558 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 16 02:13:07.405949 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.401487    2558 flags.go:64] FLAG: --enable-load-reader="false"
Sep 16 02:13:07.405949 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.401490    2558 flags.go:64] FLAG: --enable-server="true"
Sep 16 02:13:07.405949 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.401493    2558 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 16 02:13:07.405949 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.401498    2558 flags.go:64] FLAG: --event-burst="100"
Sep 16 02:13:07.405949 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.401501    2558 flags.go:64] FLAG: --event-qps="50"
Sep 16 02:13:07.405949 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.401504    2558 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 16 02:13:07.405949 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.401507    2558 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 16 02:13:07.405949 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.401510    2558 flags.go:64] FLAG: --eviction-hard=""
Sep 16 02:13:07.405949 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.401514    2558 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 16 02:13:07.405949 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.401516    2558 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 16 02:13:07.406563 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.401519    2558 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 16 02:13:07.406563 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.401522    2558 flags.go:64] FLAG: --eviction-soft=""
Sep 16 02:13:07.406563 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.401525    2558 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 16 02:13:07.406563 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.401528    2558 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 16 02:13:07.406563 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.401533    2558 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 16 02:13:07.406563 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.401537    2558 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 16 02:13:07.406563 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.401540    2558 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 16 02:13:07.406563 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.401542    2558 flags.go:64] FLAG: --fail-swap-on="true"
Sep 16 02:13:07.406563 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.401545    2558 flags.go:64] FLAG: --feature-gates=""
Sep 16 02:13:07.406563 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.401549    2558 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 16 02:13:07.406563 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.401552    2558 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 16 02:13:07.406563 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.401555    2558 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 16 02:13:07.406563 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.401559    2558 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 16 02:13:07.406563 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.401562    2558 flags.go:64] FLAG: --healthz-port="10248"
Sep 16 02:13:07.406563 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.401565    2558 flags.go:64] FLAG: --help="false"
Sep 16 02:13:07.406563 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.401567    2558 flags.go:64] FLAG: --hostname-override="ip-10-0-140-34.ec2.internal"
Sep 16 02:13:07.406563 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.401571    2558 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 16 02:13:07.406563 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.401573    2558 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 16 02:13:07.406563 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.401576    2558 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 16 02:13:07.406563 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.401580    2558 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 16 02:13:07.406563 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.401583    2558 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 16 02:13:07.406563 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.401586    2558 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 16 02:13:07.406563 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.401588    2558 flags.go:64] FLAG: --image-service-endpoint=""
Sep 16 02:13:07.407188 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.401591    2558 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 16 02:13:07.407188 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.401594    2558 flags.go:64] FLAG: --kube-api-burst="100"
Sep 16 02:13:07.407188 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.401597    2558 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 16 02:13:07.407188 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.401600    2558 flags.go:64] FLAG: --kube-api-qps="50"
Sep 16 02:13:07.407188 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.401606    2558 flags.go:64] FLAG: --kube-reserved=""
Sep 16 02:13:07.407188 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.401609    2558 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 16 02:13:07.407188 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.401612    2558 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 16 02:13:07.407188 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.401615    2558 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 16 02:13:07.407188 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.401618    2558 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 16 02:13:07.407188 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.401621    2558 flags.go:64] FLAG: --lock-file=""
Sep 16 02:13:07.407188 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.401624    2558 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 16 02:13:07.407188 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.401627    2558 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 16 02:13:07.407188 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.401630    2558 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 16 02:13:07.407188 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.401635    2558 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 16 02:13:07.407188 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.401639    2558 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 16 02:13:07.407188 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.401642    2558 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 16 02:13:07.407188 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.401645    2558 flags.go:64] FLAG: --logging-format="text"
Sep 16 02:13:07.407188 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.401647    2558 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 16 02:13:07.407188 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.401651    2558 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 16 02:13:07.407188 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.401654    2558 flags.go:64] FLAG: --manifest-url=""
Sep 16 02:13:07.407188 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.401657    2558 flags.go:64] FLAG: --manifest-url-header=""
Sep 16 02:13:07.407188 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.401661    2558 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 16 02:13:07.407188 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.401664    2558 flags.go:64] FLAG: --max-open-files="1000000"
Sep 16 02:13:07.407188 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.401668    2558 flags.go:64] FLAG: --max-pods="110"
Sep 16 02:13:07.407188 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.401671    2558 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 16 02:13:07.407840 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.401674    2558 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 16 02:13:07.407840 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.401677    2558 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 16 02:13:07.407840 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.401679    2558 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 16 02:13:07.407840 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.401682    2558 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 16 02:13:07.407840 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.401686    2558 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 16 02:13:07.407840 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.401689    2558 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 16 02:13:07.407840 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.401713    2558 flags.go:64] FLAG: --node-status-max-images="50"
Sep 16 02:13:07.407840 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.401716    2558 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 16 02:13:07.407840 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.401720    2558 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 16 02:13:07.407840 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.401723    2558 flags.go:64] FLAG: --pod-cidr=""
Sep 16 02:13:07.407840 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.401725    2558 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10"
Sep 16 02:13:07.407840 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.401732    2558 flags.go:64] FLAG: --pod-manifest-path=""
Sep 16 02:13:07.407840 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.401736    2558 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 16 02:13:07.407840 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.401739    2558 flags.go:64] FLAG: --pods-per-core="0"
Sep 16 02:13:07.407840 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.401742    2558 flags.go:64] FLAG: --port="10250"
Sep 16 02:13:07.407840 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.401746    2558 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 16 02:13:07.407840 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.401749    2558 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0454b8a4e9cb2c59e"
Sep 16 02:13:07.407840 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.401752    2558 flags.go:64] FLAG: --qos-reserved=""
Sep 16 02:13:07.407840 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.401755    2558 flags.go:64] FLAG: --read-only-port="10255"
Sep 16 02:13:07.407840 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.401758    2558 flags.go:64] FLAG: --register-node="true"
Sep 16 02:13:07.407840 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.401762    2558 flags.go:64] FLAG: --register-with-taints=""
Sep 16 02:13:07.407840 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.401765    2558 flags.go:64] FLAG: --registry-burst="10"
Sep 16 02:13:07.407840 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.401769    2558 flags.go:64] FLAG: --registry-qps="5"
Sep 16 02:13:07.407840 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.401772    2558 flags.go:64] FLAG: --reserved-cpus=""
Sep 16 02:13:07.408426 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.401775    2558 flags.go:64] FLAG: --reserved-memory=""
Sep 16 02:13:07.408426 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.401779    2558 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 16 02:13:07.408426 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.401782    2558 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 16 02:13:07.408426 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.401785    2558 flags.go:64] FLAG: --rotate-certificates="false"
Sep 16 02:13:07.408426 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.401788    2558 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 16 02:13:07.408426 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.401791    2558 flags.go:64] FLAG: --runonce="false"
Sep 16 02:13:07.408426 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.401793    2558 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 16 02:13:07.408426 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.401797    2558 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 16 02:13:07.408426 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.401800    2558 flags.go:64] FLAG: --seccomp-default="false"
Sep 16 02:13:07.408426 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.401802    2558 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 16 02:13:07.408426 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.401805    2558 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 16 02:13:07.408426 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.401808    2558 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 16 02:13:07.408426 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.401811    2558 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 16 02:13:07.408426 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.401815    2558 flags.go:64] FLAG: --storage-driver-password="root"
Sep 16 02:13:07.408426 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.401818    2558 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 16 02:13:07.408426 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.401820    2558 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 16 02:13:07.408426 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.401823    2558 flags.go:64] FLAG: --storage-driver-user="root"
Sep 16 02:13:07.408426 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.401826    2558 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 16 02:13:07.408426 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.401829    2558 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 16 02:13:07.408426 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.401832    2558 flags.go:64] FLAG: --system-cgroups=""
Sep 16 02:13:07.408426 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.401836    2558 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 16 02:13:07.408426 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.401842    2558 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 16 02:13:07.408426 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.401845    2558 flags.go:64] FLAG: --tls-cert-file=""
Sep 16 02:13:07.408426 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.401848    2558 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 16 02:13:07.408426 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.401852    2558 flags.go:64] FLAG: --tls-min-version=""
Sep 16 02:13:07.409060 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.401855    2558 flags.go:64] FLAG: --tls-private-key-file=""
Sep 16 02:13:07.409060 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.401857    2558 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 16 02:13:07.409060 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.401860    2558 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 16 02:13:07.409060 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.401864    2558 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 16 02:13:07.409060 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.401867    2558 flags.go:64] FLAG: --v="2"
Sep 16 02:13:07.409060 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.401871    2558 flags.go:64] FLAG: --version="false"
Sep 16 02:13:07.409060 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.401875    2558 flags.go:64] FLAG: --vmodule=""
Sep 16 02:13:07.409060 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.401879    2558 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 16 02:13:07.409060 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.401882    2558 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 16 02:13:07.409060 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.402001    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 02:13:07.409060 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.402005    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 02:13:07.409060 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.402009    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 02:13:07.409060 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.402012    2558 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 02:13:07.409060 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.402015    2558 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 02:13:07.409060 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.402018    2558 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 02:13:07.409060 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.402021    2558 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 02:13:07.409060 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.402023    2558 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 02:13:07.409060 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.402026    2558 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 02:13:07.409060 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.402028    2558 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 02:13:07.409060 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.402031    2558 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 02:13:07.409060 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.402034    2558 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 02:13:07.409060 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.402036    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 02:13:07.409669 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.402039    2558 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 02:13:07.409669 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.402041    2558 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 02:13:07.409669 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.402044    2558 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 02:13:07.409669 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.402047    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 02:13:07.409669 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.402049    2558 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 02:13:07.409669 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.402055    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 02:13:07.409669 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.402057    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 02:13:07.409669 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.402060    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 02:13:07.409669 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.402063    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 02:13:07.409669 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.402065    2558 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 02:13:07.409669 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.402068    2558 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 02:13:07.409669 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.402071    2558 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 02:13:07.409669 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.402074    2558 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 02:13:07.409669 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.402077    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 02:13:07.409669 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.402079    2558 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 02:13:07.409669 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.402082    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 02:13:07.409669 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.402084    2558 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 02:13:07.409669 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.402089    2558 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 02:13:07.409669 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.402092    2558 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 02:13:07.410159 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.402096    2558 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 02:13:07.410159 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.402099    2558 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 02:13:07.410159 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.402102    2558 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 02:13:07.410159 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.402106    2558 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 02:13:07.410159 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.402109    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 02:13:07.410159 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.402112    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 02:13:07.410159 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.402114    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 02:13:07.410159 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.402117    2558 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 02:13:07.410159 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.402120    2558 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 02:13:07.410159 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.402123    2558 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 02:13:07.410159 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.402125    2558 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 02:13:07.410159 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.402128    2558 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 02:13:07.410159 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.402130    2558 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 02:13:07.410159 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.402133    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 02:13:07.410159 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.402136    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 02:13:07.410159 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.402138    2558 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 02:13:07.410159 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.402141    2558 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 02:13:07.410159 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.402144    2558 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 02:13:07.410159 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.402148    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 02:13:07.410159 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.402150    2558 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 02:13:07.410662 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.402153    2558 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 02:13:07.410662 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.402155    2558 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 02:13:07.410662 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.402158    2558 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 02:13:07.410662 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.402162    2558 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 02:13:07.410662 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.402165    2558 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 02:13:07.410662 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.402169    2558 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 02:13:07.410662 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.402172    2558 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 02:13:07.410662 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.402174    2558 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 02:13:07.410662 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.402177    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 02:13:07.410662 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.402179    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 02:13:07.410662 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.402182    2558 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 02:13:07.410662 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.402190    2558 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 02:13:07.410662 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.402193    2558 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 02:13:07.410662 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.402196    2558 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 02:13:07.410662 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.402198    2558 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 02:13:07.410662 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.402201    2558 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 02:13:07.410662 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.402203    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 02:13:07.410662 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.402206    2558 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 02:13:07.410662 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.402208    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 02:13:07.410662 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.402211    2558 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 02:13:07.411168 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.402213    2558 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 02:13:07.411168 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.402216    2558 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 02:13:07.411168 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.402219    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 02:13:07.411168 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.402221    2558 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 02:13:07.411168 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.402224    2558 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 02:13:07.411168 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.402226    2558 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 02:13:07.411168 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.402230    2558 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 02:13:07.411168 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.402232    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 02:13:07.411168 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.402235    2558 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 02:13:07.411168 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.402237    2558 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 02:13:07.411168 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.402241    2558 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 02:13:07.411168 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.402244    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 02:13:07.411168 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.402246    2558 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 02:13:07.411168 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.402249    2558 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 02:13:07.411168 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.402252    2558 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 02:13:07.411168 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.402255    2558 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 02:13:07.411168 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.402259    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 02:13:07.411168 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.402262    2558 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 02:13:07.411168 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.402264    2558 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 02:13:07.411168 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.402267    2558 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 02:13:07.411655 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.402269    2558 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 02:13:07.411655 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.402272    2558 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 02:13:07.411655 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.402274    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 02:13:07.411655 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.402277    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 02:13:07.411655 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.402281    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 02:13:07.411655 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.402283    2558 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 02:13:07.411655 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.402286    2558 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 02:13:07.411655 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.403153    2558 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 02:13:07.411655 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.410656    2558 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 16 02:13:07.411655 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.410671    2558 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 16 02:13:07.411989 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.411734    2558 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 16 02:13:07.411989 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.411742    2558 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 16 02:13:07.411989 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.411803    2558 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 02:13:07.411989 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.411812    2558 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 02:13:07.411989 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.411818    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 02:13:07.411989 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.411822    2558 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 02:13:07.411989 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.411826    2558 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 02:13:07.411989 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.411830    2558 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 02:13:07.411989 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.411834    2558 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 02:13:07.411989 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.411839    2558 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 02:13:07.411989 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.411841    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 02:13:07.411989 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.411844    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 02:13:07.411989 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.411847    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 02:13:07.411989 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.411850    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 02:13:07.411989 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.411853    2558 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 02:13:07.411989 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.411855    2558 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 02:13:07.411989 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.411858    2558 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 02:13:07.411989 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.411862    2558 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 02:13:07.411989 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.411864    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 02:13:07.412462 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.411867    2558 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 02:13:07.412462 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.411869    2558 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 02:13:07.412462 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.411872    2558 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 02:13:07.412462 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.411875    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 02:13:07.412462 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.411877    2558 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 02:13:07.412462 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.411880    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 02:13:07.412462 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.411882    2558 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 02:13:07.412462 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.411884    2558 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 02:13:07.412462 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.411887    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 02:13:07.412462 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.411889    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 02:13:07.412462 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.411892    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 02:13:07.412462 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.411894    2558 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 02:13:07.412462 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.411898    2558 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 02:13:07.412462 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.411901    2558 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 02:13:07.412462 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.411904    2558 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 02:13:07.412462 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.411907    2558 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 02:13:07.412462 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.411910    2558 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 02:13:07.412462 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.411912    2558 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 02:13:07.412462 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.411915    2558 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 02:13:07.412462 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.411917    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 02:13:07.412975 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.411920    2558 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 02:13:07.412975 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.411922    2558 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 02:13:07.412975 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.411925    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 02:13:07.412975 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.411927    2558 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 02:13:07.412975 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.411930    2558 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 02:13:07.412975 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.411933    2558 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 02:13:07.412975 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.411936    2558 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 02:13:07.412975 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.411938    2558 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 02:13:07.412975 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.411941    2558 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 02:13:07.412975 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.411945    2558 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 02:13:07.412975 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.411948    2558 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 02:13:07.412975 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.411952    2558 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 02:13:07.412975 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.411955    2558 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 02:13:07.412975 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.411958    2558 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 02:13:07.412975 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.411961    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 02:13:07.412975 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.411964    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 02:13:07.412975 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.411967    2558 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 02:13:07.412975 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.411969    2558 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 02:13:07.412975 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.411972    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 02:13:07.413459 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.411975    2558 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 02:13:07.413459 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.411978    2558 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 02:13:07.413459 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.411980    2558 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 02:13:07.413459 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.411983    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 02:13:07.413459 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.411985    2558 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 02:13:07.413459 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.411988    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 02:13:07.413459 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.411991    2558 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 02:13:07.413459 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.411994    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 02:13:07.413459 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.411998    2558 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 02:13:07.413459 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.412000    2558 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 02:13:07.413459 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.412003    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 02:13:07.413459 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.412006    2558 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 02:13:07.413459 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.412008    2558 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 02:13:07.413459 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.412011    2558 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 02:13:07.413459 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.412013    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 02:13:07.413459 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.412016    2558 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 02:13:07.413459 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.412018    2558 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 02:13:07.413459 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.412021    2558 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 02:13:07.413459 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.412023    2558 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 02:13:07.413459 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.412026    2558 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 02:13:07.414002 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.412029    2558 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 02:13:07.414002 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.412031    2558 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 02:13:07.414002 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.412033    2558 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 02:13:07.414002 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.412036    2558 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 02:13:07.414002 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.412039    2558 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 02:13:07.414002 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.412041    2558 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 02:13:07.414002 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.412044    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 02:13:07.414002 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.412047    2558 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 02:13:07.414002 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.412050    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 02:13:07.414002 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.412052    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 02:13:07.414002 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.412055    2558 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 02:13:07.414002 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.412057    2558 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 02:13:07.414002 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.412060    2558 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 02:13:07.414002 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.412062    2558 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 02:13:07.414002 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.412065    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 02:13:07.414002 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.412068    2558 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 02:13:07.414002 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.412072    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 02:13:07.414002 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.412075    2558 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 02:13:07.414002 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.412078    2558 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 02:13:07.414002 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.412082    2558 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 02:13:07.414491 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.412084    2558 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 02:13:07.414491 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.412087    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 02:13:07.414491 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.412090    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 02:13:07.414491 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.412094    2558 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 02:13:07.414491 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.412210    2558 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 02:13:07.414491 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.412215    2558 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 02:13:07.414491 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.412218    2558 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 02:13:07.414491 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.412221    2558 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 02:13:07.414491 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.412224    2558 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 02:13:07.414491 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.412226    2558 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 02:13:07.414491 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.412229    2558 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 02:13:07.414491 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.412233    2558 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 02:13:07.414491 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.412236    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 02:13:07.414491 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.412238    2558 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 02:13:07.414491 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.412241    2558 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 02:13:07.414877 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.412244    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 02:13:07.414877 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.412246    2558 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 02:13:07.414877 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.412249    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 02:13:07.414877 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.412252    2558 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 02:13:07.414877 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.412254    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 02:13:07.414877 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.412258    2558 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 02:13:07.414877 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.412262    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 02:13:07.414877 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.412265    2558 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 02:13:07.414877 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.412267    2558 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 02:13:07.414877 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.412270    2558 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 02:13:07.414877 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.412272    2558 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 02:13:07.414877 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.412275    2558 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 02:13:07.414877 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.412278    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 02:13:07.414877 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.412280    2558 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 02:13:07.414877 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.412283    2558 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 02:13:07.414877 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.412286    2558 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 02:13:07.414877 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.412289    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 02:13:07.414877 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.412292    2558 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 02:13:07.414877 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.412294    2558 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 02:13:07.414877 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.412297    2558 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 02:13:07.415406 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.412300    2558 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 02:13:07.415406 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.412302    2558 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 02:13:07.415406 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.412305    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 02:13:07.415406 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.412307    2558 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 02:13:07.415406 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.412310    2558 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 02:13:07.415406 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.412312    2558 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 02:13:07.415406 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.412315    2558 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 02:13:07.415406 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.412317    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 02:13:07.415406 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.412320    2558 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 02:13:07.415406 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.412322    2558 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 02:13:07.415406 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.412325    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 02:13:07.415406 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.412328    2558 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 02:13:07.415406 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.412330    2558 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 02:13:07.415406 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.412332    2558 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 02:13:07.415406 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.412335    2558 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 02:13:07.415406 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.412337    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 02:13:07.415406 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.412341    2558 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 02:13:07.415406 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.412345    2558 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 02:13:07.415406 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.412348    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 02:13:07.415950 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.412350    2558 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 02:13:07.415950 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.412353    2558 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 02:13:07.415950 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.412355    2558 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 02:13:07.415950 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.412358    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 02:13:07.415950 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.412360    2558 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 02:13:07.415950 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.412363    2558 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 02:13:07.415950 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.412366    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 02:13:07.415950 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.412368    2558 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 02:13:07.415950 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.412371    2558 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 02:13:07.415950 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.412374    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 02:13:07.415950 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.412377    2558 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 02:13:07.415950 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.412379    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 02:13:07.415950 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.412382    2558 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 02:13:07.415950 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.412384    2558 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 02:13:07.415950 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.412387    2558 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 02:13:07.415950 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.412389    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 02:13:07.415950 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.412392    2558 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 02:13:07.415950 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.412394    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 02:13:07.415950 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.412397    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 02:13:07.415950 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.412399    2558 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 02:13:07.416426 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.412402    2558 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 02:13:07.416426 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.412404    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 02:13:07.416426 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.412407    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 02:13:07.416426 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.412409    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 02:13:07.416426 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.412412    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 02:13:07.416426 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.412414    2558 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 02:13:07.416426 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.412416    2558 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 02:13:07.416426 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.412419    2558 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 02:13:07.416426 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.412421    2558 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 02:13:07.416426 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.412424    2558 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 02:13:07.416426 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.412426    2558 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 02:13:07.416426 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.412429    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 02:13:07.416426 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.412431    2558 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 02:13:07.416426 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.412434    2558 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 02:13:07.416426 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.412436    2558 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 02:13:07.416426 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.412439    2558 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 02:13:07.416426 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.412441    2558 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 02:13:07.416426 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.412444    2558 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 02:13:07.416426 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.412446    2558 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 02:13:07.416426 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.412449    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 02:13:07.416934 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.412451    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 02:13:07.416934 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.412454    2558 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 02:13:07.416934 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.412457    2558 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 02:13:07.416934 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.412459    2558 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 02:13:07.416934 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.412462    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 02:13:07.416934 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.412465    2558 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 02:13:07.416934 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.412467    2558 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 02:13:07.416934 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.412471    2558 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 02:13:07.416934 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:07.412475    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 02:13:07.416934 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.412480    2558 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 02:13:07.416934 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.412590    2558 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 16 02:13:07.416934 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.414949    2558 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 16 02:13:07.416934 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.416003    2558 server.go:1019] "Starting client certificate rotation"
Sep 16 02:13:07.416934 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.416106    2558 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 16 02:13:07.417300 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.416147    2558 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 16 02:13:07.418876 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.418862    2558 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 16 02:13:07.420833 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.420815    2558 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 16 02:13:07.433452 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.433433    2558 log.go:25] "Validated CRI v1 runtime API"
Sep 16 02:13:07.437183 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.437169    2558 log.go:25] "Validated CRI v1 image API"
Sep 16 02:13:07.439484 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.439467    2558 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 16 02:13:07.443656 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.443628    2558 fs.go:135] Filesystem UUIDs: map[6a2f2d57-7075-4a26-8e4a-f2d0b5fe1f65:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 bd8d7f6d-8d82-4c4b-8a3c-71577c4ec6ab:/dev/nvme0n1p4]
Sep 16 02:13:07.443656 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.443650    2558 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 16 02:13:07.447977 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.447960    2558 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 16 02:13:07.449926 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.449807    2558 manager.go:216] Machine: {Timestamp:2026-09-16 02:13:07.447269069 +0000 UTC m=+0.365758310 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3096878 MemoryCapacity:33164468224 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2e9d8b6c0bdba6c5f62db229c5bd91 SystemUUID:ec2e9d8b-6c0b-dba6-c5f6-2db229c5bd91 BootID:bf600a45-af19-4e21-938f-ec7bfff47b7c Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582234112 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582234112 Type:vfs Inodes:4048397 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632894464 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:e8:6e:72:23 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:e8:6e:72:23 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ca:2f:ce:bd:08:dc Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164468224 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 16 02:13:07.449926 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.449919    2558 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 16 02:13:07.450067 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.450003    2558 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 16 02:13:07.453588 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.453561    2558 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 16 02:13:07.453739 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.453590    2558 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-140-34.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 16 02:13:07.454441 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.454430    2558 topology_manager.go:138] "Creating topology manager with none policy"
Sep 16 02:13:07.454483 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.454443    2558 container_manager_linux.go:309] "Creating device plugin manager"
Sep 16 02:13:07.454483 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.454451    2558 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 16 02:13:07.455278 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.455267    2558 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 16 02:13:07.456456 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.456446    2558 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 16 02:13:07.461136 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.461123    2558 state_mem.go:36] "Initialized new in-memory state store"
Sep 16 02:13:07.461253 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.461244    2558 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 16 02:13:07.461305 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.461274    2558 kubelet.go:486] "Attempting to sync node with API server"
Sep 16 02:13:07.461305 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.461284    2558 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 16 02:13:07.461305 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.461296    2558 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 16 02:13:07.461305 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.461306    2558 kubelet.go:398] "Adding apiserver pod source"
Sep 16 02:13:07.461458 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.461315    2558 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 16 02:13:07.461727 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.461716    2558 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 16 02:13:07.461807 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.461737    2558 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 16 02:13:07.464172 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.464152    2558 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 16 02:13:07.466202 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.466181    2558 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-8klbr"
Sep 16 02:13:07.467209 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.467195    2558 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 16 02:13:07.467252 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.467226    2558 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 16 02:13:07.470064 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.470033    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 16 02:13:07.470265 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.470253    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 16 02:13:07.470353 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.470335    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 16 02:13:07.470353 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.470351    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 16 02:13:07.470489 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.470367    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 16 02:13:07.470489 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.470376    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 16 02:13:07.470489 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.470400    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 16 02:13:07.470489 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.470409    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 16 02:13:07.470489 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.470424    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 16 02:13:07.470489 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.470442    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 16 02:13:07.470489 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.470471    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 16 02:13:07.470489 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.470478    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 16 02:13:07.470872 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:13:07.470544    2558 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 16 02:13:07.471379 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.471361    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 16 02:13:07.471442 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.471382    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 16 02:13:07.472131 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:13:07.472097    2558 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-140-34.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 16 02:13:07.473371 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.473352    2558 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-8klbr"
Sep 16 02:13:07.476538 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.476522    2558 server.go:1268] "Started kubelet"
Sep 16 02:13:07.476632 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.476542    2558 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 16 02:13:07.476722 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.476667    2558 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 16 02:13:07.476776 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.476744    2558 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 16 02:13:07.477561 ip-10-0-140-34 systemd[1]: Started Kubernetes Kubelet.
Sep 16 02:13:07.478279 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.477771    2558 server.go:310] "Adding debug handlers to kubelet server"
Sep 16 02:13:07.478279 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.478221    2558 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 16 02:13:07.479936 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.479908    2558 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-140-34.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 16 02:13:07.482555 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:13:07.482520    2558 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 16 02:13:07.485460 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.485446    2558 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 16 02:13:07.485542 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.485472    2558 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 16 02:13:07.485590 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:13:07.485559    2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-34.ec2.internal\" not found"
Sep 16 02:13:07.486252 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.486094    2558 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 16 02:13:07.486252 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.486096    2558 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 16 02:13:07.486386 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.486277    2558 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 16 02:13:07.486438 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.486401    2558 reconstruct.go:97] "Volume reconstruction finished"
Sep 16 02:13:07.486438 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.486410    2558 reconciler.go:29] "Reconciler: start to sync state"
Sep 16 02:13:07.488096 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.488081    2558 factory.go:55] Registering systemd factory
Sep 16 02:13:07.488189 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.488099    2558 factory.go:223] Registration of the systemd container factory successfully
Sep 16 02:13:07.488347 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.488330    2558 factory.go:153] Registering CRI-O factory
Sep 16 02:13:07.488347 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.488349    2558 factory.go:223] Registration of the crio container factory successfully
Sep 16 02:13:07.488483 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.488442    2558 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 16 02:13:07.488483 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.488459    2558 factory.go:103] Registering Raw factory
Sep 16 02:13:07.488483 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.488475    2558 manager.go:1196] Started watching for new ooms in manager
Sep 16 02:13:07.488606 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.488529    2558 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 02:13:07.488972 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.488958    2558 manager.go:349] Starting recovery of all containers
Sep 16 02:13:07.491989 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:13:07.491970    2558 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-140-34.ec2.internal\" not found" node="ip-10-0-140-34.ec2.internal"
Sep 16 02:13:07.501924 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.501900    2558 manager.go:354] Recovery completed
Sep 16 02:13:07.504250 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:13:07.504222    2558 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory
Sep 16 02:13:07.507294 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.507282    2558 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 02:13:07.508877 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.508831    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-34.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 02:13:07.508877 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.508865    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-34.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 02:13:07.508990 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.508888    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-34.ec2.internal" event="NodeHasSufficientPID"
Sep 16 02:13:07.509476 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.509453    2558 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 16 02:13:07.509476 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.509465    2558 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 16 02:13:07.509599 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.509480    2558 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 02:13:07.509599 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.509485    2558 state_mem.go:36] "Initialized new in-memory state store"
Sep 16 02:13:07.510228 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.510211    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-34.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 02:13:07.510303 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.510241    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-34.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 02:13:07.510303 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.510255    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-34.ec2.internal" event="NodeHasSufficientPID"
Sep 16 02:13:07.512785 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.512769    2558 policy_none.go:49] "None policy: Start"
Sep 16 02:13:07.512785 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.512787    2558 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 16 02:13:07.512904 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.512798    2558 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 16 02:13:07.514789 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.514778    2558 policy_none.go:47] "Start"
Sep 16 02:13:07.553641 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.541971    2558 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 16 02:13:07.553641 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.543192    2558 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 16 02:13:07.553641 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.543247    2558 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 16 02:13:07.553641 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.543273    2558 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 16 02:13:07.553641 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:13:07.543309    2558 kubelet.go:2472] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Sep 16 02:13:07.553641 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.545144    2558 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 02:13:07.553641 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.553271    2558 manager.go:337] "Starting Device Plugin manager"
Sep 16 02:13:07.553641 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:13:07.553320    2558 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 16 02:13:07.553641 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.553332    2558 server.go:87] "Starting device plugin registration server"
Sep 16 02:13:07.554109 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.553773    2558 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 16 02:13:07.554109 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.553786    2558 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 16 02:13:07.554109 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.553875    2558 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 16 02:13:07.554109 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.553957    2558 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 16 02:13:07.554109 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.553965    2558 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 16 02:13:07.554420 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:13:07.554403    2558 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 16 02:13:07.554491 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:13:07.554447    2558 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-140-34.ec2.internal\" not found"
Sep 16 02:13:07.643845 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.643723    2558 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-34.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-140-34.ec2.internal"]
Sep 16 02:13:07.643845 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.643831    2558 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 02:13:07.644820 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.644805    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-34.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 02:13:07.644895 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.644835    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-34.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 02:13:07.644895 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.644846    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-34.ec2.internal" event="NodeHasSufficientPID"
Sep 16 02:13:07.647287 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.647274    2558 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 02:13:07.647453 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.647438    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-34.ec2.internal"
Sep 16 02:13:07.647495 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.647469    2558 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 02:13:07.647998 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.647972    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-34.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 02:13:07.647998 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.648000    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-34.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 02:13:07.648123 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.648013    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-34.ec2.internal" event="NodeHasSufficientPID"
Sep 16 02:13:07.648123 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.647976    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-34.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 02:13:07.648123 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.648047    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-34.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 02:13:07.648123 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.648057    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-34.ec2.internal" event="NodeHasSufficientPID"
Sep 16 02:13:07.650332 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.650319    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-34.ec2.internal"
Sep 16 02:13:07.650384 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.650344    2558 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 02:13:07.651041 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.651022    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-34.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 02:13:07.651112 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.651055    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-34.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 02:13:07.651112 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.651066    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-34.ec2.internal" event="NodeHasSufficientPID"
Sep 16 02:13:07.654030 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.654006    2558 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 02:13:07.654687 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.654672    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-34.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 02:13:07.654768 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.654710    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-34.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 02:13:07.654768 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.654724    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-34.ec2.internal" event="NodeHasSufficientPID"
Sep 16 02:13:07.654768 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.654740    2558 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-140-34.ec2.internal"
Sep 16 02:13:07.662750 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.662737    2558 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-140-34.ec2.internal"
Sep 16 02:13:07.662807 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:13:07.662756    2558 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-140-34.ec2.internal\": node \"ip-10-0-140-34.ec2.internal\" not found"
Sep 16 02:13:07.673322 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:13:07.673302    2558 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-34.ec2.internal\" not found" node="ip-10-0-140-34.ec2.internal"
Sep 16 02:13:07.676174 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:13:07.676088    2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-34.ec2.internal\" not found"
Sep 16 02:13:07.677786 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:13:07.677771    2558 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-34.ec2.internal\" not found" node="ip-10-0-140-34.ec2.internal"
Sep 16 02:13:07.776594 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:13:07.776560    2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-34.ec2.internal\" not found"
Sep 16 02:13:07.787771 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.787748    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/809d69af053393799cf47db49da323fb-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-34.ec2.internal\" (UID: \"809d69af053393799cf47db49da323fb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-34.ec2.internal"
Sep 16 02:13:07.787771 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.787773    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/809d69af053393799cf47db49da323fb-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-34.ec2.internal\" (UID: \"809d69af053393799cf47db49da323fb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-34.ec2.internal"
Sep 16 02:13:07.787934 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.787800    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/74fa655d1549fe38db1b28b2b10dd892-config\") pod \"kube-apiserver-proxy-ip-10-0-140-34.ec2.internal\" (UID: \"74fa655d1549fe38db1b28b2b10dd892\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-34.ec2.internal"
Sep 16 02:13:07.876920 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:13:07.876876    2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-34.ec2.internal\" not found"
Sep 16 02:13:07.888230 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.888203    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/809d69af053393799cf47db49da323fb-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-34.ec2.internal\" (UID: \"809d69af053393799cf47db49da323fb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-34.ec2.internal"
Sep 16 02:13:07.888297 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.888237    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/809d69af053393799cf47db49da323fb-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-34.ec2.internal\" (UID: \"809d69af053393799cf47db49da323fb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-34.ec2.internal"
Sep 16 02:13:07.888297 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.888255    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/74fa655d1549fe38db1b28b2b10dd892-config\") pod \"kube-apiserver-proxy-ip-10-0-140-34.ec2.internal\" (UID: \"74fa655d1549fe38db1b28b2b10dd892\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-34.ec2.internal"
Sep 16 02:13:07.888359 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.888317    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/74fa655d1549fe38db1b28b2b10dd892-config\") pod \"kube-apiserver-proxy-ip-10-0-140-34.ec2.internal\" (UID: \"74fa655d1549fe38db1b28b2b10dd892\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-34.ec2.internal"
Sep 16 02:13:07.888359 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.888318    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/809d69af053393799cf47db49da323fb-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-34.ec2.internal\" (UID: \"809d69af053393799cf47db49da323fb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-34.ec2.internal"
Sep 16 02:13:07.888359 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.888334    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/809d69af053393799cf47db49da323fb-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-34.ec2.internal\" (UID: \"809d69af053393799cf47db49da323fb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-34.ec2.internal"
Sep 16 02:13:07.976930 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.976876    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-34.ec2.internal"
Sep 16 02:13:07.978055 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:13:07.978033    2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-34.ec2.internal\" not found"
Sep 16 02:13:07.981543 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:07.981524    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-34.ec2.internal"
Sep 16 02:13:08.080491 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:13:08.080450    2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-34.ec2.internal\" not found"
Sep 16 02:13:08.188244 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:13:08.188155    2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-34.ec2.internal\" not found"
Sep 16 02:13:08.288905 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:13:08.288867    2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-34.ec2.internal\" not found"
Sep 16 02:13:08.307759 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.307735    2558 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 02:13:08.349347 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.349321    2558 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 02:13:08.387035 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.386983    2558 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-34.ec2.internal"
Sep 16 02:13:08.397907 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.397880    2558 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 16 02:13:08.399295 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.399281    2558 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-34.ec2.internal"
Sep 16 02:13:08.411142 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.411116    2558 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 16 02:13:08.415439 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.415422    2558 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 16 02:13:08.415563 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.415541    2558 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 16 02:13:08.415610 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.415585    2558 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 16 02:13:08.415643 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.415617    2558 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 16 02:13:08.415685 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.415654    2558 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 16 02:13:08.463773 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.463697    2558 apiserver.go:52] "Watching apiserver"
Sep 16 02:13:08.479195 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.479123    2558 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-15 02:08:08 +0000 UTC" deadline="2028-03-01 06:36:14.629852179 +0000 UTC"
Sep 16 02:13:08.479304 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.479196    2558 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12772h23m6.15066415s"
Sep 16 02:13:08.483075 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.483053    2558 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 16 02:13:08.485440 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.485416    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/network-metrics-daemon-ck2vw","kube-system/global-pull-secret-syncer-f227r","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n4n6g","openshift-multus/multus-84qhm","openshift-multus/multus-additional-cni-plugins-h7czj","openshift-network-diagnostics/network-check-target-mfd49","openshift-network-operator/iptables-alerter-ks4kv","openshift-ovn-kubernetes/ovnkube-node-jf4lp","kube-system/konnectivity-agent-gmvp8","kube-system/kube-apiserver-proxy-ip-10-0-140-34.ec2.internal","openshift-cluster-node-tuning-operator/tuned-bshnm","openshift-dns/node-resolver-vrf75","openshift-image-registry/node-ca-vn8jb","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-34.ec2.internal"]
Sep 16 02:13:08.485602 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.485586    2558 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 16 02:13:08.488046 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.488032    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-ks4kv"
Sep 16 02:13:08.490274 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.490257    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-vn8jb"
Sep 16 02:13:08.490529 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.490512    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 16 02:13:08.490678 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.490658    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 16 02:13:08.490790 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.490767    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 16 02:13:08.490975 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.490958    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-6hzhd\""
Sep 16 02:13:08.492580 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.492560    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n4n6g"
Sep 16 02:13:08.492580 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.492568    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c91732fe-96fc-454c-b468-e3acde4ec5a0-iptables-alerter-script\") pod \"iptables-alerter-ks4kv\" (UID: \"c91732fe-96fc-454c-b468-e3acde4ec5a0\") " pod="openshift-network-operator/iptables-alerter-ks4kv"
Sep 16 02:13:08.492737 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.492592    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c91732fe-96fc-454c-b468-e3acde4ec5a0-host-slash\") pod \"iptables-alerter-ks4kv\" (UID: \"c91732fe-96fc-454c-b468-e3acde4ec5a0\") " pod="openshift-network-operator/iptables-alerter-ks4kv"
Sep 16 02:13:08.492737 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.492608    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkv5q\" (UniqueName: \"kubernetes.io/projected/c91732fe-96fc-454c-b468-e3acde4ec5a0-kube-api-access-jkv5q\") pod \"iptables-alerter-ks4kv\" (UID: \"c91732fe-96fc-454c-b468-e3acde4ec5a0\") " pod="openshift-network-operator/iptables-alerter-ks4kv"
Sep 16 02:13:08.493070 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.493052    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-djzp6\""
Sep 16 02:13:08.493114 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.493052    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 16 02:13:08.493114 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.493094    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 16 02:13:08.493185 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.493120    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 16 02:13:08.494852 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.494838    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-84qhm"
Sep 16 02:13:08.494971 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.494950    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 16 02:13:08.495042 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.495008    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 16 02:13:08.495099 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.495071    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 16 02:13:08.495395 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.495377    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-9nb5k\""
Sep 16 02:13:08.497110 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.497096    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 16 02:13:08.497197 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.497183    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-h7czj"
Sep 16 02:13:08.497333 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.497316    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 16 02:13:08.497393 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.497353    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-2fl9k\""
Sep 16 02:13:08.497928 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.497914    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 16 02:13:08.497968 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.497917    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 16 02:13:08.498285 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.498271    2558 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 16 02:13:08.499326 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.499312    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-tjwdl\""
Sep 16 02:13:08.499435 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.499419    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mfd49"
Sep 16 02:13:08.499522 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:13:08.499497    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mfd49" podUID="6ce8b9ee-ca3a-4f48-8e6d-64a8221d2369"
Sep 16 02:13:08.499758 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.499744    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 16 02:13:08.499758 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.499753    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 16 02:13:08.501747 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.501730    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-gmvp8"
Sep 16 02:13:08.503918 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.503900    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 16 02:13:08.504119 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.504097    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ck2vw"
Sep 16 02:13:08.504197 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:13:08.504161    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ck2vw" podUID="25e85d58-77e2-4428-b099-a5075363d56f"
Sep 16 02:13:08.504538 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.504521    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-9tckg\""
Sep 16 02:13:08.504620 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.504603    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 16 02:13:08.506491 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.506473    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f227r"
Sep 16 02:13:08.506565 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:13:08.506546    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f227r" podUID="f5cc1e46-3424-49f8-8f2c-ec44086643d9"
Sep 16 02:13:08.508972 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.508955    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-jf4lp"
Sep 16 02:13:08.511022 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.511007    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 16 02:13:08.511410 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.511395    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-bshnm"
Sep 16 02:13:08.511849 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.511828    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 16 02:13:08.511849 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.511840    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 16 02:13:08.511950 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.511846    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 16 02:13:08.512320 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.511837    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 16 02:13:08.512555 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.512531    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-2l64p\""
Sep 16 02:13:08.514592 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.512861    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 16 02:13:08.515378 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.515320    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 16 02:13:08.515378 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.515351    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 16 02:13:08.515547 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.515405    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-zffb4\""
Sep 16 02:13:08.515746 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.515730    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-vrf75"
Sep 16 02:13:08.517966 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.517950    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 16 02:13:08.518332 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.518316    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-xv64h\""
Sep 16 02:13:08.518381 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.518323    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 16 02:13:08.521978 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.521958    2558 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-kxl66"
Sep 16 02:13:08.529471 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.529449    2558 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-kxl66"
Sep 16 02:13:08.587999 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.587973    2558 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 16 02:13:08.592767 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.592744    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9049e903-7671-400a-a25a-e2cc8a151f79-var-lib-kubelet\") pod \"tuned-bshnm\" (UID: \"9049e903-7671-400a-a25a-e2cc8a151f79\") " pod="openshift-cluster-node-tuning-operator/tuned-bshnm"
Sep 16 02:13:08.592956 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.592815    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/06c470fe-f6bf-4016-957c-56969805ddd0-run-ovn\") pod \"ovnkube-node-jf4lp\" (UID: \"06c470fe-f6bf-4016-957c-56969805ddd0\") " pod="openshift-ovn-kubernetes/ovnkube-node-jf4lp"
Sep 16 02:13:08.592956 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.592855    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/58f0674c-b5b4-4d0e-8d14-3b18a232d365-host-run-k8s-cni-cncf-io\") pod \"multus-84qhm\" (UID: \"58f0674c-b5b4-4d0e-8d14-3b18a232d365\") " pod="openshift-multus/multus-84qhm"
Sep 16 02:13:08.592956 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.592874    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/58f0674c-b5b4-4d0e-8d14-3b18a232d365-host-var-lib-cni-bin\") pod \"multus-84qhm\" (UID: \"58f0674c-b5b4-4d0e-8d14-3b18a232d365\") " pod="openshift-multus/multus-84qhm"
Sep 16 02:13:08.592956 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.592889    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a5be83aa-c2b7-4808-a32d-80a845f3435a-cnibin\") pod \"multus-additional-cni-plugins-h7czj\" (UID: \"a5be83aa-c2b7-4808-a32d-80a845f3435a\") " pod="openshift-multus/multus-additional-cni-plugins-h7czj"
Sep 16 02:13:08.592956 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.592907    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a5be83aa-c2b7-4808-a32d-80a845f3435a-cni-binary-copy\") pod \"multus-additional-cni-plugins-h7czj\" (UID: \"a5be83aa-c2b7-4808-a32d-80a845f3435a\") " pod="openshift-multus/multus-additional-cni-plugins-h7czj"
Sep 16 02:13:08.593179 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.592973    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/28767adf-af94-4851-947e-f0f2b21ef658-konnectivity-ca\") pod \"konnectivity-agent-gmvp8\" (UID: \"28767adf-af94-4851-947e-f0f2b21ef658\") " pod="kube-system/konnectivity-agent-gmvp8"
Sep 16 02:13:08.593179 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.593001    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/06c470fe-f6bf-4016-957c-56969805ddd0-run-systemd\") pod \"ovnkube-node-jf4lp\" (UID: \"06c470fe-f6bf-4016-957c-56969805ddd0\") " pod="openshift-ovn-kubernetes/ovnkube-node-jf4lp"
Sep 16 02:13:08.593179 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.593029    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9049e903-7671-400a-a25a-e2cc8a151f79-etc-sysctl-d\") pod \"tuned-bshnm\" (UID: \"9049e903-7671-400a-a25a-e2cc8a151f79\") " pod="openshift-cluster-node-tuning-operator/tuned-bshnm"
Sep 16 02:13:08.593179 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.593053    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/58f0674c-b5b4-4d0e-8d14-3b18a232d365-multus-daemon-config\") pod \"multus-84qhm\" (UID: \"58f0674c-b5b4-4d0e-8d14-3b18a232d365\") " pod="openshift-multus/multus-84qhm"
Sep 16 02:13:08.593179 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.593070    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/06c470fe-f6bf-4016-957c-56969805ddd0-env-overrides\") pod \"ovnkube-node-jf4lp\" (UID: \"06c470fe-f6bf-4016-957c-56969805ddd0\") " pod="openshift-ovn-kubernetes/ovnkube-node-jf4lp"
Sep 16 02:13:08.593179 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.593095    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9049e903-7671-400a-a25a-e2cc8a151f79-host\") pod \"tuned-bshnm\" (UID: \"9049e903-7671-400a-a25a-e2cc8a151f79\") " pod="openshift-cluster-node-tuning-operator/tuned-bshnm"
Sep 16 02:13:08.593179 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.593121    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9562b5b9-856c-4a81-89d3-8faed08f792c-host\") pod \"node-ca-vn8jb\" (UID: \"9562b5b9-856c-4a81-89d3-8faed08f792c\") " pod="openshift-image-registry/node-ca-vn8jb"
Sep 16 02:13:08.593179 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.593142    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/06c470fe-f6bf-4016-957c-56969805ddd0-run-openvswitch\") pod \"ovnkube-node-jf4lp\" (UID: \"06c470fe-f6bf-4016-957c-56969805ddd0\") " pod="openshift-ovn-kubernetes/ovnkube-node-jf4lp"
Sep 16 02:13:08.593179 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.593168    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79nwn\" (UniqueName: \"kubernetes.io/projected/9049e903-7671-400a-a25a-e2cc8a151f79-kube-api-access-79nwn\") pod \"tuned-bshnm\" (UID: \"9049e903-7671-400a-a25a-e2cc8a151f79\") " pod="openshift-cluster-node-tuning-operator/tuned-bshnm"
Sep 16 02:13:08.593553 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.593191    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e0800020-f9de-4dfa-9a7b-d7c88ef0561f-hosts-file\") pod \"node-resolver-vrf75\" (UID: \"e0800020-f9de-4dfa-9a7b-d7c88ef0561f\") " pod="openshift-dns/node-resolver-vrf75"
Sep 16 02:13:08.593553 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.593221    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkv5q\" (UniqueName: \"kubernetes.io/projected/c91732fe-96fc-454c-b468-e3acde4ec5a0-kube-api-access-jkv5q\") pod \"iptables-alerter-ks4kv\" (UID: \"c91732fe-96fc-454c-b468-e3acde4ec5a0\") " pod="openshift-network-operator/iptables-alerter-ks4kv"
Sep 16 02:13:08.593553 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.593247    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/06c470fe-f6bf-4016-957c-56969805ddd0-ovn-node-metrics-cert\") pod \"ovnkube-node-jf4lp\" (UID: \"06c470fe-f6bf-4016-957c-56969805ddd0\") " pod="openshift-ovn-kubernetes/ovnkube-node-jf4lp"
Sep 16 02:13:08.593553 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.593272    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9049e903-7671-400a-a25a-e2cc8a151f79-etc-kubernetes\") pod \"tuned-bshnm\" (UID: \"9049e903-7671-400a-a25a-e2cc8a151f79\") " pod="openshift-cluster-node-tuning-operator/tuned-bshnm"
Sep 16 02:13:08.593553 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.593319    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9049e903-7671-400a-a25a-e2cc8a151f79-lib-modules\") pod \"tuned-bshnm\" (UID: \"9049e903-7671-400a-a25a-e2cc8a151f79\") " pod="openshift-cluster-node-tuning-operator/tuned-bshnm"
Sep 16 02:13:08.593553 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.593350    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9049e903-7671-400a-a25a-e2cc8a151f79-tmp\") pod \"tuned-bshnm\" (UID: \"9049e903-7671-400a-a25a-e2cc8a151f79\") " pod="openshift-cluster-node-tuning-operator/tuned-bshnm"
Sep 16 02:13:08.593553 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.593374    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/58f0674c-b5b4-4d0e-8d14-3b18a232d365-system-cni-dir\") pod \"multus-84qhm\" (UID: \"58f0674c-b5b4-4d0e-8d14-3b18a232d365\") " pod="openshift-multus/multus-84qhm"
Sep 16 02:13:08.593553 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.593419    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/58f0674c-b5b4-4d0e-8d14-3b18a232d365-multus-conf-dir\") pod \"multus-84qhm\" (UID: \"58f0674c-b5b4-4d0e-8d14-3b18a232d365\") " pod="openshift-multus/multus-84qhm"
Sep 16 02:13:08.593553 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.593444    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a5be83aa-c2b7-4808-a32d-80a845f3435a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-h7czj\" (UID: \"a5be83aa-c2b7-4808-a32d-80a845f3435a\") " pod="openshift-multus/multus-additional-cni-plugins-h7czj"
Sep 16 02:13:08.593553 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.593473    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/06c470fe-f6bf-4016-957c-56969805ddd0-systemd-units\") pod \"ovnkube-node-jf4lp\" (UID: \"06c470fe-f6bf-4016-957c-56969805ddd0\") " pod="openshift-ovn-kubernetes/ovnkube-node-jf4lp"
Sep 16 02:13:08.593553 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.593497    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/06c470fe-f6bf-4016-957c-56969805ddd0-host-run-netns\") pod \"ovnkube-node-jf4lp\" (UID: \"06c470fe-f6bf-4016-957c-56969805ddd0\") " pod="openshift-ovn-kubernetes/ovnkube-node-jf4lp"
Sep 16 02:13:08.593553 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.593521    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/06c470fe-f6bf-4016-957c-56969805ddd0-ovnkube-script-lib\") pod \"ovnkube-node-jf4lp\" (UID: \"06c470fe-f6bf-4016-957c-56969805ddd0\") " pod="openshift-ovn-kubernetes/ovnkube-node-jf4lp"
Sep 16 02:13:08.594052 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.593579    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/58f0674c-b5b4-4d0e-8d14-3b18a232d365-multus-cni-dir\") pod \"multus-84qhm\" (UID: \"58f0674c-b5b4-4d0e-8d14-3b18a232d365\") " pod="openshift-multus/multus-84qhm"
Sep 16 02:13:08.594052 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.593605    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/58f0674c-b5b4-4d0e-8d14-3b18a232d365-cnibin\") pod \"multus-84qhm\" (UID: \"58f0674c-b5b4-4d0e-8d14-3b18a232d365\") " pod="openshift-multus/multus-84qhm"
Sep 16 02:13:08.594052 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.593621    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a5be83aa-c2b7-4808-a32d-80a845f3435a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-h7czj\" (UID: \"a5be83aa-c2b7-4808-a32d-80a845f3435a\") " pod="openshift-multus/multus-additional-cni-plugins-h7czj"
Sep 16 02:13:08.594052 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.593641    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f5cc1e46-3424-49f8-8f2c-ec44086643d9-original-pull-secret\") pod \"global-pull-secret-syncer-f227r\" (UID: \"f5cc1e46-3424-49f8-8f2c-ec44086643d9\") " pod="kube-system/global-pull-secret-syncer-f227r"
Sep 16 02:13:08.594052 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.593674    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/06c470fe-f6bf-4016-957c-56969805ddd0-host-cni-netd\") pod \"ovnkube-node-jf4lp\" (UID: \"06c470fe-f6bf-4016-957c-56969805ddd0\") " pod="openshift-ovn-kubernetes/ovnkube-node-jf4lp"
Sep 16 02:13:08.594052 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.593695    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/58f0674c-b5b4-4d0e-8d14-3b18a232d365-host-run-netns\") pod \"multus-84qhm\" (UID: \"58f0674c-b5b4-4d0e-8d14-3b18a232d365\") " pod="openshift-multus/multus-84qhm"
Sep 16 02:13:08.594052 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.593724    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/58f0674c-b5b4-4d0e-8d14-3b18a232d365-etc-kubernetes\") pod \"multus-84qhm\" (UID: \"58f0674c-b5b4-4d0e-8d14-3b18a232d365\") " pod="openshift-multus/multus-84qhm"
Sep 16 02:13:08.594052 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.593741    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a5be83aa-c2b7-4808-a32d-80a845f3435a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-h7czj\" (UID: \"a5be83aa-c2b7-4808-a32d-80a845f3435a\") " pod="openshift-multus/multus-additional-cni-plugins-h7czj"
Sep 16 02:13:08.594052 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.593757    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/868ab659-0198-446b-b794-5d0d998d69bf-device-dir\") pod \"aws-ebs-csi-driver-node-n4n6g\" (UID: \"868ab659-0198-446b-b794-5d0d998d69bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n4n6g"
Sep 16 02:13:08.594052 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.593771    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/868ab659-0198-446b-b794-5d0d998d69bf-etc-selinux\") pod \"aws-ebs-csi-driver-node-n4n6g\" (UID: \"868ab659-0198-446b-b794-5d0d998d69bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n4n6g"
Sep 16 02:13:08.594052 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.593789    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c91732fe-96fc-454c-b468-e3acde4ec5a0-iptables-alerter-script\") pod \"iptables-alerter-ks4kv\" (UID: \"c91732fe-96fc-454c-b468-e3acde4ec5a0\") " pod="openshift-network-operator/iptables-alerter-ks4kv"
Sep 16 02:13:08.594052 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.593815    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/06c470fe-f6bf-4016-957c-56969805ddd0-var-lib-openvswitch\") pod \"ovnkube-node-jf4lp\" (UID: \"06c470fe-f6bf-4016-957c-56969805ddd0\") " pod="openshift-ovn-kubernetes/ovnkube-node-jf4lp"
Sep 16 02:13:08.594052 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.593834    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/868ab659-0198-446b-b794-5d0d998d69bf-kubelet-dir\") pod \"aws-ebs-csi-driver-node-n4n6g\" (UID: \"868ab659-0198-446b-b794-5d0d998d69bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n4n6g"
Sep 16 02:13:08.594052 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.593848    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/868ab659-0198-446b-b794-5d0d998d69bf-socket-dir\") pod \"aws-ebs-csi-driver-node-n4n6g\" (UID: \"868ab659-0198-446b-b794-5d0d998d69bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n4n6g"
Sep 16 02:13:08.594052 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.593863    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f5cc1e46-3424-49f8-8f2c-ec44086643d9-dbus\") pod \"global-pull-secret-syncer-f227r\" (UID: \"f5cc1e46-3424-49f8-8f2c-ec44086643d9\") " pod="kube-system/global-pull-secret-syncer-f227r"
Sep 16 02:13:08.594052 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.593900    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/06c470fe-f6bf-4016-957c-56969805ddd0-host-slash\") pod \"ovnkube-node-jf4lp\" (UID: \"06c470fe-f6bf-4016-957c-56969805ddd0\") " pod="openshift-ovn-kubernetes/ovnkube-node-jf4lp"
Sep 16 02:13:08.594817 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.593938    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/06c470fe-f6bf-4016-957c-56969805ddd0-host-run-ovn-kubernetes\") pod \"ovnkube-node-jf4lp\" (UID: \"06c470fe-f6bf-4016-957c-56969805ddd0\") " pod="openshift-ovn-kubernetes/ovnkube-node-jf4lp"
Sep 16 02:13:08.594817 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.593968    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9049e903-7671-400a-a25a-e2cc8a151f79-etc-sysconfig\") pod \"tuned-bshnm\" (UID: \"9049e903-7671-400a-a25a-e2cc8a151f79\") " pod="openshift-cluster-node-tuning-operator/tuned-bshnm"
Sep 16 02:13:08.594817 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.593994    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9049e903-7671-400a-a25a-e2cc8a151f79-sys\") pod \"tuned-bshnm\" (UID: \"9049e903-7671-400a-a25a-e2cc8a151f79\") " pod="openshift-cluster-node-tuning-operator/tuned-bshnm"
Sep 16 02:13:08.594817 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.594028    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9049e903-7671-400a-a25a-e2cc8a151f79-etc-tuned\") pod \"tuned-bshnm\" (UID: \"9049e903-7671-400a-a25a-e2cc8a151f79\") " pod="openshift-cluster-node-tuning-operator/tuned-bshnm"
Sep 16 02:13:08.594817 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.594054    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/25e85d58-77e2-4428-b099-a5075363d56f-metrics-certs\") pod \"network-metrics-daemon-ck2vw\" (UID: \"25e85d58-77e2-4428-b099-a5075363d56f\") " pod="openshift-multus/network-metrics-daemon-ck2vw"
Sep 16 02:13:08.594817 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.594073    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzqpm\" (UniqueName: \"kubernetes.io/projected/868ab659-0198-446b-b794-5d0d998d69bf-kube-api-access-hzqpm\") pod \"aws-ebs-csi-driver-node-n4n6g\" (UID: \"868ab659-0198-446b-b794-5d0d998d69bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n4n6g"
Sep 16 02:13:08.594817 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.594089    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f5cc1e46-3424-49f8-8f2c-ec44086643d9-kubelet-config\") pod \"global-pull-secret-syncer-f227r\" (UID: \"f5cc1e46-3424-49f8-8f2c-ec44086643d9\") " pod="kube-system/global-pull-secret-syncer-f227r"
Sep 16 02:13:08.594817 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.594116    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/06c470fe-f6bf-4016-957c-56969805ddd0-node-log\") pod \"ovnkube-node-jf4lp\" (UID: \"06c470fe-f6bf-4016-957c-56969805ddd0\") " pod="openshift-ovn-kubernetes/ovnkube-node-jf4lp"
Sep 16 02:13:08.594817 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.594131    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/06c470fe-f6bf-4016-957c-56969805ddd0-log-socket\") pod \"ovnkube-node-jf4lp\" (UID: \"06c470fe-f6bf-4016-957c-56969805ddd0\") " pod="openshift-ovn-kubernetes/ovnkube-node-jf4lp"
Sep 16 02:13:08.594817 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.594150    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9049e903-7671-400a-a25a-e2cc8a151f79-etc-modprobe-d\") pod \"tuned-bshnm\" (UID: \"9049e903-7671-400a-a25a-e2cc8a151f79\") " pod="openshift-cluster-node-tuning-operator/tuned-bshnm"
Sep 16 02:13:08.594817 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.594165    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9049e903-7671-400a-a25a-e2cc8a151f79-etc-systemd\") pod \"tuned-bshnm\" (UID: \"9049e903-7671-400a-a25a-e2cc8a151f79\") " pod="openshift-cluster-node-tuning-operator/tuned-bshnm"
Sep 16 02:13:08.594817 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.594227    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nchfw\" (UniqueName: \"kubernetes.io/projected/9562b5b9-856c-4a81-89d3-8faed08f792c-kube-api-access-nchfw\") pod \"node-ca-vn8jb\" (UID: \"9562b5b9-856c-4a81-89d3-8faed08f792c\") " pod="openshift-image-registry/node-ca-vn8jb"
Sep 16 02:13:08.594817 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.594256    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/58f0674c-b5b4-4d0e-8d14-3b18a232d365-cni-binary-copy\") pod \"multus-84qhm\" (UID: \"58f0674c-b5b4-4d0e-8d14-3b18a232d365\") " pod="openshift-multus/multus-84qhm"
Sep 16 02:13:08.594817 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.594281    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/58f0674c-b5b4-4d0e-8d14-3b18a232d365-host-run-multus-certs\") pod \"multus-84qhm\" (UID: \"58f0674c-b5b4-4d0e-8d14-3b18a232d365\") " pod="openshift-multus/multus-84qhm"
Sep 16 02:13:08.594817 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.594305    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/28767adf-af94-4851-947e-f0f2b21ef658-agent-certs\") pod \"konnectivity-agent-gmvp8\" (UID: \"28767adf-af94-4851-947e-f0f2b21ef658\") " pod="kube-system/konnectivity-agent-gmvp8"
Sep 16 02:13:08.594817 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.594281    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c91732fe-96fc-454c-b468-e3acde4ec5a0-iptables-alerter-script\") pod \"iptables-alerter-ks4kv\" (UID: \"c91732fe-96fc-454c-b468-e3acde4ec5a0\") " pod="openshift-network-operator/iptables-alerter-ks4kv"
Sep 16 02:13:08.595378 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.594329    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/06c470fe-f6bf-4016-957c-56969805ddd0-etc-openvswitch\") pod \"ovnkube-node-jf4lp\" (UID: \"06c470fe-f6bf-4016-957c-56969805ddd0\") " pod="openshift-ovn-kubernetes/ovnkube-node-jf4lp"
Sep 16 02:13:08.595378 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.594356    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/58f0674c-b5b4-4d0e-8d14-3b18a232d365-host-var-lib-kubelet\") pod \"multus-84qhm\" (UID: \"58f0674c-b5b4-4d0e-8d14-3b18a232d365\") " pod="openshift-multus/multus-84qhm"
Sep 16 02:13:08.595378 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.594380    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ptzr\" (UniqueName: \"kubernetes.io/projected/25e85d58-77e2-4428-b099-a5075363d56f-kube-api-access-6ptzr\") pod \"network-metrics-daemon-ck2vw\" (UID: \"25e85d58-77e2-4428-b099-a5075363d56f\") " pod="openshift-multus/network-metrics-daemon-ck2vw"
Sep 16 02:13:08.595378 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.594407    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/58f0674c-b5b4-4d0e-8d14-3b18a232d365-multus-socket-dir-parent\") pod \"multus-84qhm\" (UID: \"58f0674c-b5b4-4d0e-8d14-3b18a232d365\") " pod="openshift-multus/multus-84qhm"
Sep 16 02:13:08.595378 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.594434    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/06c470fe-f6bf-4016-957c-56969805ddd0-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-jf4lp\" (UID: \"06c470fe-f6bf-4016-957c-56969805ddd0\") " pod="openshift-ovn-kubernetes/ovnkube-node-jf4lp"
Sep 16 02:13:08.595378 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.594461    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/58f0674c-b5b4-4d0e-8d14-3b18a232d365-host-var-lib-cni-multus\") pod \"multus-84qhm\" (UID: \"58f0674c-b5b4-4d0e-8d14-3b18a232d365\") " pod="openshift-multus/multus-84qhm"
Sep 16 02:13:08.595378 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.594492    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/868ab659-0198-446b-b794-5d0d998d69bf-sys-fs\") pod \"aws-ebs-csi-driver-node-n4n6g\" (UID: \"868ab659-0198-446b-b794-5d0d998d69bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n4n6g"
Sep 16 02:13:08.595378 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.594519    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c91732fe-96fc-454c-b468-e3acde4ec5a0-host-slash\") pod \"iptables-alerter-ks4kv\" (UID: \"c91732fe-96fc-454c-b468-e3acde4ec5a0\") " pod="openshift-network-operator/iptables-alerter-ks4kv"
Sep 16 02:13:08.595378 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.594545    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdqgf\" (UniqueName: \"kubernetes.io/projected/6ce8b9ee-ca3a-4f48-8e6d-64a8221d2369-kube-api-access-vdqgf\") pod \"network-check-target-mfd49\" (UID: \"6ce8b9ee-ca3a-4f48-8e6d-64a8221d2369\") " pod="openshift-network-diagnostics/network-check-target-mfd49"
Sep 16 02:13:08.595378 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.594572    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwllp\" (UniqueName: \"kubernetes.io/projected/06c470fe-f6bf-4016-957c-56969805ddd0-kube-api-access-mwllp\") pod \"ovnkube-node-jf4lp\" (UID: \"06c470fe-f6bf-4016-957c-56969805ddd0\") " pod="openshift-ovn-kubernetes/ovnkube-node-jf4lp"
Sep 16 02:13:08.595378 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.594582    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c91732fe-96fc-454c-b468-e3acde4ec5a0-host-slash\") pod \"iptables-alerter-ks4kv\" (UID: \"c91732fe-96fc-454c-b468-e3acde4ec5a0\") " pod="openshift-network-operator/iptables-alerter-ks4kv"
Sep 16 02:13:08.595378 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.594597    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/58f0674c-b5b4-4d0e-8d14-3b18a232d365-hostroot\") pod \"multus-84qhm\" (UID: \"58f0674c-b5b4-4d0e-8d14-3b18a232d365\") " pod="openshift-multus/multus-84qhm"
Sep 16 02:13:08.595378 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.594621    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwscn\" (UniqueName: \"kubernetes.io/projected/58f0674c-b5b4-4d0e-8d14-3b18a232d365-kube-api-access-hwscn\") pod \"multus-84qhm\" (UID: \"58f0674c-b5b4-4d0e-8d14-3b18a232d365\") " pod="openshift-multus/multus-84qhm"
Sep 16 02:13:08.595378 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.594648    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a5be83aa-c2b7-4808-a32d-80a845f3435a-system-cni-dir\") pod \"multus-additional-cni-plugins-h7czj\" (UID: \"a5be83aa-c2b7-4808-a32d-80a845f3435a\") " pod="openshift-multus/multus-additional-cni-plugins-h7czj"
Sep 16 02:13:08.595378 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.594672    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0800020-f9de-4dfa-9a7b-d7c88ef0561f-tmp-dir\") pod \"node-resolver-vrf75\" (UID: \"e0800020-f9de-4dfa-9a7b-d7c88ef0561f\") " pod="openshift-dns/node-resolver-vrf75"
Sep 16 02:13:08.595378 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.594696    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4cxfd\" (UniqueName: \"kubernetes.io/projected/e0800020-f9de-4dfa-9a7b-d7c88ef0561f-kube-api-access-4cxfd\") pod \"node-resolver-vrf75\" (UID: \"e0800020-f9de-4dfa-9a7b-d7c88ef0561f\") " pod="openshift-dns/node-resolver-vrf75"
Sep 16 02:13:08.595931 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.594719    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/06c470fe-f6bf-4016-957c-56969805ddd0-host-kubelet\") pod \"ovnkube-node-jf4lp\" (UID: \"06c470fe-f6bf-4016-957c-56969805ddd0\") " pod="openshift-ovn-kubernetes/ovnkube-node-jf4lp"
Sep 16 02:13:08.595931 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.594777    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/06c470fe-f6bf-4016-957c-56969805ddd0-host-cni-bin\") pod \"ovnkube-node-jf4lp\" (UID: \"06c470fe-f6bf-4016-957c-56969805ddd0\") " pod="openshift-ovn-kubernetes/ovnkube-node-jf4lp"
Sep 16 02:13:08.595931 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.594802    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/06c470fe-f6bf-4016-957c-56969805ddd0-ovnkube-config\") pod \"ovnkube-node-jf4lp\" (UID: \"06c470fe-f6bf-4016-957c-56969805ddd0\") " pod="openshift-ovn-kubernetes/ovnkube-node-jf4lp"
Sep 16 02:13:08.595931 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.594854    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9049e903-7671-400a-a25a-e2cc8a151f79-etc-sysctl-conf\") pod \"tuned-bshnm\" (UID: \"9049e903-7671-400a-a25a-e2cc8a151f79\") " pod="openshift-cluster-node-tuning-operator/tuned-bshnm"
Sep 16 02:13:08.595931 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.594882    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9562b5b9-856c-4a81-89d3-8faed08f792c-serviceca\") pod \"node-ca-vn8jb\" (UID: \"9562b5b9-856c-4a81-89d3-8faed08f792c\") " pod="openshift-image-registry/node-ca-vn8jb"
Sep 16 02:13:08.595931 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.594909    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gdk6t\" (UniqueName: \"kubernetes.io/projected/a5be83aa-c2b7-4808-a32d-80a845f3435a-kube-api-access-gdk6t\") pod \"multus-additional-cni-plugins-h7czj\" (UID: \"a5be83aa-c2b7-4808-a32d-80a845f3435a\") " pod="openshift-multus/multus-additional-cni-plugins-h7czj"
Sep 16 02:13:08.595931 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.594938    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/868ab659-0198-446b-b794-5d0d998d69bf-registration-dir\") pod \"aws-ebs-csi-driver-node-n4n6g\" (UID: \"868ab659-0198-446b-b794-5d0d998d69bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n4n6g"
Sep 16 02:13:08.595931 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.594969    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9049e903-7671-400a-a25a-e2cc8a151f79-run\") pod \"tuned-bshnm\" (UID: \"9049e903-7671-400a-a25a-e2cc8a151f79\") " pod="openshift-cluster-node-tuning-operator/tuned-bshnm"
Sep 16 02:13:08.600040 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.600020    2558 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 16 02:13:08.603518 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.603494    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkv5q\" (UniqueName: \"kubernetes.io/projected/c91732fe-96fc-454c-b468-e3acde4ec5a0-kube-api-access-jkv5q\") pod \"iptables-alerter-ks4kv\" (UID: \"c91732fe-96fc-454c-b468-e3acde4ec5a0\") " pod="openshift-network-operator/iptables-alerter-ks4kv"
Sep 16 02:13:08.695190 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.695148    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/06c470fe-f6bf-4016-957c-56969805ddd0-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-jf4lp\" (UID: \"06c470fe-f6bf-4016-957c-56969805ddd0\") " pod="openshift-ovn-kubernetes/ovnkube-node-jf4lp"
Sep 16 02:13:08.695190 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.695179    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/58f0674c-b5b4-4d0e-8d14-3b18a232d365-host-var-lib-cni-multus\") pod \"multus-84qhm\" (UID: \"58f0674c-b5b4-4d0e-8d14-3b18a232d365\") " pod="openshift-multus/multus-84qhm"
Sep 16 02:13:08.695416 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.695202    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/868ab659-0198-446b-b794-5d0d998d69bf-sys-fs\") pod \"aws-ebs-csi-driver-node-n4n6g\" (UID: \"868ab659-0198-446b-b794-5d0d998d69bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n4n6g"
Sep 16 02:13:08.695416 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.695230    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdqgf\" (UniqueName: \"kubernetes.io/projected/6ce8b9ee-ca3a-4f48-8e6d-64a8221d2369-kube-api-access-vdqgf\") pod \"network-check-target-mfd49\" (UID: \"6ce8b9ee-ca3a-4f48-8e6d-64a8221d2369\") " pod="openshift-network-diagnostics/network-check-target-mfd49"
Sep 16 02:13:08.695416 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.695250    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwllp\" (UniqueName: \"kubernetes.io/projected/06c470fe-f6bf-4016-957c-56969805ddd0-kube-api-access-mwllp\") pod \"ovnkube-node-jf4lp\" (UID: \"06c470fe-f6bf-4016-957c-56969805ddd0\") " pod="openshift-ovn-kubernetes/ovnkube-node-jf4lp"
Sep 16 02:13:08.695416 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.695273    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/58f0674c-b5b4-4d0e-8d14-3b18a232d365-hostroot\") pod \"multus-84qhm\" (UID: \"58f0674c-b5b4-4d0e-8d14-3b18a232d365\") " pod="openshift-multus/multus-84qhm"
Sep 16 02:13:08.695416 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.695281    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/58f0674c-b5b4-4d0e-8d14-3b18a232d365-host-var-lib-cni-multus\") pod \"multus-84qhm\" (UID: \"58f0674c-b5b4-4d0e-8d14-3b18a232d365\") " pod="openshift-multus/multus-84qhm"
Sep 16 02:13:08.695416 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.695288    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/868ab659-0198-446b-b794-5d0d998d69bf-sys-fs\") pod \"aws-ebs-csi-driver-node-n4n6g\" (UID: \"868ab659-0198-446b-b794-5d0d998d69bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n4n6g"
Sep 16 02:13:08.695416 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.695245    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/06c470fe-f6bf-4016-957c-56969805ddd0-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-jf4lp\" (UID: \"06c470fe-f6bf-4016-957c-56969805ddd0\") " pod="openshift-ovn-kubernetes/ovnkube-node-jf4lp"
Sep 16 02:13:08.695416 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.695316    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwscn\" (UniqueName: \"kubernetes.io/projected/58f0674c-b5b4-4d0e-8d14-3b18a232d365-kube-api-access-hwscn\") pod \"multus-84qhm\" (UID: \"58f0674c-b5b4-4d0e-8d14-3b18a232d365\") " pod="openshift-multus/multus-84qhm"
Sep 16 02:13:08.695416 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.695342    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a5be83aa-c2b7-4808-a32d-80a845f3435a-system-cni-dir\") pod \"multus-additional-cni-plugins-h7czj\" (UID: \"a5be83aa-c2b7-4808-a32d-80a845f3435a\") " pod="openshift-multus/multus-additional-cni-plugins-h7czj"
Sep 16 02:13:08.695416 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.695344    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/58f0674c-b5b4-4d0e-8d14-3b18a232d365-hostroot\") pod \"multus-84qhm\" (UID: \"58f0674c-b5b4-4d0e-8d14-3b18a232d365\") " pod="openshift-multus/multus-84qhm"
Sep 16 02:13:08.695914 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.695435    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a5be83aa-c2b7-4808-a32d-80a845f3435a-system-cni-dir\") pod \"multus-additional-cni-plugins-h7czj\" (UID: \"a5be83aa-c2b7-4808-a32d-80a845f3435a\") " pod="openshift-multus/multus-additional-cni-plugins-h7czj"
Sep 16 02:13:08.695914 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.695436    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0800020-f9de-4dfa-9a7b-d7c88ef0561f-tmp-dir\") pod \"node-resolver-vrf75\" (UID: \"e0800020-f9de-4dfa-9a7b-d7c88ef0561f\") " pod="openshift-dns/node-resolver-vrf75"
Sep 16 02:13:08.695914 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.695550    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4cxfd\" (UniqueName: \"kubernetes.io/projected/e0800020-f9de-4dfa-9a7b-d7c88ef0561f-kube-api-access-4cxfd\") pod \"node-resolver-vrf75\" (UID: \"e0800020-f9de-4dfa-9a7b-d7c88ef0561f\") " pod="openshift-dns/node-resolver-vrf75"
Sep 16 02:13:08.695914 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.695620    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/06c470fe-f6bf-4016-957c-56969805ddd0-host-kubelet\") pod \"ovnkube-node-jf4lp\" (UID: \"06c470fe-f6bf-4016-957c-56969805ddd0\") " pod="openshift-ovn-kubernetes/ovnkube-node-jf4lp"
Sep 16 02:13:08.695914 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.695736    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/06c470fe-f6bf-4016-957c-56969805ddd0-host-cni-bin\") pod \"ovnkube-node-jf4lp\" (UID: \"06c470fe-f6bf-4016-957c-56969805ddd0\") " pod="openshift-ovn-kubernetes/ovnkube-node-jf4lp"
Sep 16 02:13:08.695914 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.695755    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/06c470fe-f6bf-4016-957c-56969805ddd0-ovnkube-config\") pod \"ovnkube-node-jf4lp\" (UID: \"06c470fe-f6bf-4016-957c-56969805ddd0\") " pod="openshift-ovn-kubernetes/ovnkube-node-jf4lp"
Sep 16 02:13:08.695914 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.695753    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/06c470fe-f6bf-4016-957c-56969805ddd0-host-kubelet\") pod \"ovnkube-node-jf4lp\" (UID: \"06c470fe-f6bf-4016-957c-56969805ddd0\") " pod="openshift-ovn-kubernetes/ovnkube-node-jf4lp"
Sep 16 02:13:08.695914 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.695778    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9049e903-7671-400a-a25a-e2cc8a151f79-etc-sysctl-conf\") pod \"tuned-bshnm\" (UID: \"9049e903-7671-400a-a25a-e2cc8a151f79\") " pod="openshift-cluster-node-tuning-operator/tuned-bshnm"
Sep 16 02:13:08.695914 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.695795    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9562b5b9-856c-4a81-89d3-8faed08f792c-serviceca\") pod \"node-ca-vn8jb\" (UID: \"9562b5b9-856c-4a81-89d3-8faed08f792c\") " pod="openshift-image-registry/node-ca-vn8jb"
Sep 16 02:13:08.695914 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.695813    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdk6t\" (UniqueName: \"kubernetes.io/projected/a5be83aa-c2b7-4808-a32d-80a845f3435a-kube-api-access-gdk6t\") pod \"multus-additional-cni-plugins-h7czj\" (UID: \"a5be83aa-c2b7-4808-a32d-80a845f3435a\") " pod="openshift-multus/multus-additional-cni-plugins-h7czj"
Sep 16 02:13:08.695914 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.695830    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/868ab659-0198-446b-b794-5d0d998d69bf-registration-dir\") pod \"aws-ebs-csi-driver-node-n4n6g\" (UID: \"868ab659-0198-446b-b794-5d0d998d69bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n4n6g"
Sep 16 02:13:08.695914 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.695845    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9049e903-7671-400a-a25a-e2cc8a151f79-run\") pod \"tuned-bshnm\" (UID: \"9049e903-7671-400a-a25a-e2cc8a151f79\") " pod="openshift-cluster-node-tuning-operator/tuned-bshnm"
Sep 16 02:13:08.695914 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.695794    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/06c470fe-f6bf-4016-957c-56969805ddd0-host-cni-bin\") pod \"ovnkube-node-jf4lp\" (UID: \"06c470fe-f6bf-4016-957c-56969805ddd0\") " pod="openshift-ovn-kubernetes/ovnkube-node-jf4lp"
Sep 16 02:13:08.695914 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.695868    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9049e903-7671-400a-a25a-e2cc8a151f79-var-lib-kubelet\") pod \"tuned-bshnm\" (UID: \"9049e903-7671-400a-a25a-e2cc8a151f79\") " pod="openshift-cluster-node-tuning-operator/tuned-bshnm"
Sep 16 02:13:08.695914 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.695902    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9049e903-7671-400a-a25a-e2cc8a151f79-etc-sysctl-conf\") pod \"tuned-bshnm\" (UID: \"9049e903-7671-400a-a25a-e2cc8a151f79\") " pod="openshift-cluster-node-tuning-operator/tuned-bshnm"
Sep 16 02:13:08.695914 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.695921    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/868ab659-0198-446b-b794-5d0d998d69bf-registration-dir\") pod \"aws-ebs-csi-driver-node-n4n6g\" (UID: \"868ab659-0198-446b-b794-5d0d998d69bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n4n6g"
Sep 16 02:13:08.696721 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.695926    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9049e903-7671-400a-a25a-e2cc8a151f79-var-lib-kubelet\") pod \"tuned-bshnm\" (UID: \"9049e903-7671-400a-a25a-e2cc8a151f79\") " pod="openshift-cluster-node-tuning-operator/tuned-bshnm"
Sep 16 02:13:08.696721 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.695962    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/06c470fe-f6bf-4016-957c-56969805ddd0-run-ovn\") pod \"ovnkube-node-jf4lp\" (UID: \"06c470fe-f6bf-4016-957c-56969805ddd0\") " pod="openshift-ovn-kubernetes/ovnkube-node-jf4lp"
Sep 16 02:13:08.696721 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.695972    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9049e903-7671-400a-a25a-e2cc8a151f79-run\") pod \"tuned-bshnm\" (UID: \"9049e903-7671-400a-a25a-e2cc8a151f79\") " pod="openshift-cluster-node-tuning-operator/tuned-bshnm"
Sep 16 02:13:08.696721 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.695989    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/58f0674c-b5b4-4d0e-8d14-3b18a232d365-host-run-k8s-cni-cncf-io\") pod \"multus-84qhm\" (UID: \"58f0674c-b5b4-4d0e-8d14-3b18a232d365\") " pod="openshift-multus/multus-84qhm"
Sep 16 02:13:08.696721 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.696014    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/58f0674c-b5b4-4d0e-8d14-3b18a232d365-host-var-lib-cni-bin\") pod \"multus-84qhm\" (UID: \"58f0674c-b5b4-4d0e-8d14-3b18a232d365\") " pod="openshift-multus/multus-84qhm"
Sep 16 02:13:08.696721 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.696025    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/06c470fe-f6bf-4016-957c-56969805ddd0-run-ovn\") pod \"ovnkube-node-jf4lp\" (UID: \"06c470fe-f6bf-4016-957c-56969805ddd0\") " pod="openshift-ovn-kubernetes/ovnkube-node-jf4lp"
Sep 16 02:13:08.696721 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.696037    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a5be83aa-c2b7-4808-a32d-80a845f3435a-cnibin\") pod \"multus-additional-cni-plugins-h7czj\" (UID: \"a5be83aa-c2b7-4808-a32d-80a845f3435a\") " pod="openshift-multus/multus-additional-cni-plugins-h7czj"
Sep 16 02:13:08.696721 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.696048    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/58f0674c-b5b4-4d0e-8d14-3b18a232d365-host-run-k8s-cni-cncf-io\") pod \"multus-84qhm\" (UID: \"58f0674c-b5b4-4d0e-8d14-3b18a232d365\") " pod="openshift-multus/multus-84qhm"
Sep 16 02:13:08.696721 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.696054    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0800020-f9de-4dfa-9a7b-d7c88ef0561f-tmp-dir\") pod \"node-resolver-vrf75\" (UID: \"e0800020-f9de-4dfa-9a7b-d7c88ef0561f\") " pod="openshift-dns/node-resolver-vrf75"
Sep 16 02:13:08.696721 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.696101    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/58f0674c-b5b4-4d0e-8d14-3b18a232d365-host-var-lib-cni-bin\") pod \"multus-84qhm\" (UID: \"58f0674c-b5b4-4d0e-8d14-3b18a232d365\") " pod="openshift-multus/multus-84qhm"
Sep 16 02:13:08.696721 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.696110    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a5be83aa-c2b7-4808-a32d-80a845f3435a-cnibin\") pod \"multus-additional-cni-plugins-h7czj\" (UID: \"a5be83aa-c2b7-4808-a32d-80a845f3435a\") " pod="openshift-multus/multus-additional-cni-plugins-h7czj"
Sep 16 02:13:08.696721 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.696062    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a5be83aa-c2b7-4808-a32d-80a845f3435a-cni-binary-copy\") pod \"multus-additional-cni-plugins-h7czj\" (UID: \"a5be83aa-c2b7-4808-a32d-80a845f3435a\") " pod="openshift-multus/multus-additional-cni-plugins-h7czj"
Sep 16 02:13:08.696721 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.696155    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/28767adf-af94-4851-947e-f0f2b21ef658-konnectivity-ca\") pod \"konnectivity-agent-gmvp8\" (UID: \"28767adf-af94-4851-947e-f0f2b21ef658\") " pod="kube-system/konnectivity-agent-gmvp8"
Sep 16 02:13:08.696721 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.696179    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/06c470fe-f6bf-4016-957c-56969805ddd0-run-systemd\") pod \"ovnkube-node-jf4lp\" (UID: \"06c470fe-f6bf-4016-957c-56969805ddd0\") " pod="openshift-ovn-kubernetes/ovnkube-node-jf4lp"
Sep 16 02:13:08.696721 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.696204    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9049e903-7671-400a-a25a-e2cc8a151f79-etc-sysctl-d\") pod \"tuned-bshnm\" (UID: \"9049e903-7671-400a-a25a-e2cc8a151f79\") " pod="openshift-cluster-node-tuning-operator/tuned-bshnm"
Sep 16 02:13:08.696721 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.696249    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/58f0674c-b5b4-4d0e-8d14-3b18a232d365-multus-daemon-config\") pod \"multus-84qhm\" (UID: \"58f0674c-b5b4-4d0e-8d14-3b18a232d365\") " pod="openshift-multus/multus-84qhm"
Sep 16 02:13:08.696721 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.696261    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/06c470fe-f6bf-4016-957c-56969805ddd0-run-systemd\") pod \"ovnkube-node-jf4lp\" (UID: \"06c470fe-f6bf-4016-957c-56969805ddd0\") " pod="openshift-ovn-kubernetes/ovnkube-node-jf4lp"
Sep 16 02:13:08.696721 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.696278    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/06c470fe-f6bf-4016-957c-56969805ddd0-env-overrides\") pod \"ovnkube-node-jf4lp\" (UID: \"06c470fe-f6bf-4016-957c-56969805ddd0\") " pod="openshift-ovn-kubernetes/ovnkube-node-jf4lp"
Sep 16 02:13:08.697795 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.696311    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9049e903-7671-400a-a25a-e2cc8a151f79-host\") pod \"tuned-bshnm\" (UID: \"9049e903-7671-400a-a25a-e2cc8a151f79\") " pod="openshift-cluster-node-tuning-operator/tuned-bshnm"
Sep 16 02:13:08.697795 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.696336    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9562b5b9-856c-4a81-89d3-8faed08f792c-host\") pod \"node-ca-vn8jb\" (UID: \"9562b5b9-856c-4a81-89d3-8faed08f792c\") " pod="openshift-image-registry/node-ca-vn8jb"
Sep 16 02:13:08.697795 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.696364    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/06c470fe-f6bf-4016-957c-56969805ddd0-run-openvswitch\") pod \"ovnkube-node-jf4lp\" (UID: \"06c470fe-f6bf-4016-957c-56969805ddd0\") " pod="openshift-ovn-kubernetes/ovnkube-node-jf4lp"
Sep 16 02:13:08.697795 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.696379    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9049e903-7671-400a-a25a-e2cc8a151f79-etc-sysctl-d\") pod \"tuned-bshnm\" (UID: \"9049e903-7671-400a-a25a-e2cc8a151f79\") " pod="openshift-cluster-node-tuning-operator/tuned-bshnm"
Sep 16 02:13:08.697795 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.696389    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79nwn\" (UniqueName: \"kubernetes.io/projected/9049e903-7671-400a-a25a-e2cc8a151f79-kube-api-access-79nwn\") pod \"tuned-bshnm\" (UID: \"9049e903-7671-400a-a25a-e2cc8a151f79\") " pod="openshift-cluster-node-tuning-operator/tuned-bshnm"
Sep 16 02:13:08.697795 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.696428    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e0800020-f9de-4dfa-9a7b-d7c88ef0561f-hosts-file\") pod \"node-resolver-vrf75\" (UID: \"e0800020-f9de-4dfa-9a7b-d7c88ef0561f\") " pod="openshift-dns/node-resolver-vrf75"
Sep 16 02:13:08.697795 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.696452    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9562b5b9-856c-4a81-89d3-8faed08f792c-serviceca\") pod \"node-ca-vn8jb\" (UID: \"9562b5b9-856c-4a81-89d3-8faed08f792c\") " pod="openshift-image-registry/node-ca-vn8jb"
Sep 16 02:13:08.697795 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.696460    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/06c470fe-f6bf-4016-957c-56969805ddd0-ovn-node-metrics-cert\") pod \"ovnkube-node-jf4lp\" (UID: \"06c470fe-f6bf-4016-957c-56969805ddd0\") " pod="openshift-ovn-kubernetes/ovnkube-node-jf4lp"
Sep 16 02:13:08.697795 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.696490    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9049e903-7671-400a-a25a-e2cc8a151f79-etc-kubernetes\") pod \"tuned-bshnm\" (UID: \"9049e903-7671-400a-a25a-e2cc8a151f79\") " pod="openshift-cluster-node-tuning-operator/tuned-bshnm"
Sep 16 02:13:08.697795 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.696518    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9049e903-7671-400a-a25a-e2cc8a151f79-lib-modules\") pod \"tuned-bshnm\" (UID: \"9049e903-7671-400a-a25a-e2cc8a151f79\") " pod="openshift-cluster-node-tuning-operator/tuned-bshnm"
Sep 16 02:13:08.697795 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.696543    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9049e903-7671-400a-a25a-e2cc8a151f79-tmp\") pod \"tuned-bshnm\" (UID: \"9049e903-7671-400a-a25a-e2cc8a151f79\") " pod="openshift-cluster-node-tuning-operator/tuned-bshnm"
Sep 16 02:13:08.697795 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.696546    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/06c470fe-f6bf-4016-957c-56969805ddd0-ovnkube-config\") pod \"ovnkube-node-jf4lp\" (UID: \"06c470fe-f6bf-4016-957c-56969805ddd0\") " pod="openshift-ovn-kubernetes/ovnkube-node-jf4lp"
Sep 16 02:13:08.697795 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.696568    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/58f0674c-b5b4-4d0e-8d14-3b18a232d365-system-cni-dir\") pod \"multus-84qhm\" (UID: \"58f0674c-b5b4-4d0e-8d14-3b18a232d365\") " pod="openshift-multus/multus-84qhm"
Sep 16 02:13:08.697795 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.696560    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9049e903-7671-400a-a25a-e2cc8a151f79-host\") pod \"tuned-bshnm\" (UID: \"9049e903-7671-400a-a25a-e2cc8a151f79\") " pod="openshift-cluster-node-tuning-operator/tuned-bshnm"
Sep 16 02:13:08.697795 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.696519    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9562b5b9-856c-4a81-89d3-8faed08f792c-host\") pod \"node-ca-vn8jb\" (UID: \"9562b5b9-856c-4a81-89d3-8faed08f792c\") " pod="openshift-image-registry/node-ca-vn8jb"
Sep 16 02:13:08.697795 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.696618    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/06c470fe-f6bf-4016-957c-56969805ddd0-run-openvswitch\") pod \"ovnkube-node-jf4lp\" (UID: \"06c470fe-f6bf-4016-957c-56969805ddd0\") " pod="openshift-ovn-kubernetes/ovnkube-node-jf4lp"
Sep 16 02:13:08.697795 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.696703    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9049e903-7671-400a-a25a-e2cc8a151f79-lib-modules\") pod \"tuned-bshnm\" (UID: \"9049e903-7671-400a-a25a-e2cc8a151f79\") " pod="openshift-cluster-node-tuning-operator/tuned-bshnm"
Sep 16 02:13:08.697795 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.696684    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a5be83aa-c2b7-4808-a32d-80a845f3435a-cni-binary-copy\") pod \"multus-additional-cni-plugins-h7czj\" (UID: \"a5be83aa-c2b7-4808-a32d-80a845f3435a\") " pod="openshift-multus/multus-additional-cni-plugins-h7czj"
Sep 16 02:13:08.698837 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.696720    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/28767adf-af94-4851-947e-f0f2b21ef658-konnectivity-ca\") pod \"konnectivity-agent-gmvp8\" (UID: \"28767adf-af94-4851-947e-f0f2b21ef658\") " pod="kube-system/konnectivity-agent-gmvp8"
Sep 16 02:13:08.698837 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.696707    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/06c470fe-f6bf-4016-957c-56969805ddd0-env-overrides\") pod \"ovnkube-node-jf4lp\" (UID: \"06c470fe-f6bf-4016-957c-56969805ddd0\") " pod="openshift-ovn-kubernetes/ovnkube-node-jf4lp"
Sep 16 02:13:08.698837 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.696790    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/58f0674c-b5b4-4d0e-8d14-3b18a232d365-multus-conf-dir\") pod \"multus-84qhm\" (UID: \"58f0674c-b5b4-4d0e-8d14-3b18a232d365\") " pod="openshift-multus/multus-84qhm"
Sep 16 02:13:08.698837 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.696821    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e0800020-f9de-4dfa-9a7b-d7c88ef0561f-hosts-file\") pod \"node-resolver-vrf75\" (UID: \"e0800020-f9de-4dfa-9a7b-d7c88ef0561f\") " pod="openshift-dns/node-resolver-vrf75"
Sep 16 02:13:08.698837 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.696828    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/58f0674c-b5b4-4d0e-8d14-3b18a232d365-system-cni-dir\") pod \"multus-84qhm\" (UID: \"58f0674c-b5b4-4d0e-8d14-3b18a232d365\") " pod="openshift-multus/multus-84qhm"
Sep 16 02:13:08.698837 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.696822    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a5be83aa-c2b7-4808-a32d-80a845f3435a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-h7czj\" (UID: \"a5be83aa-c2b7-4808-a32d-80a845f3435a\") " pod="openshift-multus/multus-additional-cni-plugins-h7czj"
Sep 16 02:13:08.698837 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.696870    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/58f0674c-b5b4-4d0e-8d14-3b18a232d365-multus-conf-dir\") pod \"multus-84qhm\" (UID: \"58f0674c-b5b4-4d0e-8d14-3b18a232d365\") " pod="openshift-multus/multus-84qhm"
Sep 16 02:13:08.698837 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.696870    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/06c470fe-f6bf-4016-957c-56969805ddd0-systemd-units\") pod \"ovnkube-node-jf4lp\" (UID: \"06c470fe-f6bf-4016-957c-56969805ddd0\") " pod="openshift-ovn-kubernetes/ovnkube-node-jf4lp"
Sep 16 02:13:08.698837 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.696904    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/06c470fe-f6bf-4016-957c-56969805ddd0-systemd-units\") pod \"ovnkube-node-jf4lp\" (UID: \"06c470fe-f6bf-4016-957c-56969805ddd0\") " pod="openshift-ovn-kubernetes/ovnkube-node-jf4lp"
Sep 16 02:13:08.698837 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.696837    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9049e903-7671-400a-a25a-e2cc8a151f79-etc-kubernetes\") pod \"tuned-bshnm\" (UID: \"9049e903-7671-400a-a25a-e2cc8a151f79\") " pod="openshift-cluster-node-tuning-operator/tuned-bshnm"
Sep 16 02:13:08.698837 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.696905    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/06c470fe-f6bf-4016-957c-56969805ddd0-host-run-netns\") pod \"ovnkube-node-jf4lp\" (UID: \"06c470fe-f6bf-4016-957c-56969805ddd0\") " pod="openshift-ovn-kubernetes/ovnkube-node-jf4lp"
Sep 16 02:13:08.698837 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.696947    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/06c470fe-f6bf-4016-957c-56969805ddd0-ovnkube-script-lib\") pod \"ovnkube-node-jf4lp\" (UID: \"06c470fe-f6bf-4016-957c-56969805ddd0\") " pod="openshift-ovn-kubernetes/ovnkube-node-jf4lp"
Sep 16 02:13:08.698837 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.696956    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/06c470fe-f6bf-4016-957c-56969805ddd0-host-run-netns\") pod \"ovnkube-node-jf4lp\" (UID: \"06c470fe-f6bf-4016-957c-56969805ddd0\") " pod="openshift-ovn-kubernetes/ovnkube-node-jf4lp"
Sep 16 02:13:08.698837 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.696972    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/58f0674c-b5b4-4d0e-8d14-3b18a232d365-multus-cni-dir\") pod \"multus-84qhm\" (UID: \"58f0674c-b5b4-4d0e-8d14-3b18a232d365\") " pod="openshift-multus/multus-84qhm"
Sep 16 02:13:08.698837 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.697026    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/58f0674c-b5b4-4d0e-8d14-3b18a232d365-multus-cni-dir\") pod \"multus-84qhm\" (UID: \"58f0674c-b5b4-4d0e-8d14-3b18a232d365\") " pod="openshift-multus/multus-84qhm"
Sep 16 02:13:08.698837 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.697056    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/58f0674c-b5b4-4d0e-8d14-3b18a232d365-cnibin\") pod \"multus-84qhm\" (UID: \"58f0674c-b5b4-4d0e-8d14-3b18a232d365\") " pod="openshift-multus/multus-84qhm"
Sep 16 02:13:08.698837 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.697083    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a5be83aa-c2b7-4808-a32d-80a845f3435a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-h7czj\" (UID: \"a5be83aa-c2b7-4808-a32d-80a845f3435a\") " pod="openshift-multus/multus-additional-cni-plugins-h7czj"
Sep 16 02:13:08.698837 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.697104    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/58f0674c-b5b4-4d0e-8d14-3b18a232d365-cnibin\") pod \"multus-84qhm\" (UID: \"58f0674c-b5b4-4d0e-8d14-3b18a232d365\") " pod="openshift-multus/multus-84qhm"
Sep 16 02:13:08.699685 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.697106    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/58f0674c-b5b4-4d0e-8d14-3b18a232d365-multus-daemon-config\") pod \"multus-84qhm\" (UID: \"58f0674c-b5b4-4d0e-8d14-3b18a232d365\") " pod="openshift-multus/multus-84qhm"
Sep 16 02:13:08.699685 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.697114    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f5cc1e46-3424-49f8-8f2c-ec44086643d9-original-pull-secret\") pod \"global-pull-secret-syncer-f227r\" (UID: \"f5cc1e46-3424-49f8-8f2c-ec44086643d9\") " pod="kube-system/global-pull-secret-syncer-f227r"
Sep 16 02:13:08.699685 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:13:08.697191    2558 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 02:13:08.699685 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.697245    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/06c470fe-f6bf-4016-957c-56969805ddd0-host-cni-netd\") pod \"ovnkube-node-jf4lp\" (UID: \"06c470fe-f6bf-4016-957c-56969805ddd0\") " pod="openshift-ovn-kubernetes/ovnkube-node-jf4lp"
Sep 16 02:13:08.699685 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:13:08.697266    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f5cc1e46-3424-49f8-8f2c-ec44086643d9-original-pull-secret podName:f5cc1e46-3424-49f8-8f2c-ec44086643d9 nodeName:}" failed. No retries permitted until 2026-09-16 02:13:09.197243071 +0000 UTC m=+2.115732324 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f5cc1e46-3424-49f8-8f2c-ec44086643d9-original-pull-secret") pod "global-pull-secret-syncer-f227r" (UID: "f5cc1e46-3424-49f8-8f2c-ec44086643d9") : object "kube-system"/"original-pull-secret" not registered
Sep 16 02:13:08.699685 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.697275    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/06c470fe-f6bf-4016-957c-56969805ddd0-host-cni-netd\") pod \"ovnkube-node-jf4lp\" (UID: \"06c470fe-f6bf-4016-957c-56969805ddd0\") " pod="openshift-ovn-kubernetes/ovnkube-node-jf4lp"
Sep 16 02:13:08.699685 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.697376    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/58f0674c-b5b4-4d0e-8d14-3b18a232d365-host-run-netns\") pod \"multus-84qhm\" (UID: \"58f0674c-b5b4-4d0e-8d14-3b18a232d365\") " pod="openshift-multus/multus-84qhm"
Sep 16 02:13:08.699685 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.697436    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/58f0674c-b5b4-4d0e-8d14-3b18a232d365-host-run-netns\") pod \"multus-84qhm\" (UID: \"58f0674c-b5b4-4d0e-8d14-3b18a232d365\") " pod="openshift-multus/multus-84qhm"
Sep 16 02:13:08.699685 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.697457    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/06c470fe-f6bf-4016-957c-56969805ddd0-ovnkube-script-lib\") pod \"ovnkube-node-jf4lp\" (UID: \"06c470fe-f6bf-4016-957c-56969805ddd0\") " pod="openshift-ovn-kubernetes/ovnkube-node-jf4lp"
Sep 16 02:13:08.699685 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.697477    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/58f0674c-b5b4-4d0e-8d14-3b18a232d365-etc-kubernetes\") pod \"multus-84qhm\" (UID: \"58f0674c-b5b4-4d0e-8d14-3b18a232d365\") " pod="openshift-multus/multus-84qhm"
Sep 16 02:13:08.699685 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.697506    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a5be83aa-c2b7-4808-a32d-80a845f3435a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-h7czj\" (UID: \"a5be83aa-c2b7-4808-a32d-80a845f3435a\") " pod="openshift-multus/multus-additional-cni-plugins-h7czj"
Sep 16 02:13:08.699685 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.697523    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/58f0674c-b5b4-4d0e-8d14-3b18a232d365-etc-kubernetes\") pod \"multus-84qhm\" (UID: \"58f0674c-b5b4-4d0e-8d14-3b18a232d365\") " pod="openshift-multus/multus-84qhm"
Sep 16 02:13:08.699685 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.697535    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/868ab659-0198-446b-b794-5d0d998d69bf-device-dir\") pod \"aws-ebs-csi-driver-node-n4n6g\" (UID: \"868ab659-0198-446b-b794-5d0d998d69bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n4n6g"
Sep 16 02:13:08.699685 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.697565    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a5be83aa-c2b7-4808-a32d-80a845f3435a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-h7czj\" (UID: \"a5be83aa-c2b7-4808-a32d-80a845f3435a\") " pod="openshift-multus/multus-additional-cni-plugins-h7czj"
Sep 16 02:13:08.699685 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.697561    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/868ab659-0198-446b-b794-5d0d998d69bf-etc-selinux\") pod \"aws-ebs-csi-driver-node-n4n6g\" (UID: \"868ab659-0198-446b-b794-5d0d998d69bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n4n6g"
Sep 16 02:13:08.699685 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.697613    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/868ab659-0198-446b-b794-5d0d998d69bf-device-dir\") pod \"aws-ebs-csi-driver-node-n4n6g\" (UID: \"868ab659-0198-446b-b794-5d0d998d69bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n4n6g"
Sep 16 02:13:08.699685 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.697624    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/06c470fe-f6bf-4016-957c-56969805ddd0-var-lib-openvswitch\") pod \"ovnkube-node-jf4lp\" (UID: \"06c470fe-f6bf-4016-957c-56969805ddd0\") " pod="openshift-ovn-kubernetes/ovnkube-node-jf4lp"
Sep 16 02:13:08.700319 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.697660    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/06c470fe-f6bf-4016-957c-56969805ddd0-var-lib-openvswitch\") pod \"ovnkube-node-jf4lp\" (UID: \"06c470fe-f6bf-4016-957c-56969805ddd0\") " pod="openshift-ovn-kubernetes/ovnkube-node-jf4lp"
Sep 16 02:13:08.700319 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.697663    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/868ab659-0198-446b-b794-5d0d998d69bf-kubelet-dir\") pod \"aws-ebs-csi-driver-node-n4n6g\" (UID: \"868ab659-0198-446b-b794-5d0d998d69bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n4n6g"
Sep 16 02:13:08.700319 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.697682    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a5be83aa-c2b7-4808-a32d-80a845f3435a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-h7czj\" (UID: \"a5be83aa-c2b7-4808-a32d-80a845f3435a\") " pod="openshift-multus/multus-additional-cni-plugins-h7czj"
Sep 16 02:13:08.700319 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.697694    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/868ab659-0198-446b-b794-5d0d998d69bf-socket-dir\") pod \"aws-ebs-csi-driver-node-n4n6g\" (UID: \"868ab659-0198-446b-b794-5d0d998d69bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n4n6g"
Sep 16 02:13:08.700319 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.697704    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/868ab659-0198-446b-b794-5d0d998d69bf-kubelet-dir\") pod \"aws-ebs-csi-driver-node-n4n6g\" (UID: \"868ab659-0198-446b-b794-5d0d998d69bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n4n6g"
Sep 16 02:13:08.700319 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.697685    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/868ab659-0198-446b-b794-5d0d998d69bf-etc-selinux\") pod \"aws-ebs-csi-driver-node-n4n6g\" (UID: \"868ab659-0198-446b-b794-5d0d998d69bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n4n6g"
Sep 16 02:13:08.700319 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.697737    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f5cc1e46-3424-49f8-8f2c-ec44086643d9-dbus\") pod \"global-pull-secret-syncer-f227r\" (UID: \"f5cc1e46-3424-49f8-8f2c-ec44086643d9\") " pod="kube-system/global-pull-secret-syncer-f227r"
Sep 16 02:13:08.700319 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.697754    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/06c470fe-f6bf-4016-957c-56969805ddd0-host-slash\") pod \"ovnkube-node-jf4lp\" (UID: \"06c470fe-f6bf-4016-957c-56969805ddd0\") " pod="openshift-ovn-kubernetes/ovnkube-node-jf4lp"
Sep 16 02:13:08.700319 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.697778    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/06c470fe-f6bf-4016-957c-56969805ddd0-host-run-ovn-kubernetes\") pod \"ovnkube-node-jf4lp\" (UID: \"06c470fe-f6bf-4016-957c-56969805ddd0\") " pod="openshift-ovn-kubernetes/ovnkube-node-jf4lp"
Sep 16 02:13:08.700319 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.697806    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a5be83aa-c2b7-4808-a32d-80a845f3435a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-h7czj\" (UID: \"a5be83aa-c2b7-4808-a32d-80a845f3435a\") " pod="openshift-multus/multus-additional-cni-plugins-h7czj"
Sep 16 02:13:08.700319 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.697835    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/06c470fe-f6bf-4016-957c-56969805ddd0-host-run-ovn-kubernetes\") pod \"ovnkube-node-jf4lp\" (UID: \"06c470fe-f6bf-4016-957c-56969805ddd0\") " pod="openshift-ovn-kubernetes/ovnkube-node-jf4lp"
Sep 16 02:13:08.700319 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.697849    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9049e903-7671-400a-a25a-e2cc8a151f79-etc-sysconfig\") pod \"tuned-bshnm\" (UID: \"9049e903-7671-400a-a25a-e2cc8a151f79\") " pod="openshift-cluster-node-tuning-operator/tuned-bshnm"
Sep 16 02:13:08.700319 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.697802    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9049e903-7671-400a-a25a-e2cc8a151f79-etc-sysconfig\") pod \"tuned-bshnm\" (UID: \"9049e903-7671-400a-a25a-e2cc8a151f79\") " pod="openshift-cluster-node-tuning-operator/tuned-bshnm"
Sep 16 02:13:08.700319 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.697835    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/06c470fe-f6bf-4016-957c-56969805ddd0-host-slash\") pod \"ovnkube-node-jf4lp\" (UID: \"06c470fe-f6bf-4016-957c-56969805ddd0\") " pod="openshift-ovn-kubernetes/ovnkube-node-jf4lp"
Sep 16 02:13:08.700319 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.697875    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9049e903-7671-400a-a25a-e2cc8a151f79-sys\") pod \"tuned-bshnm\" (UID: \"9049e903-7671-400a-a25a-e2cc8a151f79\") " pod="openshift-cluster-node-tuning-operator/tuned-bshnm"
Sep 16 02:13:08.700319 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.697894    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f5cc1e46-3424-49f8-8f2c-ec44086643d9-dbus\") pod \"global-pull-secret-syncer-f227r\" (UID: \"f5cc1e46-3424-49f8-8f2c-ec44086643d9\") " pod="kube-system/global-pull-secret-syncer-f227r"
Sep 16 02:13:08.700319 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.697913    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9049e903-7671-400a-a25a-e2cc8a151f79-sys\") pod \"tuned-bshnm\" (UID: \"9049e903-7671-400a-a25a-e2cc8a151f79\") " pod="openshift-cluster-node-tuning-operator/tuned-bshnm"
Sep 16 02:13:08.700854 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.697919    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9049e903-7671-400a-a25a-e2cc8a151f79-etc-tuned\") pod \"tuned-bshnm\" (UID: \"9049e903-7671-400a-a25a-e2cc8a151f79\") " pod="openshift-cluster-node-tuning-operator/tuned-bshnm"
Sep 16 02:13:08.700854 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.697936    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/868ab659-0198-446b-b794-5d0d998d69bf-socket-dir\") pod \"aws-ebs-csi-driver-node-n4n6g\" (UID: \"868ab659-0198-446b-b794-5d0d998d69bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n4n6g"
Sep 16 02:13:08.700854 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.697951    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/25e85d58-77e2-4428-b099-a5075363d56f-metrics-certs\") pod \"network-metrics-daemon-ck2vw\" (UID: \"25e85d58-77e2-4428-b099-a5075363d56f\") " pod="openshift-multus/network-metrics-daemon-ck2vw"
Sep 16 02:13:08.700854 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.697969    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzqpm\" (UniqueName: \"kubernetes.io/projected/868ab659-0198-446b-b794-5d0d998d69bf-kube-api-access-hzqpm\") pod \"aws-ebs-csi-driver-node-n4n6g\" (UID: \"868ab659-0198-446b-b794-5d0d998d69bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n4n6g"
Sep 16 02:13:08.700854 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.697985    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f5cc1e46-3424-49f8-8f2c-ec44086643d9-kubelet-config\") pod \"global-pull-secret-syncer-f227r\" (UID: \"f5cc1e46-3424-49f8-8f2c-ec44086643d9\") " pod="kube-system/global-pull-secret-syncer-f227r"
Sep 16 02:13:08.700854 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.698013    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/06c470fe-f6bf-4016-957c-56969805ddd0-node-log\") pod \"ovnkube-node-jf4lp\" (UID: \"06c470fe-f6bf-4016-957c-56969805ddd0\") " pod="openshift-ovn-kubernetes/ovnkube-node-jf4lp"
Sep 16 02:13:08.700854 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.698041    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/06c470fe-f6bf-4016-957c-56969805ddd0-log-socket\") pod \"ovnkube-node-jf4lp\" (UID: \"06c470fe-f6bf-4016-957c-56969805ddd0\") " pod="openshift-ovn-kubernetes/ovnkube-node-jf4lp"
Sep 16 02:13:08.700854 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.698068    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9049e903-7671-400a-a25a-e2cc8a151f79-etc-modprobe-d\") pod \"tuned-bshnm\" (UID: \"9049e903-7671-400a-a25a-e2cc8a151f79\") " pod="openshift-cluster-node-tuning-operator/tuned-bshnm"
Sep 16 02:13:08.700854 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.698074    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f5cc1e46-3424-49f8-8f2c-ec44086643d9-kubelet-config\") pod \"global-pull-secret-syncer-f227r\" (UID: \"f5cc1e46-3424-49f8-8f2c-ec44086643d9\") " pod="kube-system/global-pull-secret-syncer-f227r"
Sep 16 02:13:08.700854 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.698091    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9049e903-7671-400a-a25a-e2cc8a151f79-etc-systemd\") pod \"tuned-bshnm\" (UID: \"9049e903-7671-400a-a25a-e2cc8a151f79\") " pod="openshift-cluster-node-tuning-operator/tuned-bshnm"
Sep 16 02:13:08.700854 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.698116    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nchfw\" (UniqueName: \"kubernetes.io/projected/9562b5b9-856c-4a81-89d3-8faed08f792c-kube-api-access-nchfw\") pod \"node-ca-vn8jb\" (UID: \"9562b5b9-856c-4a81-89d3-8faed08f792c\") " pod="openshift-image-registry/node-ca-vn8jb"
Sep 16 02:13:08.700854 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.698123    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/06c470fe-f6bf-4016-957c-56969805ddd0-log-socket\") pod \"ovnkube-node-jf4lp\" (UID: \"06c470fe-f6bf-4016-957c-56969805ddd0\") " pod="openshift-ovn-kubernetes/ovnkube-node-jf4lp"
Sep 16 02:13:08.700854 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.698140    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/58f0674c-b5b4-4d0e-8d14-3b18a232d365-cni-binary-copy\") pod \"multus-84qhm\" (UID: \"58f0674c-b5b4-4d0e-8d14-3b18a232d365\") " pod="openshift-multus/multus-84qhm"
Sep 16 02:13:08.700854 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.698165    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/58f0674c-b5b4-4d0e-8d14-3b18a232d365-host-run-multus-certs\") pod \"multus-84qhm\" (UID: \"58f0674c-b5b4-4d0e-8d14-3b18a232d365\") " pod="openshift-multus/multus-84qhm"
Sep 16 02:13:08.700854 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.698175    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/06c470fe-f6bf-4016-957c-56969805ddd0-node-log\") pod \"ovnkube-node-jf4lp\" (UID: \"06c470fe-f6bf-4016-957c-56969805ddd0\") " pod="openshift-ovn-kubernetes/ovnkube-node-jf4lp"
Sep 16 02:13:08.700854 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.698189    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/28767adf-af94-4851-947e-f0f2b21ef658-agent-certs\") pod \"konnectivity-agent-gmvp8\" (UID: \"28767adf-af94-4851-947e-f0f2b21ef658\") " pod="kube-system/konnectivity-agent-gmvp8"
Sep 16 02:13:08.700854 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.698216    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/06c470fe-f6bf-4016-957c-56969805ddd0-etc-openvswitch\") pod \"ovnkube-node-jf4lp\" (UID: \"06c470fe-f6bf-4016-957c-56969805ddd0\") " pod="openshift-ovn-kubernetes/ovnkube-node-jf4lp"
Sep 16 02:13:08.700854 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:13:08.698267    2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 02:13:08.701456 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.698242    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/58f0674c-b5b4-4d0e-8d14-3b18a232d365-host-var-lib-kubelet\") pod \"multus-84qhm\" (UID: \"58f0674c-b5b4-4d0e-8d14-3b18a232d365\") " pod="openshift-multus/multus-84qhm"
Sep 16 02:13:08.701456 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.698316    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ptzr\" (UniqueName: \"kubernetes.io/projected/25e85d58-77e2-4428-b099-a5075363d56f-kube-api-access-6ptzr\") pod \"network-metrics-daemon-ck2vw\" (UID: \"25e85d58-77e2-4428-b099-a5075363d56f\") " pod="openshift-multus/network-metrics-daemon-ck2vw"
Sep 16 02:13:08.701456 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:13:08.698330    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/25e85d58-77e2-4428-b099-a5075363d56f-metrics-certs podName:25e85d58-77e2-4428-b099-a5075363d56f nodeName:}" failed. No retries permitted until 2026-09-16 02:13:09.198312619 +0000 UTC m=+2.116801863 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/25e85d58-77e2-4428-b099-a5075363d56f-metrics-certs") pod "network-metrics-daemon-ck2vw" (UID: "25e85d58-77e2-4428-b099-a5075363d56f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 02:13:08.701456 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.698359    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/58f0674c-b5b4-4d0e-8d14-3b18a232d365-multus-socket-dir-parent\") pod \"multus-84qhm\" (UID: \"58f0674c-b5b4-4d0e-8d14-3b18a232d365\") " pod="openshift-multus/multus-84qhm"
Sep 16 02:13:08.701456 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.698438    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/58f0674c-b5b4-4d0e-8d14-3b18a232d365-multus-socket-dir-parent\") pod \"multus-84qhm\" (UID: \"58f0674c-b5b4-4d0e-8d14-3b18a232d365\") " pod="openshift-multus/multus-84qhm"
Sep 16 02:13:08.701456 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.698541    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/06c470fe-f6bf-4016-957c-56969805ddd0-etc-openvswitch\") pod \"ovnkube-node-jf4lp\" (UID: \"06c470fe-f6bf-4016-957c-56969805ddd0\") " pod="openshift-ovn-kubernetes/ovnkube-node-jf4lp"
Sep 16 02:13:08.701456 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.698589    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/58f0674c-b5b4-4d0e-8d14-3b18a232d365-host-var-lib-kubelet\") pod \"multus-84qhm\" (UID: \"58f0674c-b5b4-4d0e-8d14-3b18a232d365\") " pod="openshift-multus/multus-84qhm"
Sep 16 02:13:08.701456 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.698700    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/58f0674c-b5b4-4d0e-8d14-3b18a232d365-cni-binary-copy\") pod \"multus-84qhm\" (UID: \"58f0674c-b5b4-4d0e-8d14-3b18a232d365\") " pod="openshift-multus/multus-84qhm"
Sep 16 02:13:08.701456 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.698787    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/58f0674c-b5b4-4d0e-8d14-3b18a232d365-host-run-multus-certs\") pod \"multus-84qhm\" (UID: \"58f0674c-b5b4-4d0e-8d14-3b18a232d365\") " pod="openshift-multus/multus-84qhm"
Sep 16 02:13:08.701456 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.698819    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9049e903-7671-400a-a25a-e2cc8a151f79-etc-modprobe-d\") pod \"tuned-bshnm\" (UID: \"9049e903-7671-400a-a25a-e2cc8a151f79\") " pod="openshift-cluster-node-tuning-operator/tuned-bshnm"
Sep 16 02:13:08.701456 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.698860    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9049e903-7671-400a-a25a-e2cc8a151f79-etc-systemd\") pod \"tuned-bshnm\" (UID: \"9049e903-7671-400a-a25a-e2cc8a151f79\") " pod="openshift-cluster-node-tuning-operator/tuned-bshnm"
Sep 16 02:13:08.701456 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.699607    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/06c470fe-f6bf-4016-957c-56969805ddd0-ovn-node-metrics-cert\") pod \"ovnkube-node-jf4lp\" (UID: \"06c470fe-f6bf-4016-957c-56969805ddd0\") " pod="openshift-ovn-kubernetes/ovnkube-node-jf4lp"
Sep 16 02:13:08.701456 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.699625    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9049e903-7671-400a-a25a-e2cc8a151f79-tmp\") pod \"tuned-bshnm\" (UID: \"9049e903-7671-400a-a25a-e2cc8a151f79\") " pod="openshift-cluster-node-tuning-operator/tuned-bshnm"
Sep 16 02:13:08.701456 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.700259    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9049e903-7671-400a-a25a-e2cc8a151f79-etc-tuned\") pod \"tuned-bshnm\" (UID: \"9049e903-7671-400a-a25a-e2cc8a151f79\") " pod="openshift-cluster-node-tuning-operator/tuned-bshnm"
Sep 16 02:13:08.701456 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.701036    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/28767adf-af94-4851-947e-f0f2b21ef658-agent-certs\") pod \"konnectivity-agent-gmvp8\" (UID: \"28767adf-af94-4851-947e-f0f2b21ef658\") " pod="kube-system/konnectivity-agent-gmvp8"
Sep 16 02:13:08.702902 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:13:08.702867    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 02:13:08.702902 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:13:08.702895    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 02:13:08.703058 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:13:08.702910    2558 projected.go:196] Error preparing data for projected volume kube-api-access-vdqgf for pod openshift-network-diagnostics/network-check-target-mfd49: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 02:13:08.703058 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:13:08.703001    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6ce8b9ee-ca3a-4f48-8e6d-64a8221d2369-kube-api-access-vdqgf podName:6ce8b9ee-ca3a-4f48-8e6d-64a8221d2369 nodeName:}" failed. No retries permitted until 2026-09-16 02:13:09.202959647 +0000 UTC m=+2.121448898 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-vdqgf" (UniqueName: "kubernetes.io/projected/6ce8b9ee-ca3a-4f48-8e6d-64a8221d2369-kube-api-access-vdqgf") pod "network-check-target-mfd49" (UID: "6ce8b9ee-ca3a-4f48-8e6d-64a8221d2369") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 02:13:08.705583 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.705553    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4cxfd\" (UniqueName: \"kubernetes.io/projected/e0800020-f9de-4dfa-9a7b-d7c88ef0561f-kube-api-access-4cxfd\") pod \"node-resolver-vrf75\" (UID: \"e0800020-f9de-4dfa-9a7b-d7c88ef0561f\") " pod="openshift-dns/node-resolver-vrf75"
Sep 16 02:13:08.705695 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.705674    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwscn\" (UniqueName: \"kubernetes.io/projected/58f0674c-b5b4-4d0e-8d14-3b18a232d365-kube-api-access-hwscn\") pod \"multus-84qhm\" (UID: \"58f0674c-b5b4-4d0e-8d14-3b18a232d365\") " pod="openshift-multus/multus-84qhm"
Sep 16 02:13:08.706003 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.705983    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwllp\" (UniqueName: \"kubernetes.io/projected/06c470fe-f6bf-4016-957c-56969805ddd0-kube-api-access-mwllp\") pod \"ovnkube-node-jf4lp\" (UID: \"06c470fe-f6bf-4016-957c-56969805ddd0\") " pod="openshift-ovn-kubernetes/ovnkube-node-jf4lp"
Sep 16 02:13:08.706490 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.706461    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdk6t\" (UniqueName: \"kubernetes.io/projected/a5be83aa-c2b7-4808-a32d-80a845f3435a-kube-api-access-gdk6t\") pod \"multus-additional-cni-plugins-h7czj\" (UID: \"a5be83aa-c2b7-4808-a32d-80a845f3435a\") " pod="openshift-multus/multus-additional-cni-plugins-h7czj"
Sep 16 02:13:08.706752 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.706729    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-79nwn\" (UniqueName: \"kubernetes.io/projected/9049e903-7671-400a-a25a-e2cc8a151f79-kube-api-access-79nwn\") pod \"tuned-bshnm\" (UID: \"9049e903-7671-400a-a25a-e2cc8a151f79\") " pod="openshift-cluster-node-tuning-operator/tuned-bshnm"
Sep 16 02:13:08.707105 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.707076    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzqpm\" (UniqueName: \"kubernetes.io/projected/868ab659-0198-446b-b794-5d0d998d69bf-kube-api-access-hzqpm\") pod \"aws-ebs-csi-driver-node-n4n6g\" (UID: \"868ab659-0198-446b-b794-5d0d998d69bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n4n6g"
Sep 16 02:13:08.707684 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.707657    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nchfw\" (UniqueName: \"kubernetes.io/projected/9562b5b9-856c-4a81-89d3-8faed08f792c-kube-api-access-nchfw\") pod \"node-ca-vn8jb\" (UID: \"9562b5b9-856c-4a81-89d3-8faed08f792c\") " pod="openshift-image-registry/node-ca-vn8jb"
Sep 16 02:13:08.707792 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.707754    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ptzr\" (UniqueName: \"kubernetes.io/projected/25e85d58-77e2-4428-b099-a5075363d56f-kube-api-access-6ptzr\") pod \"network-metrics-daemon-ck2vw\" (UID: \"25e85d58-77e2-4428-b099-a5075363d56f\") " pod="openshift-multus/network-metrics-daemon-ck2vw"
Sep 16 02:13:08.725352 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:08.725174    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74fa655d1549fe38db1b28b2b10dd892.slice/crio-a789e3f584ee787ebe8059330a4f9201cd250b003e2092da083244febeff4948 WatchSource:0}: Error finding container a789e3f584ee787ebe8059330a4f9201cd250b003e2092da083244febeff4948: Status 404 returned error can't find the container with id a789e3f584ee787ebe8059330a4f9201cd250b003e2092da083244febeff4948
Sep 16 02:13:08.729169 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.729143    2558 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 02:13:08.730767 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:08.730748    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod809d69af053393799cf47db49da323fb.slice/crio-c6c11e00a28d0d918b8ef2d84dab043abf22b79613405a709d24797a3b469645 WatchSource:0}: Error finding container c6c11e00a28d0d918b8ef2d84dab043abf22b79613405a709d24797a3b469645: Status 404 returned error can't find the container with id c6c11e00a28d0d918b8ef2d84dab043abf22b79613405a709d24797a3b469645
Sep 16 02:13:08.818927 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.818875    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-ks4kv"
Sep 16 02:13:08.829947 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:08.829920    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc91732fe_96fc_454c_b468_e3acde4ec5a0.slice/crio-8a63264d882ad5c725d4b4af6bc41ade535296cdcd91ae3778df7a99d15f39cc WatchSource:0}: Error finding container 8a63264d882ad5c725d4b4af6bc41ade535296cdcd91ae3778df7a99d15f39cc: Status 404 returned error can't find the container with id 8a63264d882ad5c725d4b4af6bc41ade535296cdcd91ae3778df7a99d15f39cc
Sep 16 02:13:08.835441 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.835422    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-vn8jb"
Sep 16 02:13:08.844382 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:08.844357    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9562b5b9_856c_4a81_89d3_8faed08f792c.slice/crio-a1aeaed8e0dbb02bbcc83f4301823b20f5a13cf6af1b57f34eca1dce00a6f44b WatchSource:0}: Error finding container a1aeaed8e0dbb02bbcc83f4301823b20f5a13cf6af1b57f34eca1dce00a6f44b: Status 404 returned error can't find the container with id a1aeaed8e0dbb02bbcc83f4301823b20f5a13cf6af1b57f34eca1dce00a6f44b
Sep 16 02:13:08.856820 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.856797    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n4n6g"
Sep 16 02:13:08.856904 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.856885    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-84qhm"
Sep 16 02:13:08.868058 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:08.868033    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod868ab659_0198_446b_b794_5d0d998d69bf.slice/crio-887f67604691e2a605096798ffeb571c1f6f9f95bb05679cc212a74273480d39 WatchSource:0}: Error finding container 887f67604691e2a605096798ffeb571c1f6f9f95bb05679cc212a74273480d39: Status 404 returned error can't find the container with id 887f67604691e2a605096798ffeb571c1f6f9f95bb05679cc212a74273480d39
Sep 16 02:13:08.868924 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:08.868904    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58f0674c_b5b4_4d0e_8d14_3b18a232d365.slice/crio-6c9703bff59094d3afe8dc61c258dfc6f0fec70b63c7817f8905228d468071e4 WatchSource:0}: Error finding container 6c9703bff59094d3afe8dc61c258dfc6f0fec70b63c7817f8905228d468071e4: Status 404 returned error can't find the container with id 6c9703bff59094d3afe8dc61c258dfc6f0fec70b63c7817f8905228d468071e4
Sep 16 02:13:08.887620 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.887593    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-h7czj"
Sep 16 02:13:08.899003 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:08.898980    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5be83aa_c2b7_4808_a32d_80a845f3435a.slice/crio-34e3b2d46b8fe04a3a4eac653fc3752fe47bf3213d7840ddba5390049b5d87d7 WatchSource:0}: Error finding container 34e3b2d46b8fe04a3a4eac653fc3752fe47bf3213d7840ddba5390049b5d87d7: Status 404 returned error can't find the container with id 34e3b2d46b8fe04a3a4eac653fc3752fe47bf3213d7840ddba5390049b5d87d7
Sep 16 02:13:08.900488 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.900471    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-gmvp8"
Sep 16 02:13:08.908679 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:08.908659    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28767adf_af94_4851_947e_f0f2b21ef658.slice/crio-616ed3329ed9f621a8a19ccf44d86a9d9175b5dd2b0f6a7276540cead7f6b4fb WatchSource:0}: Error finding container 616ed3329ed9f621a8a19ccf44d86a9d9175b5dd2b0f6a7276540cead7f6b4fb: Status 404 returned error can't find the container with id 616ed3329ed9f621a8a19ccf44d86a9d9175b5dd2b0f6a7276540cead7f6b4fb
Sep 16 02:13:08.923914 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.923887    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-jf4lp"
Sep 16 02:13:08.932186 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:08.932164    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06c470fe_f6bf_4016_957c_56969805ddd0.slice/crio-512a395650777efe5ccba0c97e0a70eec047718d98cce81a3819c4ff47e339b1 WatchSource:0}: Error finding container 512a395650777efe5ccba0c97e0a70eec047718d98cce81a3819c4ff47e339b1: Status 404 returned error can't find the container with id 512a395650777efe5ccba0c97e0a70eec047718d98cce81a3819c4ff47e339b1
Sep 16 02:13:08.939424 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.939403    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-bshnm"
Sep 16 02:13:08.945393 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:08.945378    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-vrf75"
Sep 16 02:13:08.949215 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:08.949192    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9049e903_7671_400a_a25a_e2cc8a151f79.slice/crio-fb27385c09679ee4fa4306b4f70f5141c34f56ff971c0c22b97fe7561968b6cf WatchSource:0}: Error finding container fb27385c09679ee4fa4306b4f70f5141c34f56ff971c0c22b97fe7561968b6cf: Status 404 returned error can't find the container with id fb27385c09679ee4fa4306b4f70f5141c34f56ff971c0c22b97fe7561968b6cf
Sep 16 02:13:08.954246 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:08.954226    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0800020_f9de_4dfa_9a7b_d7c88ef0561f.slice/crio-96a8397535e9f6b15ff2228bb286e3277f9fe2876a393b66604b3133a5b8bb7d WatchSource:0}: Error finding container 96a8397535e9f6b15ff2228bb286e3277f9fe2876a393b66604b3133a5b8bb7d: Status 404 returned error can't find the container with id 96a8397535e9f6b15ff2228bb286e3277f9fe2876a393b66604b3133a5b8bb7d
Sep 16 02:13:09.201068 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:09.200977    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f5cc1e46-3424-49f8-8f2c-ec44086643d9-original-pull-secret\") pod \"global-pull-secret-syncer-f227r\" (UID: \"f5cc1e46-3424-49f8-8f2c-ec44086643d9\") " pod="kube-system/global-pull-secret-syncer-f227r"
Sep 16 02:13:09.201068 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:09.201019    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/25e85d58-77e2-4428-b099-a5075363d56f-metrics-certs\") pod \"network-metrics-daemon-ck2vw\" (UID: \"25e85d58-77e2-4428-b099-a5075363d56f\") " pod="openshift-multus/network-metrics-daemon-ck2vw"
Sep 16 02:13:09.201260 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:13:09.201114    2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 02:13:09.201260 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:13:09.201132    2558 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 02:13:09.201260 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:13:09.201172    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/25e85d58-77e2-4428-b099-a5075363d56f-metrics-certs podName:25e85d58-77e2-4428-b099-a5075363d56f nodeName:}" failed. No retries permitted until 2026-09-16 02:13:10.201155549 +0000 UTC m=+3.119644783 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/25e85d58-77e2-4428-b099-a5075363d56f-metrics-certs") pod "network-metrics-daemon-ck2vw" (UID: "25e85d58-77e2-4428-b099-a5075363d56f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 02:13:09.201260 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:13:09.201185    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f5cc1e46-3424-49f8-8f2c-ec44086643d9-original-pull-secret podName:f5cc1e46-3424-49f8-8f2c-ec44086643d9 nodeName:}" failed. No retries permitted until 2026-09-16 02:13:10.201178582 +0000 UTC m=+3.119667811 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f5cc1e46-3424-49f8-8f2c-ec44086643d9-original-pull-secret") pod "global-pull-secret-syncer-f227r" (UID: "f5cc1e46-3424-49f8-8f2c-ec44086643d9") : object "kube-system"/"original-pull-secret" not registered
Sep 16 02:13:09.302490 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:09.302448    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdqgf\" (UniqueName: \"kubernetes.io/projected/6ce8b9ee-ca3a-4f48-8e6d-64a8221d2369-kube-api-access-vdqgf\") pod \"network-check-target-mfd49\" (UID: \"6ce8b9ee-ca3a-4f48-8e6d-64a8221d2369\") " pod="openshift-network-diagnostics/network-check-target-mfd49"
Sep 16 02:13:09.302684 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:13:09.302640    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 02:13:09.302684 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:13:09.302662    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 02:13:09.302684 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:13:09.302676    2558 projected.go:196] Error preparing data for projected volume kube-api-access-vdqgf for pod openshift-network-diagnostics/network-check-target-mfd49: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 02:13:09.302863 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:13:09.302759    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6ce8b9ee-ca3a-4f48-8e6d-64a8221d2369-kube-api-access-vdqgf podName:6ce8b9ee-ca3a-4f48-8e6d-64a8221d2369 nodeName:}" failed. No retries permitted until 2026-09-16 02:13:10.302724406 +0000 UTC m=+3.221213655 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-vdqgf" (UniqueName: "kubernetes.io/projected/6ce8b9ee-ca3a-4f48-8e6d-64a8221d2369-kube-api-access-vdqgf") pod "network-check-target-mfd49" (UID: "6ce8b9ee-ca3a-4f48-8e6d-64a8221d2369") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 02:13:09.532645 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:09.532480    2558 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-15 02:08:09 +0000 UTC" deadline="2028-05-17 09:34:16.339804105 +0000 UTC"
Sep 16 02:13:09.532645 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:09.532547    2558 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14623h21m6.807261345s"
Sep 16 02:13:09.543925 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:09.543901    2558 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 02:13:09.558751 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:09.558688    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-gmvp8" event={"ID":"28767adf-af94-4851-947e-f0f2b21ef658","Type":"ContainerStarted","Data":"616ed3329ed9f621a8a19ccf44d86a9d9175b5dd2b0f6a7276540cead7f6b4fb"}
Sep 16 02:13:09.562124 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:09.562094    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-ks4kv" event={"ID":"c91732fe-96fc-454c-b468-e3acde4ec5a0","Type":"ContainerStarted","Data":"8a63264d882ad5c725d4b4af6bc41ade535296cdcd91ae3778df7a99d15f39cc"}
Sep 16 02:13:09.577416 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:09.577382    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-34.ec2.internal" event={"ID":"809d69af053393799cf47db49da323fb","Type":"ContainerStarted","Data":"c6c11e00a28d0d918b8ef2d84dab043abf22b79613405a709d24797a3b469645"}
Sep 16 02:13:09.597650 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:09.597615    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-34.ec2.internal" event={"ID":"74fa655d1549fe38db1b28b2b10dd892","Type":"ContainerStarted","Data":"a789e3f584ee787ebe8059330a4f9201cd250b003e2092da083244febeff4948"}
Sep 16 02:13:09.611643 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:09.611527    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-vrf75" event={"ID":"e0800020-f9de-4dfa-9a7b-d7c88ef0561f","Type":"ContainerStarted","Data":"96a8397535e9f6b15ff2228bb286e3277f9fe2876a393b66604b3133a5b8bb7d"}
Sep 16 02:13:09.635189 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:09.635157    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-h7czj" event={"ID":"a5be83aa-c2b7-4808-a32d-80a845f3435a","Type":"ContainerStarted","Data":"34e3b2d46b8fe04a3a4eac653fc3752fe47bf3213d7840ddba5390049b5d87d7"}
Sep 16 02:13:09.658979 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:09.658945    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-84qhm" event={"ID":"58f0674c-b5b4-4d0e-8d14-3b18a232d365","Type":"ContainerStarted","Data":"6c9703bff59094d3afe8dc61c258dfc6f0fec70b63c7817f8905228d468071e4"}
Sep 16 02:13:09.672880 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:09.672759    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n4n6g" event={"ID":"868ab659-0198-446b-b794-5d0d998d69bf","Type":"ContainerStarted","Data":"887f67604691e2a605096798ffeb571c1f6f9f95bb05679cc212a74273480d39"}
Sep 16 02:13:09.696225 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:09.696190    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-vn8jb" event={"ID":"9562b5b9-856c-4a81-89d3-8faed08f792c","Type":"ContainerStarted","Data":"a1aeaed8e0dbb02bbcc83f4301823b20f5a13cf6af1b57f34eca1dce00a6f44b"}
Sep 16 02:13:09.724967 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:09.724931    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-bshnm" event={"ID":"9049e903-7671-400a-a25a-e2cc8a151f79","Type":"ContainerStarted","Data":"fb27385c09679ee4fa4306b4f70f5141c34f56ff971c0c22b97fe7561968b6cf"}
Sep 16 02:13:09.745331 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:09.745287    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jf4lp" event={"ID":"06c470fe-f6bf-4016-957c-56969805ddd0","Type":"ContainerStarted","Data":"512a395650777efe5ccba0c97e0a70eec047718d98cce81a3819c4ff47e339b1"}
Sep 16 02:13:09.874590 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:09.874471    2558 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 02:13:10.211867 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:10.211459    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f5cc1e46-3424-49f8-8f2c-ec44086643d9-original-pull-secret\") pod \"global-pull-secret-syncer-f227r\" (UID: \"f5cc1e46-3424-49f8-8f2c-ec44086643d9\") " pod="kube-system/global-pull-secret-syncer-f227r"
Sep 16 02:13:10.211867 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:10.211515    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/25e85d58-77e2-4428-b099-a5075363d56f-metrics-certs\") pod \"network-metrics-daemon-ck2vw\" (UID: \"25e85d58-77e2-4428-b099-a5075363d56f\") " pod="openshift-multus/network-metrics-daemon-ck2vw"
Sep 16 02:13:10.211867 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:13:10.211683    2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 02:13:10.211867 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:13:10.211690    2558 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 02:13:10.211867 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:13:10.211770    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/25e85d58-77e2-4428-b099-a5075363d56f-metrics-certs podName:25e85d58-77e2-4428-b099-a5075363d56f nodeName:}" failed. No retries permitted until 2026-09-16 02:13:12.211731201 +0000 UTC m=+5.130220446 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/25e85d58-77e2-4428-b099-a5075363d56f-metrics-certs") pod "network-metrics-daemon-ck2vw" (UID: "25e85d58-77e2-4428-b099-a5075363d56f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 02:13:10.211867 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:13:10.211790    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f5cc1e46-3424-49f8-8f2c-ec44086643d9-original-pull-secret podName:f5cc1e46-3424-49f8-8f2c-ec44086643d9 nodeName:}" failed. No retries permitted until 2026-09-16 02:13:12.211779844 +0000 UTC m=+5.130269075 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f5cc1e46-3424-49f8-8f2c-ec44086643d9-original-pull-secret") pod "global-pull-secret-syncer-f227r" (UID: "f5cc1e46-3424-49f8-8f2c-ec44086643d9") : object "kube-system"/"original-pull-secret" not registered
Sep 16 02:13:10.312663 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:10.312626    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdqgf\" (UniqueName: \"kubernetes.io/projected/6ce8b9ee-ca3a-4f48-8e6d-64a8221d2369-kube-api-access-vdqgf\") pod \"network-check-target-mfd49\" (UID: \"6ce8b9ee-ca3a-4f48-8e6d-64a8221d2369\") " pod="openshift-network-diagnostics/network-check-target-mfd49"
Sep 16 02:13:10.312876 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:13:10.312844    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 02:13:10.312876 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:13:10.312868    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 02:13:10.312992 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:13:10.312881    2558 projected.go:196] Error preparing data for projected volume kube-api-access-vdqgf for pod openshift-network-diagnostics/network-check-target-mfd49: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 02:13:10.312992 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:13:10.312940    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6ce8b9ee-ca3a-4f48-8e6d-64a8221d2369-kube-api-access-vdqgf podName:6ce8b9ee-ca3a-4f48-8e6d-64a8221d2369 nodeName:}" failed. No retries permitted until 2026-09-16 02:13:12.312920698 +0000 UTC m=+5.231409948 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-vdqgf" (UniqueName: "kubernetes.io/projected/6ce8b9ee-ca3a-4f48-8e6d-64a8221d2369-kube-api-access-vdqgf") pod "network-check-target-mfd49" (UID: "6ce8b9ee-ca3a-4f48-8e6d-64a8221d2369") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 02:13:10.533177 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:10.533087    2558 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-15 02:08:09 +0000 UTC" deadline="2028-05-23 11:56:44.856791296 +0000 UTC"
Sep 16 02:13:10.533177 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:10.533127    2558 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14769h43m34.323669321s"
Sep 16 02:13:10.543691 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:10.543493    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ck2vw"
Sep 16 02:13:10.543691 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:13:10.543638    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ck2vw" podUID="25e85d58-77e2-4428-b099-a5075363d56f"
Sep 16 02:13:10.544091 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:10.544070    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mfd49"
Sep 16 02:13:10.544198 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:13:10.544177    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mfd49" podUID="6ce8b9ee-ca3a-4f48-8e6d-64a8221d2369"
Sep 16 02:13:10.544276 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:10.544262    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f227r"
Sep 16 02:13:10.544374 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:13:10.544355    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f227r" podUID="f5cc1e46-3424-49f8-8f2c-ec44086643d9"
Sep 16 02:13:10.645458 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:10.645419    2558 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 02:13:10.908143 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:10.908058    2558 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 02:13:12.230684 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:12.230601    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f5cc1e46-3424-49f8-8f2c-ec44086643d9-original-pull-secret\") pod \"global-pull-secret-syncer-f227r\" (UID: \"f5cc1e46-3424-49f8-8f2c-ec44086643d9\") " pod="kube-system/global-pull-secret-syncer-f227r"
Sep 16 02:13:12.230684 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:12.230670    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/25e85d58-77e2-4428-b099-a5075363d56f-metrics-certs\") pod \"network-metrics-daemon-ck2vw\" (UID: \"25e85d58-77e2-4428-b099-a5075363d56f\") " pod="openshift-multus/network-metrics-daemon-ck2vw"
Sep 16 02:13:12.231410 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:13:12.230833    2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 02:13:12.231410 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:13:12.230848    2558 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 02:13:12.231410 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:13:12.230907    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/25e85d58-77e2-4428-b099-a5075363d56f-metrics-certs podName:25e85d58-77e2-4428-b099-a5075363d56f nodeName:}" failed. No retries permitted until 2026-09-16 02:13:16.230887031 +0000 UTC m=+9.149376281 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/25e85d58-77e2-4428-b099-a5075363d56f-metrics-certs") pod "network-metrics-daemon-ck2vw" (UID: "25e85d58-77e2-4428-b099-a5075363d56f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 02:13:12.231410 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:13:12.230931    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f5cc1e46-3424-49f8-8f2c-ec44086643d9-original-pull-secret podName:f5cc1e46-3424-49f8-8f2c-ec44086643d9 nodeName:}" failed. No retries permitted until 2026-09-16 02:13:16.230917869 +0000 UTC m=+9.149407113 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f5cc1e46-3424-49f8-8f2c-ec44086643d9-original-pull-secret") pod "global-pull-secret-syncer-f227r" (UID: "f5cc1e46-3424-49f8-8f2c-ec44086643d9") : object "kube-system"/"original-pull-secret" not registered
Sep 16 02:13:12.332092 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:12.331223    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdqgf\" (UniqueName: \"kubernetes.io/projected/6ce8b9ee-ca3a-4f48-8e6d-64a8221d2369-kube-api-access-vdqgf\") pod \"network-check-target-mfd49\" (UID: \"6ce8b9ee-ca3a-4f48-8e6d-64a8221d2369\") " pod="openshift-network-diagnostics/network-check-target-mfd49"
Sep 16 02:13:12.332092 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:13:12.331454    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 02:13:12.332092 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:13:12.331518    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 02:13:12.332092 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:13:12.331531    2558 projected.go:196] Error preparing data for projected volume kube-api-access-vdqgf for pod openshift-network-diagnostics/network-check-target-mfd49: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 02:13:12.332092 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:13:12.331599    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6ce8b9ee-ca3a-4f48-8e6d-64a8221d2369-kube-api-access-vdqgf podName:6ce8b9ee-ca3a-4f48-8e6d-64a8221d2369 nodeName:}" failed. No retries permitted until 2026-09-16 02:13:16.33157631 +0000 UTC m=+9.250065560 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-vdqgf" (UniqueName: "kubernetes.io/projected/6ce8b9ee-ca3a-4f48-8e6d-64a8221d2369-kube-api-access-vdqgf") pod "network-check-target-mfd49" (UID: "6ce8b9ee-ca3a-4f48-8e6d-64a8221d2369") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 02:13:12.544014 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:12.543915    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mfd49"
Sep 16 02:13:12.544188 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:13:12.544072    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mfd49" podUID="6ce8b9ee-ca3a-4f48-8e6d-64a8221d2369"
Sep 16 02:13:12.544519 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:12.544492    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f227r"
Sep 16 02:13:12.544616 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:13:12.544591    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f227r" podUID="f5cc1e46-3424-49f8-8f2c-ec44086643d9"
Sep 16 02:13:12.544676 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:12.544655    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ck2vw"
Sep 16 02:13:12.544830 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:13:12.544746    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ck2vw" podUID="25e85d58-77e2-4428-b099-a5075363d56f"
Sep 16 02:13:14.544226 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:14.543773    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ck2vw"
Sep 16 02:13:14.544226 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:14.543802    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mfd49"
Sep 16 02:13:14.544226 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:14.543923    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f227r"
Sep 16 02:13:14.544226 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:13:14.544069    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f227r" podUID="f5cc1e46-3424-49f8-8f2c-ec44086643d9"
Sep 16 02:13:14.544226 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:13:14.543925    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mfd49" podUID="6ce8b9ee-ca3a-4f48-8e6d-64a8221d2369"
Sep 16 02:13:14.544226 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:13:14.544184    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ck2vw" podUID="25e85d58-77e2-4428-b099-a5075363d56f"
Sep 16 02:13:16.268405 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:16.268353    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f5cc1e46-3424-49f8-8f2c-ec44086643d9-original-pull-secret\") pod \"global-pull-secret-syncer-f227r\" (UID: \"f5cc1e46-3424-49f8-8f2c-ec44086643d9\") " pod="kube-system/global-pull-secret-syncer-f227r"
Sep 16 02:13:16.268982 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:16.268419    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/25e85d58-77e2-4428-b099-a5075363d56f-metrics-certs\") pod \"network-metrics-daemon-ck2vw\" (UID: \"25e85d58-77e2-4428-b099-a5075363d56f\") " pod="openshift-multus/network-metrics-daemon-ck2vw"
Sep 16 02:13:16.268982 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:13:16.268546    2558 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 02:13:16.268982 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:13:16.268636    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f5cc1e46-3424-49f8-8f2c-ec44086643d9-original-pull-secret podName:f5cc1e46-3424-49f8-8f2c-ec44086643d9 nodeName:}" failed. No retries permitted until 2026-09-16 02:13:24.268613426 +0000 UTC m=+17.187102676 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f5cc1e46-3424-49f8-8f2c-ec44086643d9-original-pull-secret") pod "global-pull-secret-syncer-f227r" (UID: "f5cc1e46-3424-49f8-8f2c-ec44086643d9") : object "kube-system"/"original-pull-secret" not registered
Sep 16 02:13:16.268982 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:13:16.268546    2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 02:13:16.268982 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:13:16.268692    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/25e85d58-77e2-4428-b099-a5075363d56f-metrics-certs podName:25e85d58-77e2-4428-b099-a5075363d56f nodeName:}" failed. No retries permitted until 2026-09-16 02:13:24.268678943 +0000 UTC m=+17.187168187 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/25e85d58-77e2-4428-b099-a5075363d56f-metrics-certs") pod "network-metrics-daemon-ck2vw" (UID: "25e85d58-77e2-4428-b099-a5075363d56f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 02:13:16.369973 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:16.369876    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdqgf\" (UniqueName: \"kubernetes.io/projected/6ce8b9ee-ca3a-4f48-8e6d-64a8221d2369-kube-api-access-vdqgf\") pod \"network-check-target-mfd49\" (UID: \"6ce8b9ee-ca3a-4f48-8e6d-64a8221d2369\") " pod="openshift-network-diagnostics/network-check-target-mfd49"
Sep 16 02:13:16.370135 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:13:16.370067    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 02:13:16.370135 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:13:16.370092    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 02:13:16.370135 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:13:16.370119    2558 projected.go:196] Error preparing data for projected volume kube-api-access-vdqgf for pod openshift-network-diagnostics/network-check-target-mfd49: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 02:13:16.370323 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:13:16.370182    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6ce8b9ee-ca3a-4f48-8e6d-64a8221d2369-kube-api-access-vdqgf podName:6ce8b9ee-ca3a-4f48-8e6d-64a8221d2369 nodeName:}" failed. No retries permitted until 2026-09-16 02:13:24.370162863 +0000 UTC m=+17.288652117 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-vdqgf" (UniqueName: "kubernetes.io/projected/6ce8b9ee-ca3a-4f48-8e6d-64a8221d2369-kube-api-access-vdqgf") pod "network-check-target-mfd49" (UID: "6ce8b9ee-ca3a-4f48-8e6d-64a8221d2369") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 02:13:16.543873 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:16.543832    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mfd49"
Sep 16 02:13:16.544081 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:16.543896    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ck2vw"
Sep 16 02:13:16.544081 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:16.544028    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f227r"
Sep 16 02:13:16.544081 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:13:16.544049    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mfd49" podUID="6ce8b9ee-ca3a-4f48-8e6d-64a8221d2369"
Sep 16 02:13:16.544256 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:13:16.544126    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f227r" podUID="f5cc1e46-3424-49f8-8f2c-ec44086643d9"
Sep 16 02:13:16.544256 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:13:16.544219    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ck2vw" podUID="25e85d58-77e2-4428-b099-a5075363d56f"
Sep 16 02:13:18.544049 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:18.543983    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f227r"
Sep 16 02:13:18.544049 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:18.544025    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mfd49"
Sep 16 02:13:18.544049 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:18.544026    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ck2vw"
Sep 16 02:13:18.544662 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:13:18.544133    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f227r" podUID="f5cc1e46-3424-49f8-8f2c-ec44086643d9"
Sep 16 02:13:18.544662 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:13:18.544233    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mfd49" podUID="6ce8b9ee-ca3a-4f48-8e6d-64a8221d2369"
Sep 16 02:13:18.544662 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:13:18.544345    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ck2vw" podUID="25e85d58-77e2-4428-b099-a5075363d56f"
Sep 16 02:13:20.543679 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:20.543637    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f227r"
Sep 16 02:13:20.543679 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:20.543663    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mfd49"
Sep 16 02:13:20.544173 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:20.543760    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ck2vw"
Sep 16 02:13:20.544173 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:13:20.543777    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f227r" podUID="f5cc1e46-3424-49f8-8f2c-ec44086643d9"
Sep 16 02:13:20.544173 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:13:20.543867    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ck2vw" podUID="25e85d58-77e2-4428-b099-a5075363d56f"
Sep 16 02:13:20.544173 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:13:20.543935    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mfd49" podUID="6ce8b9ee-ca3a-4f48-8e6d-64a8221d2369"
Sep 16 02:13:22.544172 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:22.544125    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mfd49"
Sep 16 02:13:22.544172 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:22.544151    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ck2vw"
Sep 16 02:13:22.544673 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:22.544141    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f227r"
Sep 16 02:13:22.544673 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:13:22.544254    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mfd49" podUID="6ce8b9ee-ca3a-4f48-8e6d-64a8221d2369"
Sep 16 02:13:22.544673 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:13:22.544350    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ck2vw" podUID="25e85d58-77e2-4428-b099-a5075363d56f"
Sep 16 02:13:22.544673 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:13:22.544442    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f227r" podUID="f5cc1e46-3424-49f8-8f2c-ec44086643d9"
Sep 16 02:13:24.330970 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:24.330928    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f5cc1e46-3424-49f8-8f2c-ec44086643d9-original-pull-secret\") pod \"global-pull-secret-syncer-f227r\" (UID: \"f5cc1e46-3424-49f8-8f2c-ec44086643d9\") " pod="kube-system/global-pull-secret-syncer-f227r"
Sep 16 02:13:24.331478 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:24.330989    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/25e85d58-77e2-4428-b099-a5075363d56f-metrics-certs\") pod \"network-metrics-daemon-ck2vw\" (UID: \"25e85d58-77e2-4428-b099-a5075363d56f\") " pod="openshift-multus/network-metrics-daemon-ck2vw"
Sep 16 02:13:24.331478 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:13:24.331098    2558 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 02:13:24.331478 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:13:24.331147    2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 02:13:24.331478 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:13:24.331185    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f5cc1e46-3424-49f8-8f2c-ec44086643d9-original-pull-secret podName:f5cc1e46-3424-49f8-8f2c-ec44086643d9 nodeName:}" failed. No retries permitted until 2026-09-16 02:13:40.331164575 +0000 UTC m=+33.249653826 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f5cc1e46-3424-49f8-8f2c-ec44086643d9-original-pull-secret") pod "global-pull-secret-syncer-f227r" (UID: "f5cc1e46-3424-49f8-8f2c-ec44086643d9") : object "kube-system"/"original-pull-secret" not registered
Sep 16 02:13:24.331478 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:13:24.331201    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/25e85d58-77e2-4428-b099-a5075363d56f-metrics-certs podName:25e85d58-77e2-4428-b099-a5075363d56f nodeName:}" failed. No retries permitted until 2026-09-16 02:13:40.33119421 +0000 UTC m=+33.249683437 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/25e85d58-77e2-4428-b099-a5075363d56f-metrics-certs") pod "network-metrics-daemon-ck2vw" (UID: "25e85d58-77e2-4428-b099-a5075363d56f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 02:13:24.431515 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:24.431478    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdqgf\" (UniqueName: \"kubernetes.io/projected/6ce8b9ee-ca3a-4f48-8e6d-64a8221d2369-kube-api-access-vdqgf\") pod \"network-check-target-mfd49\" (UID: \"6ce8b9ee-ca3a-4f48-8e6d-64a8221d2369\") " pod="openshift-network-diagnostics/network-check-target-mfd49"
Sep 16 02:13:24.431651 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:13:24.431637    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 02:13:24.431716 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:13:24.431660    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 02:13:24.431716 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:13:24.431669    2558 projected.go:196] Error preparing data for projected volume kube-api-access-vdqgf for pod openshift-network-diagnostics/network-check-target-mfd49: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 02:13:24.431780 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:13:24.431740    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6ce8b9ee-ca3a-4f48-8e6d-64a8221d2369-kube-api-access-vdqgf podName:6ce8b9ee-ca3a-4f48-8e6d-64a8221d2369 nodeName:}" failed. No retries permitted until 2026-09-16 02:13:40.431727139 +0000 UTC m=+33.350216371 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-vdqgf" (UniqueName: "kubernetes.io/projected/6ce8b9ee-ca3a-4f48-8e6d-64a8221d2369-kube-api-access-vdqgf") pod "network-check-target-mfd49" (UID: "6ce8b9ee-ca3a-4f48-8e6d-64a8221d2369") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 02:13:24.543675 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:24.543638    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f227r"
Sep 16 02:13:24.543835 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:24.543638    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mfd49"
Sep 16 02:13:24.543835 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:13:24.543777    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f227r" podUID="f5cc1e46-3424-49f8-8f2c-ec44086643d9"
Sep 16 02:13:24.543835 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:24.543643    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ck2vw"
Sep 16 02:13:24.543835 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:13:24.543820    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mfd49" podUID="6ce8b9ee-ca3a-4f48-8e6d-64a8221d2369"
Sep 16 02:13:24.543968 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:13:24.543907    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ck2vw" podUID="25e85d58-77e2-4428-b099-a5075363d56f"
Sep 16 02:13:26.543629 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:26.543597    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mfd49"
Sep 16 02:13:26.544159 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:26.543597    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f227r"
Sep 16 02:13:26.544159 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:13:26.543739    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mfd49" podUID="6ce8b9ee-ca3a-4f48-8e6d-64a8221d2369"
Sep 16 02:13:26.544159 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:26.543597    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ck2vw"
Sep 16 02:13:26.544159 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:13:26.543814    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f227r" podUID="f5cc1e46-3424-49f8-8f2c-ec44086643d9"
Sep 16 02:13:26.544159 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:13:26.543925    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ck2vw" podUID="25e85d58-77e2-4428-b099-a5075363d56f"
Sep 16 02:13:28.543706 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:28.543669    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ck2vw"
Sep 16 02:13:28.544204 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:28.543692    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mfd49"
Sep 16 02:13:28.544204 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:28.543691    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f227r"
Sep 16 02:13:28.544204 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:13:28.543822    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ck2vw" podUID="25e85d58-77e2-4428-b099-a5075363d56f"
Sep 16 02:13:28.544204 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:13:28.543904    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mfd49" podUID="6ce8b9ee-ca3a-4f48-8e6d-64a8221d2369"
Sep 16 02:13:28.544204 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:13:28.543982    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f227r" podUID="f5cc1e46-3424-49f8-8f2c-ec44086643d9"
Sep 16 02:13:28.796457 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:28.796241    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-84qhm" event={"ID":"58f0674c-b5b4-4d0e-8d14-3b18a232d365","Type":"ContainerStarted","Data":"b06dd71b10a8a6ff43efff3704269b96310c27799d5ba052eb4dd028889e0226"}
Sep 16 02:13:28.813241 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:28.813138    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-84qhm" podStartSLOduration=1.2096602 podStartE2EDuration="20.813114929s" totalImagesPullingTime="19.603454729s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 02:13:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:13:28.812759073 +0000 UTC m=+21.731248324" watchObservedRunningTime="2026-09-16 02:13:28.813114929 +0000 UTC m=+21.731604179"
Sep 16 02:13:28.815340 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:28.815267    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-bshnm" event={"ID":"9049e903-7671-400a-a25a-e2cc8a151f79","Type":"ContainerStarted","Data":"c5fb7fc9514539776da6fcf4b7abce1eb885486bda1f6a9082524070a66aa721"}
Sep 16 02:13:28.826257 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:28.826227    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jf4lp" event={"ID":"06c470fe-f6bf-4016-957c-56969805ddd0","Type":"ContainerStarted","Data":"9ce8cd1dfedaf17798118f6ac0b360150a2620ceb4d196946f9b751bd4c6d3d1"}
Sep 16 02:13:28.827895 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:28.827869    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-34.ec2.internal" event={"ID":"74fa655d1549fe38db1b28b2b10dd892","Type":"ContainerStarted","Data":"f031102dda6030419c754f86a5f21a2f0bf7375f8bf34bdefa30ac3e0582b396"}
Sep 16 02:13:28.831833 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:28.831796    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-bshnm" podStartSLOduration=1.33954039 podStartE2EDuration="20.831783025s" totalImagesPullingTime="19.492242635s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 02:13:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:13:28.831357531 +0000 UTC m=+21.749846786" watchObservedRunningTime="2026-09-16 02:13:28.831783025 +0000 UTC m=+21.750272273"
Sep 16 02:13:28.844200 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:28.843857    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-34.ec2.internal" podStartSLOduration=19.843838609 podStartE2EDuration="19.843838609s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 02:13:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:13:28.842882159 +0000 UTC m=+21.761371409" watchObservedRunningTime="2026-09-16 02:13:28.843838609 +0000 UTC m=+21.762327859"
Sep 16 02:13:29.831899 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:29.831624    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-gmvp8" event={"ID":"28767adf-af94-4851-947e-f0f2b21ef658","Type":"ContainerStarted","Data":"0695d3e2092d7e8f6d66634d39f5fc6ac89ecca2f1b4fe82c3077fb5a3282fce"}
Sep 16 02:13:29.833402 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:29.833375    2558 generic.go:356] "Generic (PLEG): container finished" podID="809d69af053393799cf47db49da323fb" containerID="7afd74f7e2531661db0e94a06a847f492c9efff495ec13be4bbaffc47ad3467b" exitCode=0
Sep 16 02:13:29.833528 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:29.833467    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-34.ec2.internal" event={"ID":"809d69af053393799cf47db49da323fb","Type":"ContainerDied","Data":"7afd74f7e2531661db0e94a06a847f492c9efff495ec13be4bbaffc47ad3467b"}
Sep 16 02:13:29.837379 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:29.835959    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-vrf75" event={"ID":"e0800020-f9de-4dfa-9a7b-d7c88ef0561f","Type":"ContainerStarted","Data":"a4be0d4fae2a60c5c19ab596d5ba572c34941cc2f4844a48395bc577005a7713"}
Sep 16 02:13:29.839755 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:29.839727    2558 generic.go:356] "Generic (PLEG): container finished" podID="a5be83aa-c2b7-4808-a32d-80a845f3435a" containerID="e58e43bdf0a61b509f6a49e100d9a381f5aed1133242d7b660a5940b5da2dbd8" exitCode=0
Sep 16 02:13:29.839868 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:29.839793    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-h7czj" event={"ID":"a5be83aa-c2b7-4808-a32d-80a845f3435a","Type":"ContainerDied","Data":"e58e43bdf0a61b509f6a49e100d9a381f5aed1133242d7b660a5940b5da2dbd8"}
Sep 16 02:13:29.841396 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:29.841373    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n4n6g" event={"ID":"868ab659-0198-446b-b794-5d0d998d69bf","Type":"ContainerStarted","Data":"7f2ec23a2926565e875e607f4354afb742dad3424aa3654deb23b6ad224e28f3"}
Sep 16 02:13:29.843185 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:29.843161    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-vn8jb" event={"ID":"9562b5b9-856c-4a81-89d3-8faed08f792c","Type":"ContainerStarted","Data":"fb918ce7c6bed055bf3d8118ec3b967fe68a7d7d5b23b6b80b70b7be9f6ad80d"}
Sep 16 02:13:29.845475 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:29.845421    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-gmvp8" podStartSLOduration=2.346207299 podStartE2EDuration="21.845404904s" totalImagesPullingTime="19.499197605s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 02:13:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:13:29.845375597 +0000 UTC m=+22.763864847" watchObservedRunningTime="2026-09-16 02:13:29.845404904 +0000 UTC m=+22.763894155"
Sep 16 02:13:29.847885 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:29.847475    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jf4lp" event={"ID":"06c470fe-f6bf-4016-957c-56969805ddd0","Type":"ContainerStarted","Data":"5d208db94dd273b87f2aafedb57f95fa81e72efbea3727e7fe3d18ff6e3bb211"}
Sep 16 02:13:29.847885 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:29.847548    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jf4lp" event={"ID":"06c470fe-f6bf-4016-957c-56969805ddd0","Type":"ContainerStarted","Data":"7dc4d9af6e2c4c61ede0586d6ba35b704fc4e4898ba851af37e538c944ab492e"}
Sep 16 02:13:29.847885 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:29.847596    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jf4lp" event={"ID":"06c470fe-f6bf-4016-957c-56969805ddd0","Type":"ContainerStarted","Data":"ac408abce5d5f2f2884805ec9096b915e99516d4eac10414ae6e9622e511ef77"}
Sep 16 02:13:29.847885 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:29.847645    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jf4lp" event={"ID":"06c470fe-f6bf-4016-957c-56969805ddd0","Type":"ContainerStarted","Data":"24f632b3ca0c33bc8d56ed6b47f694617222da6944f19854b461c9c282e8a753"}
Sep 16 02:13:29.847885 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:29.847690    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jf4lp" event={"ID":"06c470fe-f6bf-4016-957c-56969805ddd0","Type":"ContainerStarted","Data":"283e2851a722a06a596d8da9865717270c959518c72b488a2ea444d68d9cc1a9"}
Sep 16 02:13:29.859935 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:29.859889    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-vn8jb" podStartSLOduration=2.280574915 podStartE2EDuration="21.859870765s" totalImagesPullingTime="19.57929585s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 02:13:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:13:29.85892144 +0000 UTC m=+22.777410717" watchObservedRunningTime="2026-09-16 02:13:29.859870765 +0000 UTC m=+22.778360015"
Sep 16 02:13:29.888495 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:29.888456    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-vrf75" podStartSLOduration=2.402788923 podStartE2EDuration="21.888440211s" totalImagesPullingTime="19.485651288s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 02:13:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:13:29.888185627 +0000 UTC m=+22.806674886" watchObservedRunningTime="2026-09-16 02:13:29.888440211 +0000 UTC m=+22.806929459"
Sep 16 02:13:30.191710 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:30.191547    2558 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 16 02:13:30.543987 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:30.543809    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f227r"
Sep 16 02:13:30.543987 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:30.543809    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mfd49"
Sep 16 02:13:30.543987 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:13:30.543941    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f227r" podUID="f5cc1e46-3424-49f8-8f2c-ec44086643d9"
Sep 16 02:13:30.543987 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:30.543809    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ck2vw"
Sep 16 02:13:30.544306 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:13:30.544002    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mfd49" podUID="6ce8b9ee-ca3a-4f48-8e6d-64a8221d2369"
Sep 16 02:13:30.544306 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:13:30.544112    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ck2vw" podUID="25e85d58-77e2-4428-b099-a5075363d56f"
Sep 16 02:13:30.569391 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:30.569291    2558 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-16T02:13:30.19168973Z","UUID":"1a5d3e7f-8535-4652-9b89-f1d4e82d36a2","Handler":null,"Name":"","Endpoint":""}
Sep 16 02:13:30.571107 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:30.571085    2558 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 16 02:13:30.571230 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:30.571120    2558 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 16 02:13:30.621712 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:30.621678    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-gmvp8"
Sep 16 02:13:30.622386 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:30.622367    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-gmvp8"
Sep 16 02:13:30.853814 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:30.853674    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n4n6g" event={"ID":"868ab659-0198-446b-b794-5d0d998d69bf","Type":"ContainerStarted","Data":"3ea20b0d153071aa90d14fdc4e1dfcda318d6e3f3d38676366b3ded888a18c06"}
Sep 16 02:13:30.855826 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:30.855777    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-ks4kv" event={"ID":"c91732fe-96fc-454c-b468-e3acde4ec5a0","Type":"ContainerStarted","Data":"a7eed22e203525e1684a5915463298a78963eb9d6f631c705df92c82dc8f95bc"}
Sep 16 02:13:30.857721 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:30.857668    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-34.ec2.internal" event={"ID":"809d69af053393799cf47db49da323fb","Type":"ContainerStarted","Data":"212f5114c130e68fc16edc5420fad9bed15c102f1e0c58deafca9e1c925f2134"}
Sep 16 02:13:30.871486 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:30.871433    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-ks4kv" podStartSLOduration=3.262046079 podStartE2EDuration="22.871414465s" totalImagesPullingTime="19.609368386s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 02:13:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:13:30.870766304 +0000 UTC m=+23.789255554" watchObservedRunningTime="2026-09-16 02:13:30.871414465 +0000 UTC m=+23.789903730"
Sep 16 02:13:30.885105 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:30.885053    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-34.ec2.internal" podStartSLOduration=21.885037707 podStartE2EDuration="21.885037707s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 02:13:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:13:30.884180431 +0000 UTC m=+23.802669680" watchObservedRunningTime="2026-09-16 02:13:30.885037707 +0000 UTC m=+23.803526955"
Sep 16 02:13:31.863624 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:31.863528    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n4n6g" event={"ID":"868ab659-0198-446b-b794-5d0d998d69bf","Type":"ContainerStarted","Data":"ba519323ecd180049d0898e919a99bb181df58301cd126d3eb479f1a734cfaf9"}
Sep 16 02:13:31.867572 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:31.867549    2558 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 16 02:13:31.867718 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:31.867563    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jf4lp" event={"ID":"06c470fe-f6bf-4016-957c-56969805ddd0","Type":"ContainerStarted","Data":"967e7ad461bdcd48cb29b9496cb44923a796c29e6e2b2f6bc50a197549f66280"}
Sep 16 02:13:31.880660 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:31.880607    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n4n6g" podStartSLOduration=1.98676202 podStartE2EDuration="23.880587193s" totalImagesPullingTime="21.893825173s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 02:13:08 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:13:31.879950779 +0000 UTC m=+24.798440043" watchObservedRunningTime="2026-09-16 02:13:31.880587193 +0000 UTC m=+24.799076443"
Sep 16 02:13:32.543805 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:32.543767    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mfd49"
Sep 16 02:13:32.543805 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:32.543794    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f227r"
Sep 16 02:13:32.544049 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:32.543798    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ck2vw"
Sep 16 02:13:32.544049 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:13:32.543887    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mfd49" podUID="6ce8b9ee-ca3a-4f48-8e6d-64a8221d2369"
Sep 16 02:13:32.544049 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:13:32.543970    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f227r" podUID="f5cc1e46-3424-49f8-8f2c-ec44086643d9"
Sep 16 02:13:32.544191 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:13:32.544068    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ck2vw" podUID="25e85d58-77e2-4428-b099-a5075363d56f"
Sep 16 02:13:34.543965 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:34.543933    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ck2vw"
Sep 16 02:13:34.543965 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:34.543955    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f227r"
Sep 16 02:13:34.543965 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:34.543960    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mfd49"
Sep 16 02:13:34.544342 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:13:34.544053    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f227r" podUID="f5cc1e46-3424-49f8-8f2c-ec44086643d9"
Sep 16 02:13:34.544342 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:13:34.544096    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mfd49" podUID="6ce8b9ee-ca3a-4f48-8e6d-64a8221d2369"
Sep 16 02:13:34.544342 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:13:34.544190    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ck2vw" podUID="25e85d58-77e2-4428-b099-a5075363d56f"
Sep 16 02:13:34.878035 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:34.877791    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jf4lp" event={"ID":"06c470fe-f6bf-4016-957c-56969805ddd0","Type":"ContainerStarted","Data":"0b21061a2415f12e688e463b0acd3de729492af04fd6944d355a1a6aaaf84440"}
Sep 16 02:13:34.878208 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:34.878112    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-jf4lp"
Sep 16 02:13:34.878208 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:34.878170    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-jf4lp"
Sep 16 02:13:34.880381 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:34.880355    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-h7czj" event={"ID":"a5be83aa-c2b7-4808-a32d-80a845f3435a","Type":"ContainerStarted","Data":"39d5ea73478cedfef1b7580632f712f93e22f8d234dfa2e10194736302fef5b6"}
Sep 16 02:13:34.893254 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:34.893224    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-jf4lp"
Sep 16 02:13:34.906225 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:34.906180    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-jf4lp" podStartSLOduration=7.210625657 podStartE2EDuration="26.906166289s" totalImagesPullingTime="19.695540632s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 02:13:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:13:34.904937868 +0000 UTC m=+27.823427130" watchObservedRunningTime="2026-09-16 02:13:34.906166289 +0000 UTC m=+27.824655538"
Sep 16 02:13:35.885091 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:35.885058    2558 generic.go:356] "Generic (PLEG): container finished" podID="a5be83aa-c2b7-4808-a32d-80a845f3435a" containerID="39d5ea73478cedfef1b7580632f712f93e22f8d234dfa2e10194736302fef5b6" exitCode=0
Sep 16 02:13:35.885764 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:35.885144    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-h7czj" event={"ID":"a5be83aa-c2b7-4808-a32d-80a845f3435a","Type":"ContainerDied","Data":"39d5ea73478cedfef1b7580632f712f93e22f8d234dfa2e10194736302fef5b6"}
Sep 16 02:13:35.885764 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:35.885549    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-jf4lp"
Sep 16 02:13:35.901847 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:35.901814    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-jf4lp"
Sep 16 02:13:36.453693 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:36.453647    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-mfd49"]
Sep 16 02:13:36.453914 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:36.453830    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mfd49"
Sep 16 02:13:36.453988 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:13:36.453933    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mfd49" podUID="6ce8b9ee-ca3a-4f48-8e6d-64a8221d2369"
Sep 16 02:13:36.454392 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:36.454369    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-ck2vw"]
Sep 16 02:13:36.454497 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:36.454476    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ck2vw"
Sep 16 02:13:36.454596 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:13:36.454573    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ck2vw" podUID="25e85d58-77e2-4428-b099-a5075363d56f"
Sep 16 02:13:36.455858 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:36.455837    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-f227r"]
Sep 16 02:13:36.455961 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:36.455915    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f227r"
Sep 16 02:13:36.456020 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:13:36.455989    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f227r" podUID="f5cc1e46-3424-49f8-8f2c-ec44086643d9"
Sep 16 02:13:37.893950 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:37.893756    2558 generic.go:356] "Generic (PLEG): container finished" podID="a5be83aa-c2b7-4808-a32d-80a845f3435a" containerID="f4a8f6ec12fa2b5aed8d8081979ed5c8e098e63c293a1a1914b26b6c4773ac0d" exitCode=0
Sep 16 02:13:37.894571 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:37.893845    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-h7czj" event={"ID":"a5be83aa-c2b7-4808-a32d-80a845f3435a","Type":"ContainerDied","Data":"f4a8f6ec12fa2b5aed8d8081979ed5c8e098e63c293a1a1914b26b6c4773ac0d"}
Sep 16 02:13:38.063889 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:38.063818    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-gmvp8"
Sep 16 02:13:38.064017 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:38.063933    2558 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 16 02:13:38.064505 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:38.064479    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-gmvp8"
Sep 16 02:13:38.544358 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:38.544327    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f227r"
Sep 16 02:13:38.544491 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:38.544327    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mfd49"
Sep 16 02:13:38.544491 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:13:38.544425    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f227r" podUID="f5cc1e46-3424-49f8-8f2c-ec44086643d9"
Sep 16 02:13:38.544568 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:38.544327    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ck2vw"
Sep 16 02:13:38.544602 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:13:38.544498    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mfd49" podUID="6ce8b9ee-ca3a-4f48-8e6d-64a8221d2369"
Sep 16 02:13:38.544635 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:13:38.544596    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ck2vw" podUID="25e85d58-77e2-4428-b099-a5075363d56f"
Sep 16 02:13:38.898531 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:38.898504    2558 generic.go:356] "Generic (PLEG): container finished" podID="a5be83aa-c2b7-4808-a32d-80a845f3435a" containerID="758d448b9223c2c7963ee945509159191c30814e0db5d0a124f232bbb23e05ff" exitCode=0
Sep 16 02:13:38.898923 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:38.898586    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-h7czj" event={"ID":"a5be83aa-c2b7-4808-a32d-80a845f3435a","Type":"ContainerDied","Data":"758d448b9223c2c7963ee945509159191c30814e0db5d0a124f232bbb23e05ff"}
Sep 16 02:13:40.346815 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:40.346781    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f5cc1e46-3424-49f8-8f2c-ec44086643d9-original-pull-secret\") pod \"global-pull-secret-syncer-f227r\" (UID: \"f5cc1e46-3424-49f8-8f2c-ec44086643d9\") " pod="kube-system/global-pull-secret-syncer-f227r"
Sep 16 02:13:40.347175 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:40.346826    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/25e85d58-77e2-4428-b099-a5075363d56f-metrics-certs\") pod \"network-metrics-daemon-ck2vw\" (UID: \"25e85d58-77e2-4428-b099-a5075363d56f\") " pod="openshift-multus/network-metrics-daemon-ck2vw"
Sep 16 02:13:40.347175 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:13:40.346918    2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 02:13:40.347175 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:13:40.346919    2558 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 02:13:40.347175 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:13:40.346968    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/25e85d58-77e2-4428-b099-a5075363d56f-metrics-certs podName:25e85d58-77e2-4428-b099-a5075363d56f nodeName:}" failed. No retries permitted until 2026-09-16 02:14:12.346951431 +0000 UTC m=+65.265440658 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/25e85d58-77e2-4428-b099-a5075363d56f-metrics-certs") pod "network-metrics-daemon-ck2vw" (UID: "25e85d58-77e2-4428-b099-a5075363d56f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 02:13:40.347175 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:13:40.346983    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f5cc1e46-3424-49f8-8f2c-ec44086643d9-original-pull-secret podName:f5cc1e46-3424-49f8-8f2c-ec44086643d9 nodeName:}" failed. No retries permitted until 2026-09-16 02:14:12.346977011 +0000 UTC m=+65.265466238 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f5cc1e46-3424-49f8-8f2c-ec44086643d9-original-pull-secret") pod "global-pull-secret-syncer-f227r" (UID: "f5cc1e46-3424-49f8-8f2c-ec44086643d9") : object "kube-system"/"original-pull-secret" not registered
Sep 16 02:13:40.447388 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:40.447354    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdqgf\" (UniqueName: \"kubernetes.io/projected/6ce8b9ee-ca3a-4f48-8e6d-64a8221d2369-kube-api-access-vdqgf\") pod \"network-check-target-mfd49\" (UID: \"6ce8b9ee-ca3a-4f48-8e6d-64a8221d2369\") " pod="openshift-network-diagnostics/network-check-target-mfd49"
Sep 16 02:13:40.447563 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:13:40.447500    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 02:13:40.447563 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:13:40.447524    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 02:13:40.447563 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:13:40.447536    2558 projected.go:196] Error preparing data for projected volume kube-api-access-vdqgf for pod openshift-network-diagnostics/network-check-target-mfd49: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 02:13:40.447720 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:13:40.447595    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6ce8b9ee-ca3a-4f48-8e6d-64a8221d2369-kube-api-access-vdqgf podName:6ce8b9ee-ca3a-4f48-8e6d-64a8221d2369 nodeName:}" failed. No retries permitted until 2026-09-16 02:14:12.44757682 +0000 UTC m=+65.366066050 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-vdqgf" (UniqueName: "kubernetes.io/projected/6ce8b9ee-ca3a-4f48-8e6d-64a8221d2369-kube-api-access-vdqgf") pod "network-check-target-mfd49" (UID: "6ce8b9ee-ca3a-4f48-8e6d-64a8221d2369") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 02:13:40.543918 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:40.543846    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ck2vw"
Sep 16 02:13:40.543918 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:40.543878    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f227r"
Sep 16 02:13:40.544164 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:40.543859    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mfd49"
Sep 16 02:13:40.544164 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:13:40.543982    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ck2vw" podUID="25e85d58-77e2-4428-b099-a5075363d56f"
Sep 16 02:13:40.544164 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:13:40.544121    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mfd49" podUID="6ce8b9ee-ca3a-4f48-8e6d-64a8221d2369"
Sep 16 02:13:40.544323 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:13:40.544216    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f227r" podUID="f5cc1e46-3424-49f8-8f2c-ec44086643d9"
Sep 16 02:13:41.357893 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:41.357820    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-34.ec2.internal" event="NodeReady"
Sep 16 02:13:41.358349 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:41.357952    2558 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 16 02:13:41.400012 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:41.399978    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-q8tpf"]
Sep 16 02:13:41.403089 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:41.403060    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-b7cxx"]
Sep 16 02:13:41.403234 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:41.403216    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-q8tpf"
Sep 16 02:13:41.406184 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:41.405960    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-b7cxx"
Sep 16 02:13:41.406184 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:41.406000    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 16 02:13:41.406184 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:41.406018    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 16 02:13:41.406184 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:41.406043    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-h6q9g\""
Sep 16 02:13:41.408476 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:41.408456    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 16 02:13:41.408590 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:41.408490    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 16 02:13:41.408590 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:41.408571    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-lfqpp\""
Sep 16 02:13:41.408766 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:41.408746    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 16 02:13:41.412291 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:41.411636    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-b7cxx"]
Sep 16 02:13:41.414626 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:41.414571    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-q8tpf"]
Sep 16 02:13:41.555247 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:41.555211    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/dd03566d-e8e3-4aad-b3b2-0a06e43e5eea-metrics-tls\") pod \"dns-default-q8tpf\" (UID: \"dd03566d-e8e3-4aad-b3b2-0a06e43e5eea\") " pod="openshift-dns/dns-default-q8tpf"
Sep 16 02:13:41.555416 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:41.555255    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-666ws\" (UniqueName: \"kubernetes.io/projected/504ae285-7f73-4ebf-b39b-192c08b6e922-kube-api-access-666ws\") pod \"ingress-canary-b7cxx\" (UID: \"504ae285-7f73-4ebf-b39b-192c08b6e922\") " pod="openshift-ingress-canary/ingress-canary-b7cxx"
Sep 16 02:13:41.555416 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:41.555284    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd03566d-e8e3-4aad-b3b2-0a06e43e5eea-tmp-dir\") pod \"dns-default-q8tpf\" (UID: \"dd03566d-e8e3-4aad-b3b2-0a06e43e5eea\") " pod="openshift-dns/dns-default-q8tpf"
Sep 16 02:13:41.555416 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:41.555315    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/504ae285-7f73-4ebf-b39b-192c08b6e922-cert\") pod \"ingress-canary-b7cxx\" (UID: \"504ae285-7f73-4ebf-b39b-192c08b6e922\") " pod="openshift-ingress-canary/ingress-canary-b7cxx"
Sep 16 02:13:41.555416 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:41.555343    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqg66\" (UniqueName: \"kubernetes.io/projected/dd03566d-e8e3-4aad-b3b2-0a06e43e5eea-kube-api-access-lqg66\") pod \"dns-default-q8tpf\" (UID: \"dd03566d-e8e3-4aad-b3b2-0a06e43e5eea\") " pod="openshift-dns/dns-default-q8tpf"
Sep 16 02:13:41.555579 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:41.555439    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/dd03566d-e8e3-4aad-b3b2-0a06e43e5eea-config-volume\") pod \"dns-default-q8tpf\" (UID: \"dd03566d-e8e3-4aad-b3b2-0a06e43e5eea\") " pod="openshift-dns/dns-default-q8tpf"
Sep 16 02:13:41.656693 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:41.656658    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/dd03566d-e8e3-4aad-b3b2-0a06e43e5eea-config-volume\") pod \"dns-default-q8tpf\" (UID: \"dd03566d-e8e3-4aad-b3b2-0a06e43e5eea\") " pod="openshift-dns/dns-default-q8tpf"
Sep 16 02:13:41.656893 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:41.656745    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/dd03566d-e8e3-4aad-b3b2-0a06e43e5eea-metrics-tls\") pod \"dns-default-q8tpf\" (UID: \"dd03566d-e8e3-4aad-b3b2-0a06e43e5eea\") " pod="openshift-dns/dns-default-q8tpf"
Sep 16 02:13:41.656893 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:41.656775    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-666ws\" (UniqueName: \"kubernetes.io/projected/504ae285-7f73-4ebf-b39b-192c08b6e922-kube-api-access-666ws\") pod \"ingress-canary-b7cxx\" (UID: \"504ae285-7f73-4ebf-b39b-192c08b6e922\") " pod="openshift-ingress-canary/ingress-canary-b7cxx"
Sep 16 02:13:41.656893 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:41.656797    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd03566d-e8e3-4aad-b3b2-0a06e43e5eea-tmp-dir\") pod \"dns-default-q8tpf\" (UID: \"dd03566d-e8e3-4aad-b3b2-0a06e43e5eea\") " pod="openshift-dns/dns-default-q8tpf"
Sep 16 02:13:41.656893 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:41.656818    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/504ae285-7f73-4ebf-b39b-192c08b6e922-cert\") pod \"ingress-canary-b7cxx\" (UID: \"504ae285-7f73-4ebf-b39b-192c08b6e922\") " pod="openshift-ingress-canary/ingress-canary-b7cxx"
Sep 16 02:13:41.656893 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:41.656844    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqg66\" (UniqueName: \"kubernetes.io/projected/dd03566d-e8e3-4aad-b3b2-0a06e43e5eea-kube-api-access-lqg66\") pod \"dns-default-q8tpf\" (UID: \"dd03566d-e8e3-4aad-b3b2-0a06e43e5eea\") " pod="openshift-dns/dns-default-q8tpf"
Sep 16 02:13:41.657140 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:13:41.656901    2558 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 02:13:41.657140 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:13:41.656984    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dd03566d-e8e3-4aad-b3b2-0a06e43e5eea-metrics-tls podName:dd03566d-e8e3-4aad-b3b2-0a06e43e5eea nodeName:}" failed. No retries permitted until 2026-09-16 02:13:42.15696488 +0000 UTC m=+35.075454121 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/dd03566d-e8e3-4aad-b3b2-0a06e43e5eea-metrics-tls") pod "dns-default-q8tpf" (UID: "dd03566d-e8e3-4aad-b3b2-0a06e43e5eea") : secret "dns-default-metrics-tls" not found
Sep 16 02:13:41.657647 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:41.657621    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd03566d-e8e3-4aad-b3b2-0a06e43e5eea-tmp-dir\") pod \"dns-default-q8tpf\" (UID: \"dd03566d-e8e3-4aad-b3b2-0a06e43e5eea\") " pod="openshift-dns/dns-default-q8tpf"
Sep 16 02:13:41.657762 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:41.657653    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/dd03566d-e8e3-4aad-b3b2-0a06e43e5eea-config-volume\") pod \"dns-default-q8tpf\" (UID: \"dd03566d-e8e3-4aad-b3b2-0a06e43e5eea\") " pod="openshift-dns/dns-default-q8tpf"
Sep 16 02:13:41.661480 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:41.661358    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/504ae285-7f73-4ebf-b39b-192c08b6e922-cert\") pod \"ingress-canary-b7cxx\" (UID: \"504ae285-7f73-4ebf-b39b-192c08b6e922\") " pod="openshift-ingress-canary/ingress-canary-b7cxx"
Sep 16 02:13:41.665113 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:41.665090    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqg66\" (UniqueName: \"kubernetes.io/projected/dd03566d-e8e3-4aad-b3b2-0a06e43e5eea-kube-api-access-lqg66\") pod \"dns-default-q8tpf\" (UID: \"dd03566d-e8e3-4aad-b3b2-0a06e43e5eea\") " pod="openshift-dns/dns-default-q8tpf"
Sep 16 02:13:41.665257 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:41.665115    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-666ws\" (UniqueName: \"kubernetes.io/projected/504ae285-7f73-4ebf-b39b-192c08b6e922-kube-api-access-666ws\") pod \"ingress-canary-b7cxx\" (UID: \"504ae285-7f73-4ebf-b39b-192c08b6e922\") " pod="openshift-ingress-canary/ingress-canary-b7cxx"
Sep 16 02:13:41.723221 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:41.723184    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-b7cxx"
Sep 16 02:13:42.006267 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:42.006177    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-b7cxx"]
Sep 16 02:13:42.161106 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:42.161065    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/dd03566d-e8e3-4aad-b3b2-0a06e43e5eea-metrics-tls\") pod \"dns-default-q8tpf\" (UID: \"dd03566d-e8e3-4aad-b3b2-0a06e43e5eea\") " pod="openshift-dns/dns-default-q8tpf"
Sep 16 02:13:42.161279 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:13:42.161218    2558 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 02:13:42.161279 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:13:42.161276    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dd03566d-e8e3-4aad-b3b2-0a06e43e5eea-metrics-tls podName:dd03566d-e8e3-4aad-b3b2-0a06e43e5eea nodeName:}" failed. No retries permitted until 2026-09-16 02:13:43.161259789 +0000 UTC m=+36.079749040 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/dd03566d-e8e3-4aad-b3b2-0a06e43e5eea-metrics-tls") pod "dns-default-q8tpf" (UID: "dd03566d-e8e3-4aad-b3b2-0a06e43e5eea") : secret "dns-default-metrics-tls" not found
Sep 16 02:13:42.543801 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:42.543760    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f227r"
Sep 16 02:13:42.544604 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:42.543802    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mfd49"
Sep 16 02:13:42.544604 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:42.543813    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ck2vw"
Sep 16 02:13:42.546765 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:42.546747    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 16 02:13:42.547461 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:42.547441    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-ksfsh\""
Sep 16 02:13:42.547562 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:42.547529    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 16 02:13:42.547626 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:42.547572    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-2ht6m\""
Sep 16 02:13:42.547750 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:42.547732    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 16 02:13:42.547852 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:42.547837    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 16 02:13:43.169791 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:43.169751    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/dd03566d-e8e3-4aad-b3b2-0a06e43e5eea-metrics-tls\") pod \"dns-default-q8tpf\" (UID: \"dd03566d-e8e3-4aad-b3b2-0a06e43e5eea\") " pod="openshift-dns/dns-default-q8tpf"
Sep 16 02:13:43.169959 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:13:43.169938    2558 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 02:13:43.170034 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:13:43.170022    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dd03566d-e8e3-4aad-b3b2-0a06e43e5eea-metrics-tls podName:dd03566d-e8e3-4aad-b3b2-0a06e43e5eea nodeName:}" failed. No retries permitted until 2026-09-16 02:13:45.169999076 +0000 UTC m=+38.088488316 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/dd03566d-e8e3-4aad-b3b2-0a06e43e5eea-metrics-tls") pod "dns-default-q8tpf" (UID: "dd03566d-e8e3-4aad-b3b2-0a06e43e5eea") : secret "dns-default-metrics-tls" not found
Sep 16 02:13:43.240936 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:13:43.240906    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod504ae285_7f73_4ebf_b39b_192c08b6e922.slice/crio-ee1a370ba6e20671c4082212fb098f177e4872c19c1796d278397956a408201d WatchSource:0}: Error finding container ee1a370ba6e20671c4082212fb098f177e4872c19c1796d278397956a408201d: Status 404 returned error can't find the container with id ee1a370ba6e20671c4082212fb098f177e4872c19c1796d278397956a408201d
Sep 16 02:13:43.913440 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:43.913409    2558 generic.go:356] "Generic (PLEG): container finished" podID="a5be83aa-c2b7-4808-a32d-80a845f3435a" containerID="e3e41cbc4100029886ab188fdfb50737eb43554db81666e153a32af3c24add91" exitCode=0
Sep 16 02:13:43.913995 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:43.913488    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-h7czj" event={"ID":"a5be83aa-c2b7-4808-a32d-80a845f3435a","Type":"ContainerDied","Data":"e3e41cbc4100029886ab188fdfb50737eb43554db81666e153a32af3c24add91"}
Sep 16 02:13:43.914640 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:43.914620    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-b7cxx" event={"ID":"504ae285-7f73-4ebf-b39b-192c08b6e922","Type":"ContainerStarted","Data":"ee1a370ba6e20671c4082212fb098f177e4872c19c1796d278397956a408201d"}
Sep 16 02:13:44.921297 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:44.921103    2558 generic.go:356] "Generic (PLEG): container finished" podID="a5be83aa-c2b7-4808-a32d-80a845f3435a" containerID="5283846c2c14fee1683588235e26b2ccdc7540c04149f96195486181f6315e91" exitCode=0
Sep 16 02:13:44.921297 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:44.921161    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-h7czj" event={"ID":"a5be83aa-c2b7-4808-a32d-80a845f3435a","Type":"ContainerDied","Data":"5283846c2c14fee1683588235e26b2ccdc7540c04149f96195486181f6315e91"}
Sep 16 02:13:45.187972 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:45.187878    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/dd03566d-e8e3-4aad-b3b2-0a06e43e5eea-metrics-tls\") pod \"dns-default-q8tpf\" (UID: \"dd03566d-e8e3-4aad-b3b2-0a06e43e5eea\") " pod="openshift-dns/dns-default-q8tpf"
Sep 16 02:13:45.188117 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:13:45.188057    2558 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 02:13:45.188203 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:13:45.188139    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dd03566d-e8e3-4aad-b3b2-0a06e43e5eea-metrics-tls podName:dd03566d-e8e3-4aad-b3b2-0a06e43e5eea nodeName:}" failed. No retries permitted until 2026-09-16 02:13:49.188117262 +0000 UTC m=+42.106606494 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/dd03566d-e8e3-4aad-b3b2-0a06e43e5eea-metrics-tls") pod "dns-default-q8tpf" (UID: "dd03566d-e8e3-4aad-b3b2-0a06e43e5eea") : secret "dns-default-metrics-tls" not found
Sep 16 02:13:45.926613 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:45.926579    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-h7czj" event={"ID":"a5be83aa-c2b7-4808-a32d-80a845f3435a","Type":"ContainerStarted","Data":"bb99402cde238c2f4eb4b8ae99551c40569636e08b5631d80b995c7e92fa6fc9"}
Sep 16 02:13:45.947525 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:45.947466    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-h7czj" podStartSLOduration=7.222926441 podStartE2EDuration="37.947446666s" totalImagesPullingTime="30.724520225s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 02:13:08 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:13:45.945847374 +0000 UTC m=+38.864336624" watchObservedRunningTime="2026-09-16 02:13:45.947446666 +0000 UTC m=+38.865935916"
Sep 16 02:13:46.929982 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:46.929942    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-b7cxx" event={"ID":"504ae285-7f73-4ebf-b39b-192c08b6e922","Type":"ContainerStarted","Data":"8a68474659f46275bbebce3fc761458ca4a7eeb230df8c8df1ac35ebbea27ae2"}
Sep 16 02:13:46.944451 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:46.944405    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-b7cxx" podStartSLOduration=3.3505818339999998 podStartE2EDuration="5.944391547s" totalImagesPullingTime="2.593809713s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 02:13:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:13:46.943563081 +0000 UTC m=+39.862052333" watchObservedRunningTime="2026-09-16 02:13:46.944391547 +0000 UTC m=+39.862880796"
Sep 16 02:13:49.218867 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:49.218810    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/dd03566d-e8e3-4aad-b3b2-0a06e43e5eea-metrics-tls\") pod \"dns-default-q8tpf\" (UID: \"dd03566d-e8e3-4aad-b3b2-0a06e43e5eea\") " pod="openshift-dns/dns-default-q8tpf"
Sep 16 02:13:49.219258 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:13:49.218963    2558 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 02:13:49.219258 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:13:49.219026    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dd03566d-e8e3-4aad-b3b2-0a06e43e5eea-metrics-tls podName:dd03566d-e8e3-4aad-b3b2-0a06e43e5eea nodeName:}" failed. No retries permitted until 2026-09-16 02:13:57.219011232 +0000 UTC m=+50.137500458 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/dd03566d-e8e3-4aad-b3b2-0a06e43e5eea-metrics-tls") pod "dns-default-q8tpf" (UID: "dd03566d-e8e3-4aad-b3b2-0a06e43e5eea") : secret "dns-default-metrics-tls" not found
Sep 16 02:13:57.271745 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:13:57.271680    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/dd03566d-e8e3-4aad-b3b2-0a06e43e5eea-metrics-tls\") pod \"dns-default-q8tpf\" (UID: \"dd03566d-e8e3-4aad-b3b2-0a06e43e5eea\") " pod="openshift-dns/dns-default-q8tpf"
Sep 16 02:13:57.272195 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:13:57.271840    2558 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 02:13:57.272195 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:13:57.271912    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dd03566d-e8e3-4aad-b3b2-0a06e43e5eea-metrics-tls podName:dd03566d-e8e3-4aad-b3b2-0a06e43e5eea nodeName:}" failed. No retries permitted until 2026-09-16 02:14:13.271896354 +0000 UTC m=+66.190385581 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/dd03566d-e8e3-4aad-b3b2-0a06e43e5eea-metrics-tls") pod "dns-default-q8tpf" (UID: "dd03566d-e8e3-4aad-b3b2-0a06e43e5eea") : secret "dns-default-metrics-tls" not found
Sep 16 02:14:07.904620 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:14:07.904583    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-jf4lp"
Sep 16 02:14:12.377712 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:14:12.377640    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f5cc1e46-3424-49f8-8f2c-ec44086643d9-original-pull-secret\") pod \"global-pull-secret-syncer-f227r\" (UID: \"f5cc1e46-3424-49f8-8f2c-ec44086643d9\") " pod="kube-system/global-pull-secret-syncer-f227r"
Sep 16 02:14:12.377712 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:14:12.377714    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/25e85d58-77e2-4428-b099-a5075363d56f-metrics-certs\") pod \"network-metrics-daemon-ck2vw\" (UID: \"25e85d58-77e2-4428-b099-a5075363d56f\") " pod="openshift-multus/network-metrics-daemon-ck2vw"
Sep 16 02:14:12.380578 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:14:12.380554    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 16 02:14:12.380656 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:14:12.380562    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 16 02:14:12.388644 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:14:12.388623    2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 16 02:14:12.388758 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:14:12.388722    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/25e85d58-77e2-4428-b099-a5075363d56f-metrics-certs podName:25e85d58-77e2-4428-b099-a5075363d56f nodeName:}" failed. No retries permitted until 2026-09-16 02:15:16.388684904 +0000 UTC m=+129.307174151 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/25e85d58-77e2-4428-b099-a5075363d56f-metrics-certs") pod "network-metrics-daemon-ck2vw" (UID: "25e85d58-77e2-4428-b099-a5075363d56f") : secret "metrics-daemon-secret" not found
Sep 16 02:14:12.391542 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:14:12.391523    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f5cc1e46-3424-49f8-8f2c-ec44086643d9-original-pull-secret\") pod \"global-pull-secret-syncer-f227r\" (UID: \"f5cc1e46-3424-49f8-8f2c-ec44086643d9\") " pod="kube-system/global-pull-secret-syncer-f227r"
Sep 16 02:14:12.478326 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:14:12.478272    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdqgf\" (UniqueName: \"kubernetes.io/projected/6ce8b9ee-ca3a-4f48-8e6d-64a8221d2369-kube-api-access-vdqgf\") pod \"network-check-target-mfd49\" (UID: \"6ce8b9ee-ca3a-4f48-8e6d-64a8221d2369\") " pod="openshift-network-diagnostics/network-check-target-mfd49"
Sep 16 02:14:12.480778 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:14:12.480759    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 16 02:14:12.492128 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:14:12.492105    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 16 02:14:12.502608 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:14:12.502586    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdqgf\" (UniqueName: \"kubernetes.io/projected/6ce8b9ee-ca3a-4f48-8e6d-64a8221d2369-kube-api-access-vdqgf\") pod \"network-check-target-mfd49\" (UID: \"6ce8b9ee-ca3a-4f48-8e6d-64a8221d2369\") " pod="openshift-network-diagnostics/network-check-target-mfd49"
Sep 16 02:14:12.556273 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:14:12.556229    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f227r"
Sep 16 02:14:12.566866 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:14:12.566849    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-ksfsh\""
Sep 16 02:14:12.574741 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:14:12.574720    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mfd49"
Sep 16 02:14:12.824677 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:14:12.824636    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-f227r"]
Sep 16 02:14:12.829170 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:14:12.829139    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5cc1e46_3424_49f8_8f2c_ec44086643d9.slice/crio-b821de3a531dbb292e700206f898d908dcce09dbb526c723d39c921942653900 WatchSource:0}: Error finding container b821de3a531dbb292e700206f898d908dcce09dbb526c723d39c921942653900: Status 404 returned error can't find the container with id b821de3a531dbb292e700206f898d908dcce09dbb526c723d39c921942653900
Sep 16 02:14:12.844775 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:14:12.844752    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-mfd49"]
Sep 16 02:14:12.847527 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:14:12.847503    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ce8b9ee_ca3a_4f48_8e6d_64a8221d2369.slice/crio-3efe4b5a4d88b58d75e705d20e56f7d4fc9eb050cbc9f2d5c326e4b581141fe2 WatchSource:0}: Error finding container 3efe4b5a4d88b58d75e705d20e56f7d4fc9eb050cbc9f2d5c326e4b581141fe2: Status 404 returned error can't find the container with id 3efe4b5a4d88b58d75e705d20e56f7d4fc9eb050cbc9f2d5c326e4b581141fe2
Sep 16 02:14:12.991577 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:14:12.991536    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-mfd49" event={"ID":"6ce8b9ee-ca3a-4f48-8e6d-64a8221d2369","Type":"ContainerStarted","Data":"3efe4b5a4d88b58d75e705d20e56f7d4fc9eb050cbc9f2d5c326e4b581141fe2"}
Sep 16 02:14:12.992647 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:14:12.992627    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-f227r" event={"ID":"f5cc1e46-3424-49f8-8f2c-ec44086643d9","Type":"ContainerStarted","Data":"b821de3a531dbb292e700206f898d908dcce09dbb526c723d39c921942653900"}
Sep 16 02:14:13.284598 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:14:13.284556    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/dd03566d-e8e3-4aad-b3b2-0a06e43e5eea-metrics-tls\") pod \"dns-default-q8tpf\" (UID: \"dd03566d-e8e3-4aad-b3b2-0a06e43e5eea\") " pod="openshift-dns/dns-default-q8tpf"
Sep 16 02:14:13.284786 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:14:13.284708    2558 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 02:14:13.284786 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:14:13.284770    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dd03566d-e8e3-4aad-b3b2-0a06e43e5eea-metrics-tls podName:dd03566d-e8e3-4aad-b3b2-0a06e43e5eea nodeName:}" failed. No retries permitted until 2026-09-16 02:14:45.284753133 +0000 UTC m=+98.203242360 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/dd03566d-e8e3-4aad-b3b2-0a06e43e5eea-metrics-tls") pod "dns-default-q8tpf" (UID: "dd03566d-e8e3-4aad-b3b2-0a06e43e5eea") : secret "dns-default-metrics-tls" not found
Sep 16 02:14:18.009051 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:14:18.009002    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-mfd49" event={"ID":"6ce8b9ee-ca3a-4f48-8e6d-64a8221d2369","Type":"ContainerStarted","Data":"1951df7a5bd54a72dd8ac0431b2aea3055d792126ea5a6310baa46cc68f6a761"}
Sep 16 02:14:18.009527 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:14:18.009122    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-mfd49"
Sep 16 02:14:18.010480 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:14:18.010451    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-f227r" event={"ID":"f5cc1e46-3424-49f8-8f2c-ec44086643d9","Type":"ContainerStarted","Data":"6e888c58f7e42e133e2407afc6a6ab74ca31fae1c9f0d94df3204e3f8be5a013"}
Sep 16 02:14:18.022999 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:14:18.022957    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-mfd49" podStartSLOduration=65.558054422 podStartE2EDuration="1m10.022944995s" totalImagesPullingTime="4.464890573s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 02:13:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:14:18.02245363 +0000 UTC m=+70.940942889" watchObservedRunningTime="2026-09-16 02:14:18.022944995 +0000 UTC m=+70.941434244"
Sep 16 02:14:29.802085 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:14:29.801954    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-f227r" podStartSLOduration=77.314453774 podStartE2EDuration="1m21.801937739s" totalImagesPullingTime="4.487483965s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 02:13:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:14:18.037808029 +0000 UTC m=+70.956297278" watchObservedRunningTime="2026-09-16 02:14:29.801937739 +0000 UTC m=+82.720427365"
Sep 16 02:14:29.802530 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:14:29.802513    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-69b45b4b7d-hcccn"]
Sep 16 02:14:29.806501 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:14:29.806483    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-69b45b4b7d-hcccn"
Sep 16 02:14:29.810828 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:14:29.810805    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 16 02:14:29.810928 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:14:29.810809    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-sjd9q\""
Sep 16 02:14:29.810928 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:14:29.810880    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 16 02:14:29.810928 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:14:29.810834    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 16 02:14:29.810928 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:14:29.810810    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Sep 16 02:14:29.817636 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:14:29.817615    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-69b45b4b7d-hcccn"]
Sep 16 02:14:29.818380 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:14:29.818361    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-588bc5dc84-p8p9z"]
Sep 16 02:14:29.821203 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:14:29.821185    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-588bc5dc84-p8p9z"
Sep 16 02:14:29.824301 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:14:29.824282    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Sep 16 02:14:29.824301 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:14:29.824292    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Sep 16 02:14:29.824894 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:14:29.824875    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Sep 16 02:14:29.825208 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:14:29.825180    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Sep 16 02:14:29.847707 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:14:29.847675    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-588bc5dc84-p8p9z"]
Sep 16 02:14:29.891734 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:14:29.891706    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r87cz\" (UniqueName: \"kubernetes.io/projected/bf8e8a44-fbcc-44bf-a8e0-0633285ae3ba-kube-api-access-r87cz\") pod \"cluster-proxy-proxy-agent-588bc5dc84-p8p9z\" (UID: \"bf8e8a44-fbcc-44bf-a8e0-0633285ae3ba\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-588bc5dc84-p8p9z"
Sep 16 02:14:29.891852 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:14:29.891743    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ea7219ae-5620-4c71-8889-c89cdfa8ed2d-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-69b45b4b7d-hcccn\" (UID: \"ea7219ae-5620-4c71-8889-c89cdfa8ed2d\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-69b45b4b7d-hcccn"
Sep 16 02:14:29.891852 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:14:29.891760    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/bf8e8a44-fbcc-44bf-a8e0-0633285ae3ba-ca\") pod \"cluster-proxy-proxy-agent-588bc5dc84-p8p9z\" (UID: \"bf8e8a44-fbcc-44bf-a8e0-0633285ae3ba\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-588bc5dc84-p8p9z"
Sep 16 02:14:29.891852 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:14:29.891778    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/bf8e8a44-fbcc-44bf-a8e0-0633285ae3ba-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-588bc5dc84-p8p9z\" (UID: \"bf8e8a44-fbcc-44bf-a8e0-0633285ae3ba\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-588bc5dc84-p8p9z"
Sep 16 02:14:29.891852 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:14:29.891794    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/bf8e8a44-fbcc-44bf-a8e0-0633285ae3ba-hub\") pod \"cluster-proxy-proxy-agent-588bc5dc84-p8p9z\" (UID: \"bf8e8a44-fbcc-44bf-a8e0-0633285ae3ba\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-588bc5dc84-p8p9z"
Sep 16 02:14:29.891852 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:14:29.891843    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf8e8a44-fbcc-44bf-a8e0-0633285ae3ba-additional-service-ca\") pod \"cluster-proxy-proxy-agent-588bc5dc84-p8p9z\" (UID: \"bf8e8a44-fbcc-44bf-a8e0-0633285ae3ba\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-588bc5dc84-p8p9z"
Sep 16 02:14:29.892012 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:14:29.891866    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/bf8e8a44-fbcc-44bf-a8e0-0633285ae3ba-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-588bc5dc84-p8p9z\" (UID: \"bf8e8a44-fbcc-44bf-a8e0-0633285ae3ba\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-588bc5dc84-p8p9z"
Sep 16 02:14:29.892012 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:14:29.891893    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlp82\" (UniqueName: \"kubernetes.io/projected/ea7219ae-5620-4c71-8889-c89cdfa8ed2d-kube-api-access-zlp82\") pod \"managed-serviceaccount-addon-agent-69b45b4b7d-hcccn\" (UID: \"ea7219ae-5620-4c71-8889-c89cdfa8ed2d\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-69b45b4b7d-hcccn"
Sep 16 02:14:29.992418 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:14:29.992393    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r87cz\" (UniqueName: \"kubernetes.io/projected/bf8e8a44-fbcc-44bf-a8e0-0633285ae3ba-kube-api-access-r87cz\") pod \"cluster-proxy-proxy-agent-588bc5dc84-p8p9z\" (UID: \"bf8e8a44-fbcc-44bf-a8e0-0633285ae3ba\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-588bc5dc84-p8p9z"
Sep 16 02:14:29.992521 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:14:29.992425    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ea7219ae-5620-4c71-8889-c89cdfa8ed2d-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-69b45b4b7d-hcccn\" (UID: \"ea7219ae-5620-4c71-8889-c89cdfa8ed2d\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-69b45b4b7d-hcccn"
Sep 16 02:14:29.992521 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:14:29.992442    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/bf8e8a44-fbcc-44bf-a8e0-0633285ae3ba-ca\") pod \"cluster-proxy-proxy-agent-588bc5dc84-p8p9z\" (UID: \"bf8e8a44-fbcc-44bf-a8e0-0633285ae3ba\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-588bc5dc84-p8p9z"
Sep 16 02:14:29.992521 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:14:29.992458    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/bf8e8a44-fbcc-44bf-a8e0-0633285ae3ba-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-588bc5dc84-p8p9z\" (UID: \"bf8e8a44-fbcc-44bf-a8e0-0633285ae3ba\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-588bc5dc84-p8p9z"
Sep 16 02:14:29.992619 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:14:29.992513    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/bf8e8a44-fbcc-44bf-a8e0-0633285ae3ba-hub\") pod \"cluster-proxy-proxy-agent-588bc5dc84-p8p9z\" (UID: \"bf8e8a44-fbcc-44bf-a8e0-0633285ae3ba\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-588bc5dc84-p8p9z"
Sep 16 02:14:29.992619 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:14:29.992558    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf8e8a44-fbcc-44bf-a8e0-0633285ae3ba-additional-service-ca\") pod \"cluster-proxy-proxy-agent-588bc5dc84-p8p9z\" (UID: \"bf8e8a44-fbcc-44bf-a8e0-0633285ae3ba\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-588bc5dc84-p8p9z"
Sep 16 02:14:29.992619 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:14:29.992599    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/bf8e8a44-fbcc-44bf-a8e0-0633285ae3ba-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-588bc5dc84-p8p9z\" (UID: \"bf8e8a44-fbcc-44bf-a8e0-0633285ae3ba\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-588bc5dc84-p8p9z"
Sep 16 02:14:29.992764 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:14:29.992626    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlp82\" (UniqueName: \"kubernetes.io/projected/ea7219ae-5620-4c71-8889-c89cdfa8ed2d-kube-api-access-zlp82\") pod \"managed-serviceaccount-addon-agent-69b45b4b7d-hcccn\" (UID: \"ea7219ae-5620-4c71-8889-c89cdfa8ed2d\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-69b45b4b7d-hcccn"
Sep 16 02:14:29.993336 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:14:29.993308    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf8e8a44-fbcc-44bf-a8e0-0633285ae3ba-additional-service-ca\") pod \"cluster-proxy-proxy-agent-588bc5dc84-p8p9z\" (UID: \"bf8e8a44-fbcc-44bf-a8e0-0633285ae3ba\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-588bc5dc84-p8p9z"
Sep 16 02:14:29.995063 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:14:29.995032    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/bf8e8a44-fbcc-44bf-a8e0-0633285ae3ba-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-588bc5dc84-p8p9z\" (UID: \"bf8e8a44-fbcc-44bf-a8e0-0633285ae3ba\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-588bc5dc84-p8p9z"
Sep 16 02:14:29.995063 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:14:29.995050    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ea7219ae-5620-4c71-8889-c89cdfa8ed2d-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-69b45b4b7d-hcccn\" (UID: \"ea7219ae-5620-4c71-8889-c89cdfa8ed2d\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-69b45b4b7d-hcccn"
Sep 16 02:14:29.995220 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:14:29.995141    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/bf8e8a44-fbcc-44bf-a8e0-0633285ae3ba-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-588bc5dc84-p8p9z\" (UID: \"bf8e8a44-fbcc-44bf-a8e0-0633285ae3ba\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-588bc5dc84-p8p9z"
Sep 16 02:14:29.995281 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:14:29.995226    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/bf8e8a44-fbcc-44bf-a8e0-0633285ae3ba-ca\") pod \"cluster-proxy-proxy-agent-588bc5dc84-p8p9z\" (UID: \"bf8e8a44-fbcc-44bf-a8e0-0633285ae3ba\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-588bc5dc84-p8p9z"
Sep 16 02:14:29.995378 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:14:29.995358    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/bf8e8a44-fbcc-44bf-a8e0-0633285ae3ba-hub\") pod \"cluster-proxy-proxy-agent-588bc5dc84-p8p9z\" (UID: \"bf8e8a44-fbcc-44bf-a8e0-0633285ae3ba\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-588bc5dc84-p8p9z"
Sep 16 02:14:30.000574 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:14:30.000552    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r87cz\" (UniqueName: \"kubernetes.io/projected/bf8e8a44-fbcc-44bf-a8e0-0633285ae3ba-kube-api-access-r87cz\") pod \"cluster-proxy-proxy-agent-588bc5dc84-p8p9z\" (UID: \"bf8e8a44-fbcc-44bf-a8e0-0633285ae3ba\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-588bc5dc84-p8p9z"
Sep 16 02:14:30.000967 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:14:30.000943    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlp82\" (UniqueName: \"kubernetes.io/projected/ea7219ae-5620-4c71-8889-c89cdfa8ed2d-kube-api-access-zlp82\") pod \"managed-serviceaccount-addon-agent-69b45b4b7d-hcccn\" (UID: \"ea7219ae-5620-4c71-8889-c89cdfa8ed2d\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-69b45b4b7d-hcccn"
Sep 16 02:14:30.124226 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:14:30.124153    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-69b45b4b7d-hcccn"
Sep 16 02:14:30.133137 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:14:30.133115    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-588bc5dc84-p8p9z"
Sep 16 02:14:30.398557 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:14:30.398531    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-69b45b4b7d-hcccn"]
Sep 16 02:14:30.401305 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:14:30.401285    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podea7219ae_5620_4c71_8889_c89cdfa8ed2d.slice/crio-6c2df0224adf552c2d766509429294578ae97e0ef6dc891c77039125e16769d1 WatchSource:0}: Error finding container 6c2df0224adf552c2d766509429294578ae97e0ef6dc891c77039125e16769d1: Status 404 returned error can't find the container with id 6c2df0224adf552c2d766509429294578ae97e0ef6dc891c77039125e16769d1
Sep 16 02:14:30.410295 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:14:30.410272    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-588bc5dc84-p8p9z"]
Sep 16 02:14:30.412681 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:14:30.412662    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbf8e8a44_fbcc_44bf_a8e0_0633285ae3ba.slice/crio-f7b783447e0ae35e370bd302aaa804e1a9c6b2bafc63bef2bbabe3662c6b41bd WatchSource:0}: Error finding container f7b783447e0ae35e370bd302aaa804e1a9c6b2bafc63bef2bbabe3662c6b41bd: Status 404 returned error can't find the container with id f7b783447e0ae35e370bd302aaa804e1a9c6b2bafc63bef2bbabe3662c6b41bd
Sep 16 02:14:31.040590 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:14:31.040554    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-69b45b4b7d-hcccn" event={"ID":"ea7219ae-5620-4c71-8889-c89cdfa8ed2d","Type":"ContainerStarted","Data":"6c2df0224adf552c2d766509429294578ae97e0ef6dc891c77039125e16769d1"}
Sep 16 02:14:31.041560 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:14:31.041539    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-588bc5dc84-p8p9z" event={"ID":"bf8e8a44-fbcc-44bf-a8e0-0633285ae3ba","Type":"ContainerStarted","Data":"f7b783447e0ae35e370bd302aaa804e1a9c6b2bafc63bef2bbabe3662c6b41bd"}
Sep 16 02:14:35.055580 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:14:35.055538    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-588bc5dc84-p8p9z" event={"ID":"bf8e8a44-fbcc-44bf-a8e0-0633285ae3ba","Type":"ContainerStarted","Data":"1b96e4d798abd016817238c583c5bb1f849ec577b3530ccd9256583cb80e94cb"}
Sep 16 02:14:35.055580 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:14:35.055583    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-588bc5dc84-p8p9z" event={"ID":"bf8e8a44-fbcc-44bf-a8e0-0633285ae3ba","Type":"ContainerStarted","Data":"118543d61f16bf6209c04b1310ee61d158c958614f42a773a4df009cd9626e14"}
Sep 16 02:14:35.055991 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:14:35.055596    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-588bc5dc84-p8p9z" event={"ID":"bf8e8a44-fbcc-44bf-a8e0-0633285ae3ba","Type":"ContainerStarted","Data":"a7a3abea6ee86576ac2d5e1fcc45009bb056bb363db67245fdb02ae767d370ad"}
Sep 16 02:14:35.073116 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:14:35.073068    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-588bc5dc84-p8p9z" podStartSLOduration=2.254168247 podStartE2EDuration="6.073053342s" totalImagesPullingTime="3.818885095s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 02:14:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:14:35.071731307 +0000 UTC m=+87.990220557" watchObservedRunningTime="2026-09-16 02:14:35.073053342 +0000 UTC m=+87.991542591"
Sep 16 02:14:37.064198 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:14:37.064154    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-69b45b4b7d-hcccn" event={"ID":"ea7219ae-5620-4c71-8889-c89cdfa8ed2d","Type":"ContainerStarted","Data":"f64de234ca14372b8222edd7b49892da53f74b51ac787868aa5a031fce1f020c"}
Sep 16 02:14:37.078465 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:14:37.078410    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-69b45b4b7d-hcccn" podStartSLOduration=1.970052939 podStartE2EDuration="8.078393573s" totalImagesPullingTime="6.108340634s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 02:14:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:14:37.077473123 +0000 UTC m=+89.995962371" watchObservedRunningTime="2026-09-16 02:14:37.078393573 +0000 UTC m=+89.996882822"
Sep 16 02:14:45.309375 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:14:45.309334    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/dd03566d-e8e3-4aad-b3b2-0a06e43e5eea-metrics-tls\") pod \"dns-default-q8tpf\" (UID: \"dd03566d-e8e3-4aad-b3b2-0a06e43e5eea\") " pod="openshift-dns/dns-default-q8tpf"
Sep 16 02:14:45.309889 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:14:45.309471    2558 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 02:14:45.309889 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:14:45.309542    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dd03566d-e8e3-4aad-b3b2-0a06e43e5eea-metrics-tls podName:dd03566d-e8e3-4aad-b3b2-0a06e43e5eea nodeName:}" failed. No retries permitted until 2026-09-16 02:15:49.309523002 +0000 UTC m=+162.228012251 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/dd03566d-e8e3-4aad-b3b2-0a06e43e5eea-metrics-tls") pod "dns-default-q8tpf" (UID: "dd03566d-e8e3-4aad-b3b2-0a06e43e5eea") : secret "dns-default-metrics-tls" not found
Sep 16 02:14:49.016292 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:14:49.016258    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-mfd49"
Sep 16 02:15:16.422798 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:15:16.422756    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/25e85d58-77e2-4428-b099-a5075363d56f-metrics-certs\") pod \"network-metrics-daemon-ck2vw\" (UID: \"25e85d58-77e2-4428-b099-a5075363d56f\") " pod="openshift-multus/network-metrics-daemon-ck2vw"
Sep 16 02:15:16.423192 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:15:16.422882    2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 16 02:15:16.423192 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:15:16.422943    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/25e85d58-77e2-4428-b099-a5075363d56f-metrics-certs podName:25e85d58-77e2-4428-b099-a5075363d56f nodeName:}" failed. No retries permitted until 2026-09-16 02:17:18.422929194 +0000 UTC m=+251.341418421 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/25e85d58-77e2-4428-b099-a5075363d56f-metrics-certs") pod "network-metrics-daemon-ck2vw" (UID: "25e85d58-77e2-4428-b099-a5075363d56f") : secret "metrics-daemon-secret" not found
Sep 16 02:15:44.416940 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:15:44.416883    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-q8tpf" podUID="dd03566d-e8e3-4aad-b3b2-0a06e43e5eea"
Sep 16 02:15:45.250123 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:15:45.250092    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-q8tpf"
Sep 16 02:15:45.569966 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:15:45.569879    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-ck2vw" podUID="25e85d58-77e2-4428-b099-a5075363d56f"
Sep 16 02:15:49.352998 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:15:49.352954    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/dd03566d-e8e3-4aad-b3b2-0a06e43e5eea-metrics-tls\") pod \"dns-default-q8tpf\" (UID: \"dd03566d-e8e3-4aad-b3b2-0a06e43e5eea\") " pod="openshift-dns/dns-default-q8tpf"
Sep 16 02:15:49.353390 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:15:49.353106    2558 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 02:15:49.353390 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:15:49.353171    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dd03566d-e8e3-4aad-b3b2-0a06e43e5eea-metrics-tls podName:dd03566d-e8e3-4aad-b3b2-0a06e43e5eea nodeName:}" failed. No retries permitted until 2026-09-16 02:17:51.353153754 +0000 UTC m=+284.271642981 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/dd03566d-e8e3-4aad-b3b2-0a06e43e5eea-metrics-tls") pod "dns-default-q8tpf" (UID: "dd03566d-e8e3-4aad-b3b2-0a06e43e5eea") : secret "dns-default-metrics-tls" not found
Sep 16 02:15:59.544390 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:15:59.544357    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ck2vw"
Sep 16 02:16:37.400461 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:16:37.400428    2558 generic.go:356] "Generic (PLEG): container finished" podID="ea7219ae-5620-4c71-8889-c89cdfa8ed2d" containerID="f64de234ca14372b8222edd7b49892da53f74b51ac787868aa5a031fce1f020c" exitCode=255
Sep 16 02:16:37.400872 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:16:37.400466    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-69b45b4b7d-hcccn" event={"ID":"ea7219ae-5620-4c71-8889-c89cdfa8ed2d","Type":"ContainerDied","Data":"f64de234ca14372b8222edd7b49892da53f74b51ac787868aa5a031fce1f020c"}
Sep 16 02:16:37.400872 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:16:37.400755    2558 scope.go:117] "RemoveContainer" containerID="f64de234ca14372b8222edd7b49892da53f74b51ac787868aa5a031fce1f020c"
Sep 16 02:16:38.404945 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:16:38.404909    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-69b45b4b7d-hcccn" event={"ID":"ea7219ae-5620-4c71-8889-c89cdfa8ed2d","Type":"ContainerStarted","Data":"0001687be02bb3343fd9ac55023b68569802672c40fd78c515159f10f257e6c1"}
Sep 16 02:17:16.200360 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:17:16.200326    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-qnjpw"]
Sep 16 02:17:16.203595 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:17:16.203571    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-qnjpw"
Sep 16 02:17:16.205958 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:17:16.205926    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 16 02:17:16.206086 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:17:16.206000    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 16 02:17:16.206883 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:17:16.206865    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-8h898\""
Sep 16 02:17:16.206953 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:17:16.206876    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 16 02:17:16.206953 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:17:16.206898    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 16 02:17:16.227788 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:17:16.227757    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-qnjpw"]
Sep 16 02:17:16.334549 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:17:16.334515    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d0cba7f9-30a5-4c62-bc04-976491dc83e4-crio-socket\") pod \"insights-runtime-extractor-qnjpw\" (UID: \"d0cba7f9-30a5-4c62-bc04-976491dc83e4\") " pod="openshift-insights/insights-runtime-extractor-qnjpw"
Sep 16 02:17:16.334549 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:17:16.334549    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjks4\" (UniqueName: \"kubernetes.io/projected/d0cba7f9-30a5-4c62-bc04-976491dc83e4-kube-api-access-zjks4\") pod \"insights-runtime-extractor-qnjpw\" (UID: \"d0cba7f9-30a5-4c62-bc04-976491dc83e4\") " pod="openshift-insights/insights-runtime-extractor-qnjpw"
Sep 16 02:17:16.334783 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:17:16.334594    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d0cba7f9-30a5-4c62-bc04-976491dc83e4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qnjpw\" (UID: \"d0cba7f9-30a5-4c62-bc04-976491dc83e4\") " pod="openshift-insights/insights-runtime-extractor-qnjpw"
Sep 16 02:17:16.334783 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:17:16.334648    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d0cba7f9-30a5-4c62-bc04-976491dc83e4-data-volume\") pod \"insights-runtime-extractor-qnjpw\" (UID: \"d0cba7f9-30a5-4c62-bc04-976491dc83e4\") " pod="openshift-insights/insights-runtime-extractor-qnjpw"
Sep 16 02:17:16.334783 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:17:16.334663    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d0cba7f9-30a5-4c62-bc04-976491dc83e4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qnjpw\" (UID: \"d0cba7f9-30a5-4c62-bc04-976491dc83e4\") " pod="openshift-insights/insights-runtime-extractor-qnjpw"
Sep 16 02:17:16.435631 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:17:16.435599    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d0cba7f9-30a5-4c62-bc04-976491dc83e4-crio-socket\") pod \"insights-runtime-extractor-qnjpw\" (UID: \"d0cba7f9-30a5-4c62-bc04-976491dc83e4\") " pod="openshift-insights/insights-runtime-extractor-qnjpw"
Sep 16 02:17:16.435768 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:17:16.435636    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjks4\" (UniqueName: \"kubernetes.io/projected/d0cba7f9-30a5-4c62-bc04-976491dc83e4-kube-api-access-zjks4\") pod \"insights-runtime-extractor-qnjpw\" (UID: \"d0cba7f9-30a5-4c62-bc04-976491dc83e4\") " pod="openshift-insights/insights-runtime-extractor-qnjpw"
Sep 16 02:17:16.435768 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:17:16.435686    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d0cba7f9-30a5-4c62-bc04-976491dc83e4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qnjpw\" (UID: \"d0cba7f9-30a5-4c62-bc04-976491dc83e4\") " pod="openshift-insights/insights-runtime-extractor-qnjpw"
Sep 16 02:17:16.435768 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:17:16.435738    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d0cba7f9-30a5-4c62-bc04-976491dc83e4-crio-socket\") pod \"insights-runtime-extractor-qnjpw\" (UID: \"d0cba7f9-30a5-4c62-bc04-976491dc83e4\") " pod="openshift-insights/insights-runtime-extractor-qnjpw"
Sep 16 02:17:16.435768 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:17:16.435747    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d0cba7f9-30a5-4c62-bc04-976491dc83e4-data-volume\") pod \"insights-runtime-extractor-qnjpw\" (UID: \"d0cba7f9-30a5-4c62-bc04-976491dc83e4\") " pod="openshift-insights/insights-runtime-extractor-qnjpw"
Sep 16 02:17:16.435966 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:17:16.435772    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d0cba7f9-30a5-4c62-bc04-976491dc83e4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qnjpw\" (UID: \"d0cba7f9-30a5-4c62-bc04-976491dc83e4\") " pod="openshift-insights/insights-runtime-extractor-qnjpw"
Sep 16 02:17:16.435966 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:17:16.435828    2558 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 02:17:16.435966 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:17:16.435835    2558 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 02:17:16.435966 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:17:16.435906    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d0cba7f9-30a5-4c62-bc04-976491dc83e4-kube-rbac-proxy-cm podName:d0cba7f9-30a5-4c62-bc04-976491dc83e4 nodeName:}" failed. No retries permitted until 2026-09-16 02:17:16.935890851 +0000 UTC m=+249.854380091 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d0cba7f9-30a5-4c62-bc04-976491dc83e4-kube-rbac-proxy-cm") pod "insights-runtime-extractor-qnjpw" (UID: "d0cba7f9-30a5-4c62-bc04-976491dc83e4") : configmap "kube-rbac-proxy" not found
Sep 16 02:17:16.435966 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:17:16.435919    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d0cba7f9-30a5-4c62-bc04-976491dc83e4-insights-runtime-extractor-tls podName:d0cba7f9-30a5-4c62-bc04-976491dc83e4 nodeName:}" failed. No retries permitted until 2026-09-16 02:17:16.935913058 +0000 UTC m=+249.854402286 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d0cba7f9-30a5-4c62-bc04-976491dc83e4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qnjpw" (UID: "d0cba7f9-30a5-4c62-bc04-976491dc83e4") : secret "insights-runtime-extractor-tls" not found
Sep 16 02:17:16.436198 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:17:16.436008    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d0cba7f9-30a5-4c62-bc04-976491dc83e4-data-volume\") pod \"insights-runtime-extractor-qnjpw\" (UID: \"d0cba7f9-30a5-4c62-bc04-976491dc83e4\") " pod="openshift-insights/insights-runtime-extractor-qnjpw"
Sep 16 02:17:16.443913 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:17:16.443889    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjks4\" (UniqueName: \"kubernetes.io/projected/d0cba7f9-30a5-4c62-bc04-976491dc83e4-kube-api-access-zjks4\") pod \"insights-runtime-extractor-qnjpw\" (UID: \"d0cba7f9-30a5-4c62-bc04-976491dc83e4\") " pod="openshift-insights/insights-runtime-extractor-qnjpw"
Sep 16 02:17:16.939094 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:17:16.939051    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d0cba7f9-30a5-4c62-bc04-976491dc83e4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qnjpw\" (UID: \"d0cba7f9-30a5-4c62-bc04-976491dc83e4\") " pod="openshift-insights/insights-runtime-extractor-qnjpw"
Sep 16 02:17:16.939270 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:17:16.939112    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d0cba7f9-30a5-4c62-bc04-976491dc83e4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qnjpw\" (UID: \"d0cba7f9-30a5-4c62-bc04-976491dc83e4\") " pod="openshift-insights/insights-runtime-extractor-qnjpw"
Sep 16 02:17:16.939270 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:17:16.939198    2558 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 02:17:16.939270 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:17:16.939229    2558 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 02:17:16.939270 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:17:16.939249    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d0cba7f9-30a5-4c62-bc04-976491dc83e4-kube-rbac-proxy-cm podName:d0cba7f9-30a5-4c62-bc04-976491dc83e4 nodeName:}" failed. No retries permitted until 2026-09-16 02:17:17.93923631 +0000 UTC m=+250.857725541 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d0cba7f9-30a5-4c62-bc04-976491dc83e4-kube-rbac-proxy-cm") pod "insights-runtime-extractor-qnjpw" (UID: "d0cba7f9-30a5-4c62-bc04-976491dc83e4") : configmap "kube-rbac-proxy" not found
Sep 16 02:17:16.939402 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:17:16.939320    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d0cba7f9-30a5-4c62-bc04-976491dc83e4-insights-runtime-extractor-tls podName:d0cba7f9-30a5-4c62-bc04-976491dc83e4 nodeName:}" failed. No retries permitted until 2026-09-16 02:17:17.939300157 +0000 UTC m=+250.857789388 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d0cba7f9-30a5-4c62-bc04-976491dc83e4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qnjpw" (UID: "d0cba7f9-30a5-4c62-bc04-976491dc83e4") : secret "insights-runtime-extractor-tls" not found
Sep 16 02:17:17.945595 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:17:17.945548    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d0cba7f9-30a5-4c62-bc04-976491dc83e4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qnjpw\" (UID: \"d0cba7f9-30a5-4c62-bc04-976491dc83e4\") " pod="openshift-insights/insights-runtime-extractor-qnjpw"
Sep 16 02:17:17.946079 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:17:17.945647    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d0cba7f9-30a5-4c62-bc04-976491dc83e4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qnjpw\" (UID: \"d0cba7f9-30a5-4c62-bc04-976491dc83e4\") " pod="openshift-insights/insights-runtime-extractor-qnjpw"
Sep 16 02:17:17.946079 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:17:17.945679    2558 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 02:17:17.946079 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:17:17.945764    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d0cba7f9-30a5-4c62-bc04-976491dc83e4-kube-rbac-proxy-cm podName:d0cba7f9-30a5-4c62-bc04-976491dc83e4 nodeName:}" failed. No retries permitted until 2026-09-16 02:17:19.945749199 +0000 UTC m=+252.864238426 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d0cba7f9-30a5-4c62-bc04-976491dc83e4-kube-rbac-proxy-cm") pod "insights-runtime-extractor-qnjpw" (UID: "d0cba7f9-30a5-4c62-bc04-976491dc83e4") : configmap "kube-rbac-proxy" not found
Sep 16 02:17:17.946079 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:17:17.945846    2558 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 02:17:17.946079 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:17:17.945944    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d0cba7f9-30a5-4c62-bc04-976491dc83e4-insights-runtime-extractor-tls podName:d0cba7f9-30a5-4c62-bc04-976491dc83e4 nodeName:}" failed. No retries permitted until 2026-09-16 02:17:19.945918378 +0000 UTC m=+252.864407627 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d0cba7f9-30a5-4c62-bc04-976491dc83e4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qnjpw" (UID: "d0cba7f9-30a5-4c62-bc04-976491dc83e4") : secret "insights-runtime-extractor-tls" not found
Sep 16 02:17:18.450179 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:17:18.450132    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/25e85d58-77e2-4428-b099-a5075363d56f-metrics-certs\") pod \"network-metrics-daemon-ck2vw\" (UID: \"25e85d58-77e2-4428-b099-a5075363d56f\") " pod="openshift-multus/network-metrics-daemon-ck2vw"
Sep 16 02:17:18.450363 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:17:18.450289    2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 16 02:17:18.450404 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:17:18.450362    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/25e85d58-77e2-4428-b099-a5075363d56f-metrics-certs podName:25e85d58-77e2-4428-b099-a5075363d56f nodeName:}" failed. No retries permitted until 2026-09-16 02:19:20.450346506 +0000 UTC m=+373.368835738 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/25e85d58-77e2-4428-b099-a5075363d56f-metrics-certs") pod "network-metrics-daemon-ck2vw" (UID: "25e85d58-77e2-4428-b099-a5075363d56f") : secret "metrics-daemon-secret" not found
Sep 16 02:17:19.164095 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:17:19.164066    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-vrf75_e0800020-f9de-4dfa-9a7b-d7c88ef0561f/dns-node-resolver/0.log"
Sep 16 02:17:19.960088 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:17:19.960037    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d0cba7f9-30a5-4c62-bc04-976491dc83e4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qnjpw\" (UID: \"d0cba7f9-30a5-4c62-bc04-976491dc83e4\") " pod="openshift-insights/insights-runtime-extractor-qnjpw"
Sep 16 02:17:19.960264 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:17:19.960109    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d0cba7f9-30a5-4c62-bc04-976491dc83e4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qnjpw\" (UID: \"d0cba7f9-30a5-4c62-bc04-976491dc83e4\") " pod="openshift-insights/insights-runtime-extractor-qnjpw"
Sep 16 02:17:19.960264 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:17:19.960190    2558 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 02:17:19.960264 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:17:19.960207    2558 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 02:17:19.960264 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:17:19.960242    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d0cba7f9-30a5-4c62-bc04-976491dc83e4-kube-rbac-proxy-cm podName:d0cba7f9-30a5-4c62-bc04-976491dc83e4 nodeName:}" failed. No retries permitted until 2026-09-16 02:17:23.960229714 +0000 UTC m=+256.878718941 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d0cba7f9-30a5-4c62-bc04-976491dc83e4-kube-rbac-proxy-cm") pod "insights-runtime-extractor-qnjpw" (UID: "d0cba7f9-30a5-4c62-bc04-976491dc83e4") : configmap "kube-rbac-proxy" not found
Sep 16 02:17:19.960401 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:17:19.960271    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d0cba7f9-30a5-4c62-bc04-976491dc83e4-insights-runtime-extractor-tls podName:d0cba7f9-30a5-4c62-bc04-976491dc83e4 nodeName:}" failed. No retries permitted until 2026-09-16 02:17:23.960257653 +0000 UTC m=+256.878746880 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d0cba7f9-30a5-4c62-bc04-976491dc83e4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qnjpw" (UID: "d0cba7f9-30a5-4c62-bc04-976491dc83e4") : secret "insights-runtime-extractor-tls" not found
Sep 16 02:17:20.164173 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:17:20.164146    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-vn8jb_9562b5b9-856c-4a81-89d3-8faed08f792c/node-ca/0.log"
Sep 16 02:17:20.764203 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:17:20.764153    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-b7cxx_504ae285-7f73-4ebf-b39b-192c08b6e922/serve-healthcheck-canary/0.log"
Sep 16 02:17:23.989875 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:17:23.989840    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d0cba7f9-30a5-4c62-bc04-976491dc83e4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qnjpw\" (UID: \"d0cba7f9-30a5-4c62-bc04-976491dc83e4\") " pod="openshift-insights/insights-runtime-extractor-qnjpw"
Sep 16 02:17:23.990338 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:17:23.989905    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d0cba7f9-30a5-4c62-bc04-976491dc83e4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qnjpw\" (UID: \"d0cba7f9-30a5-4c62-bc04-976491dc83e4\") " pod="openshift-insights/insights-runtime-extractor-qnjpw"
Sep 16 02:17:23.990338 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:17:23.990016    2558 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 02:17:23.990338 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:17:23.990081    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d0cba7f9-30a5-4c62-bc04-976491dc83e4-insights-runtime-extractor-tls podName:d0cba7f9-30a5-4c62-bc04-976491dc83e4 nodeName:}" failed. No retries permitted until 2026-09-16 02:17:31.990066497 +0000 UTC m=+264.908555724 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d0cba7f9-30a5-4c62-bc04-976491dc83e4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qnjpw" (UID: "d0cba7f9-30a5-4c62-bc04-976491dc83e4") : secret "insights-runtime-extractor-tls" not found
Sep 16 02:17:23.990338 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:17:23.990021    2558 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 02:17:23.990338 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:17:23.990141    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d0cba7f9-30a5-4c62-bc04-976491dc83e4-kube-rbac-proxy-cm podName:d0cba7f9-30a5-4c62-bc04-976491dc83e4 nodeName:}" failed. No retries permitted until 2026-09-16 02:17:31.990124374 +0000 UTC m=+264.908613616 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d0cba7f9-30a5-4c62-bc04-976491dc83e4-kube-rbac-proxy-cm") pod "insights-runtime-extractor-qnjpw" (UID: "d0cba7f9-30a5-4c62-bc04-976491dc83e4") : configmap "kube-rbac-proxy" not found
Sep 16 02:17:32.050466 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:17:32.050371    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d0cba7f9-30a5-4c62-bc04-976491dc83e4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qnjpw\" (UID: \"d0cba7f9-30a5-4c62-bc04-976491dc83e4\") " pod="openshift-insights/insights-runtime-extractor-qnjpw"
Sep 16 02:17:32.050860 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:17:32.050477    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d0cba7f9-30a5-4c62-bc04-976491dc83e4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qnjpw\" (UID: \"d0cba7f9-30a5-4c62-bc04-976491dc83e4\") " pod="openshift-insights/insights-runtime-extractor-qnjpw"
Sep 16 02:17:32.050860 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:17:32.050596    2558 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 02:17:32.050860 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:17:32.050674    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d0cba7f9-30a5-4c62-bc04-976491dc83e4-insights-runtime-extractor-tls podName:d0cba7f9-30a5-4c62-bc04-976491dc83e4 nodeName:}" failed. No retries permitted until 2026-09-16 02:17:48.050654156 +0000 UTC m=+280.969143403 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d0cba7f9-30a5-4c62-bc04-976491dc83e4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qnjpw" (UID: "d0cba7f9-30a5-4c62-bc04-976491dc83e4") : secret "insights-runtime-extractor-tls" not found
Sep 16 02:17:32.051003 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:17:32.050979    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d0cba7f9-30a5-4c62-bc04-976491dc83e4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qnjpw\" (UID: \"d0cba7f9-30a5-4c62-bc04-976491dc83e4\") " pod="openshift-insights/insights-runtime-extractor-qnjpw"
Sep 16 02:17:48.062998 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:17:48.062956    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d0cba7f9-30a5-4c62-bc04-976491dc83e4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qnjpw\" (UID: \"d0cba7f9-30a5-4c62-bc04-976491dc83e4\") " pod="openshift-insights/insights-runtime-extractor-qnjpw"
Sep 16 02:17:48.065233 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:17:48.065210    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d0cba7f9-30a5-4c62-bc04-976491dc83e4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qnjpw\" (UID: \"d0cba7f9-30a5-4c62-bc04-976491dc83e4\") " pod="openshift-insights/insights-runtime-extractor-qnjpw"
Sep 16 02:17:48.250538 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:17:48.250490    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-q8tpf" podUID="dd03566d-e8e3-4aad-b3b2-0a06e43e5eea"
Sep 16 02:17:48.312643 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:17:48.312616    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-qnjpw"
Sep 16 02:17:48.577427 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:17:48.577349    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-qnjpw"]
Sep 16 02:17:48.580523 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:17:48.580494    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0cba7f9_30a5_4c62_bc04_976491dc83e4.slice/crio-c7c668e3c288ab3a960bfc14ac3d89dc7715fa6b95bea77a2194d990ae253085 WatchSource:0}: Error finding container c7c668e3c288ab3a960bfc14ac3d89dc7715fa6b95bea77a2194d990ae253085: Status 404 returned error can't find the container with id c7c668e3c288ab3a960bfc14ac3d89dc7715fa6b95bea77a2194d990ae253085
Sep 16 02:17:48.601875 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:17:48.601849    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-qnjpw" event={"ID":"d0cba7f9-30a5-4c62-bc04-976491dc83e4","Type":"ContainerStarted","Data":"c7c668e3c288ab3a960bfc14ac3d89dc7715fa6b95bea77a2194d990ae253085"}
Sep 16 02:17:48.601977 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:17:48.601861    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-q8tpf"
Sep 16 02:17:49.610362 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:17:49.610324    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-qnjpw" event={"ID":"d0cba7f9-30a5-4c62-bc04-976491dc83e4","Type":"ContainerStarted","Data":"75a4ca57b513c8a62ba747c267b75c2d96a632218442eea75d8630588c24be89"}
Sep 16 02:17:49.610745 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:17:49.610416    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-qnjpw" event={"ID":"d0cba7f9-30a5-4c62-bc04-976491dc83e4","Type":"ContainerStarted","Data":"83e0cbe14e20575d407a04c645efcf5ea2f5ad800f33c8645a10dabe09a76c1f"}
Sep 16 02:17:51.386714 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:17:51.386602    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/dd03566d-e8e3-4aad-b3b2-0a06e43e5eea-metrics-tls\") pod \"dns-default-q8tpf\" (UID: \"dd03566d-e8e3-4aad-b3b2-0a06e43e5eea\") " pod="openshift-dns/dns-default-q8tpf"
Sep 16 02:17:51.388862 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:17:51.388839    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/dd03566d-e8e3-4aad-b3b2-0a06e43e5eea-metrics-tls\") pod \"dns-default-q8tpf\" (UID: \"dd03566d-e8e3-4aad-b3b2-0a06e43e5eea\") " pod="openshift-dns/dns-default-q8tpf"
Sep 16 02:17:51.605444 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:17:51.605413    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-h6q9g\""
Sep 16 02:17:51.613408 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:17:51.613390    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-q8tpf"
Sep 16 02:17:51.618935 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:17:51.618903    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-qnjpw" event={"ID":"d0cba7f9-30a5-4c62-bc04-976491dc83e4","Type":"ContainerStarted","Data":"b8c0266e15cde3e8862ab2d5f71b4f17238aef3c430c6d28081b0fb3df7e0f45"}
Sep 16 02:17:51.635384 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:17:51.635340    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-qnjpw" podStartSLOduration=33.38868557 podStartE2EDuration="35.635322362s" totalImagesPullingTime="2.246636792s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 02:17:16 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:17:51.634333414 +0000 UTC m=+284.552822662" watchObservedRunningTime="2026-09-16 02:17:51.635322362 +0000 UTC m=+284.553811611"
Sep 16 02:17:51.882195 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:17:51.882162    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-q8tpf"]
Sep 16 02:17:51.885846 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:17:51.885819    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd03566d_e8e3_4aad_b3b2_0a06e43e5eea.slice/crio-a3719e807ba238943070a0eacd3e00f709dcd3ff979bbdc27d9e50ed87051b57 WatchSource:0}: Error finding container a3719e807ba238943070a0eacd3e00f709dcd3ff979bbdc27d9e50ed87051b57: Status 404 returned error can't find the container with id a3719e807ba238943070a0eacd3e00f709dcd3ff979bbdc27d9e50ed87051b57
Sep 16 02:17:52.622854 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:17:52.622821    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-q8tpf" event={"ID":"dd03566d-e8e3-4aad-b3b2-0a06e43e5eea","Type":"ContainerStarted","Data":"a3719e807ba238943070a0eacd3e00f709dcd3ff979bbdc27d9e50ed87051b57"}
Sep 16 02:17:54.632031 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:17:54.631997    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-q8tpf" event={"ID":"dd03566d-e8e3-4aad-b3b2-0a06e43e5eea","Type":"ContainerStarted","Data":"70cf7234c37cbf22ca5e5e5d71ee1a99e5cfa7e76fca86b2dba96c26c1441051"}
Sep 16 02:17:54.632420 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:17:54.632045    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-q8tpf" event={"ID":"dd03566d-e8e3-4aad-b3b2-0a06e43e5eea","Type":"ContainerStarted","Data":"5a737c2bc58b4b0d916cff39cdd8f603077c7a99cb059ebc74ee585412212169"}
Sep 16 02:17:54.632420 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:17:54.632147    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-q8tpf"
Sep 16 02:17:54.649480 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:17:54.649432    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-q8tpf" podStartSLOduration=251.654353343 podStartE2EDuration="4m13.649419181s" totalImagesPullingTime="1.995065838s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 02:13:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:17:54.648288614 +0000 UTC m=+287.566777876" watchObservedRunningTime="2026-09-16 02:17:54.649419181 +0000 UTC m=+287.567908421"
Sep 16 02:17:55.158774 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:17:55.158740    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-s686b"]
Sep 16 02:17:55.161674 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:17:55.161652    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-s686b"
Sep 16 02:17:55.163382 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:17:55.163357    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-qvr4r"]
Sep 16 02:17:55.164488 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:17:55.164470    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Sep 16 02:17:55.164571 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:17:55.164497    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 16 02:17:55.166169 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:17:55.166151    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-qvr4r"
Sep 16 02:17:55.173129 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:17:55.173111    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Sep 16 02:17:55.178078 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:17:55.178057    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-9zsrh\""
Sep 16 02:17:55.178289 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:17:55.178270    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 16 02:17:55.180309 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:17:55.180288    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 16 02:17:55.180404 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:17:55.180308    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 16 02:17:55.180404 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:17:55.180292    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 16 02:17:55.181244 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:17:55.181228    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-qfp9h\""
Sep 16 02:17:55.182167 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:17:55.182151    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 16 02:17:55.194008 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:17:55.193984    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-s686b"]
Sep 16 02:17:55.311113 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:17:55.311086    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9e294a9b-44a5-499a-8c28-c88e4ef85b7e-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-s686b\" (UID: \"9e294a9b-44a5-499a-8c28-c88e4ef85b7e\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-s686b"
Sep 16 02:17:55.311113 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:17:55.311114    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/9e294a9b-44a5-499a-8c28-c88e4ef85b7e-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-s686b\" (UID: \"9e294a9b-44a5-499a-8c28-c88e4ef85b7e\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-s686b"
Sep 16 02:17:55.311322 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:17:55.311135    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czc4k\" (UniqueName: \"kubernetes.io/projected/b8b6adb3-45cc-4eab-a2f7-b1435a4ac409-kube-api-access-czc4k\") pod \"node-exporter-qvr4r\" (UID: \"b8b6adb3-45cc-4eab-a2f7-b1435a4ac409\") " pod="openshift-monitoring/node-exporter-qvr4r"
Sep 16 02:17:55.311322 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:17:55.311151    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tntvf\" (UniqueName: \"kubernetes.io/projected/9e294a9b-44a5-499a-8c28-c88e4ef85b7e-kube-api-access-tntvf\") pod \"openshift-state-metrics-67dcf8c979-s686b\" (UID: \"9e294a9b-44a5-499a-8c28-c88e4ef85b7e\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-s686b"
Sep 16 02:17:55.311322 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:17:55.311194    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b8b6adb3-45cc-4eab-a2f7-b1435a4ac409-node-exporter-textfile\") pod \"node-exporter-qvr4r\" (UID: \"b8b6adb3-45cc-4eab-a2f7-b1435a4ac409\") " pod="openshift-monitoring/node-exporter-qvr4r"
Sep 16 02:17:55.311322 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:17:55.311238    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b8b6adb3-45cc-4eab-a2f7-b1435a4ac409-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-qvr4r\" (UID: \"b8b6adb3-45cc-4eab-a2f7-b1435a4ac409\") " pod="openshift-monitoring/node-exporter-qvr4r"
Sep 16 02:17:55.311322 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:17:55.311266    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b8b6adb3-45cc-4eab-a2f7-b1435a4ac409-root\") pod \"node-exporter-qvr4r\" (UID: \"b8b6adb3-45cc-4eab-a2f7-b1435a4ac409\") " pod="openshift-monitoring/node-exporter-qvr4r"
Sep 16 02:17:55.311322 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:17:55.311298    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b8b6adb3-45cc-4eab-a2f7-b1435a4ac409-sys\") pod \"node-exporter-qvr4r\" (UID: \"b8b6adb3-45cc-4eab-a2f7-b1435a4ac409\") " pod="openshift-monitoring/node-exporter-qvr4r"
Sep 16 02:17:55.311551 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:17:55.311360    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b8b6adb3-45cc-4eab-a2f7-b1435a4ac409-node-exporter-tls\") pod \"node-exporter-qvr4r\" (UID: \"b8b6adb3-45cc-4eab-a2f7-b1435a4ac409\") " pod="openshift-monitoring/node-exporter-qvr4r"
Sep 16 02:17:55.311551 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:17:55.311382    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b8b6adb3-45cc-4eab-a2f7-b1435a4ac409-metrics-client-ca\") pod \"node-exporter-qvr4r\" (UID: \"b8b6adb3-45cc-4eab-a2f7-b1435a4ac409\") " pod="openshift-monitoring/node-exporter-qvr4r"
Sep 16 02:17:55.311551 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:17:55.311399    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b8b6adb3-45cc-4eab-a2f7-b1435a4ac409-node-exporter-accelerators-collector-config\") pod \"node-exporter-qvr4r\" (UID: \"b8b6adb3-45cc-4eab-a2f7-b1435a4ac409\") " pod="openshift-monitoring/node-exporter-qvr4r"
Sep 16 02:17:55.311551 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:17:55.311415    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9e294a9b-44a5-499a-8c28-c88e4ef85b7e-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-s686b\" (UID: \"9e294a9b-44a5-499a-8c28-c88e4ef85b7e\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-s686b"
Sep 16 02:17:55.311551 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:17:55.311463    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b8b6adb3-45cc-4eab-a2f7-b1435a4ac409-node-exporter-wtmp\") pod \"node-exporter-qvr4r\" (UID: \"b8b6adb3-45cc-4eab-a2f7-b1435a4ac409\") " pod="openshift-monitoring/node-exporter-qvr4r"
Sep 16 02:17:55.412114 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:17:55.412048    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b8b6adb3-45cc-4eab-a2f7-b1435a4ac409-node-exporter-wtmp\") pod \"node-exporter-qvr4r\" (UID: \"b8b6adb3-45cc-4eab-a2f7-b1435a4ac409\") " pod="openshift-monitoring/node-exporter-qvr4r"
Sep 16 02:17:55.412114 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:17:55.412079    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9e294a9b-44a5-499a-8c28-c88e4ef85b7e-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-s686b\" (UID: \"9e294a9b-44a5-499a-8c28-c88e4ef85b7e\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-s686b"
Sep 16 02:17:55.412114 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:17:55.412101    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/9e294a9b-44a5-499a-8c28-c88e4ef85b7e-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-s686b\" (UID: \"9e294a9b-44a5-499a-8c28-c88e4ef85b7e\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-s686b"
Sep 16 02:17:55.412325 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:17:55.412191    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czc4k\" (UniqueName: \"kubernetes.io/projected/b8b6adb3-45cc-4eab-a2f7-b1435a4ac409-kube-api-access-czc4k\") pod \"node-exporter-qvr4r\" (UID: \"b8b6adb3-45cc-4eab-a2f7-b1435a4ac409\") " pod="openshift-monitoring/node-exporter-qvr4r"
Sep 16 02:17:55.412325 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:17:55.412207    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tntvf\" (UniqueName: \"kubernetes.io/projected/9e294a9b-44a5-499a-8c28-c88e4ef85b7e-kube-api-access-tntvf\") pod \"openshift-state-metrics-67dcf8c979-s686b\" (UID: \"9e294a9b-44a5-499a-8c28-c88e4ef85b7e\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-s686b"
Sep 16 02:17:55.412325 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:17:55.412222    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b8b6adb3-45cc-4eab-a2f7-b1435a4ac409-node-exporter-textfile\") pod \"node-exporter-qvr4r\" (UID: \"b8b6adb3-45cc-4eab-a2f7-b1435a4ac409\") " pod="openshift-monitoring/node-exporter-qvr4r"
Sep 16 02:17:55.412325 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:17:55.412242    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b8b6adb3-45cc-4eab-a2f7-b1435a4ac409-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-qvr4r\" (UID: \"b8b6adb3-45cc-4eab-a2f7-b1435a4ac409\") " pod="openshift-monitoring/node-exporter-qvr4r"
Sep 16 02:17:55.412325 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:17:55.412256    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b8b6adb3-45cc-4eab-a2f7-b1435a4ac409-node-exporter-wtmp\") pod \"node-exporter-qvr4r\" (UID: \"b8b6adb3-45cc-4eab-a2f7-b1435a4ac409\") " pod="openshift-monitoring/node-exporter-qvr4r"
Sep 16 02:17:55.412325 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:17:55.412261    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b8b6adb3-45cc-4eab-a2f7-b1435a4ac409-root\") pod \"node-exporter-qvr4r\" (UID: \"b8b6adb3-45cc-4eab-a2f7-b1435a4ac409\") " pod="openshift-monitoring/node-exporter-qvr4r"
Sep 16 02:17:55.412325 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:17:55.412310    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b8b6adb3-45cc-4eab-a2f7-b1435a4ac409-sys\") pod \"node-exporter-qvr4r\" (UID: \"b8b6adb3-45cc-4eab-a2f7-b1435a4ac409\") " pod="openshift-monitoring/node-exporter-qvr4r"
Sep 16 02:17:55.412646 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:17:55.412328    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b8b6adb3-45cc-4eab-a2f7-b1435a4ac409-root\") pod \"node-exporter-qvr4r\" (UID: \"b8b6adb3-45cc-4eab-a2f7-b1435a4ac409\") " pod="openshift-monitoring/node-exporter-qvr4r"
Sep 16 02:17:55.412646 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:17:55.412355    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b8b6adb3-45cc-4eab-a2f7-b1435a4ac409-node-exporter-tls\") pod \"node-exporter-qvr4r\" (UID: \"b8b6adb3-45cc-4eab-a2f7-b1435a4ac409\") " pod="openshift-monitoring/node-exporter-qvr4r"
Sep 16 02:17:55.412646 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:17:55.412379    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b8b6adb3-45cc-4eab-a2f7-b1435a4ac409-metrics-client-ca\") pod \"node-exporter-qvr4r\" (UID: \"b8b6adb3-45cc-4eab-a2f7-b1435a4ac409\") " pod="openshift-monitoring/node-exporter-qvr4r"
Sep 16 02:17:55.412646 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:17:55.412404    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b8b6adb3-45cc-4eab-a2f7-b1435a4ac409-node-exporter-accelerators-collector-config\") pod \"node-exporter-qvr4r\" (UID: \"b8b6adb3-45cc-4eab-a2f7-b1435a4ac409\") " pod="openshift-monitoring/node-exporter-qvr4r"
Sep 16 02:17:55.412646 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:17:55.412412    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b8b6adb3-45cc-4eab-a2f7-b1435a4ac409-sys\") pod \"node-exporter-qvr4r\" (UID: \"b8b6adb3-45cc-4eab-a2f7-b1435a4ac409\") " pod="openshift-monitoring/node-exporter-qvr4r"
Sep 16 02:17:55.412646 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:17:55.412428    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9e294a9b-44a5-499a-8c28-c88e4ef85b7e-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-s686b\" (UID: \"9e294a9b-44a5-499a-8c28-c88e4ef85b7e\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-s686b"
Sep 16 02:17:55.412646 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:17:55.412607    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b8b6adb3-45cc-4eab-a2f7-b1435a4ac409-node-exporter-textfile\") pod \"node-exporter-qvr4r\" (UID: \"b8b6adb3-45cc-4eab-a2f7-b1435a4ac409\") " pod="openshift-monitoring/node-exporter-qvr4r"
Sep 16 02:17:55.413116 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:17:55.413070    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b8b6adb3-45cc-4eab-a2f7-b1435a4ac409-metrics-client-ca\") pod \"node-exporter-qvr4r\" (UID: \"b8b6adb3-45cc-4eab-a2f7-b1435a4ac409\") " pod="openshift-monitoring/node-exporter-qvr4r"
Sep 16 02:17:55.413215 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:17:55.413136    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b8b6adb3-45cc-4eab-a2f7-b1435a4ac409-node-exporter-accelerators-collector-config\") pod \"node-exporter-qvr4r\" (UID: \"b8b6adb3-45cc-4eab-a2f7-b1435a4ac409\") " pod="openshift-monitoring/node-exporter-qvr4r"
Sep 16 02:17:55.413369 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:17:55.413321    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9e294a9b-44a5-499a-8c28-c88e4ef85b7e-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-s686b\" (UID: \"9e294a9b-44a5-499a-8c28-c88e4ef85b7e\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-s686b"
Sep 16 02:17:55.415087 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:17:55.415054    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/9e294a9b-44a5-499a-8c28-c88e4ef85b7e-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-s686b\" (UID: \"9e294a9b-44a5-499a-8c28-c88e4ef85b7e\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-s686b"
Sep 16 02:17:55.415183 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:17:55.415149    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b8b6adb3-45cc-4eab-a2f7-b1435a4ac409-node-exporter-tls\") pod \"node-exporter-qvr4r\" (UID: \"b8b6adb3-45cc-4eab-a2f7-b1435a4ac409\") " pod="openshift-monitoring/node-exporter-qvr4r"
Sep 16 02:17:55.415271 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:17:55.415246    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b8b6adb3-45cc-4eab-a2f7-b1435a4ac409-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-qvr4r\" (UID: \"b8b6adb3-45cc-4eab-a2f7-b1435a4ac409\") " pod="openshift-monitoring/node-exporter-qvr4r"
Sep 16 02:17:55.415341 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:17:55.415318    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9e294a9b-44a5-499a-8c28-c88e4ef85b7e-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-s686b\" (UID: \"9e294a9b-44a5-499a-8c28-c88e4ef85b7e\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-s686b"
Sep 16 02:17:55.419389 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:17:55.419366    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-czc4k\" (UniqueName: \"kubernetes.io/projected/b8b6adb3-45cc-4eab-a2f7-b1435a4ac409-kube-api-access-czc4k\") pod \"node-exporter-qvr4r\" (UID: \"b8b6adb3-45cc-4eab-a2f7-b1435a4ac409\") " pod="openshift-monitoring/node-exporter-qvr4r"
Sep 16 02:17:55.419481 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:17:55.419456    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tntvf\" (UniqueName: \"kubernetes.io/projected/9e294a9b-44a5-499a-8c28-c88e4ef85b7e-kube-api-access-tntvf\") pod \"openshift-state-metrics-67dcf8c979-s686b\" (UID: \"9e294a9b-44a5-499a-8c28-c88e4ef85b7e\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-s686b"
Sep 16 02:17:55.471344 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:17:55.471315    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-s686b"
Sep 16 02:17:55.480516 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:17:55.480500    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-qvr4r"
Sep 16 02:17:55.490186 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:17:55.490146    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8b6adb3_45cc_4eab_a2f7_b1435a4ac409.slice/crio-10ccdb2401936018021b959b7e3967fe58dd81ed982e1ffd87671f4d7731f10c WatchSource:0}: Error finding container 10ccdb2401936018021b959b7e3967fe58dd81ed982e1ffd87671f4d7731f10c: Status 404 returned error can't find the container with id 10ccdb2401936018021b959b7e3967fe58dd81ed982e1ffd87671f4d7731f10c
Sep 16 02:17:55.636397 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:17:55.636363    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qvr4r" event={"ID":"b8b6adb3-45cc-4eab-a2f7-b1435a4ac409","Type":"ContainerStarted","Data":"10ccdb2401936018021b959b7e3967fe58dd81ed982e1ffd87671f4d7731f10c"}
Sep 16 02:17:55.739252 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:17:55.739188    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-s686b"]
Sep 16 02:17:55.742026 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:17:55.742002    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e294a9b_44a5_499a_8c28_c88e4ef85b7e.slice/crio-7575629541ab6968eca35b0ad987522bbd59808e65d11a41f1f02e45a2b4c5c2 WatchSource:0}: Error finding container 7575629541ab6968eca35b0ad987522bbd59808e65d11a41f1f02e45a2b4c5c2: Status 404 returned error can't find the container with id 7575629541ab6968eca35b0ad987522bbd59808e65d11a41f1f02e45a2b4c5c2
Sep 16 02:17:56.230621 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:17:56.230395    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 02:17:56.234498 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:17:56.234470    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 16 02:17:56.236489 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:17:56.236463    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 16 02:17:56.236859 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:17:56.236830    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 16 02:17:56.236859 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:17:56.236857    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 16 02:17:56.237171 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:17:56.237150    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 16 02:17:56.237250 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:17:56.237218    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 16 02:17:56.237318 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:17:56.237220    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 16 02:17:56.237414 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:17:56.237393    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 16 02:17:56.237500 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:17:56.237476    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-kxzjt\""
Sep 16 02:17:56.237688 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:17:56.237669    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 16 02:17:56.237876 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:17:56.237781    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 16 02:17:56.247219 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:17:56.247196    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 02:17:56.420997 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:17:56.420921    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/1629aab1-29f9-42f7-92c9-435571df7299-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"1629aab1-29f9-42f7-92c9-435571df7299\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 02:17:56.420997 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:17:56.420968    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wb8ht\" (UniqueName: \"kubernetes.io/projected/1629aab1-29f9-42f7-92c9-435571df7299-kube-api-access-wb8ht\") pod \"alertmanager-main-0\" (UID: \"1629aab1-29f9-42f7-92c9-435571df7299\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 02:17:56.420997 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:17:56.421002    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1629aab1-29f9-42f7-92c9-435571df7299-tls-assets\") pod \"alertmanager-main-0\" (UID: \"1629aab1-29f9-42f7-92c9-435571df7299\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 02:17:56.421276 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:17:56.421037    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/1629aab1-29f9-42f7-92c9-435571df7299-config-volume\") pod \"alertmanager-main-0\" (UID: \"1629aab1-29f9-42f7-92c9-435571df7299\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 02:17:56.421276 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:17:56.421091    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1629aab1-29f9-42f7-92c9-435571df7299-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"1629aab1-29f9-42f7-92c9-435571df7299\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 02:17:56.421276 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:17:56.421116    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1629aab1-29f9-42f7-92c9-435571df7299-web-config\") pod \"alertmanager-main-0\" (UID: \"1629aab1-29f9-42f7-92c9-435571df7299\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 02:17:56.421276 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:17:56.421141    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1629aab1-29f9-42f7-92c9-435571df7299-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"1629aab1-29f9-42f7-92c9-435571df7299\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 02:17:56.421276 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:17:56.421173    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1629aab1-29f9-42f7-92c9-435571df7299-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"1629aab1-29f9-42f7-92c9-435571df7299\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 02:17:56.421276 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:17:56.421222    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/1629aab1-29f9-42f7-92c9-435571df7299-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"1629aab1-29f9-42f7-92c9-435571df7299\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 02:17:56.421276 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:17:56.421261    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/1629aab1-29f9-42f7-92c9-435571df7299-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"1629aab1-29f9-42f7-92c9-435571df7299\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 02:17:56.421596 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:17:56.421286    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1629aab1-29f9-42f7-92c9-435571df7299-config-out\") pod \"alertmanager-main-0\" (UID: \"1629aab1-29f9-42f7-92c9-435571df7299\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 02:17:56.421596 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:17:56.421327    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1629aab1-29f9-42f7-92c9-435571df7299-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"1629aab1-29f9-42f7-92c9-435571df7299\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 02:17:56.421596 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:17:56.421390    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1629aab1-29f9-42f7-92c9-435571df7299-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"1629aab1-29f9-42f7-92c9-435571df7299\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 02:17:56.521722 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:17:56.521681    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/1629aab1-29f9-42f7-92c9-435571df7299-config-volume\") pod \"alertmanager-main-0\" (UID: \"1629aab1-29f9-42f7-92c9-435571df7299\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 02:17:56.521859 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:17:56.521736    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1629aab1-29f9-42f7-92c9-435571df7299-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"1629aab1-29f9-42f7-92c9-435571df7299\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 02:17:56.521859 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:17:56.521767    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1629aab1-29f9-42f7-92c9-435571df7299-web-config\") pod \"alertmanager-main-0\" (UID: \"1629aab1-29f9-42f7-92c9-435571df7299\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 02:17:56.521859 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:17:56.521799    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1629aab1-29f9-42f7-92c9-435571df7299-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"1629aab1-29f9-42f7-92c9-435571df7299\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 02:17:56.521859 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:17:56.521826    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1629aab1-29f9-42f7-92c9-435571df7299-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"1629aab1-29f9-42f7-92c9-435571df7299\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 02:17:56.522072 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:17:56.521873    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/1629aab1-29f9-42f7-92c9-435571df7299-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"1629aab1-29f9-42f7-92c9-435571df7299\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 02:17:56.522072 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:17:56.521901    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/1629aab1-29f9-42f7-92c9-435571df7299-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"1629aab1-29f9-42f7-92c9-435571df7299\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 02:17:56.522072 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:17:56.521929    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1629aab1-29f9-42f7-92c9-435571df7299-config-out\") pod \"alertmanager-main-0\" (UID: \"1629aab1-29f9-42f7-92c9-435571df7299\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 02:17:56.522072 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:17:56.521964    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1629aab1-29f9-42f7-92c9-435571df7299-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"1629aab1-29f9-42f7-92c9-435571df7299\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 02:17:56.522072 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:17:56.522016    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1629aab1-29f9-42f7-92c9-435571df7299-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"1629aab1-29f9-42f7-92c9-435571df7299\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 02:17:56.522072 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:17:56.522044    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/1629aab1-29f9-42f7-92c9-435571df7299-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"1629aab1-29f9-42f7-92c9-435571df7299\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 02:17:56.522360 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:17:56.522067    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wb8ht\" (UniqueName: \"kubernetes.io/projected/1629aab1-29f9-42f7-92c9-435571df7299-kube-api-access-wb8ht\") pod \"alertmanager-main-0\" (UID: \"1629aab1-29f9-42f7-92c9-435571df7299\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 02:17:56.522360 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:17:56.522122    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1629aab1-29f9-42f7-92c9-435571df7299-tls-assets\") pod \"alertmanager-main-0\" (UID: \"1629aab1-29f9-42f7-92c9-435571df7299\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 02:17:56.523018 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:17:56.522932    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/1629aab1-29f9-42f7-92c9-435571df7299-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"1629aab1-29f9-42f7-92c9-435571df7299\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 02:17:56.523150 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:17:56.523086    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1629aab1-29f9-42f7-92c9-435571df7299-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"1629aab1-29f9-42f7-92c9-435571df7299\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 02:17:56.524916 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:17:56.524888    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1629aab1-29f9-42f7-92c9-435571df7299-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"1629aab1-29f9-42f7-92c9-435571df7299\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 02:17:56.525685 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:17:56.525659    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1629aab1-29f9-42f7-92c9-435571df7299-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"1629aab1-29f9-42f7-92c9-435571df7299\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 02:17:56.526240 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:17:56.525791    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1629aab1-29f9-42f7-92c9-435571df7299-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"1629aab1-29f9-42f7-92c9-435571df7299\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 02:17:56.526240 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:17:56.526122    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1629aab1-29f9-42f7-92c9-435571df7299-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"1629aab1-29f9-42f7-92c9-435571df7299\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 02:17:56.526240 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:17:56.526203    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/1629aab1-29f9-42f7-92c9-435571df7299-config-volume\") pod \"alertmanager-main-0\" (UID: \"1629aab1-29f9-42f7-92c9-435571df7299\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 02:17:56.526769 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:17:56.526726    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1629aab1-29f9-42f7-92c9-435571df7299-config-out\") pod \"alertmanager-main-0\" (UID: \"1629aab1-29f9-42f7-92c9-435571df7299\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 02:17:56.526893 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:17:56.526859    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1629aab1-29f9-42f7-92c9-435571df7299-web-config\") pod \"alertmanager-main-0\" (UID: \"1629aab1-29f9-42f7-92c9-435571df7299\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 02:17:56.526988 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:17:56.526937    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1629aab1-29f9-42f7-92c9-435571df7299-tls-assets\") pod \"alertmanager-main-0\" (UID: \"1629aab1-29f9-42f7-92c9-435571df7299\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 02:17:56.527819 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:17:56.527793    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/1629aab1-29f9-42f7-92c9-435571df7299-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"1629aab1-29f9-42f7-92c9-435571df7299\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 02:17:56.527970 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:17:56.527952    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/1629aab1-29f9-42f7-92c9-435571df7299-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"1629aab1-29f9-42f7-92c9-435571df7299\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 02:17:56.533138 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:17:56.533120    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wb8ht\" (UniqueName: \"kubernetes.io/projected/1629aab1-29f9-42f7-92c9-435571df7299-kube-api-access-wb8ht\") pod \"alertmanager-main-0\" (UID: \"1629aab1-29f9-42f7-92c9-435571df7299\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 02:17:56.545037 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:17:56.545018    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 16 02:17:56.641413 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:17:56.641372    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qvr4r" event={"ID":"b8b6adb3-45cc-4eab-a2f7-b1435a4ac409","Type":"ContainerStarted","Data":"0e1539d0b29ca1f6ecfe480cda5bfb5946505620547b3cb9a4a34342dc1cd824"}
Sep 16 02:17:56.643298 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:17:56.643273    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-s686b" event={"ID":"9e294a9b-44a5-499a-8c28-c88e4ef85b7e","Type":"ContainerStarted","Data":"9b994b9d3381c10d8a2c101580120ab64218bf24a5d0b293373b2b33cf04d5e2"}
Sep 16 02:17:56.643426 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:17:56.643329    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-s686b" event={"ID":"9e294a9b-44a5-499a-8c28-c88e4ef85b7e","Type":"ContainerStarted","Data":"c86e7bbbd6c88bd4d3c0ffb0a49e836902634e0908748ec40fe59a8dfa54d8ca"}
Sep 16 02:17:56.643426 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:17:56.643358    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-s686b" event={"ID":"9e294a9b-44a5-499a-8c28-c88e4ef85b7e","Type":"ContainerStarted","Data":"7575629541ab6968eca35b0ad987522bbd59808e65d11a41f1f02e45a2b4c5c2"}
Sep 16 02:17:56.826405 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:17:56.826327    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 02:17:56.849338 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:17:56.849307    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1629aab1_29f9_42f7_92c9_435571df7299.slice/crio-9fa0e5ad650b5145bb7e3beb2c14055ec674cc29e7e7b33e02ed2bc82009d696 WatchSource:0}: Error finding container 9fa0e5ad650b5145bb7e3beb2c14055ec674cc29e7e7b33e02ed2bc82009d696: Status 404 returned error can't find the container with id 9fa0e5ad650b5145bb7e3beb2c14055ec674cc29e7e7b33e02ed2bc82009d696
Sep 16 02:17:57.646972 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:17:57.646936    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1629aab1-29f9-42f7-92c9-435571df7299","Type":"ContainerStarted","Data":"9fa0e5ad650b5145bb7e3beb2c14055ec674cc29e7e7b33e02ed2bc82009d696"}
Sep 16 02:17:57.648764 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:17:57.648740    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-s686b" event={"ID":"9e294a9b-44a5-499a-8c28-c88e4ef85b7e","Type":"ContainerStarted","Data":"094ebd21a3df66ee82a45aec8567c9c29845f3f5f436d354eedafaa046515fd3"}
Sep 16 02:17:57.650151 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:17:57.650127    2558 generic.go:356] "Generic (PLEG): container finished" podID="b8b6adb3-45cc-4eab-a2f7-b1435a4ac409" containerID="0e1539d0b29ca1f6ecfe480cda5bfb5946505620547b3cb9a4a34342dc1cd824" exitCode=0
Sep 16 02:17:57.650263 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:17:57.650164    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qvr4r" event={"ID":"b8b6adb3-45cc-4eab-a2f7-b1435a4ac409","Type":"ContainerDied","Data":"0e1539d0b29ca1f6ecfe480cda5bfb5946505620547b3cb9a4a34342dc1cd824"}
Sep 16 02:17:57.665123 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:17:57.665075    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-s686b" podStartSLOduration=1.6179551559999998 podStartE2EDuration="2.665057231s" totalImagesPullingTime="1.047102075s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 02:17:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:17:57.663148778 +0000 UTC m=+290.581638027" watchObservedRunningTime="2026-09-16 02:17:57.665057231 +0000 UTC m=+290.583546481"
Sep 16 02:17:58.656875 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:17:58.656839    2558 generic.go:356] "Generic (PLEG): container finished" podID="1629aab1-29f9-42f7-92c9-435571df7299" containerID="203e09dca42113069c030dbf29db5c11179a7cdfbb317469adeeaf24b00eb553" exitCode=0
Sep 16 02:17:58.657339 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:17:58.656932    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1629aab1-29f9-42f7-92c9-435571df7299","Type":"ContainerDied","Data":"203e09dca42113069c030dbf29db5c11179a7cdfbb317469adeeaf24b00eb553"}
Sep 16 02:17:58.659175 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:17:58.659151    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qvr4r" event={"ID":"b8b6adb3-45cc-4eab-a2f7-b1435a4ac409","Type":"ContainerStarted","Data":"109f30af375680172bb7a0edc14c4d959c7bafb7529b0f225cbb60d1a2e69533"}
Sep 16 02:17:58.659266 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:17:58.659194    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qvr4r" event={"ID":"b8b6adb3-45cc-4eab-a2f7-b1435a4ac409","Type":"ContainerStarted","Data":"1624a1199cfac1914fcff6168144c4e2701c5eda597c45d0e0efe19fa63ca8b9"}
Sep 16 02:17:58.694344 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:17:58.694297    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-qvr4r" podStartSLOduration=2.5954217120000003 podStartE2EDuration="3.694282058s" totalImagesPullingTime="1.030141234s" totalInitContainerRuntime="68.719112ms" isStatefulPod=true podCreationTimestamp="2026-09-16 02:17:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:17:58.6937311 +0000 UTC m=+291.612220350" watchObservedRunningTime="2026-09-16 02:17:58.694282058 +0000 UTC m=+291.612771306"
Sep 16 02:17:59.466378 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:17:59.466348    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-59787c5d8d-sth5n"]
Sep 16 02:17:59.469275 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:17:59.469261    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-59787c5d8d-sth5n"
Sep 16 02:17:59.471367 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:17:59.471345    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Sep 16 02:17:59.472080 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:17:59.472061    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Sep 16 02:17:59.472172 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:17:59.472069    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-x5shx\""
Sep 16 02:17:59.472172 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:17:59.472100    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Sep 16 02:17:59.472303 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:17:59.472287    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 16 02:17:59.472365 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:17:59.472352    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-fvbvm0og2ngj\""
Sep 16 02:17:59.478116 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:17:59.478095    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-59787c5d8d-sth5n"]
Sep 16 02:17:59.546975 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:17:59.546930    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/c6abece5-0093-4c2e-b2e3-e0fd93abc7f1-audit-log\") pod \"metrics-server-59787c5d8d-sth5n\" (UID: \"c6abece5-0093-4c2e-b2e3-e0fd93abc7f1\") " pod="openshift-monitoring/metrics-server-59787c5d8d-sth5n"
Sep 16 02:17:59.547115 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:17:59.547013    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c6abece5-0093-4c2e-b2e3-e0fd93abc7f1-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-59787c5d8d-sth5n\" (UID: \"c6abece5-0093-4c2e-b2e3-e0fd93abc7f1\") " pod="openshift-monitoring/metrics-server-59787c5d8d-sth5n"
Sep 16 02:17:59.547115 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:17:59.547067    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/c6abece5-0093-4c2e-b2e3-e0fd93abc7f1-secret-metrics-server-tls\") pod \"metrics-server-59787c5d8d-sth5n\" (UID: \"c6abece5-0093-4c2e-b2e3-e0fd93abc7f1\") " pod="openshift-monitoring/metrics-server-59787c5d8d-sth5n"
Sep 16 02:17:59.547115 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:17:59.547094    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/c6abece5-0093-4c2e-b2e3-e0fd93abc7f1-metrics-server-audit-profiles\") pod \"metrics-server-59787c5d8d-sth5n\" (UID: \"c6abece5-0093-4c2e-b2e3-e0fd93abc7f1\") " pod="openshift-monitoring/metrics-server-59787c5d8d-sth5n"
Sep 16 02:17:59.547271 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:17:59.547154    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/c6abece5-0093-4c2e-b2e3-e0fd93abc7f1-secret-metrics-server-client-certs\") pod \"metrics-server-59787c5d8d-sth5n\" (UID: \"c6abece5-0093-4c2e-b2e3-e0fd93abc7f1\") " pod="openshift-monitoring/metrics-server-59787c5d8d-sth5n"
Sep 16 02:17:59.547271 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:17:59.547205    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/c6abece5-0093-4c2e-b2e3-e0fd93abc7f1-client-ca-bundle\") pod \"metrics-server-59787c5d8d-sth5n\" (UID: \"c6abece5-0093-4c2e-b2e3-e0fd93abc7f1\") " pod="openshift-monitoring/metrics-server-59787c5d8d-sth5n"
Sep 16 02:17:59.547271 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:17:59.547233    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5gfw\" (UniqueName: \"kubernetes.io/projected/c6abece5-0093-4c2e-b2e3-e0fd93abc7f1-kube-api-access-v5gfw\") pod \"metrics-server-59787c5d8d-sth5n\" (UID: \"c6abece5-0093-4c2e-b2e3-e0fd93abc7f1\") " pod="openshift-monitoring/metrics-server-59787c5d8d-sth5n"
Sep 16 02:17:59.648634 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:17:59.648596    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c6abece5-0093-4c2e-b2e3-e0fd93abc7f1-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-59787c5d8d-sth5n\" (UID: \"c6abece5-0093-4c2e-b2e3-e0fd93abc7f1\") " pod="openshift-monitoring/metrics-server-59787c5d8d-sth5n"
Sep 16 02:17:59.648816 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:17:59.648647    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/c6abece5-0093-4c2e-b2e3-e0fd93abc7f1-secret-metrics-server-tls\") pod \"metrics-server-59787c5d8d-sth5n\" (UID: \"c6abece5-0093-4c2e-b2e3-e0fd93abc7f1\") " pod="openshift-monitoring/metrics-server-59787c5d8d-sth5n"
Sep 16 02:17:59.648816 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:17:59.648679    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/c6abece5-0093-4c2e-b2e3-e0fd93abc7f1-metrics-server-audit-profiles\") pod \"metrics-server-59787c5d8d-sth5n\" (UID: \"c6abece5-0093-4c2e-b2e3-e0fd93abc7f1\") " pod="openshift-monitoring/metrics-server-59787c5d8d-sth5n"
Sep 16 02:17:59.648816 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:17:59.648724    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/c6abece5-0093-4c2e-b2e3-e0fd93abc7f1-secret-metrics-server-client-certs\") pod \"metrics-server-59787c5d8d-sth5n\" (UID: \"c6abece5-0093-4c2e-b2e3-e0fd93abc7f1\") " pod="openshift-monitoring/metrics-server-59787c5d8d-sth5n"
Sep 16 02:17:59.648816 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:17:59.648750    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/c6abece5-0093-4c2e-b2e3-e0fd93abc7f1-client-ca-bundle\") pod \"metrics-server-59787c5d8d-sth5n\" (UID: \"c6abece5-0093-4c2e-b2e3-e0fd93abc7f1\") " pod="openshift-monitoring/metrics-server-59787c5d8d-sth5n"
Sep 16 02:17:59.649003 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:17:59.648805    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5gfw\" (UniqueName: \"kubernetes.io/projected/c6abece5-0093-4c2e-b2e3-e0fd93abc7f1-kube-api-access-v5gfw\") pod \"metrics-server-59787c5d8d-sth5n\" (UID: \"c6abece5-0093-4c2e-b2e3-e0fd93abc7f1\") " pod="openshift-monitoring/metrics-server-59787c5d8d-sth5n"
Sep 16 02:17:59.649003 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:17:59.648928    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/c6abece5-0093-4c2e-b2e3-e0fd93abc7f1-audit-log\") pod \"metrics-server-59787c5d8d-sth5n\" (UID: \"c6abece5-0093-4c2e-b2e3-e0fd93abc7f1\") " pod="openshift-monitoring/metrics-server-59787c5d8d-sth5n"
Sep 16 02:17:59.651349 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:17:59.651321    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/c6abece5-0093-4c2e-b2e3-e0fd93abc7f1-audit-log\") pod \"metrics-server-59787c5d8d-sth5n\" (UID: \"c6abece5-0093-4c2e-b2e3-e0fd93abc7f1\") " pod="openshift-monitoring/metrics-server-59787c5d8d-sth5n"
Sep 16 02:17:59.651649 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:17:59.651628    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/c6abece5-0093-4c2e-b2e3-e0fd93abc7f1-client-ca-bundle\") pod \"metrics-server-59787c5d8d-sth5n\" (UID: \"c6abece5-0093-4c2e-b2e3-e0fd93abc7f1\") " pod="openshift-monitoring/metrics-server-59787c5d8d-sth5n"
Sep 16 02:17:59.652627 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:17:59.652578    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c6abece5-0093-4c2e-b2e3-e0fd93abc7f1-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-59787c5d8d-sth5n\" (UID: \"c6abece5-0093-4c2e-b2e3-e0fd93abc7f1\") " pod="openshift-monitoring/metrics-server-59787c5d8d-sth5n"
Sep 16 02:17:59.652627 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:17:59.652615    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/c6abece5-0093-4c2e-b2e3-e0fd93abc7f1-secret-metrics-server-tls\") pod \"metrics-server-59787c5d8d-sth5n\" (UID: \"c6abece5-0093-4c2e-b2e3-e0fd93abc7f1\") " pod="openshift-monitoring/metrics-server-59787c5d8d-sth5n"
Sep 16 02:17:59.652931 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:17:59.652903    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/c6abece5-0093-4c2e-b2e3-e0fd93abc7f1-metrics-server-audit-profiles\") pod \"metrics-server-59787c5d8d-sth5n\" (UID: \"c6abece5-0093-4c2e-b2e3-e0fd93abc7f1\") " pod="openshift-monitoring/metrics-server-59787c5d8d-sth5n"
Sep 16 02:17:59.653231 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:17:59.653211    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/c6abece5-0093-4c2e-b2e3-e0fd93abc7f1-secret-metrics-server-client-certs\") pod \"metrics-server-59787c5d8d-sth5n\" (UID: \"c6abece5-0093-4c2e-b2e3-e0fd93abc7f1\") " pod="openshift-monitoring/metrics-server-59787c5d8d-sth5n"
Sep 16 02:17:59.656433 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:17:59.656414    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5gfw\" (UniqueName: \"kubernetes.io/projected/c6abece5-0093-4c2e-b2e3-e0fd93abc7f1-kube-api-access-v5gfw\") pod \"metrics-server-59787c5d8d-sth5n\" (UID: \"c6abece5-0093-4c2e-b2e3-e0fd93abc7f1\") " pod="openshift-monitoring/metrics-server-59787c5d8d-sth5n"
Sep 16 02:17:59.778840 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:17:59.778773    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-59787c5d8d-sth5n"
Sep 16 02:18:00.048184 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:18:00.048158    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-59787c5d8d-sth5n"]
Sep 16 02:18:00.135490 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:18:00.135463    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6abece5_0093_4c2e_b2e3_e0fd93abc7f1.slice/crio-7a943d405ac261c3304c17451fd0f2e822541c736fad3463311794c547faa5b9 WatchSource:0}: Error finding container 7a943d405ac261c3304c17451fd0f2e822541c736fad3463311794c547faa5b9: Status 404 returned error can't find the container with id 7a943d405ac261c3304c17451fd0f2e822541c736fad3463311794c547faa5b9
Sep 16 02:18:00.669030 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:18:00.668991    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-59787c5d8d-sth5n" event={"ID":"c6abece5-0093-4c2e-b2e3-e0fd93abc7f1","Type":"ContainerStarted","Data":"7a943d405ac261c3304c17451fd0f2e822541c736fad3463311794c547faa5b9"}
Sep 16 02:18:00.672637 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:18:00.672611    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1629aab1-29f9-42f7-92c9-435571df7299","Type":"ContainerStarted","Data":"53285a0a211b62c28cc8d0d43ebb49dfcf62e1a6d1b4e90fa8d76353e84514b5"}
Sep 16 02:18:00.672798 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:18:00.672685    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1629aab1-29f9-42f7-92c9-435571df7299","Type":"ContainerStarted","Data":"6da6837ae2435d333d6bd4c0ba70b194c1a71fd7477d7440a7177efb9ddaf792"}
Sep 16 02:18:00.672798 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:18:00.672753    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1629aab1-29f9-42f7-92c9-435571df7299","Type":"ContainerStarted","Data":"5e12fc82c1f59e3a1a19b398b8f7f52a1cfdd7729d5369e09c9f27ab2d1e3d5e"}
Sep 16 02:18:00.672798 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:18:00.672789    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1629aab1-29f9-42f7-92c9-435571df7299","Type":"ContainerStarted","Data":"784bb5d339d958c83f7ba3b1daad7e8c07938379f6ced6d03e268ca22927f668"}
Sep 16 02:18:00.672925 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:18:00.672828    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1629aab1-29f9-42f7-92c9-435571df7299","Type":"ContainerStarted","Data":"ad67f8e1bcdeee68381eaef69ebec0acf98f7823889f89e22acbf3dcc128c0cf"}
Sep 16 02:18:01.309996 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:18:01.309964    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 02:18:01.313894 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:18:01.313861    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:18:01.316161 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:18:01.316131    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 16 02:18:01.316293 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:18:01.316196    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 16 02:18:01.316293 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:18:01.316201    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 16 02:18:01.316780 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:18:01.316761    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 16 02:18:01.316901 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:18:01.316828    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 16 02:18:01.316901 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:18:01.316889    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 16 02:18:01.317018 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:18:01.316910    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 16 02:18:01.317018 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:18:01.316993    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 16 02:18:01.317503 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:18:01.317483    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 16 02:18:01.317583 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:18:01.317522    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 16 02:18:01.317904 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:18:01.317793    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-fp9wb\""
Sep 16 02:18:01.317904 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:18:01.317860    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-brud7bi28g55g\""
Sep 16 02:18:01.318557 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:18:01.318462    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 16 02:18:01.318557 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:18:01.318462    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 16 02:18:01.318757 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:18:01.318721    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 16 02:18:01.319951 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:18:01.319933    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 16 02:18:01.332271 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:18:01.332241    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 02:18:01.363939 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:18:01.363909    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/bbc9e62b-1357-4c44-a284-d3ac4a6c593c-config-out\") pod \"prometheus-k8s-0\" (UID: \"bbc9e62b-1357-4c44-a284-d3ac4a6c593c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:18:01.363939 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:18:01.363943    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bbc9e62b-1357-4c44-a284-d3ac4a6c593c-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bbc9e62b-1357-4c44-a284-d3ac4a6c593c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:18:01.364165 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:18:01.363960    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2mzt\" (UniqueName: \"kubernetes.io/projected/bbc9e62b-1357-4c44-a284-d3ac4a6c593c-kube-api-access-g2mzt\") pod \"prometheus-k8s-0\" (UID: \"bbc9e62b-1357-4c44-a284-d3ac4a6c593c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:18:01.364165 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:18:01.364028    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/bbc9e62b-1357-4c44-a284-d3ac4a6c593c-config\") pod \"prometheus-k8s-0\" (UID: \"bbc9e62b-1357-4c44-a284-d3ac4a6c593c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:18:01.364165 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:18:01.364125    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/bbc9e62b-1357-4c44-a284-d3ac4a6c593c-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"bbc9e62b-1357-4c44-a284-d3ac4a6c593c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:18:01.364165 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:18:01.364146    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/bbc9e62b-1357-4c44-a284-d3ac4a6c593c-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"bbc9e62b-1357-4c44-a284-d3ac4a6c593c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:18:01.364165 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:18:01.364163    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/bbc9e62b-1357-4c44-a284-d3ac4a6c593c-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"bbc9e62b-1357-4c44-a284-d3ac4a6c593c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:18:01.364403 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:18:01.364190    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/bbc9e62b-1357-4c44-a284-d3ac4a6c593c-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"bbc9e62b-1357-4c44-a284-d3ac4a6c593c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:18:01.364403 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:18:01.364212    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/bbc9e62b-1357-4c44-a284-d3ac4a6c593c-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"bbc9e62b-1357-4c44-a284-d3ac4a6c593c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:18:01.364403 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:18:01.364272    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bbc9e62b-1357-4c44-a284-d3ac4a6c593c-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"bbc9e62b-1357-4c44-a284-d3ac4a6c593c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:18:01.364403 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:18:01.364308    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/bbc9e62b-1357-4c44-a284-d3ac4a6c593c-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"bbc9e62b-1357-4c44-a284-d3ac4a6c593c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:18:01.364403 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:18:01.364334    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/bbc9e62b-1357-4c44-a284-d3ac4a6c593c-web-config\") pod \"prometheus-k8s-0\" (UID: \"bbc9e62b-1357-4c44-a284-d3ac4a6c593c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:18:01.364403 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:18:01.364363    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bbc9e62b-1357-4c44-a284-d3ac4a6c593c-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bbc9e62b-1357-4c44-a284-d3ac4a6c593c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:18:01.364638 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:18:01.364403    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/bbc9e62b-1357-4c44-a284-d3ac4a6c593c-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"bbc9e62b-1357-4c44-a284-d3ac4a6c593c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:18:01.364638 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:18:01.364437    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/bbc9e62b-1357-4c44-a284-d3ac4a6c593c-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"bbc9e62b-1357-4c44-a284-d3ac4a6c593c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:18:01.364638 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:18:01.364480    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bbc9e62b-1357-4c44-a284-d3ac4a6c593c-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bbc9e62b-1357-4c44-a284-d3ac4a6c593c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:18:01.364638 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:18:01.364544    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/bbc9e62b-1357-4c44-a284-d3ac4a6c593c-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"bbc9e62b-1357-4c44-a284-d3ac4a6c593c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:18:01.364638 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:18:01.364586    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/bbc9e62b-1357-4c44-a284-d3ac4a6c593c-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"bbc9e62b-1357-4c44-a284-d3ac4a6c593c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:18:01.364830 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:18:01.364639    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/bbc9e62b-1357-4c44-a284-d3ac4a6c593c-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"bbc9e62b-1357-4c44-a284-d3ac4a6c593c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:18:01.364830 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:18:01.364667    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/bbc9e62b-1357-4c44-a284-d3ac4a6c593c-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"bbc9e62b-1357-4c44-a284-d3ac4a6c593c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:18:01.466157 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:18:01.466117    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/bbc9e62b-1357-4c44-a284-d3ac4a6c593c-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"bbc9e62b-1357-4c44-a284-d3ac4a6c593c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:18:01.466332 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:18:01.466175    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/bbc9e62b-1357-4c44-a284-d3ac4a6c593c-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"bbc9e62b-1357-4c44-a284-d3ac4a6c593c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:18:01.466332 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:18:01.466201    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bbc9e62b-1357-4c44-a284-d3ac4a6c593c-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bbc9e62b-1357-4c44-a284-d3ac4a6c593c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:18:01.466332 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:18:01.466227    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/bbc9e62b-1357-4c44-a284-d3ac4a6c593c-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"bbc9e62b-1357-4c44-a284-d3ac4a6c593c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:18:01.466332 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:18:01.466262    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/bbc9e62b-1357-4c44-a284-d3ac4a6c593c-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"bbc9e62b-1357-4c44-a284-d3ac4a6c593c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:18:01.466332 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:18:01.466290    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/bbc9e62b-1357-4c44-a284-d3ac4a6c593c-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"bbc9e62b-1357-4c44-a284-d3ac4a6c593c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:18:01.466332 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:18:01.466316    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/bbc9e62b-1357-4c44-a284-d3ac4a6c593c-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"bbc9e62b-1357-4c44-a284-d3ac4a6c593c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:18:01.466643 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:18:01.466346    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/bbc9e62b-1357-4c44-a284-d3ac4a6c593c-config-out\") pod \"prometheus-k8s-0\" (UID: \"bbc9e62b-1357-4c44-a284-d3ac4a6c593c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:18:01.466643 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:18:01.466372    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bbc9e62b-1357-4c44-a284-d3ac4a6c593c-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bbc9e62b-1357-4c44-a284-d3ac4a6c593c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:18:01.466643 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:18:01.466395    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2mzt\" (UniqueName: \"kubernetes.io/projected/bbc9e62b-1357-4c44-a284-d3ac4a6c593c-kube-api-access-g2mzt\") pod \"prometheus-k8s-0\" (UID: \"bbc9e62b-1357-4c44-a284-d3ac4a6c593c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:18:01.466643 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:18:01.466421    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/bbc9e62b-1357-4c44-a284-d3ac4a6c593c-config\") pod \"prometheus-k8s-0\" (UID: \"bbc9e62b-1357-4c44-a284-d3ac4a6c593c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:18:01.466643 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:18:01.466493    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/bbc9e62b-1357-4c44-a284-d3ac4a6c593c-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"bbc9e62b-1357-4c44-a284-d3ac4a6c593c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:18:01.466643 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:18:01.466515    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/bbc9e62b-1357-4c44-a284-d3ac4a6c593c-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"bbc9e62b-1357-4c44-a284-d3ac4a6c593c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:18:01.466643 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:18:01.466543    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/bbc9e62b-1357-4c44-a284-d3ac4a6c593c-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"bbc9e62b-1357-4c44-a284-d3ac4a6c593c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:18:01.466643 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:18:01.466576    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/bbc9e62b-1357-4c44-a284-d3ac4a6c593c-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"bbc9e62b-1357-4c44-a284-d3ac4a6c593c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:18:01.466643 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:18:01.466603    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/bbc9e62b-1357-4c44-a284-d3ac4a6c593c-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"bbc9e62b-1357-4c44-a284-d3ac4a6c593c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:18:01.466643 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:18:01.466640    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bbc9e62b-1357-4c44-a284-d3ac4a6c593c-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"bbc9e62b-1357-4c44-a284-d3ac4a6c593c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:18:01.467057 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:18:01.466663    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/bbc9e62b-1357-4c44-a284-d3ac4a6c593c-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"bbc9e62b-1357-4c44-a284-d3ac4a6c593c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:18:01.467057 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:18:01.466687    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/bbc9e62b-1357-4c44-a284-d3ac4a6c593c-web-config\") pod \"prometheus-k8s-0\" (UID: \"bbc9e62b-1357-4c44-a284-d3ac4a6c593c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:18:01.467057 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:18:01.466740    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bbc9e62b-1357-4c44-a284-d3ac4a6c593c-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bbc9e62b-1357-4c44-a284-d3ac4a6c593c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:18:01.468777 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:18:01.467221    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/bbc9e62b-1357-4c44-a284-d3ac4a6c593c-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"bbc9e62b-1357-4c44-a284-d3ac4a6c593c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:18:01.468777 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:18:01.467264    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bbc9e62b-1357-4c44-a284-d3ac4a6c593c-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bbc9e62b-1357-4c44-a284-d3ac4a6c593c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:18:01.468777 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:18:01.467533    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/bbc9e62b-1357-4c44-a284-d3ac4a6c593c-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"bbc9e62b-1357-4c44-a284-d3ac4a6c593c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:18:01.468777 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:18:01.468037    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bbc9e62b-1357-4c44-a284-d3ac4a6c593c-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bbc9e62b-1357-4c44-a284-d3ac4a6c593c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:18:01.468777 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:18:01.468280    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bbc9e62b-1357-4c44-a284-d3ac4a6c593c-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bbc9e62b-1357-4c44-a284-d3ac4a6c593c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:18:01.468777 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:18:01.468672    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bbc9e62b-1357-4c44-a284-d3ac4a6c593c-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"bbc9e62b-1357-4c44-a284-d3ac4a6c593c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:18:01.469274 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:18:01.469175    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/bbc9e62b-1357-4c44-a284-d3ac4a6c593c-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"bbc9e62b-1357-4c44-a284-d3ac4a6c593c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:18:01.469524 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:18:01.469497    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/bbc9e62b-1357-4c44-a284-d3ac4a6c593c-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"bbc9e62b-1357-4c44-a284-d3ac4a6c593c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:18:01.469612 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:18:01.469580    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/bbc9e62b-1357-4c44-a284-d3ac4a6c593c-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"bbc9e62b-1357-4c44-a284-d3ac4a6c593c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:18:01.470321 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:18:01.470298    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/bbc9e62b-1357-4c44-a284-d3ac4a6c593c-config-out\") pod \"prometheus-k8s-0\" (UID: \"bbc9e62b-1357-4c44-a284-d3ac4a6c593c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:18:01.470714 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:18:01.470650    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/bbc9e62b-1357-4c44-a284-d3ac4a6c593c-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"bbc9e62b-1357-4c44-a284-d3ac4a6c593c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:18:01.471551 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:18:01.471528    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/bbc9e62b-1357-4c44-a284-d3ac4a6c593c-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"bbc9e62b-1357-4c44-a284-d3ac4a6c593c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:18:01.471651 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:18:01.471616    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/bbc9e62b-1357-4c44-a284-d3ac4a6c593c-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"bbc9e62b-1357-4c44-a284-d3ac4a6c593c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:18:01.472632 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:18:01.472590    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/bbc9e62b-1357-4c44-a284-d3ac4a6c593c-config\") pod \"prometheus-k8s-0\" (UID: \"bbc9e62b-1357-4c44-a284-d3ac4a6c593c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:18:01.472936 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:18:01.472911    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/bbc9e62b-1357-4c44-a284-d3ac4a6c593c-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"bbc9e62b-1357-4c44-a284-d3ac4a6c593c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:18:01.473250 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:18:01.473227    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/bbc9e62b-1357-4c44-a284-d3ac4a6c593c-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"bbc9e62b-1357-4c44-a284-d3ac4a6c593c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:18:01.473663 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:18:01.473629    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/bbc9e62b-1357-4c44-a284-d3ac4a6c593c-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"bbc9e62b-1357-4c44-a284-d3ac4a6c593c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:18:01.473874 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:18:01.473850    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/bbc9e62b-1357-4c44-a284-d3ac4a6c593c-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"bbc9e62b-1357-4c44-a284-d3ac4a6c593c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:18:01.474308 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:18:01.474279    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2mzt\" (UniqueName: \"kubernetes.io/projected/bbc9e62b-1357-4c44-a284-d3ac4a6c593c-kube-api-access-g2mzt\") pod \"prometheus-k8s-0\" (UID: \"bbc9e62b-1357-4c44-a284-d3ac4a6c593c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:18:01.475171 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:18:01.475153    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/bbc9e62b-1357-4c44-a284-d3ac4a6c593c-web-config\") pod \"prometheus-k8s-0\" (UID: \"bbc9e62b-1357-4c44-a284-d3ac4a6c593c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:18:01.625224 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:18:01.625186    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:18:01.716656 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:18:01.716600    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=1.7781428689999998 podStartE2EDuration="5.716583722s" totalImagesPullingTime="3.920708581s" totalInitContainerRuntime="17.732272ms" isStatefulPod=false podCreationTimestamp="2026-09-16 02:17:56 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:18:01.715104285 +0000 UTC m=+294.633593533" watchObservedRunningTime="2026-09-16 02:18:01.716583722 +0000 UTC m=+294.635072985"
Sep 16 02:18:01.912999 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:18:01.912971    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 02:18:01.915305 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:18:01.915277    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbc9e62b_1357_4c44_a284_d3ac4a6c593c.slice/crio-cf0d22a40df25f24d2879510e2967b0070c549f885417b0dcc60dcb4ac911ea5 WatchSource:0}: Error finding container cf0d22a40df25f24d2879510e2967b0070c549f885417b0dcc60dcb4ac911ea5: Status 404 returned error can't find the container with id cf0d22a40df25f24d2879510e2967b0070c549f885417b0dcc60dcb4ac911ea5
Sep 16 02:18:02.545797 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:18:02.545746    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-ck2vw" podUID="25e85d58-77e2-4428-b099-a5075363d56f"
Sep 16 02:18:02.700578 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:18:02.700539    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-59787c5d8d-sth5n" event={"ID":"c6abece5-0093-4c2e-b2e3-e0fd93abc7f1","Type":"ContainerStarted","Data":"8c7ca8b6c61f524f174bc7aba8bec6f51ab9e6d90c2ef39a51518fabd1e7e799"}
Sep 16 02:18:02.702141 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:18:02.702107    2558 generic.go:356] "Generic (PLEG): container finished" podID="bbc9e62b-1357-4c44-a284-d3ac4a6c593c" containerID="06d22256acb387ea4d461f898a63d96b601655840f439c08e019ae5c6d3eaf57" exitCode=0
Sep 16 02:18:02.702298 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:18:02.702198    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bbc9e62b-1357-4c44-a284-d3ac4a6c593c","Type":"ContainerDied","Data":"06d22256acb387ea4d461f898a63d96b601655840f439c08e019ae5c6d3eaf57"}
Sep 16 02:18:02.702298 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:18:02.702264    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bbc9e62b-1357-4c44-a284-d3ac4a6c593c","Type":"ContainerStarted","Data":"cf0d22a40df25f24d2879510e2967b0070c549f885417b0dcc60dcb4ac911ea5"}
Sep 16 02:18:02.706084 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:18:02.706061    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1629aab1-29f9-42f7-92c9-435571df7299","Type":"ContainerStarted","Data":"215e97d2cae581638addfec4c1dd84e38cd13660387c25c4cb028442aab42a0e"}
Sep 16 02:18:02.718557 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:18:02.718489    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-59787c5d8d-sth5n" podStartSLOduration=2.243481095 podStartE2EDuration="3.718471703s" totalImagesPullingTime="1.474990608s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 02:17:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:18:02.717046282 +0000 UTC m=+295.635535522" watchObservedRunningTime="2026-09-16 02:18:02.718471703 +0000 UTC m=+295.636960952"
Sep 16 02:18:04.639262 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:18:04.639222    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-q8tpf"
Sep 16 02:18:06.726418 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:18:06.726382    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bbc9e62b-1357-4c44-a284-d3ac4a6c593c","Type":"ContainerStarted","Data":"a5fc9315177131cb886db3f2985bbf14b089596e1caa7b2e12a9adcc70ca8721"}
Sep 16 02:18:06.726782 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:18:06.726450    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bbc9e62b-1357-4c44-a284-d3ac4a6c593c","Type":"ContainerStarted","Data":"d8fa4e821be4059e18f589bfbf6784413d70d416ac4f6cbd1ab52392dba26b0b"}
Sep 16 02:18:07.522394 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:18:07.522370    2558 kubelet.go:1648] "Image garbage collection succeeded"
Sep 16 02:18:08.737337 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:18:08.737299    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bbc9e62b-1357-4c44-a284-d3ac4a6c593c","Type":"ContainerStarted","Data":"bd24b06a02919851a8a2508cac92481a82a98e36ed734f689330686729868bf4"}
Sep 16 02:18:08.737857 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:18:08.737398    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bbc9e62b-1357-4c44-a284-d3ac4a6c593c","Type":"ContainerStarted","Data":"546c6b1fd362d5890ac34fc654937063b7b5077268a72766b9b53ec47d521ed6"}
Sep 16 02:18:08.737857 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:18:08.737428    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bbc9e62b-1357-4c44-a284-d3ac4a6c593c","Type":"ContainerStarted","Data":"9f763d7d753f7105f9d7272cf52d58cfc99913bad623a0bf8f80ef81dadcf1e4"}
Sep 16 02:18:08.737857 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:18:08.737454    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bbc9e62b-1357-4c44-a284-d3ac4a6c593c","Type":"ContainerStarted","Data":"848a9f5242e21b99c79c20be2c8a7de14d7000c19decafe0e4f64a8d7a8052b0"}
Sep 16 02:18:08.763425 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:18:08.763368    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.487648087 podStartE2EDuration="7.763349969s" totalImagesPullingTime="5.257093588s" totalInitContainerRuntime="18.608294ms" isStatefulPod=false podCreationTimestamp="2026-09-16 02:18:01 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:18:08.76082363 +0000 UTC m=+301.679312879" watchObservedRunningTime="2026-09-16 02:18:08.763349969 +0000 UTC m=+301.681839217"
Sep 16 02:18:11.625987 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:18:11.625949    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:18:15.543945 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:18:15.543901    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ck2vw"
Sep 16 02:18:19.779820 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:18:19.779783    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-59787c5d8d-sth5n"
Sep 16 02:18:19.780237 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:18:19.779842    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-59787c5d8d-sth5n"
Sep 16 02:18:20.134207 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:18:20.134122    2558 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-588bc5dc84-p8p9z" podUID="bf8e8a44-fbcc-44bf-a8e0-0633285ae3ba" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Sep 16 02:18:30.134872 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:18:30.134837    2558 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-588bc5dc84-p8p9z" podUID="bf8e8a44-fbcc-44bf-a8e0-0633285ae3ba" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Sep 16 02:18:39.784869 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:18:39.784840    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-59787c5d8d-sth5n"
Sep 16 02:18:39.788477 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:18:39.788453    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-59787c5d8d-sth5n"
Sep 16 02:18:40.135185 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:18:40.135107    2558 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-588bc5dc84-p8p9z" podUID="bf8e8a44-fbcc-44bf-a8e0-0633285ae3ba" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Sep 16 02:18:40.135185 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:18:40.135182    2558 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-588bc5dc84-p8p9z"
Sep 16 02:18:40.135646 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:18:40.135618    2558 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="service-proxy" containerStatusID={"Type":"cri-o","ID":"1b96e4d798abd016817238c583c5bb1f849ec577b3530ccd9256583cb80e94cb"} pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-588bc5dc84-p8p9z" containerMessage="Container service-proxy failed liveness probe, will be restarted"
Sep 16 02:18:40.135686 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:18:40.135666    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-588bc5dc84-p8p9z" podUID="bf8e8a44-fbcc-44bf-a8e0-0633285ae3ba" containerName="service-proxy" containerID="cri-o://1b96e4d798abd016817238c583c5bb1f849ec577b3530ccd9256583cb80e94cb" gracePeriod=30
Sep 16 02:18:40.856036 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:18:40.856004    2558 generic.go:356] "Generic (PLEG): container finished" podID="bf8e8a44-fbcc-44bf-a8e0-0633285ae3ba" containerID="1b96e4d798abd016817238c583c5bb1f849ec577b3530ccd9256583cb80e94cb" exitCode=2
Sep 16 02:18:40.856410 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:18:40.856068    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-588bc5dc84-p8p9z" event={"ID":"bf8e8a44-fbcc-44bf-a8e0-0633285ae3ba","Type":"ContainerDied","Data":"1b96e4d798abd016817238c583c5bb1f849ec577b3530ccd9256583cb80e94cb"}
Sep 16 02:18:40.856410 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:18:40.856114    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-588bc5dc84-p8p9z" event={"ID":"bf8e8a44-fbcc-44bf-a8e0-0633285ae3ba","Type":"ContainerStarted","Data":"043a8fb3b5f2f0640b9e9ee85f3f4f0c20a6335d1bab837d87ae62ebbfe46fc2"}
Sep 16 02:19:01.625761 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:01.625670    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:19:01.644544 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:01.644515    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:19:01.949146 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:01.949116    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:19:15.368674 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:15.368639    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 02:19:15.369186 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:15.368945    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="1629aab1-29f9-42f7-92c9-435571df7299" containerName="alertmanager" containerID="cri-o://ad67f8e1bcdeee68381eaef69ebec0acf98f7823889f89e22acbf3dcc128c0cf" gracePeriod=120
Sep 16 02:19:15.369186 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:15.369005    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="1629aab1-29f9-42f7-92c9-435571df7299" containerName="kube-rbac-proxy-metric" containerID="cri-o://53285a0a211b62c28cc8d0d43ebb49dfcf62e1a6d1b4e90fa8d76353e84514b5" gracePeriod=120
Sep 16 02:19:15.369186 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:15.369043    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="1629aab1-29f9-42f7-92c9-435571df7299" containerName="config-reloader" containerID="cri-o://784bb5d339d958c83f7ba3b1daad7e8c07938379f6ced6d03e268ca22927f668" gracePeriod=120
Sep 16 02:19:15.369186 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:15.369084    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="1629aab1-29f9-42f7-92c9-435571df7299" containerName="prom-label-proxy" containerID="cri-o://215e97d2cae581638addfec4c1dd84e38cd13660387c25c4cb028442aab42a0e" gracePeriod=120
Sep 16 02:19:15.369186 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:15.369108    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="1629aab1-29f9-42f7-92c9-435571df7299" containerName="kube-rbac-proxy" containerID="cri-o://6da6837ae2435d333d6bd4c0ba70b194c1a71fd7477d7440a7177efb9ddaf792" gracePeriod=120
Sep 16 02:19:15.369436 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:15.369027    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="1629aab1-29f9-42f7-92c9-435571df7299" containerName="kube-rbac-proxy-web" containerID="cri-o://5e12fc82c1f59e3a1a19b398b8f7f52a1cfdd7729d5369e09c9f27ab2d1e3d5e" gracePeriod=120
Sep 16 02:19:15.992031 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:15.991994    2558 generic.go:356] "Generic (PLEG): container finished" podID="1629aab1-29f9-42f7-92c9-435571df7299" containerID="215e97d2cae581638addfec4c1dd84e38cd13660387c25c4cb028442aab42a0e" exitCode=0
Sep 16 02:19:15.992031 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:15.992018    2558 generic.go:356] "Generic (PLEG): container finished" podID="1629aab1-29f9-42f7-92c9-435571df7299" containerID="6da6837ae2435d333d6bd4c0ba70b194c1a71fd7477d7440a7177efb9ddaf792" exitCode=0
Sep 16 02:19:15.992031 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:15.992028    2558 generic.go:356] "Generic (PLEG): container finished" podID="1629aab1-29f9-42f7-92c9-435571df7299" containerID="784bb5d339d958c83f7ba3b1daad7e8c07938379f6ced6d03e268ca22927f668" exitCode=0
Sep 16 02:19:15.992031 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:15.992037    2558 generic.go:356] "Generic (PLEG): container finished" podID="1629aab1-29f9-42f7-92c9-435571df7299" containerID="ad67f8e1bcdeee68381eaef69ebec0acf98f7823889f89e22acbf3dcc128c0cf" exitCode=0
Sep 16 02:19:15.992278 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:15.992101    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1629aab1-29f9-42f7-92c9-435571df7299","Type":"ContainerDied","Data":"215e97d2cae581638addfec4c1dd84e38cd13660387c25c4cb028442aab42a0e"}
Sep 16 02:19:15.992278 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:15.992156    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1629aab1-29f9-42f7-92c9-435571df7299","Type":"ContainerDied","Data":"6da6837ae2435d333d6bd4c0ba70b194c1a71fd7477d7440a7177efb9ddaf792"}
Sep 16 02:19:15.992278 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:15.992183    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1629aab1-29f9-42f7-92c9-435571df7299","Type":"ContainerDied","Data":"784bb5d339d958c83f7ba3b1daad7e8c07938379f6ced6d03e268ca22927f668"}
Sep 16 02:19:15.992278 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:15.992205    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1629aab1-29f9-42f7-92c9-435571df7299","Type":"ContainerDied","Data":"ad67f8e1bcdeee68381eaef69ebec0acf98f7823889f89e22acbf3dcc128c0cf"}
Sep 16 02:19:16.605451 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:16.605424    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 16 02:19:16.690463 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:16.690382    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1629aab1-29f9-42f7-92c9-435571df7299-tls-assets\") pod \"1629aab1-29f9-42f7-92c9-435571df7299\" (UID: \"1629aab1-29f9-42f7-92c9-435571df7299\") "
Sep 16 02:19:16.690463 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:16.690420    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/1629aab1-29f9-42f7-92c9-435571df7299-config-volume\") pod \"1629aab1-29f9-42f7-92c9-435571df7299\" (UID: \"1629aab1-29f9-42f7-92c9-435571df7299\") "
Sep 16 02:19:16.690463 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:16.690451    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1629aab1-29f9-42f7-92c9-435571df7299-web-config\") pod \"1629aab1-29f9-42f7-92c9-435571df7299\" (UID: \"1629aab1-29f9-42f7-92c9-435571df7299\") "
Sep 16 02:19:16.690718 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:16.690573    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1629aab1-29f9-42f7-92c9-435571df7299-secret-alertmanager-main-tls\") pod \"1629aab1-29f9-42f7-92c9-435571df7299\" (UID: \"1629aab1-29f9-42f7-92c9-435571df7299\") "
Sep 16 02:19:16.690718 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:16.690624    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1629aab1-29f9-42f7-92c9-435571df7299-config-out\") pod \"1629aab1-29f9-42f7-92c9-435571df7299\" (UID: \"1629aab1-29f9-42f7-92c9-435571df7299\") "
Sep 16 02:19:16.690718 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:16.690653    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wb8ht\" (UniqueName: \"kubernetes.io/projected/1629aab1-29f9-42f7-92c9-435571df7299-kube-api-access-wb8ht\") pod \"1629aab1-29f9-42f7-92c9-435571df7299\" (UID: \"1629aab1-29f9-42f7-92c9-435571df7299\") "
Sep 16 02:19:16.690718 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:16.690678    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1629aab1-29f9-42f7-92c9-435571df7299-alertmanager-trusted-ca-bundle\") pod \"1629aab1-29f9-42f7-92c9-435571df7299\" (UID: \"1629aab1-29f9-42f7-92c9-435571df7299\") "
Sep 16 02:19:16.690931 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:16.690732    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/1629aab1-29f9-42f7-92c9-435571df7299-secret-alertmanager-kube-rbac-proxy-metric\") pod \"1629aab1-29f9-42f7-92c9-435571df7299\" (UID: \"1629aab1-29f9-42f7-92c9-435571df7299\") "
Sep 16 02:19:16.690931 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:16.690763    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1629aab1-29f9-42f7-92c9-435571df7299-metrics-client-ca\") pod \"1629aab1-29f9-42f7-92c9-435571df7299\" (UID: \"1629aab1-29f9-42f7-92c9-435571df7299\") "
Sep 16 02:19:16.690931 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:16.690840    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/1629aab1-29f9-42f7-92c9-435571df7299-alertmanager-main-db\") pod \"1629aab1-29f9-42f7-92c9-435571df7299\" (UID: \"1629aab1-29f9-42f7-92c9-435571df7299\") "
Sep 16 02:19:16.690931 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:16.690870    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1629aab1-29f9-42f7-92c9-435571df7299-secret-alertmanager-kube-rbac-proxy\") pod \"1629aab1-29f9-42f7-92c9-435571df7299\" (UID: \"1629aab1-29f9-42f7-92c9-435571df7299\") "
Sep 16 02:19:16.690931 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:16.690913    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1629aab1-29f9-42f7-92c9-435571df7299-secret-alertmanager-kube-rbac-proxy-web\") pod \"1629aab1-29f9-42f7-92c9-435571df7299\" (UID: \"1629aab1-29f9-42f7-92c9-435571df7299\") "
Sep 16 02:19:16.691188 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:16.690937    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/1629aab1-29f9-42f7-92c9-435571df7299-cluster-tls-config\") pod \"1629aab1-29f9-42f7-92c9-435571df7299\" (UID: \"1629aab1-29f9-42f7-92c9-435571df7299\") "
Sep 16 02:19:16.692475 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:16.692435    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1629aab1-29f9-42f7-92c9-435571df7299-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "1629aab1-29f9-42f7-92c9-435571df7299" (UID: "1629aab1-29f9-42f7-92c9-435571df7299"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:19:16.693283 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:16.693251    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1629aab1-29f9-42f7-92c9-435571df7299-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "1629aab1-29f9-42f7-92c9-435571df7299" (UID: "1629aab1-29f9-42f7-92c9-435571df7299"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 02:19:16.693409 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:16.693376    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1629aab1-29f9-42f7-92c9-435571df7299-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "1629aab1-29f9-42f7-92c9-435571df7299" (UID: "1629aab1-29f9-42f7-92c9-435571df7299"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 02:19:16.693478 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:16.693453    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1629aab1-29f9-42f7-92c9-435571df7299-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "1629aab1-29f9-42f7-92c9-435571df7299" (UID: "1629aab1-29f9-42f7-92c9-435571df7299"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 02:19:16.693547 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:16.693505    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1629aab1-29f9-42f7-92c9-435571df7299-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "1629aab1-29f9-42f7-92c9-435571df7299" (UID: "1629aab1-29f9-42f7-92c9-435571df7299"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 02:19:16.693846 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:16.693822    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1629aab1-29f9-42f7-92c9-435571df7299-kube-api-access-wb8ht" (OuterVolumeSpecName: "kube-api-access-wb8ht") pod "1629aab1-29f9-42f7-92c9-435571df7299" (UID: "1629aab1-29f9-42f7-92c9-435571df7299"). InnerVolumeSpecName "kube-api-access-wb8ht". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 02:19:16.694164 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:16.694129    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1629aab1-29f9-42f7-92c9-435571df7299-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "1629aab1-29f9-42f7-92c9-435571df7299" (UID: "1629aab1-29f9-42f7-92c9-435571df7299"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 02:19:16.694314 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:16.694293    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1629aab1-29f9-42f7-92c9-435571df7299-config-out" (OuterVolumeSpecName: "config-out") pod "1629aab1-29f9-42f7-92c9-435571df7299" (UID: "1629aab1-29f9-42f7-92c9-435571df7299"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:19:16.694673 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:16.694648    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1629aab1-29f9-42f7-92c9-435571df7299-config-volume" (OuterVolumeSpecName: "config-volume") pod "1629aab1-29f9-42f7-92c9-435571df7299" (UID: "1629aab1-29f9-42f7-92c9-435571df7299"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 02:19:16.695574 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:16.695540    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1629aab1-29f9-42f7-92c9-435571df7299-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "1629aab1-29f9-42f7-92c9-435571df7299" (UID: "1629aab1-29f9-42f7-92c9-435571df7299"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 02:19:16.696076 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:16.696048    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1629aab1-29f9-42f7-92c9-435571df7299-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "1629aab1-29f9-42f7-92c9-435571df7299" (UID: "1629aab1-29f9-42f7-92c9-435571df7299"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 02:19:16.697636 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:16.697612    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1629aab1-29f9-42f7-92c9-435571df7299-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "1629aab1-29f9-42f7-92c9-435571df7299" (UID: "1629aab1-29f9-42f7-92c9-435571df7299"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 02:19:16.704178 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:16.704149    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1629aab1-29f9-42f7-92c9-435571df7299-web-config" (OuterVolumeSpecName: "web-config") pod "1629aab1-29f9-42f7-92c9-435571df7299" (UID: "1629aab1-29f9-42f7-92c9-435571df7299"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 02:19:16.792111 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:16.792059    2558 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1629aab1-29f9-42f7-92c9-435571df7299-web-config\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:19:16.792111 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:16.792107    2558 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1629aab1-29f9-42f7-92c9-435571df7299-secret-alertmanager-main-tls\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:19:16.792111 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:16.792118    2558 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1629aab1-29f9-42f7-92c9-435571df7299-config-out\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:19:16.792111 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:16.792127    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wb8ht\" (UniqueName: \"kubernetes.io/projected/1629aab1-29f9-42f7-92c9-435571df7299-kube-api-access-wb8ht\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:19:16.792329 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:16.792136    2558 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1629aab1-29f9-42f7-92c9-435571df7299-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:19:16.792329 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:16.792146    2558 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/1629aab1-29f9-42f7-92c9-435571df7299-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:19:16.792329 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:16.792157    2558 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1629aab1-29f9-42f7-92c9-435571df7299-metrics-client-ca\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:19:16.792329 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:16.792165    2558 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/1629aab1-29f9-42f7-92c9-435571df7299-alertmanager-main-db\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:19:16.792329 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:16.792173    2558 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1629aab1-29f9-42f7-92c9-435571df7299-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:19:16.792329 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:16.792181    2558 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1629aab1-29f9-42f7-92c9-435571df7299-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:19:16.792329 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:16.792189    2558 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/1629aab1-29f9-42f7-92c9-435571df7299-cluster-tls-config\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:19:16.792329 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:16.792198    2558 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1629aab1-29f9-42f7-92c9-435571df7299-tls-assets\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:19:16.792329 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:16.792205    2558 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/1629aab1-29f9-42f7-92c9-435571df7299-config-volume\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:19:16.998663 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:16.998583    2558 generic.go:356] "Generic (PLEG): container finished" podID="1629aab1-29f9-42f7-92c9-435571df7299" containerID="53285a0a211b62c28cc8d0d43ebb49dfcf62e1a6d1b4e90fa8d76353e84514b5" exitCode=0
Sep 16 02:19:16.998663 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:16.998606    2558 generic.go:356] "Generic (PLEG): container finished" podID="1629aab1-29f9-42f7-92c9-435571df7299" containerID="5e12fc82c1f59e3a1a19b398b8f7f52a1cfdd7729d5369e09c9f27ab2d1e3d5e" exitCode=0
Sep 16 02:19:16.998663 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:16.998643    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1629aab1-29f9-42f7-92c9-435571df7299","Type":"ContainerDied","Data":"53285a0a211b62c28cc8d0d43ebb49dfcf62e1a6d1b4e90fa8d76353e84514b5"}
Sep 16 02:19:16.998663 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:16.998658    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 16 02:19:16.998934 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:16.998755    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1629aab1-29f9-42f7-92c9-435571df7299","Type":"ContainerDied","Data":"5e12fc82c1f59e3a1a19b398b8f7f52a1cfdd7729d5369e09c9f27ab2d1e3d5e"}
Sep 16 02:19:16.998934 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:16.998783    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1629aab1-29f9-42f7-92c9-435571df7299","Type":"ContainerDied","Data":"9fa0e5ad650b5145bb7e3beb2c14055ec674cc29e7e7b33e02ed2bc82009d696"}
Sep 16 02:19:16.998934 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:16.998815    2558 scope.go:117] "RemoveContainer" containerID="215e97d2cae581638addfec4c1dd84e38cd13660387c25c4cb028442aab42a0e"
Sep 16 02:19:17.007109 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:17.007089    2558 scope.go:117] "RemoveContainer" containerID="53285a0a211b62c28cc8d0d43ebb49dfcf62e1a6d1b4e90fa8d76353e84514b5"
Sep 16 02:19:17.013885 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:17.013863    2558 scope.go:117] "RemoveContainer" containerID="6da6837ae2435d333d6bd4c0ba70b194c1a71fd7477d7440a7177efb9ddaf792"
Sep 16 02:19:17.021517 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:17.021484    2558 scope.go:117] "RemoveContainer" containerID="5e12fc82c1f59e3a1a19b398b8f7f52a1cfdd7729d5369e09c9f27ab2d1e3d5e"
Sep 16 02:19:17.023152 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:17.023131    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 02:19:17.026258 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:17.026234    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 02:19:17.028616 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:17.028600    2558 scope.go:117] "RemoveContainer" containerID="784bb5d339d958c83f7ba3b1daad7e8c07938379f6ced6d03e268ca22927f668"
Sep 16 02:19:17.035595 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:17.035576    2558 scope.go:117] "RemoveContainer" containerID="ad67f8e1bcdeee68381eaef69ebec0acf98f7823889f89e22acbf3dcc128c0cf"
Sep 16 02:19:17.042928 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:17.042911    2558 scope.go:117] "RemoveContainer" containerID="203e09dca42113069c030dbf29db5c11179a7cdfbb317469adeeaf24b00eb553"
Sep 16 02:19:17.049751 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:17.049732    2558 scope.go:117] "RemoveContainer" containerID="215e97d2cae581638addfec4c1dd84e38cd13660387c25c4cb028442aab42a0e"
Sep 16 02:19:17.050055 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:19:17.050029    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"215e97d2cae581638addfec4c1dd84e38cd13660387c25c4cb028442aab42a0e\": container with ID starting with 215e97d2cae581638addfec4c1dd84e38cd13660387c25c4cb028442aab42a0e not found: ID does not exist" containerID="215e97d2cae581638addfec4c1dd84e38cd13660387c25c4cb028442aab42a0e"
Sep 16 02:19:17.050229 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:17.050068    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"215e97d2cae581638addfec4c1dd84e38cd13660387c25c4cb028442aab42a0e"} err="failed to get container status \"215e97d2cae581638addfec4c1dd84e38cd13660387c25c4cb028442aab42a0e\": rpc error: code = NotFound desc = could not find container \"215e97d2cae581638addfec4c1dd84e38cd13660387c25c4cb028442aab42a0e\": container with ID starting with 215e97d2cae581638addfec4c1dd84e38cd13660387c25c4cb028442aab42a0e not found: ID does not exist"
Sep 16 02:19:17.050229 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:17.050106    2558 scope.go:117] "RemoveContainer" containerID="53285a0a211b62c28cc8d0d43ebb49dfcf62e1a6d1b4e90fa8d76353e84514b5"
Sep 16 02:19:17.050576 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:19:17.050549    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53285a0a211b62c28cc8d0d43ebb49dfcf62e1a6d1b4e90fa8d76353e84514b5\": container with ID starting with 53285a0a211b62c28cc8d0d43ebb49dfcf62e1a6d1b4e90fa8d76353e84514b5 not found: ID does not exist" containerID="53285a0a211b62c28cc8d0d43ebb49dfcf62e1a6d1b4e90fa8d76353e84514b5"
Sep 16 02:19:17.050655 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:17.050595    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53285a0a211b62c28cc8d0d43ebb49dfcf62e1a6d1b4e90fa8d76353e84514b5"} err="failed to get container status \"53285a0a211b62c28cc8d0d43ebb49dfcf62e1a6d1b4e90fa8d76353e84514b5\": rpc error: code = NotFound desc = could not find container \"53285a0a211b62c28cc8d0d43ebb49dfcf62e1a6d1b4e90fa8d76353e84514b5\": container with ID starting with 53285a0a211b62c28cc8d0d43ebb49dfcf62e1a6d1b4e90fa8d76353e84514b5 not found: ID does not exist"
Sep 16 02:19:17.050772 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:17.050660    2558 scope.go:117] "RemoveContainer" containerID="6da6837ae2435d333d6bd4c0ba70b194c1a71fd7477d7440a7177efb9ddaf792"
Sep 16 02:19:17.051114 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:19:17.051056    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6da6837ae2435d333d6bd4c0ba70b194c1a71fd7477d7440a7177efb9ddaf792\": container with ID starting with 6da6837ae2435d333d6bd4c0ba70b194c1a71fd7477d7440a7177efb9ddaf792 not found: ID does not exist" containerID="6da6837ae2435d333d6bd4c0ba70b194c1a71fd7477d7440a7177efb9ddaf792"
Sep 16 02:19:17.051293 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:17.051087    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6da6837ae2435d333d6bd4c0ba70b194c1a71fd7477d7440a7177efb9ddaf792"} err="failed to get container status \"6da6837ae2435d333d6bd4c0ba70b194c1a71fd7477d7440a7177efb9ddaf792\": rpc error: code = NotFound desc = could not find container \"6da6837ae2435d333d6bd4c0ba70b194c1a71fd7477d7440a7177efb9ddaf792\": container with ID starting with 6da6837ae2435d333d6bd4c0ba70b194c1a71fd7477d7440a7177efb9ddaf792 not found: ID does not exist"
Sep 16 02:19:17.051293 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:17.051226    2558 scope.go:117] "RemoveContainer" containerID="5e12fc82c1f59e3a1a19b398b8f7f52a1cfdd7729d5369e09c9f27ab2d1e3d5e"
Sep 16 02:19:17.051608 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:19:17.051589    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e12fc82c1f59e3a1a19b398b8f7f52a1cfdd7729d5369e09c9f27ab2d1e3d5e\": container with ID starting with 5e12fc82c1f59e3a1a19b398b8f7f52a1cfdd7729d5369e09c9f27ab2d1e3d5e not found: ID does not exist" containerID="5e12fc82c1f59e3a1a19b398b8f7f52a1cfdd7729d5369e09c9f27ab2d1e3d5e"
Sep 16 02:19:17.051666 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:17.051613    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e12fc82c1f59e3a1a19b398b8f7f52a1cfdd7729d5369e09c9f27ab2d1e3d5e"} err="failed to get container status \"5e12fc82c1f59e3a1a19b398b8f7f52a1cfdd7729d5369e09c9f27ab2d1e3d5e\": rpc error: code = NotFound desc = could not find container \"5e12fc82c1f59e3a1a19b398b8f7f52a1cfdd7729d5369e09c9f27ab2d1e3d5e\": container with ID starting with 5e12fc82c1f59e3a1a19b398b8f7f52a1cfdd7729d5369e09c9f27ab2d1e3d5e not found: ID does not exist"
Sep 16 02:19:17.051666 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:17.051628    2558 scope.go:117] "RemoveContainer" containerID="784bb5d339d958c83f7ba3b1daad7e8c07938379f6ced6d03e268ca22927f668"
Sep 16 02:19:17.051904 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:19:17.051885    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"784bb5d339d958c83f7ba3b1daad7e8c07938379f6ced6d03e268ca22927f668\": container with ID starting with 784bb5d339d958c83f7ba3b1daad7e8c07938379f6ced6d03e268ca22927f668 not found: ID does not exist" containerID="784bb5d339d958c83f7ba3b1daad7e8c07938379f6ced6d03e268ca22927f668"
Sep 16 02:19:17.051944 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:17.051910    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"784bb5d339d958c83f7ba3b1daad7e8c07938379f6ced6d03e268ca22927f668"} err="failed to get container status \"784bb5d339d958c83f7ba3b1daad7e8c07938379f6ced6d03e268ca22927f668\": rpc error: code = NotFound desc = could not find container \"784bb5d339d958c83f7ba3b1daad7e8c07938379f6ced6d03e268ca22927f668\": container with ID starting with 784bb5d339d958c83f7ba3b1daad7e8c07938379f6ced6d03e268ca22927f668 not found: ID does not exist"
Sep 16 02:19:17.051944 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:17.051926    2558 scope.go:117] "RemoveContainer" containerID="ad67f8e1bcdeee68381eaef69ebec0acf98f7823889f89e22acbf3dcc128c0cf"
Sep 16 02:19:17.052160 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:17.052142    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 02:19:17.052209 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:19:17.052185    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad67f8e1bcdeee68381eaef69ebec0acf98f7823889f89e22acbf3dcc128c0cf\": container with ID starting with ad67f8e1bcdeee68381eaef69ebec0acf98f7823889f89e22acbf3dcc128c0cf not found: ID does not exist" containerID="ad67f8e1bcdeee68381eaef69ebec0acf98f7823889f89e22acbf3dcc128c0cf"
Sep 16 02:19:17.052209 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:17.052202    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad67f8e1bcdeee68381eaef69ebec0acf98f7823889f89e22acbf3dcc128c0cf"} err="failed to get container status \"ad67f8e1bcdeee68381eaef69ebec0acf98f7823889f89e22acbf3dcc128c0cf\": rpc error: code = NotFound desc = could not find container \"ad67f8e1bcdeee68381eaef69ebec0acf98f7823889f89e22acbf3dcc128c0cf\": container with ID starting with ad67f8e1bcdeee68381eaef69ebec0acf98f7823889f89e22acbf3dcc128c0cf not found: ID does not exist"
Sep 16 02:19:17.052275 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:17.052213    2558 scope.go:117] "RemoveContainer" containerID="203e09dca42113069c030dbf29db5c11179a7cdfbb317469adeeaf24b00eb553"
Sep 16 02:19:17.052462 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:17.052448    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1629aab1-29f9-42f7-92c9-435571df7299" containerName="alertmanager"
Sep 16 02:19:17.052506 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:17.052464    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="1629aab1-29f9-42f7-92c9-435571df7299" containerName="alertmanager"
Sep 16 02:19:17.052506 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:19:17.052459    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"203e09dca42113069c030dbf29db5c11179a7cdfbb317469adeeaf24b00eb553\": container with ID starting with 203e09dca42113069c030dbf29db5c11179a7cdfbb317469adeeaf24b00eb553 not found: ID does not exist" containerID="203e09dca42113069c030dbf29db5c11179a7cdfbb317469adeeaf24b00eb553"
Sep 16 02:19:17.052506 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:17.052471    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1629aab1-29f9-42f7-92c9-435571df7299" containerName="kube-rbac-proxy-web"
Sep 16 02:19:17.052506 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:17.052479    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="1629aab1-29f9-42f7-92c9-435571df7299" containerName="kube-rbac-proxy-web"
Sep 16 02:19:17.052506 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:17.052480    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"203e09dca42113069c030dbf29db5c11179a7cdfbb317469adeeaf24b00eb553"} err="failed to get container status \"203e09dca42113069c030dbf29db5c11179a7cdfbb317469adeeaf24b00eb553\": rpc error: code = NotFound desc = could not find container \"203e09dca42113069c030dbf29db5c11179a7cdfbb317469adeeaf24b00eb553\": container with ID starting with 203e09dca42113069c030dbf29db5c11179a7cdfbb317469adeeaf24b00eb553 not found: ID does not exist"
Sep 16 02:19:17.052506 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:17.052490    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1629aab1-29f9-42f7-92c9-435571df7299" containerName="init-config-reloader"
Sep 16 02:19:17.052506 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:17.052493    2558 scope.go:117] "RemoveContainer" containerID="215e97d2cae581638addfec4c1dd84e38cd13660387c25c4cb028442aab42a0e"
Sep 16 02:19:17.052767 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:17.052496    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="1629aab1-29f9-42f7-92c9-435571df7299" containerName="init-config-reloader"
Sep 16 02:19:17.052767 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:17.052546    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1629aab1-29f9-42f7-92c9-435571df7299" containerName="config-reloader"
Sep 16 02:19:17.052767 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:17.052554    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="1629aab1-29f9-42f7-92c9-435571df7299" containerName="config-reloader"
Sep 16 02:19:17.052767 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:17.052560    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1629aab1-29f9-42f7-92c9-435571df7299" containerName="kube-rbac-proxy-metric"
Sep 16 02:19:17.052767 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:17.052566    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="1629aab1-29f9-42f7-92c9-435571df7299" containerName="kube-rbac-proxy-metric"
Sep 16 02:19:17.052767 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:17.052584    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1629aab1-29f9-42f7-92c9-435571df7299" containerName="kube-rbac-proxy"
Sep 16 02:19:17.052767 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:17.052593    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="1629aab1-29f9-42f7-92c9-435571df7299" containerName="kube-rbac-proxy"
Sep 16 02:19:17.052767 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:17.052612    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1629aab1-29f9-42f7-92c9-435571df7299" containerName="prom-label-proxy"
Sep 16 02:19:17.052767 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:17.052621    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="1629aab1-29f9-42f7-92c9-435571df7299" containerName="prom-label-proxy"
Sep 16 02:19:17.052767 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:17.052722    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="1629aab1-29f9-42f7-92c9-435571df7299" containerName="kube-rbac-proxy-web"
Sep 16 02:19:17.052767 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:17.052736    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="1629aab1-29f9-42f7-92c9-435571df7299" containerName="kube-rbac-proxy-metric"
Sep 16 02:19:17.052767 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:17.052746    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="1629aab1-29f9-42f7-92c9-435571df7299" containerName="alertmanager"
Sep 16 02:19:17.052767 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:17.052752    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="1629aab1-29f9-42f7-92c9-435571df7299" containerName="kube-rbac-proxy"
Sep 16 02:19:17.052767 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:17.052758    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="1629aab1-29f9-42f7-92c9-435571df7299" containerName="config-reloader"
Sep 16 02:19:17.052767 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:17.052764    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="1629aab1-29f9-42f7-92c9-435571df7299" containerName="prom-label-proxy"
Sep 16 02:19:17.053267 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:17.052785    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"215e97d2cae581638addfec4c1dd84e38cd13660387c25c4cb028442aab42a0e"} err="failed to get container status \"215e97d2cae581638addfec4c1dd84e38cd13660387c25c4cb028442aab42a0e\": rpc error: code = NotFound desc = could not find container \"215e97d2cae581638addfec4c1dd84e38cd13660387c25c4cb028442aab42a0e\": container with ID starting with 215e97d2cae581638addfec4c1dd84e38cd13660387c25c4cb028442aab42a0e not found: ID does not exist"
Sep 16 02:19:17.053267 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:17.052800    2558 scope.go:117] "RemoveContainer" containerID="53285a0a211b62c28cc8d0d43ebb49dfcf62e1a6d1b4e90fa8d76353e84514b5"
Sep 16 02:19:17.053267 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:17.052999    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53285a0a211b62c28cc8d0d43ebb49dfcf62e1a6d1b4e90fa8d76353e84514b5"} err="failed to get container status \"53285a0a211b62c28cc8d0d43ebb49dfcf62e1a6d1b4e90fa8d76353e84514b5\": rpc error: code = NotFound desc = could not find container \"53285a0a211b62c28cc8d0d43ebb49dfcf62e1a6d1b4e90fa8d76353e84514b5\": container with ID starting with 53285a0a211b62c28cc8d0d43ebb49dfcf62e1a6d1b4e90fa8d76353e84514b5 not found: ID does not exist"
Sep 16 02:19:17.053267 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:17.053015    2558 scope.go:117] "RemoveContainer" containerID="6da6837ae2435d333d6bd4c0ba70b194c1a71fd7477d7440a7177efb9ddaf792"
Sep 16 02:19:17.053400 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:17.053275    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6da6837ae2435d333d6bd4c0ba70b194c1a71fd7477d7440a7177efb9ddaf792"} err="failed to get container status \"6da6837ae2435d333d6bd4c0ba70b194c1a71fd7477d7440a7177efb9ddaf792\": rpc error: code = NotFound desc = could not find container \"6da6837ae2435d333d6bd4c0ba70b194c1a71fd7477d7440a7177efb9ddaf792\": container with ID starting with 6da6837ae2435d333d6bd4c0ba70b194c1a71fd7477d7440a7177efb9ddaf792 not found: ID does not exist"
Sep 16 02:19:17.053400 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:17.053291    2558 scope.go:117] "RemoveContainer" containerID="5e12fc82c1f59e3a1a19b398b8f7f52a1cfdd7729d5369e09c9f27ab2d1e3d5e"
Sep 16 02:19:17.053516 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:17.053500    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e12fc82c1f59e3a1a19b398b8f7f52a1cfdd7729d5369e09c9f27ab2d1e3d5e"} err="failed to get container status \"5e12fc82c1f59e3a1a19b398b8f7f52a1cfdd7729d5369e09c9f27ab2d1e3d5e\": rpc error: code = NotFound desc = could not find container \"5e12fc82c1f59e3a1a19b398b8f7f52a1cfdd7729d5369e09c9f27ab2d1e3d5e\": container with ID starting with 5e12fc82c1f59e3a1a19b398b8f7f52a1cfdd7729d5369e09c9f27ab2d1e3d5e not found: ID does not exist"
Sep 16 02:19:17.053563 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:17.053516    2558 scope.go:117] "RemoveContainer" containerID="784bb5d339d958c83f7ba3b1daad7e8c07938379f6ced6d03e268ca22927f668"
Sep 16 02:19:17.053765 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:17.053737    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"784bb5d339d958c83f7ba3b1daad7e8c07938379f6ced6d03e268ca22927f668"} err="failed to get container status \"784bb5d339d958c83f7ba3b1daad7e8c07938379f6ced6d03e268ca22927f668\": rpc error: code = NotFound desc = could not find container \"784bb5d339d958c83f7ba3b1daad7e8c07938379f6ced6d03e268ca22927f668\": container with ID starting with 784bb5d339d958c83f7ba3b1daad7e8c07938379f6ced6d03e268ca22927f668 not found: ID does not exist"
Sep 16 02:19:17.053765 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:17.053763    2558 scope.go:117] "RemoveContainer" containerID="ad67f8e1bcdeee68381eaef69ebec0acf98f7823889f89e22acbf3dcc128c0cf"
Sep 16 02:19:17.053984 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:17.053965    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad67f8e1bcdeee68381eaef69ebec0acf98f7823889f89e22acbf3dcc128c0cf"} err="failed to get container status \"ad67f8e1bcdeee68381eaef69ebec0acf98f7823889f89e22acbf3dcc128c0cf\": rpc error: code = NotFound desc = could not find container \"ad67f8e1bcdeee68381eaef69ebec0acf98f7823889f89e22acbf3dcc128c0cf\": container with ID starting with ad67f8e1bcdeee68381eaef69ebec0acf98f7823889f89e22acbf3dcc128c0cf not found: ID does not exist"
Sep 16 02:19:17.053984 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:17.053981    2558 scope.go:117] "RemoveContainer" containerID="203e09dca42113069c030dbf29db5c11179a7cdfbb317469adeeaf24b00eb553"
Sep 16 02:19:17.054234 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:17.054215    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"203e09dca42113069c030dbf29db5c11179a7cdfbb317469adeeaf24b00eb553"} err="failed to get container status \"203e09dca42113069c030dbf29db5c11179a7cdfbb317469adeeaf24b00eb553\": rpc error: code = NotFound desc = could not find container \"203e09dca42113069c030dbf29db5c11179a7cdfbb317469adeeaf24b00eb553\": container with ID starting with 203e09dca42113069c030dbf29db5c11179a7cdfbb317469adeeaf24b00eb553 not found: ID does not exist"
Sep 16 02:19:17.057775 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:17.057758    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 16 02:19:17.059903 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:17.059886    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 16 02:19:17.060149 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:17.060124    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 16 02:19:17.060258 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:17.060228    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-kxzjt\""
Sep 16 02:19:17.060330 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:17.060280    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 16 02:19:17.060501 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:17.060485    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 16 02:19:17.060833 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:17.060777    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 16 02:19:17.060833 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:17.060787    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 16 02:19:17.060833 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:17.060787    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 16 02:19:17.061042 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:17.060794    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 16 02:19:17.066561 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:17.066543    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 16 02:19:17.072313 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:17.072292    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 02:19:17.195564 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:17.195528    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a2de3f62-ec4e-49fc-8d7c-42c62f64dd94-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"a2de3f62-ec4e-49fc-8d7c-42c62f64dd94\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 02:19:17.195753 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:17.195572    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a2de3f62-ec4e-49fc-8d7c-42c62f64dd94-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"a2de3f62-ec4e-49fc-8d7c-42c62f64dd94\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 02:19:17.195753 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:17.195611    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a2de3f62-ec4e-49fc-8d7c-42c62f64dd94-config-volume\") pod \"alertmanager-main-0\" (UID: \"a2de3f62-ec4e-49fc-8d7c-42c62f64dd94\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 02:19:17.195753 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:17.195660    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a2de3f62-ec4e-49fc-8d7c-42c62f64dd94-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"a2de3f62-ec4e-49fc-8d7c-42c62f64dd94\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 02:19:17.195875 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:17.195781    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a2de3f62-ec4e-49fc-8d7c-42c62f64dd94-tls-assets\") pod \"alertmanager-main-0\" (UID: \"a2de3f62-ec4e-49fc-8d7c-42c62f64dd94\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 02:19:17.195875 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:17.195836    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a2de3f62-ec4e-49fc-8d7c-42c62f64dd94-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"a2de3f62-ec4e-49fc-8d7c-42c62f64dd94\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 02:19:17.195875 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:17.195871    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a2de3f62-ec4e-49fc-8d7c-42c62f64dd94-config-out\") pod \"alertmanager-main-0\" (UID: \"a2de3f62-ec4e-49fc-8d7c-42c62f64dd94\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 02:19:17.195979 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:17.195900    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a2de3f62-ec4e-49fc-8d7c-42c62f64dd94-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"a2de3f62-ec4e-49fc-8d7c-42c62f64dd94\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 02:19:17.195979 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:17.195933    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a2de3f62-ec4e-49fc-8d7c-42c62f64dd94-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"a2de3f62-ec4e-49fc-8d7c-42c62f64dd94\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 02:19:17.195979 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:17.195959    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a2de3f62-ec4e-49fc-8d7c-42c62f64dd94-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"a2de3f62-ec4e-49fc-8d7c-42c62f64dd94\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 02:19:17.196092 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:17.196040    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a2de3f62-ec4e-49fc-8d7c-42c62f64dd94-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"a2de3f62-ec4e-49fc-8d7c-42c62f64dd94\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 02:19:17.196092 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:17.196083    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a2de3f62-ec4e-49fc-8d7c-42c62f64dd94-web-config\") pod \"alertmanager-main-0\" (UID: \"a2de3f62-ec4e-49fc-8d7c-42c62f64dd94\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 02:19:17.196159 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:17.196108    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5pm5t\" (UniqueName: \"kubernetes.io/projected/a2de3f62-ec4e-49fc-8d7c-42c62f64dd94-kube-api-access-5pm5t\") pod \"alertmanager-main-0\" (UID: \"a2de3f62-ec4e-49fc-8d7c-42c62f64dd94\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 02:19:17.297548 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:17.297457    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a2de3f62-ec4e-49fc-8d7c-42c62f64dd94-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"a2de3f62-ec4e-49fc-8d7c-42c62f64dd94\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 02:19:17.297548 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:17.297501    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a2de3f62-ec4e-49fc-8d7c-42c62f64dd94-web-config\") pod \"alertmanager-main-0\" (UID: \"a2de3f62-ec4e-49fc-8d7c-42c62f64dd94\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 02:19:17.297548 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:17.297531    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5pm5t\" (UniqueName: \"kubernetes.io/projected/a2de3f62-ec4e-49fc-8d7c-42c62f64dd94-kube-api-access-5pm5t\") pod \"alertmanager-main-0\" (UID: \"a2de3f62-ec4e-49fc-8d7c-42c62f64dd94\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 02:19:17.297548 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:17.297555    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a2de3f62-ec4e-49fc-8d7c-42c62f64dd94-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"a2de3f62-ec4e-49fc-8d7c-42c62f64dd94\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 02:19:17.297916 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:17.297577    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a2de3f62-ec4e-49fc-8d7c-42c62f64dd94-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"a2de3f62-ec4e-49fc-8d7c-42c62f64dd94\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 02:19:17.297916 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:17.297595    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a2de3f62-ec4e-49fc-8d7c-42c62f64dd94-config-volume\") pod \"alertmanager-main-0\" (UID: \"a2de3f62-ec4e-49fc-8d7c-42c62f64dd94\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 02:19:17.297916 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:17.297628    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a2de3f62-ec4e-49fc-8d7c-42c62f64dd94-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"a2de3f62-ec4e-49fc-8d7c-42c62f64dd94\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 02:19:17.297916 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:17.297659    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a2de3f62-ec4e-49fc-8d7c-42c62f64dd94-tls-assets\") pod \"alertmanager-main-0\" (UID: \"a2de3f62-ec4e-49fc-8d7c-42c62f64dd94\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 02:19:17.297916 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:17.297799    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a2de3f62-ec4e-49fc-8d7c-42c62f64dd94-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"a2de3f62-ec4e-49fc-8d7c-42c62f64dd94\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 02:19:17.297916 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:17.297837    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a2de3f62-ec4e-49fc-8d7c-42c62f64dd94-config-out\") pod \"alertmanager-main-0\" (UID: \"a2de3f62-ec4e-49fc-8d7c-42c62f64dd94\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 02:19:17.297916 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:17.297866    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a2de3f62-ec4e-49fc-8d7c-42c62f64dd94-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"a2de3f62-ec4e-49fc-8d7c-42c62f64dd94\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 02:19:17.297916 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:17.297902    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a2de3f62-ec4e-49fc-8d7c-42c62f64dd94-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"a2de3f62-ec4e-49fc-8d7c-42c62f64dd94\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 02:19:17.298292 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:17.297928    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a2de3f62-ec4e-49fc-8d7c-42c62f64dd94-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"a2de3f62-ec4e-49fc-8d7c-42c62f64dd94\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 02:19:17.298520 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:17.298310    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a2de3f62-ec4e-49fc-8d7c-42c62f64dd94-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"a2de3f62-ec4e-49fc-8d7c-42c62f64dd94\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 02:19:17.298594 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:17.298429    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a2de3f62-ec4e-49fc-8d7c-42c62f64dd94-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"a2de3f62-ec4e-49fc-8d7c-42c62f64dd94\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 02:19:17.301067 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:17.300692    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a2de3f62-ec4e-49fc-8d7c-42c62f64dd94-web-config\") pod \"alertmanager-main-0\" (UID: \"a2de3f62-ec4e-49fc-8d7c-42c62f64dd94\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 02:19:17.301067 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:17.300761    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a2de3f62-ec4e-49fc-8d7c-42c62f64dd94-config-out\") pod \"alertmanager-main-0\" (UID: \"a2de3f62-ec4e-49fc-8d7c-42c62f64dd94\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 02:19:17.301067 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:17.300785    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a2de3f62-ec4e-49fc-8d7c-42c62f64dd94-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"a2de3f62-ec4e-49fc-8d7c-42c62f64dd94\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 02:19:17.301067 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:17.300796    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a2de3f62-ec4e-49fc-8d7c-42c62f64dd94-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"a2de3f62-ec4e-49fc-8d7c-42c62f64dd94\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 02:19:17.301067 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:17.300908    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a2de3f62-ec4e-49fc-8d7c-42c62f64dd94-tls-assets\") pod \"alertmanager-main-0\" (UID: \"a2de3f62-ec4e-49fc-8d7c-42c62f64dd94\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 02:19:17.301067 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:17.301028    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a2de3f62-ec4e-49fc-8d7c-42c62f64dd94-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"a2de3f62-ec4e-49fc-8d7c-42c62f64dd94\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 02:19:17.301422 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:17.301106    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a2de3f62-ec4e-49fc-8d7c-42c62f64dd94-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"a2de3f62-ec4e-49fc-8d7c-42c62f64dd94\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 02:19:17.301422 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:17.301189    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a2de3f62-ec4e-49fc-8d7c-42c62f64dd94-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"a2de3f62-ec4e-49fc-8d7c-42c62f64dd94\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 02:19:17.301422 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:17.301364    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a2de3f62-ec4e-49fc-8d7c-42c62f64dd94-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"a2de3f62-ec4e-49fc-8d7c-42c62f64dd94\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 02:19:17.302544 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:17.302526    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a2de3f62-ec4e-49fc-8d7c-42c62f64dd94-config-volume\") pod \"alertmanager-main-0\" (UID: \"a2de3f62-ec4e-49fc-8d7c-42c62f64dd94\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 02:19:17.304619 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:17.304603    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5pm5t\" (UniqueName: \"kubernetes.io/projected/a2de3f62-ec4e-49fc-8d7c-42c62f64dd94-kube-api-access-5pm5t\") pod \"alertmanager-main-0\" (UID: \"a2de3f62-ec4e-49fc-8d7c-42c62f64dd94\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 02:19:17.367894 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:17.367866    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 16 02:19:17.549196 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:17.549119    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1629aab1-29f9-42f7-92c9-435571df7299" path="/var/lib/kubelet/pods/1629aab1-29f9-42f7-92c9-435571df7299/volumes"
Sep 16 02:19:17.644832 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:17.644809    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 02:19:17.646940 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:19:17.646914    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2de3f62_ec4e_49fc_8d7c_42c62f64dd94.slice/crio-89a7dd52eb63368858221b702be308de327461508914f60c67ebe9f84981bd03 WatchSource:0}: Error finding container 89a7dd52eb63368858221b702be308de327461508914f60c67ebe9f84981bd03: Status 404 returned error can't find the container with id 89a7dd52eb63368858221b702be308de327461508914f60c67ebe9f84981bd03
Sep 16 02:19:18.006812 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:18.006773    2558 generic.go:356] "Generic (PLEG): container finished" podID="a2de3f62-ec4e-49fc-8d7c-42c62f64dd94" containerID="43c0b8921d9d8a36dba4fca31a1f60e0fe430de612357b7cdc30b26a14a2134e" exitCode=0
Sep 16 02:19:18.006951 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:18.006839    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a2de3f62-ec4e-49fc-8d7c-42c62f64dd94","Type":"ContainerDied","Data":"43c0b8921d9d8a36dba4fca31a1f60e0fe430de612357b7cdc30b26a14a2134e"}
Sep 16 02:19:18.006951 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:18.006905    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a2de3f62-ec4e-49fc-8d7c-42c62f64dd94","Type":"ContainerStarted","Data":"89a7dd52eb63368858221b702be308de327461508914f60c67ebe9f84981bd03"}
Sep 16 02:19:19.014290 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:19.014259    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a2de3f62-ec4e-49fc-8d7c-42c62f64dd94","Type":"ContainerStarted","Data":"cb3d1fd4a9d83557c40fb025793188de2e9b0fc5873d5d515d99ebeba5e4f0a1"}
Sep 16 02:19:19.014646 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:19.014322    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a2de3f62-ec4e-49fc-8d7c-42c62f64dd94","Type":"ContainerStarted","Data":"38817ad9f7e1a137c8d44e7e9d801d6d03592ff78de5dc9fedbac12556793ffa"}
Sep 16 02:19:19.014646 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:19.014349    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a2de3f62-ec4e-49fc-8d7c-42c62f64dd94","Type":"ContainerStarted","Data":"0eafc33af53d8c35a81212da99e18f03441c4e3676ba70ddd7a2815c7b77ce73"}
Sep 16 02:19:19.014646 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:19.014373    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a2de3f62-ec4e-49fc-8d7c-42c62f64dd94","Type":"ContainerStarted","Data":"a1a5e03c3552525a777db01c5caba6e5ef3657d0db17b0f347e8763348d2ceea"}
Sep 16 02:19:19.014646 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:19.014397    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a2de3f62-ec4e-49fc-8d7c-42c62f64dd94","Type":"ContainerStarted","Data":"d58f8eb1bdfadee63d6a7aa72fe737023e96979089e7cd092f0a745beaa051c7"}
Sep 16 02:19:19.014646 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:19.014418    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a2de3f62-ec4e-49fc-8d7c-42c62f64dd94","Type":"ContainerStarted","Data":"88829155cca489541997cc878a760b0b6c1ef2f6afee9022cda538eb05b6b3c5"}
Sep 16 02:19:19.039678 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:19.039613    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.025268628 podStartE2EDuration="2.039596149s" totalImagesPullingTime="0s" totalInitContainerRuntime="14.327521ms" isStatefulPod=false podCreationTimestamp="2026-09-16 02:19:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:19:19.038352674 +0000 UTC m=+371.956841923" watchObservedRunningTime="2026-09-16 02:19:19.039596149 +0000 UTC m=+371.958085398"
Sep 16 02:19:19.707209 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:19.707176    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 02:19:19.707502 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:19.707443    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="bbc9e62b-1357-4c44-a284-d3ac4a6c593c" containerName="prometheus" containerID="cri-o://d8fa4e821be4059e18f589bfbf6784413d70d416ac4f6cbd1ab52392dba26b0b" gracePeriod=600
Sep 16 02:19:19.707502 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:19.707486    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="bbc9e62b-1357-4c44-a284-d3ac4a6c593c" containerName="kube-rbac-proxy" containerID="cri-o://546c6b1fd362d5890ac34fc654937063b7b5077268a72766b9b53ec47d521ed6" gracePeriod=600
Sep 16 02:19:19.707715 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:19.707506    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="bbc9e62b-1357-4c44-a284-d3ac4a6c593c" containerName="config-reloader" containerID="cri-o://a5fc9315177131cb886db3f2985bbf14b089596e1caa7b2e12a9adcc70ca8721" gracePeriod=600
Sep 16 02:19:19.707715 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:19.707523    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="bbc9e62b-1357-4c44-a284-d3ac4a6c593c" containerName="kube-rbac-proxy-web" containerID="cri-o://9f763d7d753f7105f9d7272cf52d58cfc99913bad623a0bf8f80ef81dadcf1e4" gracePeriod=600
Sep 16 02:19:19.707715 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:19.707504    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="bbc9e62b-1357-4c44-a284-d3ac4a6c593c" containerName="thanos-sidecar" containerID="cri-o://848a9f5242e21b99c79c20be2c8a7de14d7000c19decafe0e4f64a8d7a8052b0" gracePeriod=600
Sep 16 02:19:19.707715 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:19.707533    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="bbc9e62b-1357-4c44-a284-d3ac4a6c593c" containerName="kube-rbac-proxy-thanos" containerID="cri-o://bd24b06a02919851a8a2508cac92481a82a98e36ed734f689330686729868bf4" gracePeriod=600
Sep 16 02:19:19.946895 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:19.946870    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:19:20.023233 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.023151    2558 generic.go:356] "Generic (PLEG): container finished" podID="bbc9e62b-1357-4c44-a284-d3ac4a6c593c" containerID="bd24b06a02919851a8a2508cac92481a82a98e36ed734f689330686729868bf4" exitCode=0
Sep 16 02:19:20.023233 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.023174    2558 generic.go:356] "Generic (PLEG): container finished" podID="bbc9e62b-1357-4c44-a284-d3ac4a6c593c" containerID="546c6b1fd362d5890ac34fc654937063b7b5077268a72766b9b53ec47d521ed6" exitCode=0
Sep 16 02:19:20.023233 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.023183    2558 generic.go:356] "Generic (PLEG): container finished" podID="bbc9e62b-1357-4c44-a284-d3ac4a6c593c" containerID="9f763d7d753f7105f9d7272cf52d58cfc99913bad623a0bf8f80ef81dadcf1e4" exitCode=0
Sep 16 02:19:20.023233 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.023189    2558 generic.go:356] "Generic (PLEG): container finished" podID="bbc9e62b-1357-4c44-a284-d3ac4a6c593c" containerID="848a9f5242e21b99c79c20be2c8a7de14d7000c19decafe0e4f64a8d7a8052b0" exitCode=0
Sep 16 02:19:20.023233 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.023195    2558 generic.go:356] "Generic (PLEG): container finished" podID="bbc9e62b-1357-4c44-a284-d3ac4a6c593c" containerID="a5fc9315177131cb886db3f2985bbf14b089596e1caa7b2e12a9adcc70ca8721" exitCode=0
Sep 16 02:19:20.023233 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.023200    2558 generic.go:356] "Generic (PLEG): container finished" podID="bbc9e62b-1357-4c44-a284-d3ac4a6c593c" containerID="d8fa4e821be4059e18f589bfbf6784413d70d416ac4f6cbd1ab52392dba26b0b" exitCode=0
Sep 16 02:19:20.023233 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.023229    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:19:20.023880 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.023232    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bbc9e62b-1357-4c44-a284-d3ac4a6c593c","Type":"ContainerDied","Data":"bd24b06a02919851a8a2508cac92481a82a98e36ed734f689330686729868bf4"}
Sep 16 02:19:20.023880 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.023319    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bbc9e62b-1357-4c44-a284-d3ac4a6c593c","Type":"ContainerDied","Data":"546c6b1fd362d5890ac34fc654937063b7b5077268a72766b9b53ec47d521ed6"}
Sep 16 02:19:20.023880 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.023363    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bbc9e62b-1357-4c44-a284-d3ac4a6c593c","Type":"ContainerDied","Data":"9f763d7d753f7105f9d7272cf52d58cfc99913bad623a0bf8f80ef81dadcf1e4"}
Sep 16 02:19:20.023880 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.023386    2558 scope.go:117] "RemoveContainer" containerID="bd24b06a02919851a8a2508cac92481a82a98e36ed734f689330686729868bf4"
Sep 16 02:19:20.023880 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.023406    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bbc9e62b-1357-4c44-a284-d3ac4a6c593c","Type":"ContainerDied","Data":"848a9f5242e21b99c79c20be2c8a7de14d7000c19decafe0e4f64a8d7a8052b0"}
Sep 16 02:19:20.023880 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.023448    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bbc9e62b-1357-4c44-a284-d3ac4a6c593c","Type":"ContainerDied","Data":"a5fc9315177131cb886db3f2985bbf14b089596e1caa7b2e12a9adcc70ca8721"}
Sep 16 02:19:20.023880 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.023488    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bbc9e62b-1357-4c44-a284-d3ac4a6c593c","Type":"ContainerDied","Data":"d8fa4e821be4059e18f589bfbf6784413d70d416ac4f6cbd1ab52392dba26b0b"}
Sep 16 02:19:20.023880 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.023530    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bbc9e62b-1357-4c44-a284-d3ac4a6c593c","Type":"ContainerDied","Data":"cf0d22a40df25f24d2879510e2967b0070c549f885417b0dcc60dcb4ac911ea5"}
Sep 16 02:19:20.024541 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.024525    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/bbc9e62b-1357-4c44-a284-d3ac4a6c593c-prometheus-k8s-db\") pod \"bbc9e62b-1357-4c44-a284-d3ac4a6c593c\" (UID: \"bbc9e62b-1357-4c44-a284-d3ac4a6c593c\") "
Sep 16 02:19:20.024589 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.024560    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/bbc9e62b-1357-4c44-a284-d3ac4a6c593c-thanos-prometheus-http-client-file\") pod \"bbc9e62b-1357-4c44-a284-d3ac4a6c593c\" (UID: \"bbc9e62b-1357-4c44-a284-d3ac4a6c593c\") "
Sep 16 02:19:20.024641 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.024590    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/bbc9e62b-1357-4c44-a284-d3ac4a6c593c-secret-metrics-client-certs\") pod \"bbc9e62b-1357-4c44-a284-d3ac4a6c593c\" (UID: \"bbc9e62b-1357-4c44-a284-d3ac4a6c593c\") "
Sep 16 02:19:20.024641 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.024619    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/bbc9e62b-1357-4c44-a284-d3ac4a6c593c-config\") pod \"bbc9e62b-1357-4c44-a284-d3ac4a6c593c\" (UID: \"bbc9e62b-1357-4c44-a284-d3ac4a6c593c\") "
Sep 16 02:19:20.024754 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.024642    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/bbc9e62b-1357-4c44-a284-d3ac4a6c593c-secret-prometheus-k8s-tls\") pod \"bbc9e62b-1357-4c44-a284-d3ac4a6c593c\" (UID: \"bbc9e62b-1357-4c44-a284-d3ac4a6c593c\") "
Sep 16 02:19:20.024754 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.024683    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/bbc9e62b-1357-4c44-a284-d3ac4a6c593c-prometheus-k8s-rulefiles-1\") pod \"bbc9e62b-1357-4c44-a284-d3ac4a6c593c\" (UID: \"bbc9e62b-1357-4c44-a284-d3ac4a6c593c\") "
Sep 16 02:19:20.025265 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.024896    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/bbc9e62b-1357-4c44-a284-d3ac4a6c593c-tls-assets\") pod \"bbc9e62b-1357-4c44-a284-d3ac4a6c593c\" (UID: \"bbc9e62b-1357-4c44-a284-d3ac4a6c593c\") "
Sep 16 02:19:20.025265 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.024931    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bbc9e62b-1357-4c44-a284-d3ac4a6c593c-configmap-kubelet-serving-ca-bundle\") pod \"bbc9e62b-1357-4c44-a284-d3ac4a6c593c\" (UID: \"bbc9e62b-1357-4c44-a284-d3ac4a6c593c\") "
Sep 16 02:19:20.025265 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.024966    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bbc9e62b-1357-4c44-a284-d3ac4a6c593c-configmap-serving-certs-ca-bundle\") pod \"bbc9e62b-1357-4c44-a284-d3ac4a6c593c\" (UID: \"bbc9e62b-1357-4c44-a284-d3ac4a6c593c\") "
Sep 16 02:19:20.025265 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.024993    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/bbc9e62b-1357-4c44-a284-d3ac4a6c593c-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"bbc9e62b-1357-4c44-a284-d3ac4a6c593c\" (UID: \"bbc9e62b-1357-4c44-a284-d3ac4a6c593c\") "
Sep 16 02:19:20.025265 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.025027    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/bbc9e62b-1357-4c44-a284-d3ac4a6c593c-web-config\") pod \"bbc9e62b-1357-4c44-a284-d3ac4a6c593c\" (UID: \"bbc9e62b-1357-4c44-a284-d3ac4a6c593c\") "
Sep 16 02:19:20.025265 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.025066    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bbc9e62b-1357-4c44-a284-d3ac4a6c593c-prometheus-trusted-ca-bundle\") pod \"bbc9e62b-1357-4c44-a284-d3ac4a6c593c\" (UID: \"bbc9e62b-1357-4c44-a284-d3ac4a6c593c\") "
Sep 16 02:19:20.025265 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.025098    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bbc9e62b-1357-4c44-a284-d3ac4a6c593c-configmap-metrics-client-ca\") pod \"bbc9e62b-1357-4c44-a284-d3ac4a6c593c\" (UID: \"bbc9e62b-1357-4c44-a284-d3ac4a6c593c\") "
Sep 16 02:19:20.025265 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.025127    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g2mzt\" (UniqueName: \"kubernetes.io/projected/bbc9e62b-1357-4c44-a284-d3ac4a6c593c-kube-api-access-g2mzt\") pod \"bbc9e62b-1357-4c44-a284-d3ac4a6c593c\" (UID: \"bbc9e62b-1357-4c44-a284-d3ac4a6c593c\") "
Sep 16 02:19:20.025265 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.025151    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/bbc9e62b-1357-4c44-a284-d3ac4a6c593c-secret-kube-rbac-proxy\") pod \"bbc9e62b-1357-4c44-a284-d3ac4a6c593c\" (UID: \"bbc9e62b-1357-4c44-a284-d3ac4a6c593c\") "
Sep 16 02:19:20.025265 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.025157    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbc9e62b-1357-4c44-a284-d3ac4a6c593c-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "bbc9e62b-1357-4c44-a284-d3ac4a6c593c" (UID: "bbc9e62b-1357-4c44-a284-d3ac4a6c593c"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 02:19:20.025265 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.025212    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/bbc9e62b-1357-4c44-a284-d3ac4a6c593c-prometheus-k8s-rulefiles-0\") pod \"bbc9e62b-1357-4c44-a284-d3ac4a6c593c\" (UID: \"bbc9e62b-1357-4c44-a284-d3ac4a6c593c\") "
Sep 16 02:19:20.025265 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.025246    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/bbc9e62b-1357-4c44-a284-d3ac4a6c593c-secret-grpc-tls\") pod \"bbc9e62b-1357-4c44-a284-d3ac4a6c593c\" (UID: \"bbc9e62b-1357-4c44-a284-d3ac4a6c593c\") "
Sep 16 02:19:20.025961 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.025276    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/bbc9e62b-1357-4c44-a284-d3ac4a6c593c-prometheus-k8s-rulefiles-2\") pod \"bbc9e62b-1357-4c44-a284-d3ac4a6c593c\" (UID: \"bbc9e62b-1357-4c44-a284-d3ac4a6c593c\") "
Sep 16 02:19:20.025961 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.025318    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/bbc9e62b-1357-4c44-a284-d3ac4a6c593c-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"bbc9e62b-1357-4c44-a284-d3ac4a6c593c\" (UID: \"bbc9e62b-1357-4c44-a284-d3ac4a6c593c\") "
Sep 16 02:19:20.025961 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.025347    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/bbc9e62b-1357-4c44-a284-d3ac4a6c593c-config-out\") pod \"bbc9e62b-1357-4c44-a284-d3ac4a6c593c\" (UID: \"bbc9e62b-1357-4c44-a284-d3ac4a6c593c\") "
Sep 16 02:19:20.025961 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.025582    2558 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/bbc9e62b-1357-4c44-a284-d3ac4a6c593c-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:19:20.026261 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.026227    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbc9e62b-1357-4c44-a284-d3ac4a6c593c-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "bbc9e62b-1357-4c44-a284-d3ac4a6c593c" (UID: "bbc9e62b-1357-4c44-a284-d3ac4a6c593c"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 02:19:20.026465 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.026438    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbc9e62b-1357-4c44-a284-d3ac4a6c593c-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "bbc9e62b-1357-4c44-a284-d3ac4a6c593c" (UID: "bbc9e62b-1357-4c44-a284-d3ac4a6c593c"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 02:19:20.026959 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.026599    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbc9e62b-1357-4c44-a284-d3ac4a6c593c-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "bbc9e62b-1357-4c44-a284-d3ac4a6c593c" (UID: "bbc9e62b-1357-4c44-a284-d3ac4a6c593c"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 02:19:20.027045 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.026598    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbc9e62b-1357-4c44-a284-d3ac4a6c593c-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "bbc9e62b-1357-4c44-a284-d3ac4a6c593c" (UID: "bbc9e62b-1357-4c44-a284-d3ac4a6c593c"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:19:20.027045 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.026923    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbc9e62b-1357-4c44-a284-d3ac4a6c593c-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "bbc9e62b-1357-4c44-a284-d3ac4a6c593c" (UID: "bbc9e62b-1357-4c44-a284-d3ac4a6c593c"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 02:19:20.027911 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.027884    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbc9e62b-1357-4c44-a284-d3ac4a6c593c-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "bbc9e62b-1357-4c44-a284-d3ac4a6c593c" (UID: "bbc9e62b-1357-4c44-a284-d3ac4a6c593c"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 02:19:20.028480 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.028449    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbc9e62b-1357-4c44-a284-d3ac4a6c593c-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "bbc9e62b-1357-4c44-a284-d3ac4a6c593c" (UID: "bbc9e62b-1357-4c44-a284-d3ac4a6c593c"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 02:19:20.030483 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.030449    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbc9e62b-1357-4c44-a284-d3ac4a6c593c-config-out" (OuterVolumeSpecName: "config-out") pod "bbc9e62b-1357-4c44-a284-d3ac4a6c593c" (UID: "bbc9e62b-1357-4c44-a284-d3ac4a6c593c"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:19:20.030603 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.030580    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bbc9e62b-1357-4c44-a284-d3ac4a6c593c-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "bbc9e62b-1357-4c44-a284-d3ac4a6c593c" (UID: "bbc9e62b-1357-4c44-a284-d3ac4a6c593c"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 02:19:20.030817 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.030670    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bbc9e62b-1357-4c44-a284-d3ac4a6c593c-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "bbc9e62b-1357-4c44-a284-d3ac4a6c593c" (UID: "bbc9e62b-1357-4c44-a284-d3ac4a6c593c"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 02:19:20.030817 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.030785    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bbc9e62b-1357-4c44-a284-d3ac4a6c593c-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "bbc9e62b-1357-4c44-a284-d3ac4a6c593c" (UID: "bbc9e62b-1357-4c44-a284-d3ac4a6c593c"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 02:19:20.031461 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.031427    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bbc9e62b-1357-4c44-a284-d3ac4a6c593c-config" (OuterVolumeSpecName: "config") pod "bbc9e62b-1357-4c44-a284-d3ac4a6c593c" (UID: "bbc9e62b-1357-4c44-a284-d3ac4a6c593c"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 02:19:20.031683 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.031650    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbc9e62b-1357-4c44-a284-d3ac4a6c593c-kube-api-access-g2mzt" (OuterVolumeSpecName: "kube-api-access-g2mzt") pod "bbc9e62b-1357-4c44-a284-d3ac4a6c593c" (UID: "bbc9e62b-1357-4c44-a284-d3ac4a6c593c"). InnerVolumeSpecName "kube-api-access-g2mzt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 02:19:20.031801 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.031719    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbc9e62b-1357-4c44-a284-d3ac4a6c593c-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "bbc9e62b-1357-4c44-a284-d3ac4a6c593c" (UID: "bbc9e62b-1357-4c44-a284-d3ac4a6c593c"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 02:19:20.031894 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.031870    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bbc9e62b-1357-4c44-a284-d3ac4a6c593c-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "bbc9e62b-1357-4c44-a284-d3ac4a6c593c" (UID: "bbc9e62b-1357-4c44-a284-d3ac4a6c593c"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 02:19:20.031941 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.031894    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bbc9e62b-1357-4c44-a284-d3ac4a6c593c-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "bbc9e62b-1357-4c44-a284-d3ac4a6c593c" (UID: "bbc9e62b-1357-4c44-a284-d3ac4a6c593c"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 02:19:20.032486 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.032462    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bbc9e62b-1357-4c44-a284-d3ac4a6c593c-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "bbc9e62b-1357-4c44-a284-d3ac4a6c593c" (UID: "bbc9e62b-1357-4c44-a284-d3ac4a6c593c"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 02:19:20.032646 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.032624    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bbc9e62b-1357-4c44-a284-d3ac4a6c593c-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "bbc9e62b-1357-4c44-a284-d3ac4a6c593c" (UID: "bbc9e62b-1357-4c44-a284-d3ac4a6c593c"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 02:19:20.033934 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.033917    2558 scope.go:117] "RemoveContainer" containerID="546c6b1fd362d5890ac34fc654937063b7b5077268a72766b9b53ec47d521ed6"
Sep 16 02:19:20.040190 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.040168    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bbc9e62b-1357-4c44-a284-d3ac4a6c593c-web-config" (OuterVolumeSpecName: "web-config") pod "bbc9e62b-1357-4c44-a284-d3ac4a6c593c" (UID: "bbc9e62b-1357-4c44-a284-d3ac4a6c593c"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 02:19:20.041205 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.041186    2558 scope.go:117] "RemoveContainer" containerID="9f763d7d753f7105f9d7272cf52d58cfc99913bad623a0bf8f80ef81dadcf1e4"
Sep 16 02:19:20.047918 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.047904    2558 scope.go:117] "RemoveContainer" containerID="848a9f5242e21b99c79c20be2c8a7de14d7000c19decafe0e4f64a8d7a8052b0"
Sep 16 02:19:20.054298 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.054276    2558 scope.go:117] "RemoveContainer" containerID="a5fc9315177131cb886db3f2985bbf14b089596e1caa7b2e12a9adcc70ca8721"
Sep 16 02:19:20.060574 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.060559    2558 scope.go:117] "RemoveContainer" containerID="d8fa4e821be4059e18f589bfbf6784413d70d416ac4f6cbd1ab52392dba26b0b"
Sep 16 02:19:20.067674 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.067657    2558 scope.go:117] "RemoveContainer" containerID="06d22256acb387ea4d461f898a63d96b601655840f439c08e019ae5c6d3eaf57"
Sep 16 02:19:20.074109 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.074088    2558 scope.go:117] "RemoveContainer" containerID="bd24b06a02919851a8a2508cac92481a82a98e36ed734f689330686729868bf4"
Sep 16 02:19:20.074387 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:19:20.074369    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd24b06a02919851a8a2508cac92481a82a98e36ed734f689330686729868bf4\": container with ID starting with bd24b06a02919851a8a2508cac92481a82a98e36ed734f689330686729868bf4 not found: ID does not exist" containerID="bd24b06a02919851a8a2508cac92481a82a98e36ed734f689330686729868bf4"
Sep 16 02:19:20.074449 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.074397    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd24b06a02919851a8a2508cac92481a82a98e36ed734f689330686729868bf4"} err="failed to get container status \"bd24b06a02919851a8a2508cac92481a82a98e36ed734f689330686729868bf4\": rpc error: code = NotFound desc = could not find container \"bd24b06a02919851a8a2508cac92481a82a98e36ed734f689330686729868bf4\": container with ID starting with bd24b06a02919851a8a2508cac92481a82a98e36ed734f689330686729868bf4 not found: ID does not exist"
Sep 16 02:19:20.074449 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.074417    2558 scope.go:117] "RemoveContainer" containerID="546c6b1fd362d5890ac34fc654937063b7b5077268a72766b9b53ec47d521ed6"
Sep 16 02:19:20.074643 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:19:20.074627    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"546c6b1fd362d5890ac34fc654937063b7b5077268a72766b9b53ec47d521ed6\": container with ID starting with 546c6b1fd362d5890ac34fc654937063b7b5077268a72766b9b53ec47d521ed6 not found: ID does not exist" containerID="546c6b1fd362d5890ac34fc654937063b7b5077268a72766b9b53ec47d521ed6"
Sep 16 02:19:20.074682 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.074647    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"546c6b1fd362d5890ac34fc654937063b7b5077268a72766b9b53ec47d521ed6"} err="failed to get container status \"546c6b1fd362d5890ac34fc654937063b7b5077268a72766b9b53ec47d521ed6\": rpc error: code = NotFound desc = could not find container \"546c6b1fd362d5890ac34fc654937063b7b5077268a72766b9b53ec47d521ed6\": container with ID starting with 546c6b1fd362d5890ac34fc654937063b7b5077268a72766b9b53ec47d521ed6 not found: ID does not exist"
Sep 16 02:19:20.074682 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.074661    2558 scope.go:117] "RemoveContainer" containerID="9f763d7d753f7105f9d7272cf52d58cfc99913bad623a0bf8f80ef81dadcf1e4"
Sep 16 02:19:20.074901 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:19:20.074887    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f763d7d753f7105f9d7272cf52d58cfc99913bad623a0bf8f80ef81dadcf1e4\": container with ID starting with 9f763d7d753f7105f9d7272cf52d58cfc99913bad623a0bf8f80ef81dadcf1e4 not found: ID does not exist" containerID="9f763d7d753f7105f9d7272cf52d58cfc99913bad623a0bf8f80ef81dadcf1e4"
Sep 16 02:19:20.074942 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.074906    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f763d7d753f7105f9d7272cf52d58cfc99913bad623a0bf8f80ef81dadcf1e4"} err="failed to get container status \"9f763d7d753f7105f9d7272cf52d58cfc99913bad623a0bf8f80ef81dadcf1e4\": rpc error: code = NotFound desc = could not find container \"9f763d7d753f7105f9d7272cf52d58cfc99913bad623a0bf8f80ef81dadcf1e4\": container with ID starting with 9f763d7d753f7105f9d7272cf52d58cfc99913bad623a0bf8f80ef81dadcf1e4 not found: ID does not exist"
Sep 16 02:19:20.074942 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.074919    2558 scope.go:117] "RemoveContainer" containerID="848a9f5242e21b99c79c20be2c8a7de14d7000c19decafe0e4f64a8d7a8052b0"
Sep 16 02:19:20.075128 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:19:20.075115    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"848a9f5242e21b99c79c20be2c8a7de14d7000c19decafe0e4f64a8d7a8052b0\": container with ID starting with 848a9f5242e21b99c79c20be2c8a7de14d7000c19decafe0e4f64a8d7a8052b0 not found: ID does not exist" containerID="848a9f5242e21b99c79c20be2c8a7de14d7000c19decafe0e4f64a8d7a8052b0"
Sep 16 02:19:20.075170 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.075129    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"848a9f5242e21b99c79c20be2c8a7de14d7000c19decafe0e4f64a8d7a8052b0"} err="failed to get container status \"848a9f5242e21b99c79c20be2c8a7de14d7000c19decafe0e4f64a8d7a8052b0\": rpc error: code = NotFound desc = could not find container \"848a9f5242e21b99c79c20be2c8a7de14d7000c19decafe0e4f64a8d7a8052b0\": container with ID starting with 848a9f5242e21b99c79c20be2c8a7de14d7000c19decafe0e4f64a8d7a8052b0 not found: ID does not exist"
Sep 16 02:19:20.075170 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.075139    2558 scope.go:117] "RemoveContainer" containerID="a5fc9315177131cb886db3f2985bbf14b089596e1caa7b2e12a9adcc70ca8721"
Sep 16 02:19:20.075341 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:19:20.075323    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5fc9315177131cb886db3f2985bbf14b089596e1caa7b2e12a9adcc70ca8721\": container with ID starting with a5fc9315177131cb886db3f2985bbf14b089596e1caa7b2e12a9adcc70ca8721 not found: ID does not exist" containerID="a5fc9315177131cb886db3f2985bbf14b089596e1caa7b2e12a9adcc70ca8721"
Sep 16 02:19:20.075381 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.075347    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5fc9315177131cb886db3f2985bbf14b089596e1caa7b2e12a9adcc70ca8721"} err="failed to get container status \"a5fc9315177131cb886db3f2985bbf14b089596e1caa7b2e12a9adcc70ca8721\": rpc error: code = NotFound desc = could not find container \"a5fc9315177131cb886db3f2985bbf14b089596e1caa7b2e12a9adcc70ca8721\": container with ID starting with a5fc9315177131cb886db3f2985bbf14b089596e1caa7b2e12a9adcc70ca8721 not found: ID does not exist"
Sep 16 02:19:20.075381 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.075364    2558 scope.go:117] "RemoveContainer" containerID="d8fa4e821be4059e18f589bfbf6784413d70d416ac4f6cbd1ab52392dba26b0b"
Sep 16 02:19:20.075573 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:19:20.075557    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8fa4e821be4059e18f589bfbf6784413d70d416ac4f6cbd1ab52392dba26b0b\": container with ID starting with d8fa4e821be4059e18f589bfbf6784413d70d416ac4f6cbd1ab52392dba26b0b not found: ID does not exist" containerID="d8fa4e821be4059e18f589bfbf6784413d70d416ac4f6cbd1ab52392dba26b0b"
Sep 16 02:19:20.075629 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.075581    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8fa4e821be4059e18f589bfbf6784413d70d416ac4f6cbd1ab52392dba26b0b"} err="failed to get container status \"d8fa4e821be4059e18f589bfbf6784413d70d416ac4f6cbd1ab52392dba26b0b\": rpc error: code = NotFound desc = could not find container \"d8fa4e821be4059e18f589bfbf6784413d70d416ac4f6cbd1ab52392dba26b0b\": container with ID starting with d8fa4e821be4059e18f589bfbf6784413d70d416ac4f6cbd1ab52392dba26b0b not found: ID does not exist"
Sep 16 02:19:20.075629 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.075603    2558 scope.go:117] "RemoveContainer" containerID="06d22256acb387ea4d461f898a63d96b601655840f439c08e019ae5c6d3eaf57"
Sep 16 02:19:20.075897 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:19:20.075883    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06d22256acb387ea4d461f898a63d96b601655840f439c08e019ae5c6d3eaf57\": container with ID starting with 06d22256acb387ea4d461f898a63d96b601655840f439c08e019ae5c6d3eaf57 not found: ID does not exist" containerID="06d22256acb387ea4d461f898a63d96b601655840f439c08e019ae5c6d3eaf57"
Sep 16 02:19:20.075940 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.075899    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06d22256acb387ea4d461f898a63d96b601655840f439c08e019ae5c6d3eaf57"} err="failed to get container status \"06d22256acb387ea4d461f898a63d96b601655840f439c08e019ae5c6d3eaf57\": rpc error: code = NotFound desc = could not find container \"06d22256acb387ea4d461f898a63d96b601655840f439c08e019ae5c6d3eaf57\": container with ID starting with 06d22256acb387ea4d461f898a63d96b601655840f439c08e019ae5c6d3eaf57 not found: ID does not exist"
Sep 16 02:19:20.075940 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.075911    2558 scope.go:117] "RemoveContainer" containerID="bd24b06a02919851a8a2508cac92481a82a98e36ed734f689330686729868bf4"
Sep 16 02:19:20.076139 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.076120    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd24b06a02919851a8a2508cac92481a82a98e36ed734f689330686729868bf4"} err="failed to get container status \"bd24b06a02919851a8a2508cac92481a82a98e36ed734f689330686729868bf4\": rpc error: code = NotFound desc = could not find container \"bd24b06a02919851a8a2508cac92481a82a98e36ed734f689330686729868bf4\": container with ID starting with bd24b06a02919851a8a2508cac92481a82a98e36ed734f689330686729868bf4 not found: ID does not exist"
Sep 16 02:19:20.076187 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.076140    2558 scope.go:117] "RemoveContainer" containerID="546c6b1fd362d5890ac34fc654937063b7b5077268a72766b9b53ec47d521ed6"
Sep 16 02:19:20.076343 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.076328    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"546c6b1fd362d5890ac34fc654937063b7b5077268a72766b9b53ec47d521ed6"} err="failed to get container status \"546c6b1fd362d5890ac34fc654937063b7b5077268a72766b9b53ec47d521ed6\": rpc error: code = NotFound desc = could not find container \"546c6b1fd362d5890ac34fc654937063b7b5077268a72766b9b53ec47d521ed6\": container with ID starting with 546c6b1fd362d5890ac34fc654937063b7b5077268a72766b9b53ec47d521ed6 not found: ID does not exist"
Sep 16 02:19:20.076387 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.076343    2558 scope.go:117] "RemoveContainer" containerID="9f763d7d753f7105f9d7272cf52d58cfc99913bad623a0bf8f80ef81dadcf1e4"
Sep 16 02:19:20.076523 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.076507    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f763d7d753f7105f9d7272cf52d58cfc99913bad623a0bf8f80ef81dadcf1e4"} err="failed to get container status \"9f763d7d753f7105f9d7272cf52d58cfc99913bad623a0bf8f80ef81dadcf1e4\": rpc error: code = NotFound desc = could not find container \"9f763d7d753f7105f9d7272cf52d58cfc99913bad623a0bf8f80ef81dadcf1e4\": container with ID starting with 9f763d7d753f7105f9d7272cf52d58cfc99913bad623a0bf8f80ef81dadcf1e4 not found: ID does not exist"
Sep 16 02:19:20.076564 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.076523    2558 scope.go:117] "RemoveContainer" containerID="848a9f5242e21b99c79c20be2c8a7de14d7000c19decafe0e4f64a8d7a8052b0"
Sep 16 02:19:20.076763 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.076742    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"848a9f5242e21b99c79c20be2c8a7de14d7000c19decafe0e4f64a8d7a8052b0"} err="failed to get container status \"848a9f5242e21b99c79c20be2c8a7de14d7000c19decafe0e4f64a8d7a8052b0\": rpc error: code = NotFound desc = could not find container \"848a9f5242e21b99c79c20be2c8a7de14d7000c19decafe0e4f64a8d7a8052b0\": container with ID starting with 848a9f5242e21b99c79c20be2c8a7de14d7000c19decafe0e4f64a8d7a8052b0 not found: ID does not exist"
Sep 16 02:19:20.076763 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.076762    2558 scope.go:117] "RemoveContainer" containerID="a5fc9315177131cb886db3f2985bbf14b089596e1caa7b2e12a9adcc70ca8721"
Sep 16 02:19:20.077042 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.077024    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5fc9315177131cb886db3f2985bbf14b089596e1caa7b2e12a9adcc70ca8721"} err="failed to get container status \"a5fc9315177131cb886db3f2985bbf14b089596e1caa7b2e12a9adcc70ca8721\": rpc error: code = NotFound desc = could not find container \"a5fc9315177131cb886db3f2985bbf14b089596e1caa7b2e12a9adcc70ca8721\": container with ID starting with a5fc9315177131cb886db3f2985bbf14b089596e1caa7b2e12a9adcc70ca8721 not found: ID does not exist"
Sep 16 02:19:20.077086 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.077053    2558 scope.go:117] "RemoveContainer" containerID="d8fa4e821be4059e18f589bfbf6784413d70d416ac4f6cbd1ab52392dba26b0b"
Sep 16 02:19:20.077306 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.077289    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8fa4e821be4059e18f589bfbf6784413d70d416ac4f6cbd1ab52392dba26b0b"} err="failed to get container status \"d8fa4e821be4059e18f589bfbf6784413d70d416ac4f6cbd1ab52392dba26b0b\": rpc error: code = NotFound desc = could not find container \"d8fa4e821be4059e18f589bfbf6784413d70d416ac4f6cbd1ab52392dba26b0b\": container with ID starting with d8fa4e821be4059e18f589bfbf6784413d70d416ac4f6cbd1ab52392dba26b0b not found: ID does not exist"
Sep 16 02:19:20.077306 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.077306    2558 scope.go:117] "RemoveContainer" containerID="06d22256acb387ea4d461f898a63d96b601655840f439c08e019ae5c6d3eaf57"
Sep 16 02:19:20.077538 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.077519    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06d22256acb387ea4d461f898a63d96b601655840f439c08e019ae5c6d3eaf57"} err="failed to get container status \"06d22256acb387ea4d461f898a63d96b601655840f439c08e019ae5c6d3eaf57\": rpc error: code = NotFound desc = could not find container \"06d22256acb387ea4d461f898a63d96b601655840f439c08e019ae5c6d3eaf57\": container with ID starting with 06d22256acb387ea4d461f898a63d96b601655840f439c08e019ae5c6d3eaf57 not found: ID does not exist"
Sep 16 02:19:20.077579 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.077539    2558 scope.go:117] "RemoveContainer" containerID="bd24b06a02919851a8a2508cac92481a82a98e36ed734f689330686729868bf4"
Sep 16 02:19:20.077759 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.077744    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd24b06a02919851a8a2508cac92481a82a98e36ed734f689330686729868bf4"} err="failed to get container status \"bd24b06a02919851a8a2508cac92481a82a98e36ed734f689330686729868bf4\": rpc error: code = NotFound desc = could not find container \"bd24b06a02919851a8a2508cac92481a82a98e36ed734f689330686729868bf4\": container with ID starting with bd24b06a02919851a8a2508cac92481a82a98e36ed734f689330686729868bf4 not found: ID does not exist"
Sep 16 02:19:20.077797 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.077758    2558 scope.go:117] "RemoveContainer" containerID="546c6b1fd362d5890ac34fc654937063b7b5077268a72766b9b53ec47d521ed6"
Sep 16 02:19:20.077949 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.077934    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"546c6b1fd362d5890ac34fc654937063b7b5077268a72766b9b53ec47d521ed6"} err="failed to get container status \"546c6b1fd362d5890ac34fc654937063b7b5077268a72766b9b53ec47d521ed6\": rpc error: code = NotFound desc = could not find container \"546c6b1fd362d5890ac34fc654937063b7b5077268a72766b9b53ec47d521ed6\": container with ID starting with 546c6b1fd362d5890ac34fc654937063b7b5077268a72766b9b53ec47d521ed6 not found: ID does not exist"
Sep 16 02:19:20.077987 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.077949    2558 scope.go:117] "RemoveContainer" containerID="9f763d7d753f7105f9d7272cf52d58cfc99913bad623a0bf8f80ef81dadcf1e4"
Sep 16 02:19:20.078128 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.078112    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f763d7d753f7105f9d7272cf52d58cfc99913bad623a0bf8f80ef81dadcf1e4"} err="failed to get container status \"9f763d7d753f7105f9d7272cf52d58cfc99913bad623a0bf8f80ef81dadcf1e4\": rpc error: code = NotFound desc = could not find container \"9f763d7d753f7105f9d7272cf52d58cfc99913bad623a0bf8f80ef81dadcf1e4\": container with ID starting with 9f763d7d753f7105f9d7272cf52d58cfc99913bad623a0bf8f80ef81dadcf1e4 not found: ID does not exist"
Sep 16 02:19:20.078174 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.078128    2558 scope.go:117] "RemoveContainer" containerID="848a9f5242e21b99c79c20be2c8a7de14d7000c19decafe0e4f64a8d7a8052b0"
Sep 16 02:19:20.078334 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.078316    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"848a9f5242e21b99c79c20be2c8a7de14d7000c19decafe0e4f64a8d7a8052b0"} err="failed to get container status \"848a9f5242e21b99c79c20be2c8a7de14d7000c19decafe0e4f64a8d7a8052b0\": rpc error: code = NotFound desc = could not find container \"848a9f5242e21b99c79c20be2c8a7de14d7000c19decafe0e4f64a8d7a8052b0\": container with ID starting with 848a9f5242e21b99c79c20be2c8a7de14d7000c19decafe0e4f64a8d7a8052b0 not found: ID does not exist"
Sep 16 02:19:20.078371 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.078334    2558 scope.go:117] "RemoveContainer" containerID="a5fc9315177131cb886db3f2985bbf14b089596e1caa7b2e12a9adcc70ca8721"
Sep 16 02:19:20.078511 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.078495    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5fc9315177131cb886db3f2985bbf14b089596e1caa7b2e12a9adcc70ca8721"} err="failed to get container status \"a5fc9315177131cb886db3f2985bbf14b089596e1caa7b2e12a9adcc70ca8721\": rpc error: code = NotFound desc = could not find container \"a5fc9315177131cb886db3f2985bbf14b089596e1caa7b2e12a9adcc70ca8721\": container with ID starting with a5fc9315177131cb886db3f2985bbf14b089596e1caa7b2e12a9adcc70ca8721 not found: ID does not exist"
Sep 16 02:19:20.078511 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.078510    2558 scope.go:117] "RemoveContainer" containerID="d8fa4e821be4059e18f589bfbf6784413d70d416ac4f6cbd1ab52392dba26b0b"
Sep 16 02:19:20.078692 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.078677    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8fa4e821be4059e18f589bfbf6784413d70d416ac4f6cbd1ab52392dba26b0b"} err="failed to get container status \"d8fa4e821be4059e18f589bfbf6784413d70d416ac4f6cbd1ab52392dba26b0b\": rpc error: code = NotFound desc = could not find container \"d8fa4e821be4059e18f589bfbf6784413d70d416ac4f6cbd1ab52392dba26b0b\": container with ID starting with d8fa4e821be4059e18f589bfbf6784413d70d416ac4f6cbd1ab52392dba26b0b not found: ID does not exist"
Sep 16 02:19:20.078742 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.078692    2558 scope.go:117] "RemoveContainer" containerID="06d22256acb387ea4d461f898a63d96b601655840f439c08e019ae5c6d3eaf57"
Sep 16 02:19:20.078900 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.078885    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06d22256acb387ea4d461f898a63d96b601655840f439c08e019ae5c6d3eaf57"} err="failed to get container status \"06d22256acb387ea4d461f898a63d96b601655840f439c08e019ae5c6d3eaf57\": rpc error: code = NotFound desc = could not find container \"06d22256acb387ea4d461f898a63d96b601655840f439c08e019ae5c6d3eaf57\": container with ID starting with 06d22256acb387ea4d461f898a63d96b601655840f439c08e019ae5c6d3eaf57 not found: ID does not exist"
Sep 16 02:19:20.078944 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.078900    2558 scope.go:117] "RemoveContainer" containerID="bd24b06a02919851a8a2508cac92481a82a98e36ed734f689330686729868bf4"
Sep 16 02:19:20.079087 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.079071    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd24b06a02919851a8a2508cac92481a82a98e36ed734f689330686729868bf4"} err="failed to get container status \"bd24b06a02919851a8a2508cac92481a82a98e36ed734f689330686729868bf4\": rpc error: code = NotFound desc = could not find container \"bd24b06a02919851a8a2508cac92481a82a98e36ed734f689330686729868bf4\": container with ID starting with bd24b06a02919851a8a2508cac92481a82a98e36ed734f689330686729868bf4 not found: ID does not exist"
Sep 16 02:19:20.079126 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.079088    2558 scope.go:117] "RemoveContainer" containerID="546c6b1fd362d5890ac34fc654937063b7b5077268a72766b9b53ec47d521ed6"
Sep 16 02:19:20.079264 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.079249    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"546c6b1fd362d5890ac34fc654937063b7b5077268a72766b9b53ec47d521ed6"} err="failed to get container status \"546c6b1fd362d5890ac34fc654937063b7b5077268a72766b9b53ec47d521ed6\": rpc error: code = NotFound desc = could not find container \"546c6b1fd362d5890ac34fc654937063b7b5077268a72766b9b53ec47d521ed6\": container with ID starting with 546c6b1fd362d5890ac34fc654937063b7b5077268a72766b9b53ec47d521ed6 not found: ID does not exist"
Sep 16 02:19:20.079301 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.079264    2558 scope.go:117] "RemoveContainer" containerID="9f763d7d753f7105f9d7272cf52d58cfc99913bad623a0bf8f80ef81dadcf1e4"
Sep 16 02:19:20.079477 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.079459    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f763d7d753f7105f9d7272cf52d58cfc99913bad623a0bf8f80ef81dadcf1e4"} err="failed to get container status \"9f763d7d753f7105f9d7272cf52d58cfc99913bad623a0bf8f80ef81dadcf1e4\": rpc error: code = NotFound desc = could not find container \"9f763d7d753f7105f9d7272cf52d58cfc99913bad623a0bf8f80ef81dadcf1e4\": container with ID starting with 9f763d7d753f7105f9d7272cf52d58cfc99913bad623a0bf8f80ef81dadcf1e4 not found: ID does not exist"
Sep 16 02:19:20.079519 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.079478    2558 scope.go:117] "RemoveContainer" containerID="848a9f5242e21b99c79c20be2c8a7de14d7000c19decafe0e4f64a8d7a8052b0"
Sep 16 02:19:20.079689 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.079671    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"848a9f5242e21b99c79c20be2c8a7de14d7000c19decafe0e4f64a8d7a8052b0"} err="failed to get container status \"848a9f5242e21b99c79c20be2c8a7de14d7000c19decafe0e4f64a8d7a8052b0\": rpc error: code = NotFound desc = could not find container \"848a9f5242e21b99c79c20be2c8a7de14d7000c19decafe0e4f64a8d7a8052b0\": container with ID starting with 848a9f5242e21b99c79c20be2c8a7de14d7000c19decafe0e4f64a8d7a8052b0 not found: ID does not exist"
Sep 16 02:19:20.079777 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.079690    2558 scope.go:117] "RemoveContainer" containerID="a5fc9315177131cb886db3f2985bbf14b089596e1caa7b2e12a9adcc70ca8721"
Sep 16 02:19:20.079931 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.079912    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5fc9315177131cb886db3f2985bbf14b089596e1caa7b2e12a9adcc70ca8721"} err="failed to get container status \"a5fc9315177131cb886db3f2985bbf14b089596e1caa7b2e12a9adcc70ca8721\": rpc error: code = NotFound desc = could not find container \"a5fc9315177131cb886db3f2985bbf14b089596e1caa7b2e12a9adcc70ca8721\": container with ID starting with a5fc9315177131cb886db3f2985bbf14b089596e1caa7b2e12a9adcc70ca8721 not found: ID does not exist"
Sep 16 02:19:20.079977 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.079930    2558 scope.go:117] "RemoveContainer" containerID="d8fa4e821be4059e18f589bfbf6784413d70d416ac4f6cbd1ab52392dba26b0b"
Sep 16 02:19:20.080135 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.080118    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8fa4e821be4059e18f589bfbf6784413d70d416ac4f6cbd1ab52392dba26b0b"} err="failed to get container status \"d8fa4e821be4059e18f589bfbf6784413d70d416ac4f6cbd1ab52392dba26b0b\": rpc error: code = NotFound desc = could not find container \"d8fa4e821be4059e18f589bfbf6784413d70d416ac4f6cbd1ab52392dba26b0b\": container with ID starting with d8fa4e821be4059e18f589bfbf6784413d70d416ac4f6cbd1ab52392dba26b0b not found: ID does not exist"
Sep 16 02:19:20.080192 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.080135    2558 scope.go:117] "RemoveContainer" containerID="06d22256acb387ea4d461f898a63d96b601655840f439c08e019ae5c6d3eaf57"
Sep 16 02:19:20.080335 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.080317    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06d22256acb387ea4d461f898a63d96b601655840f439c08e019ae5c6d3eaf57"} err="failed to get container status \"06d22256acb387ea4d461f898a63d96b601655840f439c08e019ae5c6d3eaf57\": rpc error: code = NotFound desc = could not find container \"06d22256acb387ea4d461f898a63d96b601655840f439c08e019ae5c6d3eaf57\": container with ID starting with 06d22256acb387ea4d461f898a63d96b601655840f439c08e019ae5c6d3eaf57 not found: ID does not exist"
Sep 16 02:19:20.080386 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.080337    2558 scope.go:117] "RemoveContainer" containerID="bd24b06a02919851a8a2508cac92481a82a98e36ed734f689330686729868bf4"
Sep 16 02:19:20.080551 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.080533    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd24b06a02919851a8a2508cac92481a82a98e36ed734f689330686729868bf4"} err="failed to get container status \"bd24b06a02919851a8a2508cac92481a82a98e36ed734f689330686729868bf4\": rpc error: code = NotFound desc = could not find container \"bd24b06a02919851a8a2508cac92481a82a98e36ed734f689330686729868bf4\": container with ID starting with bd24b06a02919851a8a2508cac92481a82a98e36ed734f689330686729868bf4 not found: ID does not exist"
Sep 16 02:19:20.080610 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.080552    2558 scope.go:117] "RemoveContainer" containerID="546c6b1fd362d5890ac34fc654937063b7b5077268a72766b9b53ec47d521ed6"
Sep 16 02:19:20.080813 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.080774    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"546c6b1fd362d5890ac34fc654937063b7b5077268a72766b9b53ec47d521ed6"} err="failed to get container status \"546c6b1fd362d5890ac34fc654937063b7b5077268a72766b9b53ec47d521ed6\": rpc error: code = NotFound desc = could not find container \"546c6b1fd362d5890ac34fc654937063b7b5077268a72766b9b53ec47d521ed6\": container with ID starting with 546c6b1fd362d5890ac34fc654937063b7b5077268a72766b9b53ec47d521ed6 not found: ID does not exist"
Sep 16 02:19:20.080813 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.080811    2558 scope.go:117] "RemoveContainer" containerID="9f763d7d753f7105f9d7272cf52d58cfc99913bad623a0bf8f80ef81dadcf1e4"
Sep 16 02:19:20.081061 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.081044    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f763d7d753f7105f9d7272cf52d58cfc99913bad623a0bf8f80ef81dadcf1e4"} err="failed to get container status \"9f763d7d753f7105f9d7272cf52d58cfc99913bad623a0bf8f80ef81dadcf1e4\": rpc error: code = NotFound desc = could not find container \"9f763d7d753f7105f9d7272cf52d58cfc99913bad623a0bf8f80ef81dadcf1e4\": container with ID starting with 9f763d7d753f7105f9d7272cf52d58cfc99913bad623a0bf8f80ef81dadcf1e4 not found: ID does not exist"
Sep 16 02:19:20.081109 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.081062    2558 scope.go:117] "RemoveContainer" containerID="848a9f5242e21b99c79c20be2c8a7de14d7000c19decafe0e4f64a8d7a8052b0"
Sep 16 02:19:20.081295 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.081276    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"848a9f5242e21b99c79c20be2c8a7de14d7000c19decafe0e4f64a8d7a8052b0"} err="failed to get container status \"848a9f5242e21b99c79c20be2c8a7de14d7000c19decafe0e4f64a8d7a8052b0\": rpc error: code = NotFound desc = could not find container \"848a9f5242e21b99c79c20be2c8a7de14d7000c19decafe0e4f64a8d7a8052b0\": container with ID starting with 848a9f5242e21b99c79c20be2c8a7de14d7000c19decafe0e4f64a8d7a8052b0 not found: ID does not exist"
Sep 16 02:19:20.081363 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.081296    2558 scope.go:117] "RemoveContainer" containerID="a5fc9315177131cb886db3f2985bbf14b089596e1caa7b2e12a9adcc70ca8721"
Sep 16 02:19:20.081521 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.081502    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5fc9315177131cb886db3f2985bbf14b089596e1caa7b2e12a9adcc70ca8721"} err="failed to get container status \"a5fc9315177131cb886db3f2985bbf14b089596e1caa7b2e12a9adcc70ca8721\": rpc error: code = NotFound desc = could not find container \"a5fc9315177131cb886db3f2985bbf14b089596e1caa7b2e12a9adcc70ca8721\": container with ID starting with a5fc9315177131cb886db3f2985bbf14b089596e1caa7b2e12a9adcc70ca8721 not found: ID does not exist"
Sep 16 02:19:20.081569 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.081522    2558 scope.go:117] "RemoveContainer" containerID="d8fa4e821be4059e18f589bfbf6784413d70d416ac4f6cbd1ab52392dba26b0b"
Sep 16 02:19:20.081739 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.081722    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8fa4e821be4059e18f589bfbf6784413d70d416ac4f6cbd1ab52392dba26b0b"} err="failed to get container status \"d8fa4e821be4059e18f589bfbf6784413d70d416ac4f6cbd1ab52392dba26b0b\": rpc error: code = NotFound desc = could not find container \"d8fa4e821be4059e18f589bfbf6784413d70d416ac4f6cbd1ab52392dba26b0b\": container with ID starting with d8fa4e821be4059e18f589bfbf6784413d70d416ac4f6cbd1ab52392dba26b0b not found: ID does not exist"
Sep 16 02:19:20.081809 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.081740    2558 scope.go:117] "RemoveContainer" containerID="06d22256acb387ea4d461f898a63d96b601655840f439c08e019ae5c6d3eaf57"
Sep 16 02:19:20.081990 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.081973    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06d22256acb387ea4d461f898a63d96b601655840f439c08e019ae5c6d3eaf57"} err="failed to get container status \"06d22256acb387ea4d461f898a63d96b601655840f439c08e019ae5c6d3eaf57\": rpc error: code = NotFound desc = could not find container \"06d22256acb387ea4d461f898a63d96b601655840f439c08e019ae5c6d3eaf57\": container with ID starting with 06d22256acb387ea4d461f898a63d96b601655840f439c08e019ae5c6d3eaf57 not found: ID does not exist"
Sep 16 02:19:20.082029 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.081991    2558 scope.go:117] "RemoveContainer" containerID="bd24b06a02919851a8a2508cac92481a82a98e36ed734f689330686729868bf4"
Sep 16 02:19:20.082215 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.082189    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd24b06a02919851a8a2508cac92481a82a98e36ed734f689330686729868bf4"} err="failed to get container status \"bd24b06a02919851a8a2508cac92481a82a98e36ed734f689330686729868bf4\": rpc error: code = NotFound desc = could not find container \"bd24b06a02919851a8a2508cac92481a82a98e36ed734f689330686729868bf4\": container with ID starting with bd24b06a02919851a8a2508cac92481a82a98e36ed734f689330686729868bf4 not found: ID does not exist"
Sep 16 02:19:20.082285 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.082216    2558 scope.go:117] "RemoveContainer" containerID="546c6b1fd362d5890ac34fc654937063b7b5077268a72766b9b53ec47d521ed6"
Sep 16 02:19:20.082426 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.082409    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"546c6b1fd362d5890ac34fc654937063b7b5077268a72766b9b53ec47d521ed6"} err="failed to get container status \"546c6b1fd362d5890ac34fc654937063b7b5077268a72766b9b53ec47d521ed6\": rpc error: code = NotFound desc = could not find container \"546c6b1fd362d5890ac34fc654937063b7b5077268a72766b9b53ec47d521ed6\": container with ID starting with 546c6b1fd362d5890ac34fc654937063b7b5077268a72766b9b53ec47d521ed6 not found: ID does not exist"
Sep 16 02:19:20.082482 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.082427    2558 scope.go:117] "RemoveContainer" containerID="9f763d7d753f7105f9d7272cf52d58cfc99913bad623a0bf8f80ef81dadcf1e4"
Sep 16 02:19:20.082652 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.082634    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f763d7d753f7105f9d7272cf52d58cfc99913bad623a0bf8f80ef81dadcf1e4"} err="failed to get container status \"9f763d7d753f7105f9d7272cf52d58cfc99913bad623a0bf8f80ef81dadcf1e4\": rpc error: code = NotFound desc = could not find container \"9f763d7d753f7105f9d7272cf52d58cfc99913bad623a0bf8f80ef81dadcf1e4\": container with ID starting with 9f763d7d753f7105f9d7272cf52d58cfc99913bad623a0bf8f80ef81dadcf1e4 not found: ID does not exist"
Sep 16 02:19:20.082693 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.082652    2558 scope.go:117] "RemoveContainer" containerID="848a9f5242e21b99c79c20be2c8a7de14d7000c19decafe0e4f64a8d7a8052b0"
Sep 16 02:19:20.082876 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.082858    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"848a9f5242e21b99c79c20be2c8a7de14d7000c19decafe0e4f64a8d7a8052b0"} err="failed to get container status \"848a9f5242e21b99c79c20be2c8a7de14d7000c19decafe0e4f64a8d7a8052b0\": rpc error: code = NotFound desc = could not find container \"848a9f5242e21b99c79c20be2c8a7de14d7000c19decafe0e4f64a8d7a8052b0\": container with ID starting with 848a9f5242e21b99c79c20be2c8a7de14d7000c19decafe0e4f64a8d7a8052b0 not found: ID does not exist"
Sep 16 02:19:20.082947 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.082878    2558 scope.go:117] "RemoveContainer" containerID="a5fc9315177131cb886db3f2985bbf14b089596e1caa7b2e12a9adcc70ca8721"
Sep 16 02:19:20.083100 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.083085    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5fc9315177131cb886db3f2985bbf14b089596e1caa7b2e12a9adcc70ca8721"} err="failed to get container status \"a5fc9315177131cb886db3f2985bbf14b089596e1caa7b2e12a9adcc70ca8721\": rpc error: code = NotFound desc = could not find container \"a5fc9315177131cb886db3f2985bbf14b089596e1caa7b2e12a9adcc70ca8721\": container with ID starting with a5fc9315177131cb886db3f2985bbf14b089596e1caa7b2e12a9adcc70ca8721 not found: ID does not exist"
Sep 16 02:19:20.083146 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.083100    2558 scope.go:117] "RemoveContainer" containerID="d8fa4e821be4059e18f589bfbf6784413d70d416ac4f6cbd1ab52392dba26b0b"
Sep 16 02:19:20.083301 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.083285    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8fa4e821be4059e18f589bfbf6784413d70d416ac4f6cbd1ab52392dba26b0b"} err="failed to get container status \"d8fa4e821be4059e18f589bfbf6784413d70d416ac4f6cbd1ab52392dba26b0b\": rpc error: code = NotFound desc = could not find container \"d8fa4e821be4059e18f589bfbf6784413d70d416ac4f6cbd1ab52392dba26b0b\": container with ID starting with d8fa4e821be4059e18f589bfbf6784413d70d416ac4f6cbd1ab52392dba26b0b not found: ID does not exist"
Sep 16 02:19:20.083359 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.083302    2558 scope.go:117] "RemoveContainer" containerID="06d22256acb387ea4d461f898a63d96b601655840f439c08e019ae5c6d3eaf57"
Sep 16 02:19:20.083504 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.083489    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06d22256acb387ea4d461f898a63d96b601655840f439c08e019ae5c6d3eaf57"} err="failed to get container status \"06d22256acb387ea4d461f898a63d96b601655840f439c08e019ae5c6d3eaf57\": rpc error: code = NotFound desc = could not find container \"06d22256acb387ea4d461f898a63d96b601655840f439c08e019ae5c6d3eaf57\": container with ID starting with 06d22256acb387ea4d461f898a63d96b601655840f439c08e019ae5c6d3eaf57 not found: ID does not exist"
Sep 16 02:19:20.127126 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.127102    2558 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/bbc9e62b-1357-4c44-a284-d3ac4a6c593c-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:19:20.127277 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.127130    2558 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/bbc9e62b-1357-4c44-a284-d3ac4a6c593c-secret-grpc-tls\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:19:20.127277 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.127145    2558 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/bbc9e62b-1357-4c44-a284-d3ac4a6c593c-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:19:20.127277 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.127160    2558 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/bbc9e62b-1357-4c44-a284-d3ac4a6c593c-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:19:20.127277 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.127173    2558 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/bbc9e62b-1357-4c44-a284-d3ac4a6c593c-config-out\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:19:20.127277 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.127185    2558 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/bbc9e62b-1357-4c44-a284-d3ac4a6c593c-prometheus-k8s-db\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:19:20.127277 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.127199    2558 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/bbc9e62b-1357-4c44-a284-d3ac4a6c593c-thanos-prometheus-http-client-file\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:19:20.127277 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.127212    2558 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/bbc9e62b-1357-4c44-a284-d3ac4a6c593c-secret-metrics-client-certs\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:19:20.127277 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.127224    2558 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/bbc9e62b-1357-4c44-a284-d3ac4a6c593c-config\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:19:20.127277 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.127236    2558 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/bbc9e62b-1357-4c44-a284-d3ac4a6c593c-secret-prometheus-k8s-tls\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:19:20.127277 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.127248    2558 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/bbc9e62b-1357-4c44-a284-d3ac4a6c593c-tls-assets\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:19:20.127277 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.127259    2558 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bbc9e62b-1357-4c44-a284-d3ac4a6c593c-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:19:20.127277 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.127272    2558 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bbc9e62b-1357-4c44-a284-d3ac4a6c593c-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:19:20.127635 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.127285    2558 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/bbc9e62b-1357-4c44-a284-d3ac4a6c593c-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:19:20.127635 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.127299    2558 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/bbc9e62b-1357-4c44-a284-d3ac4a6c593c-web-config\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:19:20.127635 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.127311    2558 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bbc9e62b-1357-4c44-a284-d3ac4a6c593c-prometheus-trusted-ca-bundle\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:19:20.127635 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.127324    2558 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bbc9e62b-1357-4c44-a284-d3ac4a6c593c-configmap-metrics-client-ca\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:19:20.127635 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.127339    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g2mzt\" (UniqueName: \"kubernetes.io/projected/bbc9e62b-1357-4c44-a284-d3ac4a6c593c-kube-api-access-g2mzt\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:19:20.127635 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.127353    2558 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/bbc9e62b-1357-4c44-a284-d3ac4a6c593c-secret-kube-rbac-proxy\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:19:20.346742 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.346712    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 02:19:20.351206 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.351179    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 02:19:20.375777 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.375752    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 02:19:20.375995 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.375983    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bbc9e62b-1357-4c44-a284-d3ac4a6c593c" containerName="thanos-sidecar"
Sep 16 02:19:20.376039 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.375996    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="bbc9e62b-1357-4c44-a284-d3ac4a6c593c" containerName="thanos-sidecar"
Sep 16 02:19:20.376039 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.376013    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bbc9e62b-1357-4c44-a284-d3ac4a6c593c" containerName="config-reloader"
Sep 16 02:19:20.376039 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.376019    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="bbc9e62b-1357-4c44-a284-d3ac4a6c593c" containerName="config-reloader"
Sep 16 02:19:20.376039 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.376027    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bbc9e62b-1357-4c44-a284-d3ac4a6c593c" containerName="kube-rbac-proxy"
Sep 16 02:19:20.376039 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.376033    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="bbc9e62b-1357-4c44-a284-d3ac4a6c593c" containerName="kube-rbac-proxy"
Sep 16 02:19:20.376039 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.376040    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bbc9e62b-1357-4c44-a284-d3ac4a6c593c" containerName="kube-rbac-proxy-thanos"
Sep 16 02:19:20.376233 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.376045    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="bbc9e62b-1357-4c44-a284-d3ac4a6c593c" containerName="kube-rbac-proxy-thanos"
Sep 16 02:19:20.376233 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.376054    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bbc9e62b-1357-4c44-a284-d3ac4a6c593c" containerName="init-config-reloader"
Sep 16 02:19:20.376233 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.376060    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="bbc9e62b-1357-4c44-a284-d3ac4a6c593c" containerName="init-config-reloader"
Sep 16 02:19:20.376233 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.376065    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bbc9e62b-1357-4c44-a284-d3ac4a6c593c" containerName="kube-rbac-proxy-web"
Sep 16 02:19:20.376233 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.376070    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="bbc9e62b-1357-4c44-a284-d3ac4a6c593c" containerName="kube-rbac-proxy-web"
Sep 16 02:19:20.376233 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.376078    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bbc9e62b-1357-4c44-a284-d3ac4a6c593c" containerName="prometheus"
Sep 16 02:19:20.376233 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.376083    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="bbc9e62b-1357-4c44-a284-d3ac4a6c593c" containerName="prometheus"
Sep 16 02:19:20.376233 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.376122    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="bbc9e62b-1357-4c44-a284-d3ac4a6c593c" containerName="config-reloader"
Sep 16 02:19:20.376233 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.376131    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="bbc9e62b-1357-4c44-a284-d3ac4a6c593c" containerName="kube-rbac-proxy-web"
Sep 16 02:19:20.376233 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.376137    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="bbc9e62b-1357-4c44-a284-d3ac4a6c593c" containerName="kube-rbac-proxy"
Sep 16 02:19:20.376233 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.376143    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="bbc9e62b-1357-4c44-a284-d3ac4a6c593c" containerName="kube-rbac-proxy-thanos"
Sep 16 02:19:20.376233 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.376148    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="bbc9e62b-1357-4c44-a284-d3ac4a6c593c" containerName="prometheus"
Sep 16 02:19:20.376233 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.376152    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="bbc9e62b-1357-4c44-a284-d3ac4a6c593c" containerName="thanos-sidecar"
Sep 16 02:19:20.380624 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.380609    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:19:20.383096 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.383078    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 16 02:19:20.383096 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.383087    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 16 02:19:20.383367 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.383340    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 16 02:19:20.383545 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.383530    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 16 02:19:20.383626 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.383609    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 16 02:19:20.383688 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.383661    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 16 02:19:20.383788 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.383769    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-fp9wb\""
Sep 16 02:19:20.384046 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.384028    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 16 02:19:20.384112 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.384069    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 16 02:19:20.384112 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.384036    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 16 02:19:20.384201 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.384114    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-brud7bi28g55g\""
Sep 16 02:19:20.384201 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.384036    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 16 02:19:20.384955 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.384934    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 16 02:19:20.385043 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.384943    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 16 02:19:20.387382 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.387360    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 16 02:19:20.389190 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.389172    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 16 02:19:20.392330 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.392310    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 02:19:20.530588 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.530545    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f4ed152f-8d15-4a7a-af3f-9e234b4cc1b8-config-out\") pod \"prometheus-k8s-0\" (UID: \"f4ed152f-8d15-4a7a-af3f-9e234b4cc1b8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:19:20.530788 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.530599    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f4ed152f-8d15-4a7a-af3f-9e234b4cc1b8-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"f4ed152f-8d15-4a7a-af3f-9e234b4cc1b8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:19:20.530788 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.530629    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f4ed152f-8d15-4a7a-af3f-9e234b4cc1b8-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"f4ed152f-8d15-4a7a-af3f-9e234b4cc1b8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:19:20.530788 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.530678    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f4ed152f-8d15-4a7a-af3f-9e234b4cc1b8-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f4ed152f-8d15-4a7a-af3f-9e234b4cc1b8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:19:20.530788 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.530747    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f4ed152f-8d15-4a7a-af3f-9e234b4cc1b8-web-config\") pod \"prometheus-k8s-0\" (UID: \"f4ed152f-8d15-4a7a-af3f-9e234b4cc1b8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:19:20.531009 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.530796    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/25e85d58-77e2-4428-b099-a5075363d56f-metrics-certs\") pod \"network-metrics-daemon-ck2vw\" (UID: \"25e85d58-77e2-4428-b099-a5075363d56f\") " pod="openshift-multus/network-metrics-daemon-ck2vw"
Sep 16 02:19:20.531009 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.530823    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f4ed152f-8d15-4a7a-af3f-9e234b4cc1b8-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"f4ed152f-8d15-4a7a-af3f-9e234b4cc1b8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:19:20.531009 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.530853    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f4ed152f-8d15-4a7a-af3f-9e234b4cc1b8-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"f4ed152f-8d15-4a7a-af3f-9e234b4cc1b8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:19:20.531009 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.530880    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f4ed152f-8d15-4a7a-af3f-9e234b4cc1b8-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"f4ed152f-8d15-4a7a-af3f-9e234b4cc1b8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:19:20.531009 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.530908    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f4ed152f-8d15-4a7a-af3f-9e234b4cc1b8-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"f4ed152f-8d15-4a7a-af3f-9e234b4cc1b8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:19:20.531009 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.530933    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f4ed152f-8d15-4a7a-af3f-9e234b4cc1b8-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f4ed152f-8d15-4a7a-af3f-9e234b4cc1b8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:19:20.531009 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.530974    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f4ed152f-8d15-4a7a-af3f-9e234b4cc1b8-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"f4ed152f-8d15-4a7a-af3f-9e234b4cc1b8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:19:20.531231 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.531012    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f4ed152f-8d15-4a7a-af3f-9e234b4cc1b8-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"f4ed152f-8d15-4a7a-af3f-9e234b4cc1b8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:19:20.531231 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.531030    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f4ed152f-8d15-4a7a-af3f-9e234b4cc1b8-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"f4ed152f-8d15-4a7a-af3f-9e234b4cc1b8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:19:20.531231 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.531049    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f4ed152f-8d15-4a7a-af3f-9e234b4cc1b8-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"f4ed152f-8d15-4a7a-af3f-9e234b4cc1b8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:19:20.531231 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.531067    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f4ed152f-8d15-4a7a-af3f-9e234b4cc1b8-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"f4ed152f-8d15-4a7a-af3f-9e234b4cc1b8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:19:20.531231 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.531104    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqk7v\" (UniqueName: \"kubernetes.io/projected/f4ed152f-8d15-4a7a-af3f-9e234b4cc1b8-kube-api-access-sqk7v\") pod \"prometheus-k8s-0\" (UID: \"f4ed152f-8d15-4a7a-af3f-9e234b4cc1b8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:19:20.531231 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.531133    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/f4ed152f-8d15-4a7a-af3f-9e234b4cc1b8-config\") pod \"prometheus-k8s-0\" (UID: \"f4ed152f-8d15-4a7a-af3f-9e234b4cc1b8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:19:20.531231 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.531185    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f4ed152f-8d15-4a7a-af3f-9e234b4cc1b8-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"f4ed152f-8d15-4a7a-af3f-9e234b4cc1b8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:19:20.531231 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.531222    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f4ed152f-8d15-4a7a-af3f-9e234b4cc1b8-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"f4ed152f-8d15-4a7a-af3f-9e234b4cc1b8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:19:20.531479 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.531247    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f4ed152f-8d15-4a7a-af3f-9e234b4cc1b8-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f4ed152f-8d15-4a7a-af3f-9e234b4cc1b8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:19:20.533165 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.533148    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/25e85d58-77e2-4428-b099-a5075363d56f-metrics-certs\") pod \"network-metrics-daemon-ck2vw\" (UID: \"25e85d58-77e2-4428-b099-a5075363d56f\") " pod="openshift-multus/network-metrics-daemon-ck2vw"
Sep 16 02:19:20.631776 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.631670    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f4ed152f-8d15-4a7a-af3f-9e234b4cc1b8-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"f4ed152f-8d15-4a7a-af3f-9e234b4cc1b8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:19:20.631776 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.631730    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f4ed152f-8d15-4a7a-af3f-9e234b4cc1b8-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"f4ed152f-8d15-4a7a-af3f-9e234b4cc1b8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:19:20.631983 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.631843    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f4ed152f-8d15-4a7a-af3f-9e234b4cc1b8-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"f4ed152f-8d15-4a7a-af3f-9e234b4cc1b8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:19:20.631983 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.631879    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f4ed152f-8d15-4a7a-af3f-9e234b4cc1b8-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"f4ed152f-8d15-4a7a-af3f-9e234b4cc1b8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:19:20.632073 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.631994    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqk7v\" (UniqueName: \"kubernetes.io/projected/f4ed152f-8d15-4a7a-af3f-9e234b4cc1b8-kube-api-access-sqk7v\") pod \"prometheus-k8s-0\" (UID: \"f4ed152f-8d15-4a7a-af3f-9e234b4cc1b8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:19:20.632073 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.632028    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/f4ed152f-8d15-4a7a-af3f-9e234b4cc1b8-config\") pod \"prometheus-k8s-0\" (UID: \"f4ed152f-8d15-4a7a-af3f-9e234b4cc1b8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:19:20.632073 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.632046    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f4ed152f-8d15-4a7a-af3f-9e234b4cc1b8-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"f4ed152f-8d15-4a7a-af3f-9e234b4cc1b8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:19:20.632073 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.632063    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f4ed152f-8d15-4a7a-af3f-9e234b4cc1b8-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"f4ed152f-8d15-4a7a-af3f-9e234b4cc1b8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:19:20.632267 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.632098    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f4ed152f-8d15-4a7a-af3f-9e234b4cc1b8-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f4ed152f-8d15-4a7a-af3f-9e234b4cc1b8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:19:20.632267 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.632134    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f4ed152f-8d15-4a7a-af3f-9e234b4cc1b8-config-out\") pod \"prometheus-k8s-0\" (UID: \"f4ed152f-8d15-4a7a-af3f-9e234b4cc1b8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:19:20.632267 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.632175    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f4ed152f-8d15-4a7a-af3f-9e234b4cc1b8-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"f4ed152f-8d15-4a7a-af3f-9e234b4cc1b8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:19:20.632267 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.632200    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f4ed152f-8d15-4a7a-af3f-9e234b4cc1b8-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"f4ed152f-8d15-4a7a-af3f-9e234b4cc1b8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:19:20.632267 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.632247    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f4ed152f-8d15-4a7a-af3f-9e234b4cc1b8-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f4ed152f-8d15-4a7a-af3f-9e234b4cc1b8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:19:20.632474 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.632277    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f4ed152f-8d15-4a7a-af3f-9e234b4cc1b8-web-config\") pod \"prometheus-k8s-0\" (UID: \"f4ed152f-8d15-4a7a-af3f-9e234b4cc1b8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:19:20.632474 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.632306    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f4ed152f-8d15-4a7a-af3f-9e234b4cc1b8-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"f4ed152f-8d15-4a7a-af3f-9e234b4cc1b8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:19:20.632474 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.632338    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f4ed152f-8d15-4a7a-af3f-9e234b4cc1b8-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"f4ed152f-8d15-4a7a-af3f-9e234b4cc1b8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:19:20.632474 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.632363    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f4ed152f-8d15-4a7a-af3f-9e234b4cc1b8-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"f4ed152f-8d15-4a7a-af3f-9e234b4cc1b8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:19:20.632474 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.632388    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f4ed152f-8d15-4a7a-af3f-9e234b4cc1b8-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"f4ed152f-8d15-4a7a-af3f-9e234b4cc1b8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:19:20.632474 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.632414    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f4ed152f-8d15-4a7a-af3f-9e234b4cc1b8-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f4ed152f-8d15-4a7a-af3f-9e234b4cc1b8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:19:20.632474 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.632441    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f4ed152f-8d15-4a7a-af3f-9e234b4cc1b8-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"f4ed152f-8d15-4a7a-af3f-9e234b4cc1b8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:19:20.632827 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.632487    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f4ed152f-8d15-4a7a-af3f-9e234b4cc1b8-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"f4ed152f-8d15-4a7a-af3f-9e234b4cc1b8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:19:20.633034 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.633007    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f4ed152f-8d15-4a7a-af3f-9e234b4cc1b8-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f4ed152f-8d15-4a7a-af3f-9e234b4cc1b8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:19:20.633311 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.633286    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f4ed152f-8d15-4a7a-af3f-9e234b4cc1b8-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"f4ed152f-8d15-4a7a-af3f-9e234b4cc1b8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:19:20.635387 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.635358    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f4ed152f-8d15-4a7a-af3f-9e234b4cc1b8-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f4ed152f-8d15-4a7a-af3f-9e234b4cc1b8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:19:20.635658 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.635634    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f4ed152f-8d15-4a7a-af3f-9e234b4cc1b8-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"f4ed152f-8d15-4a7a-af3f-9e234b4cc1b8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:19:20.636496 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.636061    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/f4ed152f-8d15-4a7a-af3f-9e234b4cc1b8-config\") pod \"prometheus-k8s-0\" (UID: \"f4ed152f-8d15-4a7a-af3f-9e234b4cc1b8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:19:20.636496 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.636258    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f4ed152f-8d15-4a7a-af3f-9e234b4cc1b8-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"f4ed152f-8d15-4a7a-af3f-9e234b4cc1b8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:19:20.636685 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.636624    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f4ed152f-8d15-4a7a-af3f-9e234b4cc1b8-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"f4ed152f-8d15-4a7a-af3f-9e234b4cc1b8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:19:20.637578 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.637217    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f4ed152f-8d15-4a7a-af3f-9e234b4cc1b8-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"f4ed152f-8d15-4a7a-af3f-9e234b4cc1b8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:19:20.637578 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.637565    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f4ed152f-8d15-4a7a-af3f-9e234b4cc1b8-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"f4ed152f-8d15-4a7a-af3f-9e234b4cc1b8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:19:20.637872 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.637844    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f4ed152f-8d15-4a7a-af3f-9e234b4cc1b8-web-config\") pod \"prometheus-k8s-0\" (UID: \"f4ed152f-8d15-4a7a-af3f-9e234b4cc1b8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:19:20.638210 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.638139    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f4ed152f-8d15-4a7a-af3f-9e234b4cc1b8-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"f4ed152f-8d15-4a7a-af3f-9e234b4cc1b8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:19:20.638319 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.638237    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f4ed152f-8d15-4a7a-af3f-9e234b4cc1b8-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"f4ed152f-8d15-4a7a-af3f-9e234b4cc1b8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:19:20.638409 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.638389    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f4ed152f-8d15-4a7a-af3f-9e234b4cc1b8-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"f4ed152f-8d15-4a7a-af3f-9e234b4cc1b8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:19:20.638920 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.638901    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f4ed152f-8d15-4a7a-af3f-9e234b4cc1b8-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"f4ed152f-8d15-4a7a-af3f-9e234b4cc1b8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:19:20.639805 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.639777    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f4ed152f-8d15-4a7a-af3f-9e234b4cc1b8-config-out\") pod \"prometheus-k8s-0\" (UID: \"f4ed152f-8d15-4a7a-af3f-9e234b4cc1b8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:19:20.639942 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.639918    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f4ed152f-8d15-4a7a-af3f-9e234b4cc1b8-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"f4ed152f-8d15-4a7a-af3f-9e234b4cc1b8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:19:20.640347 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.640324    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f4ed152f-8d15-4a7a-af3f-9e234b4cc1b8-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f4ed152f-8d15-4a7a-af3f-9e234b4cc1b8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:19:20.642279 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.642260    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f4ed152f-8d15-4a7a-af3f-9e234b4cc1b8-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"f4ed152f-8d15-4a7a-af3f-9e234b4cc1b8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:19:20.642340 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.642307    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqk7v\" (UniqueName: \"kubernetes.io/projected/f4ed152f-8d15-4a7a-af3f-9e234b4cc1b8-kube-api-access-sqk7v\") pod \"prometheus-k8s-0\" (UID: \"f4ed152f-8d15-4a7a-af3f-9e234b4cc1b8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:19:20.647198 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.647173    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-2ht6m\""
Sep 16 02:19:20.656104 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.656089    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ck2vw"
Sep 16 02:19:20.690678 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.690649    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:19:20.921865 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.921803    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-ck2vw"]
Sep 16 02:19:20.925603 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:19:20.925578    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25e85d58_77e2_4428_b099_a5075363d56f.slice/crio-5ceedf88eee95a8747962748f5814144ae854dae878e0bdc2b1e99b75345229a WatchSource:0}: Error finding container 5ceedf88eee95a8747962748f5814144ae854dae878e0bdc2b1e99b75345229a: Status 404 returned error can't find the container with id 5ceedf88eee95a8747962748f5814144ae854dae878e0bdc2b1e99b75345229a
Sep 16 02:19:20.927722 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.927690    2558 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 02:19:20.970344 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:20.970321    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 02:19:20.972138 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:19:20.972112    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4ed152f_8d15_4a7a_af3f_9e234b4cc1b8.slice/crio-14561deb8e634c9632c8ea97d8b0e54271e31eac51fbc6bf97a006c1a9cff190 WatchSource:0}: Error finding container 14561deb8e634c9632c8ea97d8b0e54271e31eac51fbc6bf97a006c1a9cff190: Status 404 returned error can't find the container with id 14561deb8e634c9632c8ea97d8b0e54271e31eac51fbc6bf97a006c1a9cff190
Sep 16 02:19:21.030127 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:21.030098    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f4ed152f-8d15-4a7a-af3f-9e234b4cc1b8","Type":"ContainerStarted","Data":"14561deb8e634c9632c8ea97d8b0e54271e31eac51fbc6bf97a006c1a9cff190"}
Sep 16 02:19:21.031425 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:21.031392    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-ck2vw" event={"ID":"25e85d58-77e2-4428-b099-a5075363d56f","Type":"ContainerStarted","Data":"5ceedf88eee95a8747962748f5814144ae854dae878e0bdc2b1e99b75345229a"}
Sep 16 02:19:21.552172 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:21.552133    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bbc9e62b-1357-4c44-a284-d3ac4a6c593c" path="/var/lib/kubelet/pods/bbc9e62b-1357-4c44-a284-d3ac4a6c593c/volumes"
Sep 16 02:19:22.038434 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:22.038385    2558 generic.go:356] "Generic (PLEG): container finished" podID="f4ed152f-8d15-4a7a-af3f-9e234b4cc1b8" containerID="6bd3df5b81f248274a576b3c36481a08bf1cbabb94b87daba1d1b8077db39ea8" exitCode=0
Sep 16 02:19:22.039140 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:22.038470    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f4ed152f-8d15-4a7a-af3f-9e234b4cc1b8","Type":"ContainerDied","Data":"6bd3df5b81f248274a576b3c36481a08bf1cbabb94b87daba1d1b8077db39ea8"}
Sep 16 02:19:22.040719 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:22.040513    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-ck2vw" event={"ID":"25e85d58-77e2-4428-b099-a5075363d56f","Type":"ContainerStarted","Data":"2a0ac9bd12d72bd85d3141dd901db49192fc980309dd3919dfb40db9419b859b"}
Sep 16 02:19:23.047682 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:23.047642    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f4ed152f-8d15-4a7a-af3f-9e234b4cc1b8","Type":"ContainerStarted","Data":"e50924de45305526e7b06e0a17ecda7a8e214db26a47e718e0f5d1a44255cb96"}
Sep 16 02:19:23.048088 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:23.047747    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f4ed152f-8d15-4a7a-af3f-9e234b4cc1b8","Type":"ContainerStarted","Data":"7a5bdc4a1cdc5fe664383705cd6e23078652cf69dbce17a94a0e210c9a4bd8d4"}
Sep 16 02:19:23.048088 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:23.047789    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f4ed152f-8d15-4a7a-af3f-9e234b4cc1b8","Type":"ContainerStarted","Data":"7c50a9f22b530c51f20a38b57d363b0dbec5e3b023abd120c09ed5568a072fe6"}
Sep 16 02:19:23.048088 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:23.047826    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f4ed152f-8d15-4a7a-af3f-9e234b4cc1b8","Type":"ContainerStarted","Data":"703cf62938b950517a219774205e98bf03b500b61468a6930f74b72c061694b7"}
Sep 16 02:19:23.048088 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:23.047860    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f4ed152f-8d15-4a7a-af3f-9e234b4cc1b8","Type":"ContainerStarted","Data":"6dae31ec8810d0fa4f16b79b87f30c62f712860d8144e7d599d7599f8e7795ca"}
Sep 16 02:19:23.048088 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:23.047896    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f4ed152f-8d15-4a7a-af3f-9e234b4cc1b8","Type":"ContainerStarted","Data":"9da2b191b859d016b29498d259bb787de3232127f2199ed5907fc9374d4ae301"}
Sep 16 02:19:23.049448 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:23.049427    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-ck2vw" event={"ID":"25e85d58-77e2-4428-b099-a5075363d56f","Type":"ContainerStarted","Data":"bc9b607ae7c860807edd4dd620fffdd86ad24070f6026c92f7d7c69f622aaa53"}
Sep 16 02:19:23.075558 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:23.075501    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.056805377 podStartE2EDuration="3.075483413s" totalImagesPullingTime="0s" totalInitContainerRuntime="18.678036ms" isStatefulPod=false podCreationTimestamp="2026-09-16 02:19:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:19:23.07327887 +0000 UTC m=+375.991768121" watchObservedRunningTime="2026-09-16 02:19:23.075483413 +0000 UTC m=+375.993972662"
Sep 16 02:19:23.088594 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:23.088544    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-ck2vw" podStartSLOduration=374.090089743 podStartE2EDuration="6m15.088527104s" totalImagesPullingTime="998.437361ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 02:13:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:19:23.086860973 +0000 UTC m=+376.005350221" watchObservedRunningTime="2026-09-16 02:19:23.088527104 +0000 UTC m=+376.007016352"
Sep 16 02:19:25.691582 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:19:25.691550    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:20:20.691560 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:20:20.691507    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:20:20.706811 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:20:20.706785    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:20:21.288724 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:20:21.288679    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 02:23:08.834249 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:23:08.834209    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-79c8b99f94-b6hpk"]
Sep 16 02:23:08.837610 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:23:08.837587    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-79c8b99f94-b6hpk"
Sep 16 02:23:08.840322 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:23:08.840295    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 16 02:23:08.841170 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:23:08.841152    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 16 02:23:08.841268 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:23:08.841227    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 16 02:23:08.841268 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:23:08.841235    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 16 02:23:08.841483 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:23:08.841470    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-t82b9\""
Sep 16 02:23:08.841930 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:23:08.841913    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 16 02:23:08.858050 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:23:08.858020    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-79c8b99f94-b6hpk"]
Sep 16 02:23:09.019030 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:23:09.018995    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/2cacabd8-aaf3-4b2b-948b-bd75714a4bf6-metrics-cert\") pod \"lws-controller-manager-79c8b99f94-b6hpk\" (UID: \"2cacabd8-aaf3-4b2b-948b-bd75714a4bf6\") " pod="openshift-lws-operator/lws-controller-manager-79c8b99f94-b6hpk"
Sep 16 02:23:09.019200 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:23:09.019044    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2cacabd8-aaf3-4b2b-948b-bd75714a4bf6-cert\") pod \"lws-controller-manager-79c8b99f94-b6hpk\" (UID: \"2cacabd8-aaf3-4b2b-948b-bd75714a4bf6\") " pod="openshift-lws-operator/lws-controller-manager-79c8b99f94-b6hpk"
Sep 16 02:23:09.019200 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:23:09.019078    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/2cacabd8-aaf3-4b2b-948b-bd75714a4bf6-manager-config\") pod \"lws-controller-manager-79c8b99f94-b6hpk\" (UID: \"2cacabd8-aaf3-4b2b-948b-bd75714a4bf6\") " pod="openshift-lws-operator/lws-controller-manager-79c8b99f94-b6hpk"
Sep 16 02:23:09.019200 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:23:09.019100    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p85pd\" (UniqueName: \"kubernetes.io/projected/2cacabd8-aaf3-4b2b-948b-bd75714a4bf6-kube-api-access-p85pd\") pod \"lws-controller-manager-79c8b99f94-b6hpk\" (UID: \"2cacabd8-aaf3-4b2b-948b-bd75714a4bf6\") " pod="openshift-lws-operator/lws-controller-manager-79c8b99f94-b6hpk"
Sep 16 02:23:09.120083 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:23:09.119975    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/2cacabd8-aaf3-4b2b-948b-bd75714a4bf6-metrics-cert\") pod \"lws-controller-manager-79c8b99f94-b6hpk\" (UID: \"2cacabd8-aaf3-4b2b-948b-bd75714a4bf6\") " pod="openshift-lws-operator/lws-controller-manager-79c8b99f94-b6hpk"
Sep 16 02:23:09.120083 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:23:09.120066    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2cacabd8-aaf3-4b2b-948b-bd75714a4bf6-cert\") pod \"lws-controller-manager-79c8b99f94-b6hpk\" (UID: \"2cacabd8-aaf3-4b2b-948b-bd75714a4bf6\") " pod="openshift-lws-operator/lws-controller-manager-79c8b99f94-b6hpk"
Sep 16 02:23:09.120310 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:23:09.120104    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/2cacabd8-aaf3-4b2b-948b-bd75714a4bf6-manager-config\") pod \"lws-controller-manager-79c8b99f94-b6hpk\" (UID: \"2cacabd8-aaf3-4b2b-948b-bd75714a4bf6\") " pod="openshift-lws-operator/lws-controller-manager-79c8b99f94-b6hpk"
Sep 16 02:23:09.120310 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:23:09.120135    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p85pd\" (UniqueName: \"kubernetes.io/projected/2cacabd8-aaf3-4b2b-948b-bd75714a4bf6-kube-api-access-p85pd\") pod \"lws-controller-manager-79c8b99f94-b6hpk\" (UID: \"2cacabd8-aaf3-4b2b-948b-bd75714a4bf6\") " pod="openshift-lws-operator/lws-controller-manager-79c8b99f94-b6hpk"
Sep 16 02:23:09.120798 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:23:09.120773    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/2cacabd8-aaf3-4b2b-948b-bd75714a4bf6-manager-config\") pod \"lws-controller-manager-79c8b99f94-b6hpk\" (UID: \"2cacabd8-aaf3-4b2b-948b-bd75714a4bf6\") " pod="openshift-lws-operator/lws-controller-manager-79c8b99f94-b6hpk"
Sep 16 02:23:09.122509 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:23:09.122487    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/2cacabd8-aaf3-4b2b-948b-bd75714a4bf6-metrics-cert\") pod \"lws-controller-manager-79c8b99f94-b6hpk\" (UID: \"2cacabd8-aaf3-4b2b-948b-bd75714a4bf6\") " pod="openshift-lws-operator/lws-controller-manager-79c8b99f94-b6hpk"
Sep 16 02:23:09.122605 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:23:09.122523    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2cacabd8-aaf3-4b2b-948b-bd75714a4bf6-cert\") pod \"lws-controller-manager-79c8b99f94-b6hpk\" (UID: \"2cacabd8-aaf3-4b2b-948b-bd75714a4bf6\") " pod="openshift-lws-operator/lws-controller-manager-79c8b99f94-b6hpk"
Sep 16 02:23:09.156009 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:23:09.155979    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p85pd\" (UniqueName: \"kubernetes.io/projected/2cacabd8-aaf3-4b2b-948b-bd75714a4bf6-kube-api-access-p85pd\") pod \"lws-controller-manager-79c8b99f94-b6hpk\" (UID: \"2cacabd8-aaf3-4b2b-948b-bd75714a4bf6\") " pod="openshift-lws-operator/lws-controller-manager-79c8b99f94-b6hpk"
Sep 16 02:23:09.447055 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:23:09.447018    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-79c8b99f94-b6hpk"
Sep 16 02:23:09.753989 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:23:09.753961    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-79c8b99f94-b6hpk"]
Sep 16 02:23:09.754350 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:23:09.754331    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cacabd8_aaf3_4b2b_948b_bd75714a4bf6.slice/crio-95d0222990766ff4f84a3c7f1f981f041641f20f00479065e564429aa2da213e WatchSource:0}: Error finding container 95d0222990766ff4f84a3c7f1f981f041641f20f00479065e564429aa2da213e: Status 404 returned error can't find the container with id 95d0222990766ff4f84a3c7f1f981f041641f20f00479065e564429aa2da213e
Sep 16 02:23:09.808332 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:23:09.808303    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-74fbd865d6-7sp4h"]
Sep 16 02:23:09.812614 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:23:09.812596    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-74fbd865d6-7sp4h"
Sep 16 02:23:09.819743 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:23:09.819721    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 16 02:23:09.820247 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:23:09.820229    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 16 02:23:09.820494 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:23:09.820472    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 16 02:23:09.820572 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:23:09.820530    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-xzpsv\""
Sep 16 02:23:09.820628 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:23:09.820597    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 16 02:23:09.849674 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:23:09.849643    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-74fbd865d6-7sp4h"]
Sep 16 02:23:09.883545 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:23:09.883515    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-79c8b99f94-b6hpk" event={"ID":"2cacabd8-aaf3-4b2b-948b-bd75714a4bf6","Type":"ContainerStarted","Data":"95d0222990766ff4f84a3c7f1f981f041641f20f00479065e564429aa2da213e"}
Sep 16 02:23:09.928248 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:23:09.928216    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f13e89aa-f62e-48e3-9f2d-137e6cc6276e-tmp\") pod \"opendatahub-operator-controller-manager-74fbd865d6-7sp4h\" (UID: \"f13e89aa-f62e-48e3-9f2d-137e6cc6276e\") " pod="opendatahub/opendatahub-operator-controller-manager-74fbd865d6-7sp4h"
Sep 16 02:23:09.928396 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:23:09.928269    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f13e89aa-f62e-48e3-9f2d-137e6cc6276e-manifests\") pod \"opendatahub-operator-controller-manager-74fbd865d6-7sp4h\" (UID: \"f13e89aa-f62e-48e3-9f2d-137e6cc6276e\") " pod="opendatahub/opendatahub-operator-controller-manager-74fbd865d6-7sp4h"
Sep 16 02:23:09.928396 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:23:09.928306    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hnrh\" (UniqueName: \"kubernetes.io/projected/f13e89aa-f62e-48e3-9f2d-137e6cc6276e-kube-api-access-8hnrh\") pod \"opendatahub-operator-controller-manager-74fbd865d6-7sp4h\" (UID: \"f13e89aa-f62e-48e3-9f2d-137e6cc6276e\") " pod="opendatahub/opendatahub-operator-controller-manager-74fbd865d6-7sp4h"
Sep 16 02:23:09.928396 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:23:09.928341    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/f13e89aa-f62e-48e3-9f2d-137e6cc6276e-apiservice-cert\") pod \"opendatahub-operator-controller-manager-74fbd865d6-7sp4h\" (UID: \"f13e89aa-f62e-48e3-9f2d-137e6cc6276e\") " pod="opendatahub/opendatahub-operator-controller-manager-74fbd865d6-7sp4h"
Sep 16 02:23:09.928503 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:23:09.928401    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f13e89aa-f62e-48e3-9f2d-137e6cc6276e-webhook-cert\") pod \"opendatahub-operator-controller-manager-74fbd865d6-7sp4h\" (UID: \"f13e89aa-f62e-48e3-9f2d-137e6cc6276e\") " pod="opendatahub/opendatahub-operator-controller-manager-74fbd865d6-7sp4h"
Sep 16 02:23:10.029725 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:23:10.029628    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hnrh\" (UniqueName: \"kubernetes.io/projected/f13e89aa-f62e-48e3-9f2d-137e6cc6276e-kube-api-access-8hnrh\") pod \"opendatahub-operator-controller-manager-74fbd865d6-7sp4h\" (UID: \"f13e89aa-f62e-48e3-9f2d-137e6cc6276e\") " pod="opendatahub/opendatahub-operator-controller-manager-74fbd865d6-7sp4h"
Sep 16 02:23:10.029725 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:23:10.029674    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/f13e89aa-f62e-48e3-9f2d-137e6cc6276e-apiservice-cert\") pod \"opendatahub-operator-controller-manager-74fbd865d6-7sp4h\" (UID: \"f13e89aa-f62e-48e3-9f2d-137e6cc6276e\") " pod="opendatahub/opendatahub-operator-controller-manager-74fbd865d6-7sp4h"
Sep 16 02:23:10.029896 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:23:10.029730    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f13e89aa-f62e-48e3-9f2d-137e6cc6276e-webhook-cert\") pod \"opendatahub-operator-controller-manager-74fbd865d6-7sp4h\" (UID: \"f13e89aa-f62e-48e3-9f2d-137e6cc6276e\") " pod="opendatahub/opendatahub-operator-controller-manager-74fbd865d6-7sp4h"
Sep 16 02:23:10.029896 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:23:10.029755    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f13e89aa-f62e-48e3-9f2d-137e6cc6276e-tmp\") pod \"opendatahub-operator-controller-manager-74fbd865d6-7sp4h\" (UID: \"f13e89aa-f62e-48e3-9f2d-137e6cc6276e\") " pod="opendatahub/opendatahub-operator-controller-manager-74fbd865d6-7sp4h"
Sep 16 02:23:10.029896 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:23:10.029798    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f13e89aa-f62e-48e3-9f2d-137e6cc6276e-manifests\") pod \"opendatahub-operator-controller-manager-74fbd865d6-7sp4h\" (UID: \"f13e89aa-f62e-48e3-9f2d-137e6cc6276e\") " pod="opendatahub/opendatahub-operator-controller-manager-74fbd865d6-7sp4h"
Sep 16 02:23:10.030297 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:23:10.030272    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f13e89aa-f62e-48e3-9f2d-137e6cc6276e-manifests\") pod \"opendatahub-operator-controller-manager-74fbd865d6-7sp4h\" (UID: \"f13e89aa-f62e-48e3-9f2d-137e6cc6276e\") " pod="opendatahub/opendatahub-operator-controller-manager-74fbd865d6-7sp4h"
Sep 16 02:23:10.032084 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:23:10.032062    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f13e89aa-f62e-48e3-9f2d-137e6cc6276e-tmp\") pod \"opendatahub-operator-controller-manager-74fbd865d6-7sp4h\" (UID: \"f13e89aa-f62e-48e3-9f2d-137e6cc6276e\") " pod="opendatahub/opendatahub-operator-controller-manager-74fbd865d6-7sp4h"
Sep 16 02:23:10.032218 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:23:10.032202    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f13e89aa-f62e-48e3-9f2d-137e6cc6276e-webhook-cert\") pod \"opendatahub-operator-controller-manager-74fbd865d6-7sp4h\" (UID: \"f13e89aa-f62e-48e3-9f2d-137e6cc6276e\") " pod="opendatahub/opendatahub-operator-controller-manager-74fbd865d6-7sp4h"
Sep 16 02:23:10.032283 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:23:10.032267    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/f13e89aa-f62e-48e3-9f2d-137e6cc6276e-apiservice-cert\") pod \"opendatahub-operator-controller-manager-74fbd865d6-7sp4h\" (UID: \"f13e89aa-f62e-48e3-9f2d-137e6cc6276e\") " pod="opendatahub/opendatahub-operator-controller-manager-74fbd865d6-7sp4h"
Sep 16 02:23:10.049988 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:23:10.049966    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hnrh\" (UniqueName: \"kubernetes.io/projected/f13e89aa-f62e-48e3-9f2d-137e6cc6276e-kube-api-access-8hnrh\") pod \"opendatahub-operator-controller-manager-74fbd865d6-7sp4h\" (UID: \"f13e89aa-f62e-48e3-9f2d-137e6cc6276e\") " pod="opendatahub/opendatahub-operator-controller-manager-74fbd865d6-7sp4h"
Sep 16 02:23:10.122630 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:23:10.122598    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-74fbd865d6-7sp4h"
Sep 16 02:23:10.420206 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:23:10.419977    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-74fbd865d6-7sp4h"]
Sep 16 02:23:10.890020 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:23:10.889939    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-74fbd865d6-7sp4h" event={"ID":"f13e89aa-f62e-48e3-9f2d-137e6cc6276e","Type":"ContainerStarted","Data":"0f5bc13f529c4252dc06a717b0784d860af1b85e5b964624c823b5a881d0fb09"}
Sep 16 02:23:14.908020 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:23:14.907988    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-79c8b99f94-b6hpk" event={"ID":"2cacabd8-aaf3-4b2b-948b-bd75714a4bf6","Type":"ContainerStarted","Data":"5229f603c90f2bf9a5d7c56e014f8b18920bb5844e0f6d8c8eecf867f8f192c1"}
Sep 16 02:23:14.908490 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:23:14.908088    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-79c8b99f94-b6hpk"
Sep 16 02:23:14.909733 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:23:14.909712    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-74fbd865d6-7sp4h" event={"ID":"f13e89aa-f62e-48e3-9f2d-137e6cc6276e","Type":"ContainerStarted","Data":"6b81de8d37bdefd737d4e0fb9ee8d29f28a7241c4e972239bb5813eddced280b"}
Sep 16 02:23:14.932508 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:23:14.932456    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-79c8b99f94-b6hpk" podStartSLOduration=2.736068745 podStartE2EDuration="6.932439181s" totalImagesPullingTime="4.196370436s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 02:23:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:23:14.931325829 +0000 UTC m=+607.849815079" watchObservedRunningTime="2026-09-16 02:23:14.932439181 +0000 UTC m=+607.850928430"
Sep 16 02:23:15.914592 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:23:15.914558    2558 generic.go:356] "Generic (PLEG): container finished" podID="f13e89aa-f62e-48e3-9f2d-137e6cc6276e" containerID="6b81de8d37bdefd737d4e0fb9ee8d29f28a7241c4e972239bb5813eddced280b" exitCode=0
Sep 16 02:23:15.915049 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:23:15.914645    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-74fbd865d6-7sp4h" event={"ID":"f13e89aa-f62e-48e3-9f2d-137e6cc6276e","Type":"ContainerDied","Data":"6b81de8d37bdefd737d4e0fb9ee8d29f28a7241c4e972239bb5813eddced280b"}
Sep 16 02:23:16.924997 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:23:16.924951    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-74fbd865d6-7sp4h" event={"ID":"f13e89aa-f62e-48e3-9f2d-137e6cc6276e","Type":"ContainerStarted","Data":"75ec907f88f0f98f8823f5fdce8668f64bff49a0970e3fa97e7dd3af6b40f4b2"}
Sep 16 02:23:16.925385 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:23:16.925102    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-74fbd865d6-7sp4h"
Sep 16 02:23:16.949198 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:23:16.949145    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-74fbd865d6-7sp4h" podStartSLOduration=2.92446408 podStartE2EDuration="7.949130862s" totalImagesPullingTime="3.930726894s" totalInitContainerRuntime="1.093939888s" isStatefulPod=false podCreationTimestamp="2026-09-16 02:23:09 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:23:16.947337149 +0000 UTC m=+609.865826400" watchObservedRunningTime="2026-09-16 02:23:16.949130862 +0000 UTC m=+609.867620110"
Sep 16 02:23:25.917530 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:23:25.917496    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-79c8b99f94-b6hpk"
Sep 16 02:23:27.930647 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:23:27.930615    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-74fbd865d6-7sp4h"
Sep 16 02:23:38.274132 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:23:38.274060    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-z9zbw"]
Sep 16 02:23:38.277316 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:23:38.277299    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-z9zbw"
Sep 16 02:23:38.280667 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:23:38.280648    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 16 02:23:38.280962 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:23:38.280943    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-shpjc\""
Sep 16 02:23:38.281218 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:23:38.281196    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 16 02:23:38.281365 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:23:38.281351    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 16 02:23:38.298628 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:23:38.298595    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-z9zbw"]
Sep 16 02:23:38.366243 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:23:38.366199    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d128e57b-91ba-4ed8-ab92-837ead863c2d-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-z9zbw\" (UID: \"d128e57b-91ba-4ed8-ab92-837ead863c2d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-z9zbw"
Sep 16 02:23:38.366421 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:23:38.366258    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d128e57b-91ba-4ed8-ab92-837ead863c2d-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-z9zbw\" (UID: \"d128e57b-91ba-4ed8-ab92-837ead863c2d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-z9zbw"
Sep 16 02:23:38.366421 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:23:38.366312    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d128e57b-91ba-4ed8-ab92-837ead863c2d-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-z9zbw\" (UID: \"d128e57b-91ba-4ed8-ab92-837ead863c2d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-z9zbw"
Sep 16 02:23:38.366421 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:23:38.366409    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d128e57b-91ba-4ed8-ab92-837ead863c2d-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-z9zbw\" (UID: \"d128e57b-91ba-4ed8-ab92-837ead863c2d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-z9zbw"
Sep 16 02:23:38.366556 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:23:38.366445    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d128e57b-91ba-4ed8-ab92-837ead863c2d-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-z9zbw\" (UID: \"d128e57b-91ba-4ed8-ab92-837ead863c2d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-z9zbw"
Sep 16 02:23:38.366556 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:23:38.366470    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hx2v2\" (UniqueName: \"kubernetes.io/projected/d128e57b-91ba-4ed8-ab92-837ead863c2d-kube-api-access-hx2v2\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-z9zbw\" (UID: \"d128e57b-91ba-4ed8-ab92-837ead863c2d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-z9zbw"
Sep 16 02:23:38.366556 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:23:38.366492    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d128e57b-91ba-4ed8-ab92-837ead863c2d-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-z9zbw\" (UID: \"d128e57b-91ba-4ed8-ab92-837ead863c2d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-z9zbw"
Sep 16 02:23:38.366556 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:23:38.366525    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d128e57b-91ba-4ed8-ab92-837ead863c2d-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-z9zbw\" (UID: \"d128e57b-91ba-4ed8-ab92-837ead863c2d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-z9zbw"
Sep 16 02:23:38.366707 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:23:38.366556    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d128e57b-91ba-4ed8-ab92-837ead863c2d-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-z9zbw\" (UID: \"d128e57b-91ba-4ed8-ab92-837ead863c2d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-z9zbw"
Sep 16 02:23:38.467342 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:23:38.467304    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d128e57b-91ba-4ed8-ab92-837ead863c2d-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-z9zbw\" (UID: \"d128e57b-91ba-4ed8-ab92-837ead863c2d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-z9zbw"
Sep 16 02:23:38.467342 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:23:38.467348    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d128e57b-91ba-4ed8-ab92-837ead863c2d-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-z9zbw\" (UID: \"d128e57b-91ba-4ed8-ab92-837ead863c2d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-z9zbw"
Sep 16 02:23:38.467546 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:23:38.467372    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hx2v2\" (UniqueName: \"kubernetes.io/projected/d128e57b-91ba-4ed8-ab92-837ead863c2d-kube-api-access-hx2v2\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-z9zbw\" (UID: \"d128e57b-91ba-4ed8-ab92-837ead863c2d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-z9zbw"
Sep 16 02:23:38.467546 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:23:38.467397    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d128e57b-91ba-4ed8-ab92-837ead863c2d-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-z9zbw\" (UID: \"d128e57b-91ba-4ed8-ab92-837ead863c2d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-z9zbw"
Sep 16 02:23:38.467546 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:23:38.467519    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d128e57b-91ba-4ed8-ab92-837ead863c2d-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-z9zbw\" (UID: \"d128e57b-91ba-4ed8-ab92-837ead863c2d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-z9zbw"
Sep 16 02:23:38.467691 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:23:38.467566    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d128e57b-91ba-4ed8-ab92-837ead863c2d-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-z9zbw\" (UID: \"d128e57b-91ba-4ed8-ab92-837ead863c2d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-z9zbw"
Sep 16 02:23:38.467691 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:23:38.467614    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d128e57b-91ba-4ed8-ab92-837ead863c2d-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-z9zbw\" (UID: \"d128e57b-91ba-4ed8-ab92-837ead863c2d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-z9zbw"
Sep 16 02:23:38.467691 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:23:38.467646    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d128e57b-91ba-4ed8-ab92-837ead863c2d-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-z9zbw\" (UID: \"d128e57b-91ba-4ed8-ab92-837ead863c2d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-z9zbw"
Sep 16 02:23:38.467691 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:23:38.467682    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d128e57b-91ba-4ed8-ab92-837ead863c2d-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-z9zbw\" (UID: \"d128e57b-91ba-4ed8-ab92-837ead863c2d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-z9zbw"
Sep 16 02:23:38.467924 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:23:38.467755    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d128e57b-91ba-4ed8-ab92-837ead863c2d-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-z9zbw\" (UID: \"d128e57b-91ba-4ed8-ab92-837ead863c2d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-z9zbw"
Sep 16 02:23:38.467924 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:23:38.467878    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d128e57b-91ba-4ed8-ab92-837ead863c2d-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-z9zbw\" (UID: \"d128e57b-91ba-4ed8-ab92-837ead863c2d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-z9zbw"
Sep 16 02:23:38.468054 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:23:38.468032    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d128e57b-91ba-4ed8-ab92-837ead863c2d-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-z9zbw\" (UID: \"d128e57b-91ba-4ed8-ab92-837ead863c2d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-z9zbw"
Sep 16 02:23:38.468092 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:23:38.468070    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d128e57b-91ba-4ed8-ab92-837ead863c2d-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-z9zbw\" (UID: \"d128e57b-91ba-4ed8-ab92-837ead863c2d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-z9zbw"
Sep 16 02:23:38.468339 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:23:38.468318    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d128e57b-91ba-4ed8-ab92-837ead863c2d-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-z9zbw\" (UID: \"d128e57b-91ba-4ed8-ab92-837ead863c2d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-z9zbw"
Sep 16 02:23:38.469772 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:23:38.469754    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d128e57b-91ba-4ed8-ab92-837ead863c2d-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-z9zbw\" (UID: \"d128e57b-91ba-4ed8-ab92-837ead863c2d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-z9zbw"
Sep 16 02:23:38.469830 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:23:38.469816    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d128e57b-91ba-4ed8-ab92-837ead863c2d-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-z9zbw\" (UID: \"d128e57b-91ba-4ed8-ab92-837ead863c2d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-z9zbw"
Sep 16 02:23:38.484152 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:23:38.484132    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d128e57b-91ba-4ed8-ab92-837ead863c2d-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-z9zbw\" (UID: \"d128e57b-91ba-4ed8-ab92-837ead863c2d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-z9zbw"
Sep 16 02:23:38.486791 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:23:38.486774    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hx2v2\" (UniqueName: \"kubernetes.io/projected/d128e57b-91ba-4ed8-ab92-837ead863c2d-kube-api-access-hx2v2\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-z9zbw\" (UID: \"d128e57b-91ba-4ed8-ab92-837ead863c2d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-z9zbw"
Sep 16 02:23:38.586767 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:23:38.586664    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-z9zbw"
Sep 16 02:23:38.860163 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:23:38.860105    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-z9zbw"]
Sep 16 02:23:38.862304 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:23:38.862272    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd128e57b_91ba_4ed8_ab92_837ead863c2d.slice/crio-465164be7c688094fd768fd424cf18df5a6e1795fff1b29952b7e88242e6827e WatchSource:0}: Error finding container 465164be7c688094fd768fd424cf18df5a6e1795fff1b29952b7e88242e6827e: Status 404 returned error can't find the container with id 465164be7c688094fd768fd424cf18df5a6e1795fff1b29952b7e88242e6827e
Sep 16 02:23:39.009614 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:23:39.009578    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-z9zbw" event={"ID":"d128e57b-91ba-4ed8-ab92-837ead863c2d","Type":"ContainerStarted","Data":"465164be7c688094fd768fd424cf18df5a6e1795fff1b29952b7e88242e6827e"}
Sep 16 02:23:40.744256 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:23:40.744221    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-zcg5h"]
Sep 16 02:23:40.747767 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:23:40.747744    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-zcg5h"
Sep 16 02:23:40.750129 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:23:40.750103    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\""
Sep 16 02:23:40.750262 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:23:40.750148    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-rxfc7\""
Sep 16 02:23:40.759444 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:23:40.759409    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-zcg5h"]
Sep 16 02:23:40.890253 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:23:40.890218    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frtr8\" (UniqueName: \"kubernetes.io/projected/09e730e6-3b3f-488c-99d9-d285e5ff195e-kube-api-access-frtr8\") pod \"odh-model-controller-7998c67dc4-zcg5h\" (UID: \"09e730e6-3b3f-488c-99d9-d285e5ff195e\") " pod="opendatahub/odh-model-controller-7998c67dc4-zcg5h"
Sep 16 02:23:40.890430 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:23:40.890281    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/09e730e6-3b3f-488c-99d9-d285e5ff195e-cert\") pod \"odh-model-controller-7998c67dc4-zcg5h\" (UID: \"09e730e6-3b3f-488c-99d9-d285e5ff195e\") " pod="opendatahub/odh-model-controller-7998c67dc4-zcg5h"
Sep 16 02:23:40.991146 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:23:40.991106    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/09e730e6-3b3f-488c-99d9-d285e5ff195e-cert\") pod \"odh-model-controller-7998c67dc4-zcg5h\" (UID: \"09e730e6-3b3f-488c-99d9-d285e5ff195e\") " pod="opendatahub/odh-model-controller-7998c67dc4-zcg5h"
Sep 16 02:23:40.991387 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:23:40.991211    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frtr8\" (UniqueName: \"kubernetes.io/projected/09e730e6-3b3f-488c-99d9-d285e5ff195e-kube-api-access-frtr8\") pod \"odh-model-controller-7998c67dc4-zcg5h\" (UID: \"09e730e6-3b3f-488c-99d9-d285e5ff195e\") " pod="opendatahub/odh-model-controller-7998c67dc4-zcg5h"
Sep 16 02:23:40.991387 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:23:40.991299    2558 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
Sep 16 02:23:40.991387 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:23:40.991382    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/09e730e6-3b3f-488c-99d9-d285e5ff195e-cert podName:09e730e6-3b3f-488c-99d9-d285e5ff195e nodeName:}" failed. No retries permitted until 2026-09-16 02:23:41.491359891 +0000 UTC m=+634.409849120 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/09e730e6-3b3f-488c-99d9-d285e5ff195e-cert") pod "odh-model-controller-7998c67dc4-zcg5h" (UID: "09e730e6-3b3f-488c-99d9-d285e5ff195e") : secret "odh-model-controller-webhook-cert" not found
Sep 16 02:23:41.000294 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:23:41.000226    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-frtr8\" (UniqueName: \"kubernetes.io/projected/09e730e6-3b3f-488c-99d9-d285e5ff195e-kube-api-access-frtr8\") pod \"odh-model-controller-7998c67dc4-zcg5h\" (UID: \"09e730e6-3b3f-488c-99d9-d285e5ff195e\") " pod="opendatahub/odh-model-controller-7998c67dc4-zcg5h"
Sep 16 02:23:41.495266 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:23:41.495225    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/09e730e6-3b3f-488c-99d9-d285e5ff195e-cert\") pod \"odh-model-controller-7998c67dc4-zcg5h\" (UID: \"09e730e6-3b3f-488c-99d9-d285e5ff195e\") " pod="opendatahub/odh-model-controller-7998c67dc4-zcg5h"
Sep 16 02:23:41.498063 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:23:41.498032    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/09e730e6-3b3f-488c-99d9-d285e5ff195e-cert\") pod \"odh-model-controller-7998c67dc4-zcg5h\" (UID: \"09e730e6-3b3f-488c-99d9-d285e5ff195e\") " pod="opendatahub/odh-model-controller-7998c67dc4-zcg5h"
Sep 16 02:23:41.663353 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:23:41.663316    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-zcg5h"
Sep 16 02:23:41.915589 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:23:41.915541    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 16 02:23:41.915892 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:23:41.915640    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 16 02:23:41.915892 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:23:41.915671    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 16 02:23:42.024638 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:23:42.024601    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-z9zbw" event={"ID":"d128e57b-91ba-4ed8-ab92-837ead863c2d","Type":"ContainerStarted","Data":"d2f8f4b6bdbdb22517461f1c879f802622cd47fea91ea510851136767e5f9b3d"}
Sep 16 02:23:42.045681 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:23:42.045630    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-z9zbw" podStartSLOduration=0.994690403 podStartE2EDuration="4.045612766s" totalImagesPullingTime="3.050922363s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 02:23:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:23:42.045289793 +0000 UTC m=+634.963779044" watchObservedRunningTime="2026-09-16 02:23:42.045612766 +0000 UTC m=+634.964102018"
Sep 16 02:23:42.116133 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:23:42.116107    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-zcg5h"]
Sep 16 02:23:42.118018 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:23:42.117989    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09e730e6_3b3f_488c_99d9_d285e5ff195e.slice/crio-6705ba0b3783555a7f6d78a7094aca76815ca0a1974b13a9d2949bfe5fef40b4 WatchSource:0}: Error finding container 6705ba0b3783555a7f6d78a7094aca76815ca0a1974b13a9d2949bfe5fef40b4: Status 404 returned error can't find the container with id 6705ba0b3783555a7f6d78a7094aca76815ca0a1974b13a9d2949bfe5fef40b4
Sep 16 02:23:42.587567 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:23:42.587531    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-z9zbw"
Sep 16 02:23:42.591855 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:23:42.591831    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-z9zbw"
Sep 16 02:23:43.031363 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:23:43.031320    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-zcg5h" event={"ID":"09e730e6-3b3f-488c-99d9-d285e5ff195e","Type":"ContainerStarted","Data":"6705ba0b3783555a7f6d78a7094aca76815ca0a1974b13a9d2949bfe5fef40b4"}
Sep 16 02:23:43.031848 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:23:43.031566    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-z9zbw"
Sep 16 02:23:43.032931 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:23:43.032909    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-z9zbw"
Sep 16 02:23:45.042077 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:23:45.041994    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-zcg5h" event={"ID":"09e730e6-3b3f-488c-99d9-d285e5ff195e","Type":"ContainerStarted","Data":"8e2c41cab2da27199f900643c149aaf735978cd6b86af505be7ed582aff1759d"}
Sep 16 02:23:45.057350 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:23:45.057290    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-zcg5h" podStartSLOduration=2.502855492 podStartE2EDuration="5.057268393s" totalImagesPullingTime="2.554412901s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 02:23:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:23:45.057252953 +0000 UTC m=+637.975742202" watchObservedRunningTime="2026-09-16 02:23:45.057268393 +0000 UTC m=+637.975757645"
Sep 16 02:23:46.046279 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:23:46.046250    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-zcg5h"
Sep 16 02:23:47.549898 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:23:47.549870    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-6nqh9"]
Sep 16 02:23:47.553332 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:23:47.553316    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-6nqh9"
Sep 16 02:23:47.555805 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:23:47.555774    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Sep 16 02:23:47.555805 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:23:47.555796    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-mrtw8\""
Sep 16 02:23:47.555975 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:23:47.555780    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Sep 16 02:23:47.555975 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:23:47.555834    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Sep 16 02:23:47.567046 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:23:47.567015    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-6nqh9"]
Sep 16 02:23:47.654475 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:23:47.654433    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/dfc5e005-4a80-42db-b611-e4c01f9bbb76-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-6nqh9\" (UID: \"dfc5e005-4a80-42db-b611-e4c01f9bbb76\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-6nqh9"
Sep 16 02:23:47.654475 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:23:47.654472    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/dfc5e005-4a80-42db-b611-e4c01f9bbb76-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-6nqh9\" (UID: \"dfc5e005-4a80-42db-b611-e4c01f9bbb76\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-6nqh9"
Sep 16 02:23:47.654682 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:23:47.654492    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfc5e005-4a80-42db-b611-e4c01f9bbb76-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-6nqh9\" (UID: \"dfc5e005-4a80-42db-b611-e4c01f9bbb76\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-6nqh9"
Sep 16 02:23:47.654682 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:23:47.654507    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbrvs\" (UniqueName: \"kubernetes.io/projected/dfc5e005-4a80-42db-b611-e4c01f9bbb76-kube-api-access-bbrvs\") pod \"istiod-openshift-gateway-5677ccb49b-6nqh9\" (UID: \"dfc5e005-4a80-42db-b611-e4c01f9bbb76\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-6nqh9"
Sep 16 02:23:47.654682 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:23:47.654596    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/dfc5e005-4a80-42db-b611-e4c01f9bbb76-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-6nqh9\" (UID: \"dfc5e005-4a80-42db-b611-e4c01f9bbb76\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-6nqh9"
Sep 16 02:23:47.654682 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:23:47.654626    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/dfc5e005-4a80-42db-b611-e4c01f9bbb76-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-6nqh9\" (UID: \"dfc5e005-4a80-42db-b611-e4c01f9bbb76\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-6nqh9"
Sep 16 02:23:47.654857 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:23:47.654717    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dfc5e005-4a80-42db-b611-e4c01f9bbb76-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-6nqh9\" (UID: \"dfc5e005-4a80-42db-b611-e4c01f9bbb76\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-6nqh9"
Sep 16 02:23:47.755546 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:23:47.755512    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/dfc5e005-4a80-42db-b611-e4c01f9bbb76-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-6nqh9\" (UID: \"dfc5e005-4a80-42db-b611-e4c01f9bbb76\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-6nqh9"
Sep 16 02:23:47.755546 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:23:47.755542    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/dfc5e005-4a80-42db-b611-e4c01f9bbb76-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-6nqh9\" (UID: \"dfc5e005-4a80-42db-b611-e4c01f9bbb76\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-6nqh9"
Sep 16 02:23:47.755798 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:23:47.755578    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dfc5e005-4a80-42db-b611-e4c01f9bbb76-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-6nqh9\" (UID: \"dfc5e005-4a80-42db-b611-e4c01f9bbb76\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-6nqh9"
Sep 16 02:23:47.755798 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:23:47.755611    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/dfc5e005-4a80-42db-b611-e4c01f9bbb76-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-6nqh9\" (UID: \"dfc5e005-4a80-42db-b611-e4c01f9bbb76\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-6nqh9"
Sep 16 02:23:47.755798 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:23:47.755644    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/dfc5e005-4a80-42db-b611-e4c01f9bbb76-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-6nqh9\" (UID: \"dfc5e005-4a80-42db-b611-e4c01f9bbb76\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-6nqh9"
Sep 16 02:23:47.755798 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:23:47.755660    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfc5e005-4a80-42db-b611-e4c01f9bbb76-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-6nqh9\" (UID: \"dfc5e005-4a80-42db-b611-e4c01f9bbb76\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-6nqh9"
Sep 16 02:23:47.755798 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:23:47.755677    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbrvs\" (UniqueName: \"kubernetes.io/projected/dfc5e005-4a80-42db-b611-e4c01f9bbb76-kube-api-access-bbrvs\") pod \"istiod-openshift-gateway-5677ccb49b-6nqh9\" (UID: \"dfc5e005-4a80-42db-b611-e4c01f9bbb76\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-6nqh9"
Sep 16 02:23:47.756474 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:23:47.756441    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/dfc5e005-4a80-42db-b611-e4c01f9bbb76-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-6nqh9\" (UID: \"dfc5e005-4a80-42db-b611-e4c01f9bbb76\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-6nqh9"
Sep 16 02:23:47.757968 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:23:47.757947    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfc5e005-4a80-42db-b611-e4c01f9bbb76-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-6nqh9\" (UID: \"dfc5e005-4a80-42db-b611-e4c01f9bbb76\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-6nqh9"
Sep 16 02:23:47.758068 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:23:47.757985    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/dfc5e005-4a80-42db-b611-e4c01f9bbb76-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-6nqh9\" (UID: \"dfc5e005-4a80-42db-b611-e4c01f9bbb76\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-6nqh9"
Sep 16 02:23:47.758146 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:23:47.758129    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/dfc5e005-4a80-42db-b611-e4c01f9bbb76-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-6nqh9\" (UID: \"dfc5e005-4a80-42db-b611-e4c01f9bbb76\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-6nqh9"
Sep 16 02:23:47.758184 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:23:47.758139    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/dfc5e005-4a80-42db-b611-e4c01f9bbb76-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-6nqh9\" (UID: \"dfc5e005-4a80-42db-b611-e4c01f9bbb76\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-6nqh9"
Sep 16 02:23:47.769398 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:23:47.769378    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dfc5e005-4a80-42db-b611-e4c01f9bbb76-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-6nqh9\" (UID: \"dfc5e005-4a80-42db-b611-e4c01f9bbb76\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-6nqh9"
Sep 16 02:23:47.774468 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:23:47.774445    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbrvs\" (UniqueName: \"kubernetes.io/projected/dfc5e005-4a80-42db-b611-e4c01f9bbb76-kube-api-access-bbrvs\") pod \"istiod-openshift-gateway-5677ccb49b-6nqh9\" (UID: \"dfc5e005-4a80-42db-b611-e4c01f9bbb76\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-6nqh9"
Sep 16 02:23:47.862468 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:23:47.862391    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-6nqh9"
Sep 16 02:23:48.136355 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:23:48.136267    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-6nqh9"]
Sep 16 02:23:48.140135 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:23:48.140095    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfc5e005_4a80_42db_b611_e4c01f9bbb76.slice/crio-cddead51533b7d12dea662fc8c6b57f328e3a8f5e00748fa8bf10456b4d17d63 WatchSource:0}: Error finding container cddead51533b7d12dea662fc8c6b57f328e3a8f5e00748fa8bf10456b4d17d63: Status 404 returned error can't find the container with id cddead51533b7d12dea662fc8c6b57f328e3a8f5e00748fa8bf10456b4d17d63
Sep 16 02:23:49.059269 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:23:49.059227    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-6nqh9" event={"ID":"dfc5e005-4a80-42db-b611-e4c01f9bbb76","Type":"ContainerStarted","Data":"cddead51533b7d12dea662fc8c6b57f328e3a8f5e00748fa8bf10456b4d17d63"}
Sep 16 02:23:50.445448 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:23:50.445411    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 16 02:23:50.445674 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:23:50.445482    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 16 02:23:51.071789 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:23:51.071753    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-6nqh9" event={"ID":"dfc5e005-4a80-42db-b611-e4c01f9bbb76","Type":"ContainerStarted","Data":"32fe019a7b6f66cfadbcf215aed8dc78bdc84461b0d0a4ecc0911d0b881ef121"}
Sep 16 02:23:51.071991 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:23:51.071902    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-6nqh9"
Sep 16 02:23:51.074031 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:23:51.073996    2558 patch_prober.go:28] interesting pod/istiod-openshift-gateway-5677ccb49b-6nqh9 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 16 02:23:51.074175 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:23:51.074059    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-6nqh9" podUID="dfc5e005-4a80-42db-b611-e4c01f9bbb76" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 02:23:51.094366 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:23:51.094292    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-6nqh9" podStartSLOduration=1.791230321 podStartE2EDuration="4.094270289s" totalImagesPullingTime="2.303039968s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 02:23:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:23:51.093442311 +0000 UTC m=+644.011931617" watchObservedRunningTime="2026-09-16 02:23:51.094270289 +0000 UTC m=+644.012759538"
Sep 16 02:23:52.076444 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:23:52.076416    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-6nqh9"
Sep 16 02:23:57.052411 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:23:57.052375    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-zcg5h"
Sep 16 02:25:53.926229 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:25:53.926195    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tc8md"]
Sep 16 02:25:53.930212 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:25:53.930190    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tc8md"
Sep 16 02:25:53.932272 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:25:53.932247    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-zqnvh\""
Sep 16 02:25:53.940873 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:25:53.940847    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tc8md"]
Sep 16 02:25:54.050181 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:25:54.050149    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9bpb4\" (UniqueName: \"kubernetes.io/projected/98d6db2c-d022-45af-89d5-5ba2eb8518b7-kube-api-access-9bpb4\") pod \"maas-default-gateway-openshift-default-78bb78ff78-tc8md\" (UID: \"98d6db2c-d022-45af-89d5-5ba2eb8518b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tc8md"
Sep 16 02:25:54.050181 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:25:54.050185    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98d6db2c-d022-45af-89d5-5ba2eb8518b7-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-tc8md\" (UID: \"98d6db2c-d022-45af-89d5-5ba2eb8518b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tc8md"
Sep 16 02:25:54.050383 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:25:54.050222    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98d6db2c-d022-45af-89d5-5ba2eb8518b7-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-tc8md\" (UID: \"98d6db2c-d022-45af-89d5-5ba2eb8518b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tc8md"
Sep 16 02:25:54.050383 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:25:54.050244    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98d6db2c-d022-45af-89d5-5ba2eb8518b7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-tc8md\" (UID: \"98d6db2c-d022-45af-89d5-5ba2eb8518b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tc8md"
Sep 16 02:25:54.050383 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:25:54.050270    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98d6db2c-d022-45af-89d5-5ba2eb8518b7-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-tc8md\" (UID: \"98d6db2c-d022-45af-89d5-5ba2eb8518b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tc8md"
Sep 16 02:25:54.050383 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:25:54.050345    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98d6db2c-d022-45af-89d5-5ba2eb8518b7-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-tc8md\" (UID: \"98d6db2c-d022-45af-89d5-5ba2eb8518b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tc8md"
Sep 16 02:25:54.050383 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:25:54.050378    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98d6db2c-d022-45af-89d5-5ba2eb8518b7-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-tc8md\" (UID: \"98d6db2c-d022-45af-89d5-5ba2eb8518b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tc8md"
Sep 16 02:25:54.050540 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:25:54.050413    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98d6db2c-d022-45af-89d5-5ba2eb8518b7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-tc8md\" (UID: \"98d6db2c-d022-45af-89d5-5ba2eb8518b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tc8md"
Sep 16 02:25:54.050540 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:25:54.050433    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98d6db2c-d022-45af-89d5-5ba2eb8518b7-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-tc8md\" (UID: \"98d6db2c-d022-45af-89d5-5ba2eb8518b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tc8md"
Sep 16 02:25:54.151360 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:25:54.151316    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98d6db2c-d022-45af-89d5-5ba2eb8518b7-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-tc8md\" (UID: \"98d6db2c-d022-45af-89d5-5ba2eb8518b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tc8md"
Sep 16 02:25:54.151360 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:25:54.151366    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98d6db2c-d022-45af-89d5-5ba2eb8518b7-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-tc8md\" (UID: \"98d6db2c-d022-45af-89d5-5ba2eb8518b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tc8md"
Sep 16 02:25:54.151591 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:25:54.151399    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98d6db2c-d022-45af-89d5-5ba2eb8518b7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-tc8md\" (UID: \"98d6db2c-d022-45af-89d5-5ba2eb8518b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tc8md"
Sep 16 02:25:54.151591 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:25:54.151425    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98d6db2c-d022-45af-89d5-5ba2eb8518b7-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-tc8md\" (UID: \"98d6db2c-d022-45af-89d5-5ba2eb8518b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tc8md"
Sep 16 02:25:54.151591 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:25:54.151456    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bpb4\" (UniqueName: \"kubernetes.io/projected/98d6db2c-d022-45af-89d5-5ba2eb8518b7-kube-api-access-9bpb4\") pod \"maas-default-gateway-openshift-default-78bb78ff78-tc8md\" (UID: \"98d6db2c-d022-45af-89d5-5ba2eb8518b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tc8md"
Sep 16 02:25:54.151591 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:25:54.151484    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98d6db2c-d022-45af-89d5-5ba2eb8518b7-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-tc8md\" (UID: \"98d6db2c-d022-45af-89d5-5ba2eb8518b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tc8md"
Sep 16 02:25:54.151591 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:25:54.151532    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98d6db2c-d022-45af-89d5-5ba2eb8518b7-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-tc8md\" (UID: \"98d6db2c-d022-45af-89d5-5ba2eb8518b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tc8md"
Sep 16 02:25:54.151870 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:25:54.151652    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98d6db2c-d022-45af-89d5-5ba2eb8518b7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-tc8md\" (UID: \"98d6db2c-d022-45af-89d5-5ba2eb8518b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tc8md"
Sep 16 02:25:54.151870 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:25:54.151727    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98d6db2c-d022-45af-89d5-5ba2eb8518b7-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-tc8md\" (UID: \"98d6db2c-d022-45af-89d5-5ba2eb8518b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tc8md"
Sep 16 02:25:54.151870 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:25:54.151845    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98d6db2c-d022-45af-89d5-5ba2eb8518b7-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-tc8md\" (UID: \"98d6db2c-d022-45af-89d5-5ba2eb8518b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tc8md"
Sep 16 02:25:54.151984 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:25:54.151971    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98d6db2c-d022-45af-89d5-5ba2eb8518b7-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-tc8md\" (UID: \"98d6db2c-d022-45af-89d5-5ba2eb8518b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tc8md"
Sep 16 02:25:54.152088 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:25:54.152063    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98d6db2c-d022-45af-89d5-5ba2eb8518b7-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-tc8md\" (UID: \"98d6db2c-d022-45af-89d5-5ba2eb8518b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tc8md"
Sep 16 02:25:54.152156 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:25:54.152140    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98d6db2c-d022-45af-89d5-5ba2eb8518b7-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-tc8md\" (UID: \"98d6db2c-d022-45af-89d5-5ba2eb8518b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tc8md"
Sep 16 02:25:54.152296 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:25:54.152278    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98d6db2c-d022-45af-89d5-5ba2eb8518b7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-tc8md\" (UID: \"98d6db2c-d022-45af-89d5-5ba2eb8518b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tc8md"
Sep 16 02:25:54.153828 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:25:54.153804    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98d6db2c-d022-45af-89d5-5ba2eb8518b7-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-tc8md\" (UID: \"98d6db2c-d022-45af-89d5-5ba2eb8518b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tc8md"
Sep 16 02:25:54.154119 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:25:54.154099    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98d6db2c-d022-45af-89d5-5ba2eb8518b7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-tc8md\" (UID: \"98d6db2c-d022-45af-89d5-5ba2eb8518b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tc8md"
Sep 16 02:25:54.159194 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:25:54.159169    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bpb4\" (UniqueName: \"kubernetes.io/projected/98d6db2c-d022-45af-89d5-5ba2eb8518b7-kube-api-access-9bpb4\") pod \"maas-default-gateway-openshift-default-78bb78ff78-tc8md\" (UID: \"98d6db2c-d022-45af-89d5-5ba2eb8518b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tc8md"
Sep 16 02:25:54.159279 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:25:54.159195    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98d6db2c-d022-45af-89d5-5ba2eb8518b7-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-tc8md\" (UID: \"98d6db2c-d022-45af-89d5-5ba2eb8518b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tc8md"
Sep 16 02:25:54.241345 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:25:54.241262    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tc8md"
Sep 16 02:25:54.513567 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:25:54.513462    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tc8md"]
Sep 16 02:25:54.516136 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:25:54.516111    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98d6db2c_d022_45af_89d5_5ba2eb8518b7.slice/crio-d9488f5d1c926d0719412c08793b31c62dd625afd06d2ad5399c5f538afdf8ff WatchSource:0}: Error finding container d9488f5d1c926d0719412c08793b31c62dd625afd06d2ad5399c5f538afdf8ff: Status 404 returned error can't find the container with id d9488f5d1c926d0719412c08793b31c62dd625afd06d2ad5399c5f538afdf8ff
Sep 16 02:25:54.518148 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:25:54.518106    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 16 02:25:54.518249 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:25:54.518181    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 16 02:25:54.518249 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:25:54.518208    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 16 02:25:54.586363 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:25:54.586333    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tc8md" event={"ID":"98d6db2c-d022-45af-89d5-5ba2eb8518b7","Type":"ContainerStarted","Data":"4a7410c09971f39242dcbae03acd2f1a31ccde0bfaed9c92f2bc27eb66eda2b2"}
Sep 16 02:25:54.586505 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:25:54.586382    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tc8md" event={"ID":"98d6db2c-d022-45af-89d5-5ba2eb8518b7","Type":"ContainerStarted","Data":"d9488f5d1c926d0719412c08793b31c62dd625afd06d2ad5399c5f538afdf8ff"}
Sep 16 02:25:54.605063 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:25:54.605001    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tc8md" podStartSLOduration=1.604981743 podStartE2EDuration="1.604981743s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 02:25:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:25:54.603811201 +0000 UTC m=+767.522300451" watchObservedRunningTime="2026-09-16 02:25:54.604981743 +0000 UTC m=+767.523470993"
Sep 16 02:25:55.241607 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:25:55.241569    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tc8md"
Sep 16 02:25:55.242730 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:25:55.242689    2558 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-tc8md container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.22:15021/healthz/ready\": dial tcp 10.133.0.22:15021: connect: connection refused" start-of-body=
Sep 16 02:25:55.242847 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:25:55.242751    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tc8md" podUID="98d6db2c-d022-45af-89d5-5ba2eb8518b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.22:15021/healthz/ready\": dial tcp 10.133.0.22:15021: connect: connection refused"
Sep 16 02:25:56.245966 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:25:56.245934    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tc8md"
Sep 16 02:25:56.594874 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:25:56.594794    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tc8md"
Sep 16 02:25:56.595634 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:25:56.595616    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tc8md"
Sep 16 02:27:25.261369 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:27:25.261332    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-f5q9h"]
Sep 16 02:27:25.264603 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:27:25.264586    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-f5q9h"
Sep 16 02:27:25.266869 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:27:25.266847    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-zzx6j\""
Sep 16 02:27:25.266995 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:27:25.266876    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Sep 16 02:27:25.276126 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:27:25.276100    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-f5q9h"]
Sep 16 02:27:25.348710 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:27:25.348661    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2vx4\" (UniqueName: \"kubernetes.io/projected/95e64058-6ede-4561-8f90-5f08592ccb6e-kube-api-access-t2vx4\") pod \"payload-processing-5b54b7fbb7-f5q9h\" (UID: \"95e64058-6ede-4561-8f90-5f08592ccb6e\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-f5q9h"
Sep 16 02:27:25.348870 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:27:25.348804    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95e64058-6ede-4561-8f90-5f08592ccb6e-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-f5q9h\" (UID: \"95e64058-6ede-4561-8f90-5f08592ccb6e\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-f5q9h"
Sep 16 02:27:25.450000 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:27:25.449961    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95e64058-6ede-4561-8f90-5f08592ccb6e-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-f5q9h\" (UID: \"95e64058-6ede-4561-8f90-5f08592ccb6e\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-f5q9h"
Sep 16 02:27:25.450173 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:27:25.450016    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2vx4\" (UniqueName: \"kubernetes.io/projected/95e64058-6ede-4561-8f90-5f08592ccb6e-kube-api-access-t2vx4\") pod \"payload-processing-5b54b7fbb7-f5q9h\" (UID: \"95e64058-6ede-4561-8f90-5f08592ccb6e\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-f5q9h"
Sep 16 02:27:25.450688 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:27:25.450669    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95e64058-6ede-4561-8f90-5f08592ccb6e-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-f5q9h\" (UID: \"95e64058-6ede-4561-8f90-5f08592ccb6e\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-f5q9h"
Sep 16 02:27:25.459886 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:27:25.459863    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2vx4\" (UniqueName: \"kubernetes.io/projected/95e64058-6ede-4561-8f90-5f08592ccb6e-kube-api-access-t2vx4\") pod \"payload-processing-5b54b7fbb7-f5q9h\" (UID: \"95e64058-6ede-4561-8f90-5f08592ccb6e\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-f5q9h"
Sep 16 02:27:25.574327 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:27:25.574232    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-f5q9h"
Sep 16 02:27:25.842672 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:27:25.842601    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-f5q9h"]
Sep 16 02:27:25.845654 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:27:25.845626    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95e64058_6ede_4561_8f90_5f08592ccb6e.slice/crio-f945a9404f0807ad2ec0aa3698369dd54615b9f13a705a552ff408f53af8cfd6 WatchSource:0}: Error finding container f945a9404f0807ad2ec0aa3698369dd54615b9f13a705a552ff408f53af8cfd6: Status 404 returned error can't find the container with id f945a9404f0807ad2ec0aa3698369dd54615b9f13a705a552ff408f53af8cfd6
Sep 16 02:27:25.847623 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:27:25.847604    2558 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 02:27:25.989823 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:27:25.989788    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-f5q9h" event={"ID":"95e64058-6ede-4561-8f90-5f08592ccb6e","Type":"ContainerStarted","Data":"f945a9404f0807ad2ec0aa3698369dd54615b9f13a705a552ff408f53af8cfd6"}
Sep 16 02:27:29.006833 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:27:29.006788    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-f5q9h" event={"ID":"95e64058-6ede-4561-8f90-5f08592ccb6e","Type":"ContainerStarted","Data":"2bfcd859905a97dbdec65c8b2b6db5ab08b4c084ac852374eb922bfb6ad42311"}
Sep 16 02:27:29.007228 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:27:29.006902    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-f5q9h"
Sep 16 02:27:29.023493 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:27:29.023444    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5b54b7fbb7-f5q9h" podStartSLOduration=1.28700974 podStartE2EDuration="4.023428032s" totalImagesPullingTime="2.736418292s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 02:27:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:27:29.022283663 +0000 UTC m=+861.940772912" watchObservedRunningTime="2026-09-16 02:27:29.023428032 +0000 UTC m=+861.941917330"
Sep 16 02:27:35.033399 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:27:35.033363    2558 generic.go:356] "Generic (PLEG): container finished" podID="98d6db2c-d022-45af-89d5-5ba2eb8518b7" containerID="4a7410c09971f39242dcbae03acd2f1a31ccde0bfaed9c92f2bc27eb66eda2b2" exitCode=0
Sep 16 02:27:35.033839 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:27:35.033439    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tc8md" event={"ID":"98d6db2c-d022-45af-89d5-5ba2eb8518b7","Type":"ContainerDied","Data":"4a7410c09971f39242dcbae03acd2f1a31ccde0bfaed9c92f2bc27eb66eda2b2"}
Sep 16 02:27:35.033886 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:27:35.033838    2558 scope.go:117] "RemoveContainer" containerID="4a7410c09971f39242dcbae03acd2f1a31ccde0bfaed9c92f2bc27eb66eda2b2"
Sep 16 02:27:35.034836 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:27:35.034806    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 16 02:27:35.034911 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:27:35.034890    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 16 02:27:35.034949 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:27:35.034934    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 16 02:27:36.040224 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:27:36.040185    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tc8md" event={"ID":"98d6db2c-d022-45af-89d5-5ba2eb8518b7","Type":"ContainerStarted","Data":"ca128bdb82a8b1ff24571280dbd319d80588f65e51e2162924eaa7e88886dd11"}
Sep 16 02:27:36.242251 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:27:36.242208    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tc8md"
Sep 16 02:27:36.242441 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:27:36.242418    2558 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-tc8md container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.22:15021/healthz/ready\": dial tcp 10.133.0.22:15021: connect: connection refused" start-of-body=
Sep 16 02:27:36.242494 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:27:36.242474    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tc8md" podUID="98d6db2c-d022-45af-89d5-5ba2eb8518b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.22:15021/healthz/ready\": dial tcp 10.133.0.22:15021: connect: connection refused"
Sep 16 02:27:37.242608 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:27:37.242573    2558 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-tc8md container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.22:15021/healthz/ready\": dial tcp 10.133.0.22:15021: connect: connection refused" start-of-body=
Sep 16 02:27:37.242996 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:27:37.242632    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tc8md" podUID="98d6db2c-d022-45af-89d5-5ba2eb8518b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.22:15021/healthz/ready\": dial tcp 10.133.0.22:15021: connect: connection refused"
Sep 16 02:27:38.241870 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:27:38.241838    2558 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-tc8md container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.22:15021/healthz/ready\": dial tcp 10.133.0.22:15021: connect: connection refused" start-of-body=
Sep 16 02:27:38.242029 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:27:38.241895    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tc8md" podUID="98d6db2c-d022-45af-89d5-5ba2eb8518b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.22:15021/healthz/ready\": dial tcp 10.133.0.22:15021: connect: connection refused"
Sep 16 02:27:39.241782 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:27:39.241743    2558 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-tc8md container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.22:15021/healthz/ready\": dial tcp 10.133.0.22:15021: connect: connection refused" start-of-body=
Sep 16 02:27:39.242181 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:27:39.241803    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tc8md" podUID="98d6db2c-d022-45af-89d5-5ba2eb8518b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.22:15021/healthz/ready\": dial tcp 10.133.0.22:15021: connect: connection refused"
Sep 16 02:27:40.012924 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:27:40.012888    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-f5q9h"
Sep 16 02:27:40.242532 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:27:40.242494    2558 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-tc8md container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.22:15021/healthz/ready\": dial tcp 10.133.0.22:15021: connect: connection refused" start-of-body=
Sep 16 02:27:40.242915 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:27:40.242554    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tc8md" podUID="98d6db2c-d022-45af-89d5-5ba2eb8518b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.22:15021/healthz/ready\": dial tcp 10.133.0.22:15021: connect: connection refused"
Sep 16 02:27:41.246391 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:27:41.246364    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tc8md"
Sep 16 02:27:41.246841 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:27:41.246636    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tc8md"
Sep 16 02:27:41.247284 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:27:41.247268    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tc8md"
Sep 16 02:28:15.271270 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:28:15.271236    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-2hbhv"]
Sep 16 02:28:15.274591 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:28:15.274567    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-2hbhv"
Sep 16 02:28:15.276890 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:28:15.276866    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 16 02:28:15.277000 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:28:15.276878    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 16 02:28:15.277684 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:28:15.277662    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\""
Sep 16 02:28:15.277754 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:28:15.277719    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-lrwjw\""
Sep 16 02:28:15.283152 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:28:15.283126    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-2hbhv"]
Sep 16 02:28:15.303918 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:28:15.303886    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8fec50ca-ea58-4aba-b72a-3db8bf0a0401-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-2hbhv\" (UID: \"8fec50ca-ea58-4aba-b72a-3db8bf0a0401\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-2hbhv"
Sep 16 02:28:15.303918 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:28:15.303923    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8fec50ca-ea58-4aba-b72a-3db8bf0a0401-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-2hbhv\" (UID: \"8fec50ca-ea58-4aba-b72a-3db8bf0a0401\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-2hbhv"
Sep 16 02:28:15.304115 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:28:15.303996    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9d8s\" (UniqueName: \"kubernetes.io/projected/8fec50ca-ea58-4aba-b72a-3db8bf0a0401-kube-api-access-g9d8s\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-2hbhv\" (UID: \"8fec50ca-ea58-4aba-b72a-3db8bf0a0401\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-2hbhv"
Sep 16 02:28:15.304115 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:28:15.304042    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8fec50ca-ea58-4aba-b72a-3db8bf0a0401-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-2hbhv\" (UID: \"8fec50ca-ea58-4aba-b72a-3db8bf0a0401\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-2hbhv"
Sep 16 02:28:15.304115 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:28:15.304061    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8fec50ca-ea58-4aba-b72a-3db8bf0a0401-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-2hbhv\" (UID: \"8fec50ca-ea58-4aba-b72a-3db8bf0a0401\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-2hbhv"
Sep 16 02:28:15.304115 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:28:15.304080    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8fec50ca-ea58-4aba-b72a-3db8bf0a0401-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-2hbhv\" (UID: \"8fec50ca-ea58-4aba-b72a-3db8bf0a0401\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-2hbhv"
Sep 16 02:28:15.405151 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:28:15.405110    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9d8s\" (UniqueName: \"kubernetes.io/projected/8fec50ca-ea58-4aba-b72a-3db8bf0a0401-kube-api-access-g9d8s\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-2hbhv\" (UID: \"8fec50ca-ea58-4aba-b72a-3db8bf0a0401\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-2hbhv"
Sep 16 02:28:15.405327 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:28:15.405190    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8fec50ca-ea58-4aba-b72a-3db8bf0a0401-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-2hbhv\" (UID: \"8fec50ca-ea58-4aba-b72a-3db8bf0a0401\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-2hbhv"
Sep 16 02:28:15.405327 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:28:15.405220    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8fec50ca-ea58-4aba-b72a-3db8bf0a0401-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-2hbhv\" (UID: \"8fec50ca-ea58-4aba-b72a-3db8bf0a0401\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-2hbhv"
Sep 16 02:28:15.405327 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:28:15.405252    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8fec50ca-ea58-4aba-b72a-3db8bf0a0401-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-2hbhv\" (UID: \"8fec50ca-ea58-4aba-b72a-3db8bf0a0401\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-2hbhv"
Sep 16 02:28:15.405327 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:28:15.405292    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8fec50ca-ea58-4aba-b72a-3db8bf0a0401-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-2hbhv\" (UID: \"8fec50ca-ea58-4aba-b72a-3db8bf0a0401\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-2hbhv"
Sep 16 02:28:15.405327 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:28:15.405318    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8fec50ca-ea58-4aba-b72a-3db8bf0a0401-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-2hbhv\" (UID: \"8fec50ca-ea58-4aba-b72a-3db8bf0a0401\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-2hbhv"
Sep 16 02:28:15.405661 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:28:15.405639    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8fec50ca-ea58-4aba-b72a-3db8bf0a0401-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-2hbhv\" (UID: \"8fec50ca-ea58-4aba-b72a-3db8bf0a0401\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-2hbhv"
Sep 16 02:28:15.405661 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:28:15.405654    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8fec50ca-ea58-4aba-b72a-3db8bf0a0401-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-2hbhv\" (UID: \"8fec50ca-ea58-4aba-b72a-3db8bf0a0401\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-2hbhv"
Sep 16 02:28:15.405822 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:28:15.405762    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8fec50ca-ea58-4aba-b72a-3db8bf0a0401-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-2hbhv\" (UID: \"8fec50ca-ea58-4aba-b72a-3db8bf0a0401\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-2hbhv"
Sep 16 02:28:15.407521 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:28:15.407503    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8fec50ca-ea58-4aba-b72a-3db8bf0a0401-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-2hbhv\" (UID: \"8fec50ca-ea58-4aba-b72a-3db8bf0a0401\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-2hbhv"
Sep 16 02:28:15.407884 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:28:15.407862    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8fec50ca-ea58-4aba-b72a-3db8bf0a0401-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-2hbhv\" (UID: \"8fec50ca-ea58-4aba-b72a-3db8bf0a0401\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-2hbhv"
Sep 16 02:28:15.412524 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:28:15.412495    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9d8s\" (UniqueName: \"kubernetes.io/projected/8fec50ca-ea58-4aba-b72a-3db8bf0a0401-kube-api-access-g9d8s\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-2hbhv\" (UID: \"8fec50ca-ea58-4aba-b72a-3db8bf0a0401\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-2hbhv"
Sep 16 02:28:15.586540 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:28:15.586432    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-2hbhv"
Sep 16 02:28:15.856421 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:28:15.856387    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-2hbhv"]
Sep 16 02:28:16.228667 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:28:16.228629    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-2hbhv" event={"ID":"8fec50ca-ea58-4aba-b72a-3db8bf0a0401","Type":"ContainerStarted","Data":"e60ee6dd94ae3698ff7afb904d9f7b5c0f427b87c44c79af244b3c31483c23bc"}
Sep 16 02:28:17.234411 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:28:17.234364    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-2hbhv" event={"ID":"8fec50ca-ea58-4aba-b72a-3db8bf0a0401","Type":"ContainerStarted","Data":"6d6c2a252bbed8893cfefe041f1779ca84474a296ace36bb4aa4485307190de8"}
Sep 16 02:28:25.587160 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:28:25.587126    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-2hbhv"
Sep 16 02:28:25.587677 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:28:25.587187    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-2hbhv"
Sep 16 02:28:25.599277 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:28:25.599256    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-2hbhv"
Sep 16 02:28:25.617086 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:28:25.617037    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-2hbhv" podStartSLOduration=9.587291732 podStartE2EDuration="10.617023853s" totalImagesPullingTime="1.029732121s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 02:28:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:28:17.252534966 +0000 UTC m=+910.171024218" watchObservedRunningTime="2026-09-16 02:28:25.617023853 +0000 UTC m=+918.535513120"
Sep 16 02:28:26.285559 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:28:26.285528    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-2hbhv"
Sep 16 02:28:29.870285 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:28:29.870246    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fx7rrz"]
Sep 16 02:28:29.874253 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:28:29.874221    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fx7rrz"
Sep 16 02:28:29.876526 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:28:29.876500    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Sep 16 02:28:29.884045 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:28:29.884018    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fx7rrz"]
Sep 16 02:28:29.939574 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:28:29.939542    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17b8995b-6c0c-4748-a4e9-9690961ef651-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fx7rrz\" (UID: \"17b8995b-6c0c-4748-a4e9-9690961ef651\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fx7rrz"
Sep 16 02:28:29.939750 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:28:29.939579    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5mzb\" (UniqueName: \"kubernetes.io/projected/17b8995b-6c0c-4748-a4e9-9690961ef651-kube-api-access-b5mzb\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fx7rrz\" (UID: \"17b8995b-6c0c-4748-a4e9-9690961ef651\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fx7rrz"
Sep 16 02:28:29.939750 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:28:29.939602    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17b8995b-6c0c-4748-a4e9-9690961ef651-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fx7rrz\" (UID: \"17b8995b-6c0c-4748-a4e9-9690961ef651\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fx7rrz"
Sep 16 02:28:29.939750 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:28:29.939660    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/17b8995b-6c0c-4748-a4e9-9690961ef651-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fx7rrz\" (UID: \"17b8995b-6c0c-4748-a4e9-9690961ef651\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fx7rrz"
Sep 16 02:28:29.939750 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:28:29.939719    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17b8995b-6c0c-4748-a4e9-9690961ef651-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fx7rrz\" (UID: \"17b8995b-6c0c-4748-a4e9-9690961ef651\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fx7rrz"
Sep 16 02:28:29.939902 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:28:29.939767    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/17b8995b-6c0c-4748-a4e9-9690961ef651-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fx7rrz\" (UID: \"17b8995b-6c0c-4748-a4e9-9690961ef651\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fx7rrz"
Sep 16 02:28:30.040649 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:28:30.040616    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17b8995b-6c0c-4748-a4e9-9690961ef651-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fx7rrz\" (UID: \"17b8995b-6c0c-4748-a4e9-9690961ef651\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fx7rrz"
Sep 16 02:28:30.040840 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:28:30.040653    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5mzb\" (UniqueName: \"kubernetes.io/projected/17b8995b-6c0c-4748-a4e9-9690961ef651-kube-api-access-b5mzb\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fx7rrz\" (UID: \"17b8995b-6c0c-4748-a4e9-9690961ef651\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fx7rrz"
Sep 16 02:28:30.040840 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:28:30.040684    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17b8995b-6c0c-4748-a4e9-9690961ef651-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fx7rrz\" (UID: \"17b8995b-6c0c-4748-a4e9-9690961ef651\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fx7rrz"
Sep 16 02:28:30.040840 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:28:30.040738    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/17b8995b-6c0c-4748-a4e9-9690961ef651-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fx7rrz\" (UID: \"17b8995b-6c0c-4748-a4e9-9690961ef651\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fx7rrz"
Sep 16 02:28:30.040840 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:28:30.040785    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17b8995b-6c0c-4748-a4e9-9690961ef651-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fx7rrz\" (UID: \"17b8995b-6c0c-4748-a4e9-9690961ef651\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fx7rrz"
Sep 16 02:28:30.041041 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:28:30.040845    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/17b8995b-6c0c-4748-a4e9-9690961ef651-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fx7rrz\" (UID: \"17b8995b-6c0c-4748-a4e9-9690961ef651\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fx7rrz"
Sep 16 02:28:30.041106 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:28:30.041047    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17b8995b-6c0c-4748-a4e9-9690961ef651-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fx7rrz\" (UID: \"17b8995b-6c0c-4748-a4e9-9690961ef651\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fx7rrz"
Sep 16 02:28:30.041191 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:28:30.041171    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/17b8995b-6c0c-4748-a4e9-9690961ef651-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fx7rrz\" (UID: \"17b8995b-6c0c-4748-a4e9-9690961ef651\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fx7rrz"
Sep 16 02:28:30.041245 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:28:30.041174    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17b8995b-6c0c-4748-a4e9-9690961ef651-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fx7rrz\" (UID: \"17b8995b-6c0c-4748-a4e9-9690961ef651\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fx7rrz"
Sep 16 02:28:30.043021 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:28:30.043002    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/17b8995b-6c0c-4748-a4e9-9690961ef651-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fx7rrz\" (UID: \"17b8995b-6c0c-4748-a4e9-9690961ef651\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fx7rrz"
Sep 16 02:28:30.043397 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:28:30.043374    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17b8995b-6c0c-4748-a4e9-9690961ef651-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fx7rrz\" (UID: \"17b8995b-6c0c-4748-a4e9-9690961ef651\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fx7rrz"
Sep 16 02:28:30.048267 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:28:30.048244    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5mzb\" (UniqueName: \"kubernetes.io/projected/17b8995b-6c0c-4748-a4e9-9690961ef651-kube-api-access-b5mzb\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fx7rrz\" (UID: \"17b8995b-6c0c-4748-a4e9-9690961ef651\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fx7rrz"
Sep 16 02:28:30.186680 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:28:30.186650    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fx7rrz"
Sep 16 02:28:30.455341 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:28:30.455320    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fx7rrz"]
Sep 16 02:28:30.457554 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:28:30.457524    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17b8995b_6c0c_4748_a4e9_9690961ef651.slice/crio-a086e929d7b231b138cd648aab4ebe070a9b1b88a68704a81b073251af50cbed WatchSource:0}: Error finding container a086e929d7b231b138cd648aab4ebe070a9b1b88a68704a81b073251af50cbed: Status 404 returned error can't find the container with id a086e929d7b231b138cd648aab4ebe070a9b1b88a68704a81b073251af50cbed
Sep 16 02:28:31.297081 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:28:31.297047    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fx7rrz" event={"ID":"17b8995b-6c0c-4748-a4e9-9690961ef651","Type":"ContainerStarted","Data":"72d3bf0aea1798447aef7be4b1a776066e3e576e54f7642ec5c9447fe2a71f73"}
Sep 16 02:28:31.297448 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:28:31.297104    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fx7rrz" event={"ID":"17b8995b-6c0c-4748-a4e9-9690961ef651","Type":"ContainerStarted","Data":"a086e929d7b231b138cd648aab4ebe070a9b1b88a68704a81b073251af50cbed"}
Sep 16 02:28:31.312017 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:28:31.311967    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fx7rrz" podStartSLOduration=2.152840439 podStartE2EDuration="2.311928301s" totalImagesPullingTime="159.087862ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 02:28:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:28:31.311245845 +0000 UTC m=+924.229735093" watchObservedRunningTime="2026-09-16 02:28:31.311928301 +0000 UTC m=+924.230417562"
Sep 16 02:28:40.187436 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:28:40.187401    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fx7rrz"
Sep 16 02:28:40.187830 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:28:40.187480    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fx7rrz"
Sep 16 02:28:40.199788 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:28:40.199760    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fx7rrz"
Sep 16 02:28:40.351964 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:28:40.351934    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fx7rrz"
Sep 16 02:29:59.749811 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:29:59.749732    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-f5q9h"]
Sep 16 02:29:59.750323 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:29:59.749945    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-5b54b7fbb7-f5q9h" podUID="95e64058-6ede-4561-8f90-5f08592ccb6e" containerName="payload-processing" containerID="cri-o://2bfcd859905a97dbdec65c8b2b6db5ab08b4c084ac852374eb922bfb6ad42311" gracePeriod=30
Sep 16 02:29:59.991109 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:29:59.991085    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-f5q9h"
Sep 16 02:30:00.041665 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:30:00.041582    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t2vx4\" (UniqueName: \"kubernetes.io/projected/95e64058-6ede-4561-8f90-5f08592ccb6e-kube-api-access-t2vx4\") pod \"95e64058-6ede-4561-8f90-5f08592ccb6e\" (UID: \"95e64058-6ede-4561-8f90-5f08592ccb6e\") "
Sep 16 02:30:00.041665 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:30:00.041616    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95e64058-6ede-4561-8f90-5f08592ccb6e-plugins-config-volume\") pod \"95e64058-6ede-4561-8f90-5f08592ccb6e\" (UID: \"95e64058-6ede-4561-8f90-5f08592ccb6e\") "
Sep 16 02:30:00.042078 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:30:00.042047    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95e64058-6ede-4561-8f90-5f08592ccb6e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "95e64058-6ede-4561-8f90-5f08592ccb6e" (UID: "95e64058-6ede-4561-8f90-5f08592ccb6e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 02:30:00.043886 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:30:00.043859    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95e64058-6ede-4561-8f90-5f08592ccb6e-kube-api-access-t2vx4" (OuterVolumeSpecName: "kube-api-access-t2vx4") pod "95e64058-6ede-4561-8f90-5f08592ccb6e" (UID: "95e64058-6ede-4561-8f90-5f08592ccb6e"). InnerVolumeSpecName "kube-api-access-t2vx4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 02:30:00.142775 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:30:00.142738    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29825430-gvllf"]
Sep 16 02:30:00.143839 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:30:00.143812    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="95e64058-6ede-4561-8f90-5f08592ccb6e" containerName="payload-processing"
Sep 16 02:30:00.151962 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:30:00.151662    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="95e64058-6ede-4561-8f90-5f08592ccb6e" containerName="payload-processing"
Sep 16 02:30:00.151962 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:30:00.146736    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t2vx4\" (UniqueName: \"kubernetes.io/projected/95e64058-6ede-4561-8f90-5f08592ccb6e-kube-api-access-t2vx4\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:30:00.151962 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:30:00.151841    2558 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95e64058-6ede-4561-8f90-5f08592ccb6e-plugins-config-volume\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:30:00.152216 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:30:00.151981    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="95e64058-6ede-4561-8f90-5f08592ccb6e" containerName="payload-processing"
Sep 16 02:30:00.156021 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:30:00.155993    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29825430-gvllf"]
Sep 16 02:30:00.156272 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:30:00.156249    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29825430-gvllf"
Sep 16 02:30:00.158647 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:30:00.158621    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jpd4f\""
Sep 16 02:30:00.168307 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:30:00.168282    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29825430-gvllf"
Sep 16 02:30:00.450441 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:30:00.450413    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29825430-gvllf"]
Sep 16 02:30:00.453096 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:30:00.453069    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bf4750b_8209_41b1_ab14_c96399f6996f.slice/crio-2788d3e3781f9ef8bddee7fae00b6bfc890b569ac08be76db7f3402933b4c878 WatchSource:0}: Error finding container 2788d3e3781f9ef8bddee7fae00b6bfc890b569ac08be76db7f3402933b4c878: Status 404 returned error can't find the container with id 2788d3e3781f9ef8bddee7fae00b6bfc890b569ac08be76db7f3402933b4c878
Sep 16 02:30:00.681761 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:30:00.681722    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29825430-gvllf" event={"ID":"3bf4750b-8209-41b1-ab14-c96399f6996f","Type":"ContainerStarted","Data":"2788d3e3781f9ef8bddee7fae00b6bfc890b569ac08be76db7f3402933b4c878"}
Sep 16 02:30:00.682889 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:30:00.682864    2558 generic.go:356] "Generic (PLEG): container finished" podID="95e64058-6ede-4561-8f90-5f08592ccb6e" containerID="2bfcd859905a97dbdec65c8b2b6db5ab08b4c084ac852374eb922bfb6ad42311" exitCode=0
Sep 16 02:30:00.682965 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:30:00.682900    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-f5q9h" event={"ID":"95e64058-6ede-4561-8f90-5f08592ccb6e","Type":"ContainerDied","Data":"2bfcd859905a97dbdec65c8b2b6db5ab08b4c084ac852374eb922bfb6ad42311"}
Sep 16 02:30:00.682965 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:30:00.682921    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-f5q9h"
Sep 16 02:30:00.682965 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:30:00.682935    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-f5q9h" event={"ID":"95e64058-6ede-4561-8f90-5f08592ccb6e","Type":"ContainerDied","Data":"f945a9404f0807ad2ec0aa3698369dd54615b9f13a705a552ff408f53af8cfd6"}
Sep 16 02:30:00.682965 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:30:00.682956    2558 scope.go:117] "RemoveContainer" containerID="2bfcd859905a97dbdec65c8b2b6db5ab08b4c084ac852374eb922bfb6ad42311"
Sep 16 02:30:00.691557 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:30:00.691529    2558 scope.go:117] "RemoveContainer" containerID="2bfcd859905a97dbdec65c8b2b6db5ab08b4c084ac852374eb922bfb6ad42311"
Sep 16 02:30:00.691918 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:30:00.691896    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2bfcd859905a97dbdec65c8b2b6db5ab08b4c084ac852374eb922bfb6ad42311\": container with ID starting with 2bfcd859905a97dbdec65c8b2b6db5ab08b4c084ac852374eb922bfb6ad42311 not found: ID does not exist" containerID="2bfcd859905a97dbdec65c8b2b6db5ab08b4c084ac852374eb922bfb6ad42311"
Sep 16 02:30:00.691996 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:30:00.691926    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2bfcd859905a97dbdec65c8b2b6db5ab08b4c084ac852374eb922bfb6ad42311"} err="failed to get container status \"2bfcd859905a97dbdec65c8b2b6db5ab08b4c084ac852374eb922bfb6ad42311\": rpc error: code = NotFound desc = could not find container \"2bfcd859905a97dbdec65c8b2b6db5ab08b4c084ac852374eb922bfb6ad42311\": container with ID starting with 2bfcd859905a97dbdec65c8b2b6db5ab08b4c084ac852374eb922bfb6ad42311 not found: ID does not exist"
Sep 16 02:30:00.703346 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:30:00.703269    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-f5q9h"]
Sep 16 02:30:00.707212 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:30:00.707177    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-f5q9h"]
Sep 16 02:30:01.549814 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:30:01.549782    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="95e64058-6ede-4561-8f90-5f08592ccb6e" path="/var/lib/kubelet/pods/95e64058-6ede-4561-8f90-5f08592ccb6e/volumes"
Sep 16 02:30:03.698879 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:30:03.698837    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29825430-gvllf" event={"ID":"3bf4750b-8209-41b1-ab14-c96399f6996f","Type":"ContainerStarted","Data":"e42099955077bf7ea811f46415554e7c39301fb13a2e8c2bac1950b6d9feecdd"}
Sep 16 02:30:03.712267 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:30:03.712145    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29825430-gvllf" podStartSLOduration=0.866664319 podStartE2EDuration="3.712123781s" totalImagesPullingTime="2.845459462s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 02:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:30:03.711946925 +0000 UTC m=+1016.630436174" watchObservedRunningTime="2026-09-16 02:30:03.712123781 +0000 UTC m=+1016.630613128"
Sep 16 02:30:04.705908 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:30:04.705870    2558 generic.go:356] "Generic (PLEG): container finished" podID="3bf4750b-8209-41b1-ab14-c96399f6996f" containerID="e42099955077bf7ea811f46415554e7c39301fb13a2e8c2bac1950b6d9feecdd" exitCode=0
Sep 16 02:30:04.706307 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:30:04.705915    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29825430-gvllf" event={"ID":"3bf4750b-8209-41b1-ab14-c96399f6996f","Type":"ContainerDied","Data":"e42099955077bf7ea811f46415554e7c39301fb13a2e8c2bac1950b6d9feecdd"}
Sep 16 02:30:05.843460 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:30:05.843432    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29825430-gvllf"
Sep 16 02:30:06.716010 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:30:06.715979    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29825430-gvllf"
Sep 16 02:30:06.716210 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:30:06.716008    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29825430-gvllf" event={"ID":"3bf4750b-8209-41b1-ab14-c96399f6996f","Type":"ContainerDied","Data":"2788d3e3781f9ef8bddee7fae00b6bfc890b569ac08be76db7f3402933b4c878"}
Sep 16 02:30:06.716210 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:30:06.716052    2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2788d3e3781f9ef8bddee7fae00b6bfc890b569ac08be76db7f3402933b4c878"
Sep 16 02:30:14.495178 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:30:14.495144    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-857878477c-5mzgh"]
Sep 16 02:30:14.495547 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:30:14.495463    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3bf4750b-8209-41b1-ab14-c96399f6996f" containerName="cleanup"
Sep 16 02:30:14.495547 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:30:14.495474    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="3bf4750b-8209-41b1-ab14-c96399f6996f" containerName="cleanup"
Sep 16 02:30:14.495620 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:30:14.495557    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="3bf4750b-8209-41b1-ab14-c96399f6996f" containerName="cleanup"
Sep 16 02:30:14.499567 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:30:14.499550    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-857878477c-5mzgh"
Sep 16 02:30:14.501893 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:30:14.501872    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\""
Sep 16 02:30:14.502002 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:30:14.501916    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-tbxsd\""
Sep 16 02:30:14.508952 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:30:14.508928    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-857878477c-5mzgh"]
Sep 16 02:30:14.583167 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:30:14.583133    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7dbb\" (UniqueName: \"kubernetes.io/projected/936ca1de-ac51-45d1-916d-3c2cdef147b6-kube-api-access-w7dbb\") pod \"ai-gateway-controller-857878477c-5mzgh\" (UID: \"936ca1de-ac51-45d1-916d-3c2cdef147b6\") " pod="opendatahub/ai-gateway-controller-857878477c-5mzgh"
Sep 16 02:30:14.684513 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:30:14.684475    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7dbb\" (UniqueName: \"kubernetes.io/projected/936ca1de-ac51-45d1-916d-3c2cdef147b6-kube-api-access-w7dbb\") pod \"ai-gateway-controller-857878477c-5mzgh\" (UID: \"936ca1de-ac51-45d1-916d-3c2cdef147b6\") " pod="opendatahub/ai-gateway-controller-857878477c-5mzgh"
Sep 16 02:30:14.692878 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:30:14.692856    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7dbb\" (UniqueName: \"kubernetes.io/projected/936ca1de-ac51-45d1-916d-3c2cdef147b6-kube-api-access-w7dbb\") pod \"ai-gateway-controller-857878477c-5mzgh\" (UID: \"936ca1de-ac51-45d1-916d-3c2cdef147b6\") " pod="opendatahub/ai-gateway-controller-857878477c-5mzgh"
Sep 16 02:30:14.810424 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:30:14.810346    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-857878477c-5mzgh"
Sep 16 02:30:15.076910 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:30:15.076884    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-857878477c-5mzgh"]
Sep 16 02:30:15.079445 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:30:15.079418    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod936ca1de_ac51_45d1_916d_3c2cdef147b6.slice/crio-fdfbccc02ae865cdb92c5f9de145cdf8b496477a45cd83e5f1edea4c5b81ad9f WatchSource:0}: Error finding container fdfbccc02ae865cdb92c5f9de145cdf8b496477a45cd83e5f1edea4c5b81ad9f: Status 404 returned error can't find the container with id fdfbccc02ae865cdb92c5f9de145cdf8b496477a45cd83e5f1edea4c5b81ad9f
Sep 16 02:30:15.760523 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:30:15.760479    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-857878477c-5mzgh" event={"ID":"936ca1de-ac51-45d1-916d-3c2cdef147b6","Type":"ContainerStarted","Data":"fdfbccc02ae865cdb92c5f9de145cdf8b496477a45cd83e5f1edea4c5b81ad9f"}
Sep 16 02:30:17.772826 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:30:17.772785    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-857878477c-5mzgh" event={"ID":"936ca1de-ac51-45d1-916d-3c2cdef147b6","Type":"ContainerStarted","Data":"3ae7d7e3fd301718ca11988875dc374f4aae5ff36df29c91f60d8b8f4ae5231c"}
Sep 16 02:30:17.773237 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:30:17.772908    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-857878477c-5mzgh"
Sep 16 02:30:17.786605 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:30:17.786569    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5d7dbbd56b-pltkr"]
Sep 16 02:30:17.790630 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:30:17.790602    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5d7dbbd56b-pltkr"
Sep 16 02:30:17.792907 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:30:17.792881    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Sep 16 02:30:17.793651 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:30:17.793627    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-zzx6j\""
Sep 16 02:30:17.794569 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:30:17.794516    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-857878477c-5mzgh" podStartSLOduration=1.501041513 podStartE2EDuration="3.794502683s" totalImagesPullingTime="2.29346117s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 02:30:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:30:17.793666425 +0000 UTC m=+1030.712155704" watchObservedRunningTime="2026-09-16 02:30:17.794502683 +0000 UTC m=+1030.712991932"
Sep 16 02:30:17.803512 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:30:17.803489    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5d7dbbd56b-pltkr"]
Sep 16 02:30:17.918671 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:30:17.918636    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4189cfcd-c3ba-4bad-813f-5a6851cb5317-plugins-config-volume\") pod \"payload-processing-5d7dbbd56b-pltkr\" (UID: \"4189cfcd-c3ba-4bad-813f-5a6851cb5317\") " pod="openshift-ingress/payload-processing-5d7dbbd56b-pltkr"
Sep 16 02:30:17.918836 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:30:17.918781    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-csnvf\" (UniqueName: \"kubernetes.io/projected/4189cfcd-c3ba-4bad-813f-5a6851cb5317-kube-api-access-csnvf\") pod \"payload-processing-5d7dbbd56b-pltkr\" (UID: \"4189cfcd-c3ba-4bad-813f-5a6851cb5317\") " pod="openshift-ingress/payload-processing-5d7dbbd56b-pltkr"
Sep 16 02:30:18.020159 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:30:18.020121    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-csnvf\" (UniqueName: \"kubernetes.io/projected/4189cfcd-c3ba-4bad-813f-5a6851cb5317-kube-api-access-csnvf\") pod \"payload-processing-5d7dbbd56b-pltkr\" (UID: \"4189cfcd-c3ba-4bad-813f-5a6851cb5317\") " pod="openshift-ingress/payload-processing-5d7dbbd56b-pltkr"
Sep 16 02:30:18.020332 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:30:18.020189    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4189cfcd-c3ba-4bad-813f-5a6851cb5317-plugins-config-volume\") pod \"payload-processing-5d7dbbd56b-pltkr\" (UID: \"4189cfcd-c3ba-4bad-813f-5a6851cb5317\") " pod="openshift-ingress/payload-processing-5d7dbbd56b-pltkr"
Sep 16 02:30:18.021118 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:30:18.021092    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4189cfcd-c3ba-4bad-813f-5a6851cb5317-plugins-config-volume\") pod \"payload-processing-5d7dbbd56b-pltkr\" (UID: \"4189cfcd-c3ba-4bad-813f-5a6851cb5317\") " pod="openshift-ingress/payload-processing-5d7dbbd56b-pltkr"
Sep 16 02:30:18.028108 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:30:18.028036    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-csnvf\" (UniqueName: \"kubernetes.io/projected/4189cfcd-c3ba-4bad-813f-5a6851cb5317-kube-api-access-csnvf\") pod \"payload-processing-5d7dbbd56b-pltkr\" (UID: \"4189cfcd-c3ba-4bad-813f-5a6851cb5317\") " pod="openshift-ingress/payload-processing-5d7dbbd56b-pltkr"
Sep 16 02:30:18.104112 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:30:18.104060    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5d7dbbd56b-pltkr"
Sep 16 02:30:18.373623 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:30:18.373602    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5d7dbbd56b-pltkr"]
Sep 16 02:30:18.375889 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:30:18.375860    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4189cfcd_c3ba_4bad_813f_5a6851cb5317.slice/crio-5bd0b65b7ffe9917a4101e1b6f3aa0ba1767a44185fa2fb74bb7f1dd439887ab WatchSource:0}: Error finding container 5bd0b65b7ffe9917a4101e1b6f3aa0ba1767a44185fa2fb74bb7f1dd439887ab: Status 404 returned error can't find the container with id 5bd0b65b7ffe9917a4101e1b6f3aa0ba1767a44185fa2fb74bb7f1dd439887ab
Sep 16 02:30:18.778880 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:30:18.778844    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5d7dbbd56b-pltkr" event={"ID":"4189cfcd-c3ba-4bad-813f-5a6851cb5317","Type":"ContainerStarted","Data":"5bd0b65b7ffe9917a4101e1b6f3aa0ba1767a44185fa2fb74bb7f1dd439887ab"}
Sep 16 02:30:19.784925 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:30:19.784895    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5d7dbbd56b-pltkr" event={"ID":"4189cfcd-c3ba-4bad-813f-5a6851cb5317","Type":"ContainerStarted","Data":"dea8c6184f50d4c7d5d372637e3afeff3f9d5a134440ec21488ec54e9486a179"}
Sep 16 02:30:19.785296 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:30:19.784969    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5d7dbbd56b-pltkr"
Sep 16 02:30:19.801096 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:30:19.801050    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5d7dbbd56b-pltkr" podStartSLOduration=1.814727549 podStartE2EDuration="2.801034366s" totalImagesPullingTime="986.306817ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 02:30:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:30:19.798957041 +0000 UTC m=+1032.717446290" watchObservedRunningTime="2026-09-16 02:30:19.801034366 +0000 UTC m=+1032.719523614"
Sep 16 02:30:28.780936 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:30:28.780903    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-857878477c-5mzgh"
Sep 16 02:30:28.954543 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:30:28.954509    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-ff88c4765-6bc9x"]
Sep 16 02:30:28.957852 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:30:28.957834    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-ff88c4765-6bc9x"
Sep 16 02:30:28.966871 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:30:28.966842    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-ff88c4765-6bc9x"]
Sep 16 02:30:29.124254 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:30:29.124157    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qvp8\" (UniqueName: \"kubernetes.io/projected/4a3a2d7f-ef64-4b21-b1d6-0bb19ff6eb0e-kube-api-access-9qvp8\") pod \"ai-gateway-controller-ff88c4765-6bc9x\" (UID: \"4a3a2d7f-ef64-4b21-b1d6-0bb19ff6eb0e\") " pod="opendatahub/ai-gateway-controller-ff88c4765-6bc9x"
Sep 16 02:30:29.225122 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:30:29.225082    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qvp8\" (UniqueName: \"kubernetes.io/projected/4a3a2d7f-ef64-4b21-b1d6-0bb19ff6eb0e-kube-api-access-9qvp8\") pod \"ai-gateway-controller-ff88c4765-6bc9x\" (UID: \"4a3a2d7f-ef64-4b21-b1d6-0bb19ff6eb0e\") " pod="opendatahub/ai-gateway-controller-ff88c4765-6bc9x"
Sep 16 02:30:29.233036 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:30:29.233011    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qvp8\" (UniqueName: \"kubernetes.io/projected/4a3a2d7f-ef64-4b21-b1d6-0bb19ff6eb0e-kube-api-access-9qvp8\") pod \"ai-gateway-controller-ff88c4765-6bc9x\" (UID: \"4a3a2d7f-ef64-4b21-b1d6-0bb19ff6eb0e\") " pod="opendatahub/ai-gateway-controller-ff88c4765-6bc9x"
Sep 16 02:30:29.268196 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:30:29.268172    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-ff88c4765-6bc9x"
Sep 16 02:30:29.533803 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:30:29.533777    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-ff88c4765-6bc9x"]
Sep 16 02:30:29.536089 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:30:29.536066    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a3a2d7f_ef64_4b21_b1d6_0bb19ff6eb0e.slice/crio-945402c720ea6c57ed40b2287b38a2965a14f1c416876132383657f64cf96227 WatchSource:0}: Error finding container 945402c720ea6c57ed40b2287b38a2965a14f1c416876132383657f64cf96227: Status 404 returned error can't find the container with id 945402c720ea6c57ed40b2287b38a2965a14f1c416876132383657f64cf96227
Sep 16 02:30:29.831211 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:30:29.831187    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-ff88c4765-6bc9x" event={"ID":"4a3a2d7f-ef64-4b21-b1d6-0bb19ff6eb0e","Type":"ContainerStarted","Data":"945402c720ea6c57ed40b2287b38a2965a14f1c416876132383657f64cf96227"}
Sep 16 02:30:30.793952 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:30:30.793926    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5d7dbbd56b-pltkr"
Sep 16 02:30:30.794302 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:30:30.794285    2558 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-processing-5d7dbbd56b-pltkr" secret="" err="secret \"payload-processing-dockercfg-zzx6j\" not found"
Sep 16 02:30:30.838438 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:30:30.838392    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-ff88c4765-6bc9x" event={"ID":"4a3a2d7f-ef64-4b21-b1d6-0bb19ff6eb0e","Type":"ContainerStarted","Data":"268ef666cb2b0546ddc353c01daf2e2584e8b5a3d24525f7c3cc53c907f416af"}
Sep 16 02:30:30.838981 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:30:30.838466    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-ff88c4765-6bc9x"
Sep 16 02:30:30.853797 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:30:30.853749    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-ff88c4765-6bc9x" podStartSLOduration=2.583808814 podStartE2EDuration="2.85373421s" totalImagesPullingTime="269.925396ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 02:30:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:30:30.851950366 +0000 UTC m=+1043.770439614" watchObservedRunningTime="2026-09-16 02:30:30.85373421 +0000 UTC m=+1043.772223458"
Sep 16 02:30:41.850099 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:30:41.850064    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-ff88c4765-6bc9x"
Sep 16 02:30:41.890105 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:30:41.890068    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-857878477c-5mzgh"]
Sep 16 02:30:41.890300 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:30:41.890263    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/ai-gateway-controller-857878477c-5mzgh" podUID="936ca1de-ac51-45d1-916d-3c2cdef147b6" containerName="manager" containerID="cri-o://3ae7d7e3fd301718ca11988875dc374f4aae5ff36df29c91f60d8b8f4ae5231c" gracePeriod=30
Sep 16 02:30:42.133956 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:30:42.133925    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-857878477c-5mzgh"
Sep 16 02:30:42.241555 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:30:42.241513    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w7dbb\" (UniqueName: \"kubernetes.io/projected/936ca1de-ac51-45d1-916d-3c2cdef147b6-kube-api-access-w7dbb\") pod \"936ca1de-ac51-45d1-916d-3c2cdef147b6\" (UID: \"936ca1de-ac51-45d1-916d-3c2cdef147b6\") "
Sep 16 02:30:42.243637 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:30:42.243612    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/936ca1de-ac51-45d1-916d-3c2cdef147b6-kube-api-access-w7dbb" (OuterVolumeSpecName: "kube-api-access-w7dbb") pod "936ca1de-ac51-45d1-916d-3c2cdef147b6" (UID: "936ca1de-ac51-45d1-916d-3c2cdef147b6"). InnerVolumeSpecName "kube-api-access-w7dbb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 02:30:42.342968 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:30:42.342939    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w7dbb\" (UniqueName: \"kubernetes.io/projected/936ca1de-ac51-45d1-916d-3c2cdef147b6-kube-api-access-w7dbb\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:30:42.897743 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:30:42.897687    2558 generic.go:356] "Generic (PLEG): container finished" podID="936ca1de-ac51-45d1-916d-3c2cdef147b6" containerID="3ae7d7e3fd301718ca11988875dc374f4aae5ff36df29c91f60d8b8f4ae5231c" exitCode=0
Sep 16 02:30:42.897743 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:30:42.897744    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-857878477c-5mzgh"
Sep 16 02:30:42.898183 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:30:42.897770    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-857878477c-5mzgh" event={"ID":"936ca1de-ac51-45d1-916d-3c2cdef147b6","Type":"ContainerDied","Data":"3ae7d7e3fd301718ca11988875dc374f4aae5ff36df29c91f60d8b8f4ae5231c"}
Sep 16 02:30:42.898183 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:30:42.897821    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-857878477c-5mzgh" event={"ID":"936ca1de-ac51-45d1-916d-3c2cdef147b6","Type":"ContainerDied","Data":"fdfbccc02ae865cdb92c5f9de145cdf8b496477a45cd83e5f1edea4c5b81ad9f"}
Sep 16 02:30:42.898183 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:30:42.897843    2558 scope.go:117] "RemoveContainer" containerID="3ae7d7e3fd301718ca11988875dc374f4aae5ff36df29c91f60d8b8f4ae5231c"
Sep 16 02:30:42.909813 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:30:42.909789    2558 scope.go:117] "RemoveContainer" containerID="3ae7d7e3fd301718ca11988875dc374f4aae5ff36df29c91f60d8b8f4ae5231c"
Sep 16 02:30:42.910224 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:30:42.910191    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ae7d7e3fd301718ca11988875dc374f4aae5ff36df29c91f60d8b8f4ae5231c\": container with ID starting with 3ae7d7e3fd301718ca11988875dc374f4aae5ff36df29c91f60d8b8f4ae5231c not found: ID does not exist" containerID="3ae7d7e3fd301718ca11988875dc374f4aae5ff36df29c91f60d8b8f4ae5231c"
Sep 16 02:30:42.910327 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:30:42.910234    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ae7d7e3fd301718ca11988875dc374f4aae5ff36df29c91f60d8b8f4ae5231c"} err="failed to get container status \"3ae7d7e3fd301718ca11988875dc374f4aae5ff36df29c91f60d8b8f4ae5231c\": rpc error: code = NotFound desc = could not find container \"3ae7d7e3fd301718ca11988875dc374f4aae5ff36df29c91f60d8b8f4ae5231c\": container with ID starting with 3ae7d7e3fd301718ca11988875dc374f4aae5ff36df29c91f60d8b8f4ae5231c not found: ID does not exist"
Sep 16 02:30:42.921984 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:30:42.921958    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-857878477c-5mzgh"]
Sep 16 02:30:42.925651 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:30:42.925628    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/ai-gateway-controller-857878477c-5mzgh"]
Sep 16 02:30:43.552273 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:30:43.552240    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="936ca1de-ac51-45d1-916d-3c2cdef147b6" path="/var/lib/kubelet/pods/936ca1de-ac51-45d1-916d-3c2cdef147b6/volumes"
Sep 16 02:31:01.560487 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:01.560399    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-60674e-openshift-default-6c56877b8b-qc47f"]
Sep 16 02:31:01.561145 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:01.560773    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="936ca1de-ac51-45d1-916d-3c2cdef147b6" containerName="manager"
Sep 16 02:31:01.561145 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:01.560785    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="936ca1de-ac51-45d1-916d-3c2cdef147b6" containerName="manager"
Sep 16 02:31:01.561145 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:01.560850    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="936ca1de-ac51-45d1-916d-3c2cdef147b6" containerName="manager"
Sep 16 02:31:01.567312 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:01.567287    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-60674e-openshift-default-6c56877b8b-qc47f"
Sep 16 02:31:01.569656 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:01.569630    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-60674e-openshift-default-dockercfg-mlzjz\""
Sep 16 02:31:01.577042 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:01.577019    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-60674e-openshift-default-6c56877b8b-qc47f"]
Sep 16 02:31:01.621714 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:01.621668    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-542ed5-openshift-default-7dc67cb65b-gcjgl"]
Sep 16 02:31:01.625662 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:01.625638    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-542ed5-openshift-default-7dc67cb65b-gcjgl"
Sep 16 02:31:01.628343 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:01.628316    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-542ed5-openshift-default-dockercfg-9fdrw\""
Sep 16 02:31:01.636470 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:01.636443    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-542ed5-openshift-default-7dc67cb65b-gcjgl"]
Sep 16 02:31:01.688891 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:01.688860    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-619550-openshift-default-7787b6c55b-5jrjc"]
Sep 16 02:31:01.692422 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:01.692398    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-619550-openshift-default-7787b6c55b-5jrjc"
Sep 16 02:31:01.694886 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:01.694863    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-619550-openshift-default-dockercfg-znk99\""
Sep 16 02:31:01.704653 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:01.704627    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-619550-openshift-default-7787b6c55b-5jrjc"]
Sep 16 02:31:01.712201 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:01.712169    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/556c7c3b-f739-4660-95e6-3f4348276d47-istio-data\") pod \"e2e-worker-w3-542ed5-openshift-default-7dc67cb65b-gcjgl\" (UID: \"556c7c3b-f739-4660-95e6-3f4348276d47\") " pod="openshift-ingress/e2e-worker-w3-542ed5-openshift-default-7dc67cb65b-gcjgl"
Sep 16 02:31:01.712369 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:01.712210    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pz4h7\" (UniqueName: \"kubernetes.io/projected/556c7c3b-f739-4660-95e6-3f4348276d47-kube-api-access-pz4h7\") pod \"e2e-worker-w3-542ed5-openshift-default-7dc67cb65b-gcjgl\" (UID: \"556c7c3b-f739-4660-95e6-3f4348276d47\") " pod="openshift-ingress/e2e-worker-w3-542ed5-openshift-default-7dc67cb65b-gcjgl"
Sep 16 02:31:01.712369 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:01.712247    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/556c7c3b-f739-4660-95e6-3f4348276d47-istiod-ca-cert\") pod \"e2e-worker-w3-542ed5-openshift-default-7dc67cb65b-gcjgl\" (UID: \"556c7c3b-f739-4660-95e6-3f4348276d47\") " pod="openshift-ingress/e2e-worker-w3-542ed5-openshift-default-7dc67cb65b-gcjgl"
Sep 16 02:31:01.712369 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:01.712276    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/556c7c3b-f739-4660-95e6-3f4348276d47-istio-podinfo\") pod \"e2e-worker-w3-542ed5-openshift-default-7dc67cb65b-gcjgl\" (UID: \"556c7c3b-f739-4660-95e6-3f4348276d47\") " pod="openshift-ingress/e2e-worker-w3-542ed5-openshift-default-7dc67cb65b-gcjgl"
Sep 16 02:31:01.712369 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:01.712304    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/556c7c3b-f739-4660-95e6-3f4348276d47-workload-certs\") pod \"e2e-worker-w3-542ed5-openshift-default-7dc67cb65b-gcjgl\" (UID: \"556c7c3b-f739-4660-95e6-3f4348276d47\") " pod="openshift-ingress/e2e-worker-w3-542ed5-openshift-default-7dc67cb65b-gcjgl"
Sep 16 02:31:01.712369 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:01.712333    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d81e13b-62c5-4fe2-a392-c5bac92f0ba5-workload-certs\") pod \"e2e-inf-a-60674e-openshift-default-6c56877b8b-qc47f\" (UID: \"6d81e13b-62c5-4fe2-a392-c5bac92f0ba5\") " pod="openshift-ingress/e2e-inf-a-60674e-openshift-default-6c56877b8b-qc47f"
Sep 16 02:31:01.712369 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:01.712355    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/556c7c3b-f739-4660-95e6-3f4348276d47-credential-socket\") pod \"e2e-worker-w3-542ed5-openshift-default-7dc67cb65b-gcjgl\" (UID: \"556c7c3b-f739-4660-95e6-3f4348276d47\") " pod="openshift-ingress/e2e-worker-w3-542ed5-openshift-default-7dc67cb65b-gcjgl"
Sep 16 02:31:01.712604 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:01.712383    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d81e13b-62c5-4fe2-a392-c5bac92f0ba5-istiod-ca-cert\") pod \"e2e-inf-a-60674e-openshift-default-6c56877b8b-qc47f\" (UID: \"6d81e13b-62c5-4fe2-a392-c5bac92f0ba5\") " pod="openshift-ingress/e2e-inf-a-60674e-openshift-default-6c56877b8b-qc47f"
Sep 16 02:31:01.712604 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:01.712446    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/556c7c3b-f739-4660-95e6-3f4348276d47-workload-socket\") pod \"e2e-worker-w3-542ed5-openshift-default-7dc67cb65b-gcjgl\" (UID: \"556c7c3b-f739-4660-95e6-3f4348276d47\") " pod="openshift-ingress/e2e-worker-w3-542ed5-openshift-default-7dc67cb65b-gcjgl"
Sep 16 02:31:01.712604 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:01.712489    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/556c7c3b-f739-4660-95e6-3f4348276d47-istio-token\") pod \"e2e-worker-w3-542ed5-openshift-default-7dc67cb65b-gcjgl\" (UID: \"556c7c3b-f739-4660-95e6-3f4348276d47\") " pod="openshift-ingress/e2e-worker-w3-542ed5-openshift-default-7dc67cb65b-gcjgl"
Sep 16 02:31:01.712604 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:01.712517    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d81e13b-62c5-4fe2-a392-c5bac92f0ba5-credential-socket\") pod \"e2e-inf-a-60674e-openshift-default-6c56877b8b-qc47f\" (UID: \"6d81e13b-62c5-4fe2-a392-c5bac92f0ba5\") " pod="openshift-ingress/e2e-inf-a-60674e-openshift-default-6c56877b8b-qc47f"
Sep 16 02:31:01.712604 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:01.712539    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/556c7c3b-f739-4660-95e6-3f4348276d47-istio-envoy\") pod \"e2e-worker-w3-542ed5-openshift-default-7dc67cb65b-gcjgl\" (UID: \"556c7c3b-f739-4660-95e6-3f4348276d47\") " pod="openshift-ingress/e2e-worker-w3-542ed5-openshift-default-7dc67cb65b-gcjgl"
Sep 16 02:31:01.712604 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:01.712591    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d81e13b-62c5-4fe2-a392-c5bac92f0ba5-istio-token\") pod \"e2e-inf-a-60674e-openshift-default-6c56877b8b-qc47f\" (UID: \"6d81e13b-62c5-4fe2-a392-c5bac92f0ba5\") " pod="openshift-ingress/e2e-inf-a-60674e-openshift-default-6c56877b8b-qc47f"
Sep 16 02:31:01.713025 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:01.712647    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d81e13b-62c5-4fe2-a392-c5bac92f0ba5-istio-podinfo\") pod \"e2e-inf-a-60674e-openshift-default-6c56877b8b-qc47f\" (UID: \"6d81e13b-62c5-4fe2-a392-c5bac92f0ba5\") " pod="openshift-ingress/e2e-inf-a-60674e-openshift-default-6c56877b8b-qc47f"
Sep 16 02:31:01.713025 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:01.712684    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d81e13b-62c5-4fe2-a392-c5bac92f0ba5-workload-socket\") pod \"e2e-inf-a-60674e-openshift-default-6c56877b8b-qc47f\" (UID: \"6d81e13b-62c5-4fe2-a392-c5bac92f0ba5\") " pod="openshift-ingress/e2e-inf-a-60674e-openshift-default-6c56877b8b-qc47f"
Sep 16 02:31:01.713025 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:01.712751    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d81e13b-62c5-4fe2-a392-c5bac92f0ba5-istio-envoy\") pod \"e2e-inf-a-60674e-openshift-default-6c56877b8b-qc47f\" (UID: \"6d81e13b-62c5-4fe2-a392-c5bac92f0ba5\") " pod="openshift-ingress/e2e-inf-a-60674e-openshift-default-6c56877b8b-qc47f"
Sep 16 02:31:01.713025 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:01.712816    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d81e13b-62c5-4fe2-a392-c5bac92f0ba5-istio-data\") pod \"e2e-inf-a-60674e-openshift-default-6c56877b8b-qc47f\" (UID: \"6d81e13b-62c5-4fe2-a392-c5bac92f0ba5\") " pod="openshift-ingress/e2e-inf-a-60674e-openshift-default-6c56877b8b-qc47f"
Sep 16 02:31:01.713025 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:01.712897    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ssrw2\" (UniqueName: \"kubernetes.io/projected/6d81e13b-62c5-4fe2-a392-c5bac92f0ba5-kube-api-access-ssrw2\") pod \"e2e-inf-a-60674e-openshift-default-6c56877b8b-qc47f\" (UID: \"6d81e13b-62c5-4fe2-a392-c5bac92f0ba5\") " pod="openshift-ingress/e2e-inf-a-60674e-openshift-default-6c56877b8b-qc47f"
Sep 16 02:31:01.813842 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:01.813739    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d81e13b-62c5-4fe2-a392-c5bac92f0ba5-istio-podinfo\") pod \"e2e-inf-a-60674e-openshift-default-6c56877b8b-qc47f\" (UID: \"6d81e13b-62c5-4fe2-a392-c5bac92f0ba5\") " pod="openshift-ingress/e2e-inf-a-60674e-openshift-default-6c56877b8b-qc47f"
Sep 16 02:31:01.813842 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:01.813797    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b5f2556-ebb9-4089-b0f5-9cc37357b31f-istio-envoy\") pod \"e2e-worker-w0-619550-openshift-default-7787b6c55b-5jrjc\" (UID: \"8b5f2556-ebb9-4089-b0f5-9cc37357b31f\") " pod="openshift-ingress/e2e-worker-w0-619550-openshift-default-7787b6c55b-5jrjc"
Sep 16 02:31:01.814079 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:01.813911    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d81e13b-62c5-4fe2-a392-c5bac92f0ba5-workload-socket\") pod \"e2e-inf-a-60674e-openshift-default-6c56877b8b-qc47f\" (UID: \"6d81e13b-62c5-4fe2-a392-c5bac92f0ba5\") " pod="openshift-ingress/e2e-inf-a-60674e-openshift-default-6c56877b8b-qc47f"
Sep 16 02:31:01.814079 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:01.813957    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b5f2556-ebb9-4089-b0f5-9cc37357b31f-istio-podinfo\") pod \"e2e-worker-w0-619550-openshift-default-7787b6c55b-5jrjc\" (UID: \"8b5f2556-ebb9-4089-b0f5-9cc37357b31f\") " pod="openshift-ingress/e2e-worker-w0-619550-openshift-default-7787b6c55b-5jrjc"
Sep 16 02:31:01.814079 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:01.813987    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d81e13b-62c5-4fe2-a392-c5bac92f0ba5-istio-envoy\") pod \"e2e-inf-a-60674e-openshift-default-6c56877b8b-qc47f\" (UID: \"6d81e13b-62c5-4fe2-a392-c5bac92f0ba5\") " pod="openshift-ingress/e2e-inf-a-60674e-openshift-default-6c56877b8b-qc47f"
Sep 16 02:31:01.814079 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:01.814032    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b5f2556-ebb9-4089-b0f5-9cc37357b31f-credential-socket\") pod \"e2e-worker-w0-619550-openshift-default-7787b6c55b-5jrjc\" (UID: \"8b5f2556-ebb9-4089-b0f5-9cc37357b31f\") " pod="openshift-ingress/e2e-worker-w0-619550-openshift-default-7787b6c55b-5jrjc"
Sep 16 02:31:01.814079 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:01.814065    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b5f2556-ebb9-4089-b0f5-9cc37357b31f-workload-certs\") pod \"e2e-worker-w0-619550-openshift-default-7787b6c55b-5jrjc\" (UID: \"8b5f2556-ebb9-4089-b0f5-9cc37357b31f\") " pod="openshift-ingress/e2e-worker-w0-619550-openshift-default-7787b6c55b-5jrjc"
Sep 16 02:31:01.814337 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:01.814095    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d81e13b-62c5-4fe2-a392-c5bac92f0ba5-istio-data\") pod \"e2e-inf-a-60674e-openshift-default-6c56877b8b-qc47f\" (UID: \"6d81e13b-62c5-4fe2-a392-c5bac92f0ba5\") " pod="openshift-ingress/e2e-inf-a-60674e-openshift-default-6c56877b8b-qc47f"
Sep 16 02:31:01.814337 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:01.814230    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b5f2556-ebb9-4089-b0f5-9cc37357b31f-workload-socket\") pod \"e2e-worker-w0-619550-openshift-default-7787b6c55b-5jrjc\" (UID: \"8b5f2556-ebb9-4089-b0f5-9cc37357b31f\") " pod="openshift-ingress/e2e-worker-w0-619550-openshift-default-7787b6c55b-5jrjc"
Sep 16 02:31:01.814337 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:01.814274    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b5f2556-ebb9-4089-b0f5-9cc37357b31f-istio-data\") pod \"e2e-worker-w0-619550-openshift-default-7787b6c55b-5jrjc\" (UID: \"8b5f2556-ebb9-4089-b0f5-9cc37357b31f\") " pod="openshift-ingress/e2e-worker-w0-619550-openshift-default-7787b6c55b-5jrjc"
Sep 16 02:31:01.814337 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:01.814309    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ssrw2\" (UniqueName: \"kubernetes.io/projected/6d81e13b-62c5-4fe2-a392-c5bac92f0ba5-kube-api-access-ssrw2\") pod \"e2e-inf-a-60674e-openshift-default-6c56877b8b-qc47f\" (UID: \"6d81e13b-62c5-4fe2-a392-c5bac92f0ba5\") " pod="openshift-ingress/e2e-inf-a-60674e-openshift-default-6c56877b8b-qc47f"
Sep 16 02:31:01.814517 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:01.814368    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d81e13b-62c5-4fe2-a392-c5bac92f0ba5-workload-socket\") pod \"e2e-inf-a-60674e-openshift-default-6c56877b8b-qc47f\" (UID: \"6d81e13b-62c5-4fe2-a392-c5bac92f0ba5\") " pod="openshift-ingress/e2e-inf-a-60674e-openshift-default-6c56877b8b-qc47f"
Sep 16 02:31:01.814517 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:01.814397    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/556c7c3b-f739-4660-95e6-3f4348276d47-istio-data\") pod \"e2e-worker-w3-542ed5-openshift-default-7dc67cb65b-gcjgl\" (UID: \"556c7c3b-f739-4660-95e6-3f4348276d47\") " pod="openshift-ingress/e2e-worker-w3-542ed5-openshift-default-7dc67cb65b-gcjgl"
Sep 16 02:31:01.814517 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:01.814424    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pz4h7\" (UniqueName: \"kubernetes.io/projected/556c7c3b-f739-4660-95e6-3f4348276d47-kube-api-access-pz4h7\") pod \"e2e-worker-w3-542ed5-openshift-default-7dc67cb65b-gcjgl\" (UID: \"556c7c3b-f739-4660-95e6-3f4348276d47\") " pod="openshift-ingress/e2e-worker-w3-542ed5-openshift-default-7dc67cb65b-gcjgl"
Sep 16 02:31:01.814517 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:01.814460    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/556c7c3b-f739-4660-95e6-3f4348276d47-istiod-ca-cert\") pod \"e2e-worker-w3-542ed5-openshift-default-7dc67cb65b-gcjgl\" (UID: \"556c7c3b-f739-4660-95e6-3f4348276d47\") " pod="openshift-ingress/e2e-worker-w3-542ed5-openshift-default-7dc67cb65b-gcjgl"
Sep 16 02:31:01.814517 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:01.814397    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d81e13b-62c5-4fe2-a392-c5bac92f0ba5-istio-data\") pod \"e2e-inf-a-60674e-openshift-default-6c56877b8b-qc47f\" (UID: \"6d81e13b-62c5-4fe2-a392-c5bac92f0ba5\") " pod="openshift-ingress/e2e-inf-a-60674e-openshift-default-6c56877b8b-qc47f"
Sep 16 02:31:01.814778 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:01.814579    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/556c7c3b-f739-4660-95e6-3f4348276d47-istio-podinfo\") pod \"e2e-worker-w3-542ed5-openshift-default-7dc67cb65b-gcjgl\" (UID: \"556c7c3b-f739-4660-95e6-3f4348276d47\") " pod="openshift-ingress/e2e-worker-w3-542ed5-openshift-default-7dc67cb65b-gcjgl"
Sep 16 02:31:01.814778 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:01.814635    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhmfj\" (UniqueName: \"kubernetes.io/projected/8b5f2556-ebb9-4089-b0f5-9cc37357b31f-kube-api-access-qhmfj\") pod \"e2e-worker-w0-619550-openshift-default-7787b6c55b-5jrjc\" (UID: \"8b5f2556-ebb9-4089-b0f5-9cc37357b31f\") " pod="openshift-ingress/e2e-worker-w0-619550-openshift-default-7787b6c55b-5jrjc"
Sep 16 02:31:01.814778 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:01.814675    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/556c7c3b-f739-4660-95e6-3f4348276d47-workload-certs\") pod \"e2e-worker-w3-542ed5-openshift-default-7dc67cb65b-gcjgl\" (UID: \"556c7c3b-f739-4660-95e6-3f4348276d47\") " pod="openshift-ingress/e2e-worker-w3-542ed5-openshift-default-7dc67cb65b-gcjgl"
Sep 16 02:31:01.814778 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:01.814693    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/556c7c3b-f739-4660-95e6-3f4348276d47-istio-data\") pod \"e2e-worker-w3-542ed5-openshift-default-7dc67cb65b-gcjgl\" (UID: \"556c7c3b-f739-4660-95e6-3f4348276d47\") " pod="openshift-ingress/e2e-worker-w3-542ed5-openshift-default-7dc67cb65b-gcjgl"
Sep 16 02:31:01.814778 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:01.814736    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d81e13b-62c5-4fe2-a392-c5bac92f0ba5-workload-certs\") pod \"e2e-inf-a-60674e-openshift-default-6c56877b8b-qc47f\" (UID: \"6d81e13b-62c5-4fe2-a392-c5bac92f0ba5\") " pod="openshift-ingress/e2e-inf-a-60674e-openshift-default-6c56877b8b-qc47f"
Sep 16 02:31:01.814778 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:01.814763    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/556c7c3b-f739-4660-95e6-3f4348276d47-credential-socket\") pod \"e2e-worker-w3-542ed5-openshift-default-7dc67cb65b-gcjgl\" (UID: \"556c7c3b-f739-4660-95e6-3f4348276d47\") " pod="openshift-ingress/e2e-worker-w3-542ed5-openshift-default-7dc67cb65b-gcjgl"
Sep 16 02:31:01.815080 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:01.814789    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b5f2556-ebb9-4089-b0f5-9cc37357b31f-istiod-ca-cert\") pod \"e2e-worker-w0-619550-openshift-default-7787b6c55b-5jrjc\" (UID: \"8b5f2556-ebb9-4089-b0f5-9cc37357b31f\") " pod="openshift-ingress/e2e-worker-w0-619550-openshift-default-7787b6c55b-5jrjc"
Sep 16 02:31:01.815080 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:01.814835    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d81e13b-62c5-4fe2-a392-c5bac92f0ba5-istiod-ca-cert\") pod \"e2e-inf-a-60674e-openshift-default-6c56877b8b-qc47f\" (UID: \"6d81e13b-62c5-4fe2-a392-c5bac92f0ba5\") " pod="openshift-ingress/e2e-inf-a-60674e-openshift-default-6c56877b8b-qc47f"
Sep 16 02:31:01.815080 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:01.814879    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/556c7c3b-f739-4660-95e6-3f4348276d47-workload-socket\") pod \"e2e-worker-w3-542ed5-openshift-default-7dc67cb65b-gcjgl\" (UID: \"556c7c3b-f739-4660-95e6-3f4348276d47\") " pod="openshift-ingress/e2e-worker-w3-542ed5-openshift-default-7dc67cb65b-gcjgl"
Sep 16 02:31:01.815080 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:01.814902    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/556c7c3b-f739-4660-95e6-3f4348276d47-istio-token\") pod \"e2e-worker-w3-542ed5-openshift-default-7dc67cb65b-gcjgl\" (UID: \"556c7c3b-f739-4660-95e6-3f4348276d47\") " pod="openshift-ingress/e2e-worker-w3-542ed5-openshift-default-7dc67cb65b-gcjgl"
Sep 16 02:31:01.815080 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:01.814930    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d81e13b-62c5-4fe2-a392-c5bac92f0ba5-credential-socket\") pod \"e2e-inf-a-60674e-openshift-default-6c56877b8b-qc47f\" (UID: \"6d81e13b-62c5-4fe2-a392-c5bac92f0ba5\") " pod="openshift-ingress/e2e-inf-a-60674e-openshift-default-6c56877b8b-qc47f"
Sep 16 02:31:01.815080 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:01.814955    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/556c7c3b-f739-4660-95e6-3f4348276d47-istio-envoy\") pod \"e2e-worker-w3-542ed5-openshift-default-7dc67cb65b-gcjgl\" (UID: \"556c7c3b-f739-4660-95e6-3f4348276d47\") " pod="openshift-ingress/e2e-worker-w3-542ed5-openshift-default-7dc67cb65b-gcjgl"
Sep 16 02:31:01.815080 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:01.814983    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b5f2556-ebb9-4089-b0f5-9cc37357b31f-istio-token\") pod \"e2e-worker-w0-619550-openshift-default-7787b6c55b-5jrjc\" (UID: \"8b5f2556-ebb9-4089-b0f5-9cc37357b31f\") " pod="openshift-ingress/e2e-worker-w0-619550-openshift-default-7787b6c55b-5jrjc"
Sep 16 02:31:01.815080 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:01.815031    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d81e13b-62c5-4fe2-a392-c5bac92f0ba5-istio-token\") pod \"e2e-inf-a-60674e-openshift-default-6c56877b8b-qc47f\" (UID: \"6d81e13b-62c5-4fe2-a392-c5bac92f0ba5\") " pod="openshift-ingress/e2e-inf-a-60674e-openshift-default-6c56877b8b-qc47f"
Sep 16 02:31:01.815080 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:01.815076    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d81e13b-62c5-4fe2-a392-c5bac92f0ba5-workload-certs\") pod \"e2e-inf-a-60674e-openshift-default-6c56877b8b-qc47f\" (UID: \"6d81e13b-62c5-4fe2-a392-c5bac92f0ba5\") " pod="openshift-ingress/e2e-inf-a-60674e-openshift-default-6c56877b8b-qc47f"
Sep 16 02:31:01.815565 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:01.815080    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/556c7c3b-f739-4660-95e6-3f4348276d47-credential-socket\") pod \"e2e-worker-w3-542ed5-openshift-default-7dc67cb65b-gcjgl\" (UID: \"556c7c3b-f739-4660-95e6-3f4348276d47\") " pod="openshift-ingress/e2e-worker-w3-542ed5-openshift-default-7dc67cb65b-gcjgl"
Sep 16 02:31:01.815565 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:01.815153    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/556c7c3b-f739-4660-95e6-3f4348276d47-istiod-ca-cert\") pod \"e2e-worker-w3-542ed5-openshift-default-7dc67cb65b-gcjgl\" (UID: \"556c7c3b-f739-4660-95e6-3f4348276d47\") " pod="openshift-ingress/e2e-worker-w3-542ed5-openshift-default-7dc67cb65b-gcjgl"
Sep 16 02:31:01.815565 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:01.815313    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/556c7c3b-f739-4660-95e6-3f4348276d47-workload-socket\") pod \"e2e-worker-w3-542ed5-openshift-default-7dc67cb65b-gcjgl\" (UID: \"556c7c3b-f739-4660-95e6-3f4348276d47\") " pod="openshift-ingress/e2e-worker-w3-542ed5-openshift-default-7dc67cb65b-gcjgl"
Sep 16 02:31:01.815565 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:01.815537    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d81e13b-62c5-4fe2-a392-c5bac92f0ba5-credential-socket\") pod \"e2e-inf-a-60674e-openshift-default-6c56877b8b-qc47f\" (UID: \"6d81e13b-62c5-4fe2-a392-c5bac92f0ba5\") " pod="openshift-ingress/e2e-inf-a-60674e-openshift-default-6c56877b8b-qc47f"
Sep 16 02:31:01.815811 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:01.815632    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/556c7c3b-f739-4660-95e6-3f4348276d47-workload-certs\") pod \"e2e-worker-w3-542ed5-openshift-default-7dc67cb65b-gcjgl\" (UID: \"556c7c3b-f739-4660-95e6-3f4348276d47\") " pod="openshift-ingress/e2e-worker-w3-542ed5-openshift-default-7dc67cb65b-gcjgl"
Sep 16 02:31:01.815891 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:01.815865    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d81e13b-62c5-4fe2-a392-c5bac92f0ba5-istiod-ca-cert\") pod \"e2e-inf-a-60674e-openshift-default-6c56877b8b-qc47f\" (UID: \"6d81e13b-62c5-4fe2-a392-c5bac92f0ba5\") " pod="openshift-ingress/e2e-inf-a-60674e-openshift-default-6c56877b8b-qc47f"
Sep 16 02:31:01.816820 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:01.816793    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d81e13b-62c5-4fe2-a392-c5bac92f0ba5-istio-envoy\") pod \"e2e-inf-a-60674e-openshift-default-6c56877b8b-qc47f\" (UID: \"6d81e13b-62c5-4fe2-a392-c5bac92f0ba5\") " pod="openshift-ingress/e2e-inf-a-60674e-openshift-default-6c56877b8b-qc47f"
Sep 16 02:31:01.817047 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:01.817029    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d81e13b-62c5-4fe2-a392-c5bac92f0ba5-istio-podinfo\") pod \"e2e-inf-a-60674e-openshift-default-6c56877b8b-qc47f\" (UID: \"6d81e13b-62c5-4fe2-a392-c5bac92f0ba5\") " pod="openshift-ingress/e2e-inf-a-60674e-openshift-default-6c56877b8b-qc47f"
Sep 16 02:31:01.817337 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:01.817308    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/556c7c3b-f739-4660-95e6-3f4348276d47-istio-envoy\") pod \"e2e-worker-w3-542ed5-openshift-default-7dc67cb65b-gcjgl\" (UID: \"556c7c3b-f739-4660-95e6-3f4348276d47\") " pod="openshift-ingress/e2e-worker-w3-542ed5-openshift-default-7dc67cb65b-gcjgl"
Sep 16 02:31:01.817633 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:01.817614    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/556c7c3b-f739-4660-95e6-3f4348276d47-istio-podinfo\") pod \"e2e-worker-w3-542ed5-openshift-default-7dc67cb65b-gcjgl\" (UID: \"556c7c3b-f739-4660-95e6-3f4348276d47\") " pod="openshift-ingress/e2e-worker-w3-542ed5-openshift-default-7dc67cb65b-gcjgl"
Sep 16 02:31:01.822074 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:01.822045    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/556c7c3b-f739-4660-95e6-3f4348276d47-istio-token\") pod \"e2e-worker-w3-542ed5-openshift-default-7dc67cb65b-gcjgl\" (UID: \"556c7c3b-f739-4660-95e6-3f4348276d47\") " pod="openshift-ingress/e2e-worker-w3-542ed5-openshift-default-7dc67cb65b-gcjgl"
Sep 16 02:31:01.822305 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:01.822285    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pz4h7\" (UniqueName: \"kubernetes.io/projected/556c7c3b-f739-4660-95e6-3f4348276d47-kube-api-access-pz4h7\") pod \"e2e-worker-w3-542ed5-openshift-default-7dc67cb65b-gcjgl\" (UID: \"556c7c3b-f739-4660-95e6-3f4348276d47\") " pod="openshift-ingress/e2e-worker-w3-542ed5-openshift-default-7dc67cb65b-gcjgl"
Sep 16 02:31:01.822614 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:01.822594    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d81e13b-62c5-4fe2-a392-c5bac92f0ba5-istio-token\") pod \"e2e-inf-a-60674e-openshift-default-6c56877b8b-qc47f\" (UID: \"6d81e13b-62c5-4fe2-a392-c5bac92f0ba5\") " pod="openshift-ingress/e2e-inf-a-60674e-openshift-default-6c56877b8b-qc47f"
Sep 16 02:31:01.822656 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:01.822637    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ssrw2\" (UniqueName: \"kubernetes.io/projected/6d81e13b-62c5-4fe2-a392-c5bac92f0ba5-kube-api-access-ssrw2\") pod \"e2e-inf-a-60674e-openshift-default-6c56877b8b-qc47f\" (UID: \"6d81e13b-62c5-4fe2-a392-c5bac92f0ba5\") " pod="openshift-ingress/e2e-inf-a-60674e-openshift-default-6c56877b8b-qc47f"
Sep 16 02:31:01.880722 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:01.880595    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-60674e-openshift-default-6c56877b8b-qc47f"
Sep 16 02:31:01.916516 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:01.916481    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b5f2556-ebb9-4089-b0f5-9cc37357b31f-istio-envoy\") pod \"e2e-worker-w0-619550-openshift-default-7787b6c55b-5jrjc\" (UID: \"8b5f2556-ebb9-4089-b0f5-9cc37357b31f\") " pod="openshift-ingress/e2e-worker-w0-619550-openshift-default-7787b6c55b-5jrjc"
Sep 16 02:31:01.916516 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:01.916522    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b5f2556-ebb9-4089-b0f5-9cc37357b31f-istio-podinfo\") pod \"e2e-worker-w0-619550-openshift-default-7787b6c55b-5jrjc\" (UID: \"8b5f2556-ebb9-4089-b0f5-9cc37357b31f\") " pod="openshift-ingress/e2e-worker-w0-619550-openshift-default-7787b6c55b-5jrjc"
Sep 16 02:31:01.916794 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:01.916544    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b5f2556-ebb9-4089-b0f5-9cc37357b31f-credential-socket\") pod \"e2e-worker-w0-619550-openshift-default-7787b6c55b-5jrjc\" (UID: \"8b5f2556-ebb9-4089-b0f5-9cc37357b31f\") " pod="openshift-ingress/e2e-worker-w0-619550-openshift-default-7787b6c55b-5jrjc"
Sep 16 02:31:01.916794 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:01.916560    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b5f2556-ebb9-4089-b0f5-9cc37357b31f-workload-certs\") pod \"e2e-worker-w0-619550-openshift-default-7787b6c55b-5jrjc\" (UID: \"8b5f2556-ebb9-4089-b0f5-9cc37357b31f\") " pod="openshift-ingress/e2e-worker-w0-619550-openshift-default-7787b6c55b-5jrjc"
Sep 16 02:31:01.916794 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:01.916593    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b5f2556-ebb9-4089-b0f5-9cc37357b31f-workload-socket\") pod \"e2e-worker-w0-619550-openshift-default-7787b6c55b-5jrjc\" (UID: \"8b5f2556-ebb9-4089-b0f5-9cc37357b31f\") " pod="openshift-ingress/e2e-worker-w0-619550-openshift-default-7787b6c55b-5jrjc"
Sep 16 02:31:01.916794 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:01.916608    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b5f2556-ebb9-4089-b0f5-9cc37357b31f-istio-data\") pod \"e2e-worker-w0-619550-openshift-default-7787b6c55b-5jrjc\" (UID: \"8b5f2556-ebb9-4089-b0f5-9cc37357b31f\") " pod="openshift-ingress/e2e-worker-w0-619550-openshift-default-7787b6c55b-5jrjc"
Sep 16 02:31:01.916794 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:01.916669    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhmfj\" (UniqueName: \"kubernetes.io/projected/8b5f2556-ebb9-4089-b0f5-9cc37357b31f-kube-api-access-qhmfj\") pod \"e2e-worker-w0-619550-openshift-default-7787b6c55b-5jrjc\" (UID: \"8b5f2556-ebb9-4089-b0f5-9cc37357b31f\") " pod="openshift-ingress/e2e-worker-w0-619550-openshift-default-7787b6c55b-5jrjc"
Sep 16 02:31:01.916794 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:01.916722    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b5f2556-ebb9-4089-b0f5-9cc37357b31f-istiod-ca-cert\") pod \"e2e-worker-w0-619550-openshift-default-7787b6c55b-5jrjc\" (UID: \"8b5f2556-ebb9-4089-b0f5-9cc37357b31f\") " pod="openshift-ingress/e2e-worker-w0-619550-openshift-default-7787b6c55b-5jrjc"
Sep 16 02:31:01.916794 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:01.916768    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b5f2556-ebb9-4089-b0f5-9cc37357b31f-istio-token\") pod \"e2e-worker-w0-619550-openshift-default-7787b6c55b-5jrjc\" (UID: \"8b5f2556-ebb9-4089-b0f5-9cc37357b31f\") " pod="openshift-ingress/e2e-worker-w0-619550-openshift-default-7787b6c55b-5jrjc"
Sep 16 02:31:01.917142 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:01.917007    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b5f2556-ebb9-4089-b0f5-9cc37357b31f-credential-socket\") pod \"e2e-worker-w0-619550-openshift-default-7787b6c55b-5jrjc\" (UID: \"8b5f2556-ebb9-4089-b0f5-9cc37357b31f\") " pod="openshift-ingress/e2e-worker-w0-619550-openshift-default-7787b6c55b-5jrjc"
Sep 16 02:31:01.917142 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:01.917034    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b5f2556-ebb9-4089-b0f5-9cc37357b31f-workload-certs\") pod \"e2e-worker-w0-619550-openshift-default-7787b6c55b-5jrjc\" (UID: \"8b5f2556-ebb9-4089-b0f5-9cc37357b31f\") " pod="openshift-ingress/e2e-worker-w0-619550-openshift-default-7787b6c55b-5jrjc"
Sep 16 02:31:01.917317 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:01.917291    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b5f2556-ebb9-4089-b0f5-9cc37357b31f-istio-data\") pod \"e2e-worker-w0-619550-openshift-default-7787b6c55b-5jrjc\" (UID: \"8b5f2556-ebb9-4089-b0f5-9cc37357b31f\") " pod="openshift-ingress/e2e-worker-w0-619550-openshift-default-7787b6c55b-5jrjc"
Sep 16 02:31:01.917506 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:01.917485    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b5f2556-ebb9-4089-b0f5-9cc37357b31f-workload-socket\") pod \"e2e-worker-w0-619550-openshift-default-7787b6c55b-5jrjc\" (UID: \"8b5f2556-ebb9-4089-b0f5-9cc37357b31f\") " pod="openshift-ingress/e2e-worker-w0-619550-openshift-default-7787b6c55b-5jrjc"
Sep 16 02:31:01.917550 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:01.917519    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b5f2556-ebb9-4089-b0f5-9cc37357b31f-istiod-ca-cert\") pod \"e2e-worker-w0-619550-openshift-default-7787b6c55b-5jrjc\" (UID: \"8b5f2556-ebb9-4089-b0f5-9cc37357b31f\") " pod="openshift-ingress/e2e-worker-w0-619550-openshift-default-7787b6c55b-5jrjc"
Sep 16 02:31:01.919107 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:01.919078    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b5f2556-ebb9-4089-b0f5-9cc37357b31f-istio-envoy\") pod \"e2e-worker-w0-619550-openshift-default-7787b6c55b-5jrjc\" (UID: \"8b5f2556-ebb9-4089-b0f5-9cc37357b31f\") " pod="openshift-ingress/e2e-worker-w0-619550-openshift-default-7787b6c55b-5jrjc"
Sep 16 02:31:01.919671 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:01.919648    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b5f2556-ebb9-4089-b0f5-9cc37357b31f-istio-podinfo\") pod \"e2e-worker-w0-619550-openshift-default-7787b6c55b-5jrjc\" (UID: \"8b5f2556-ebb9-4089-b0f5-9cc37357b31f\") " pod="openshift-ingress/e2e-worker-w0-619550-openshift-default-7787b6c55b-5jrjc"
Sep 16 02:31:01.925719 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:01.925669    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhmfj\" (UniqueName: \"kubernetes.io/projected/8b5f2556-ebb9-4089-b0f5-9cc37357b31f-kube-api-access-qhmfj\") pod \"e2e-worker-w0-619550-openshift-default-7787b6c55b-5jrjc\" (UID: \"8b5f2556-ebb9-4089-b0f5-9cc37357b31f\") " pod="openshift-ingress/e2e-worker-w0-619550-openshift-default-7787b6c55b-5jrjc"
Sep 16 02:31:01.925867 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:01.925848    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b5f2556-ebb9-4089-b0f5-9cc37357b31f-istio-token\") pod \"e2e-worker-w0-619550-openshift-default-7787b6c55b-5jrjc\" (UID: \"8b5f2556-ebb9-4089-b0f5-9cc37357b31f\") " pod="openshift-ingress/e2e-worker-w0-619550-openshift-default-7787b6c55b-5jrjc"
Sep 16 02:31:01.937300 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:01.936692    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-542ed5-openshift-default-7dc67cb65b-gcjgl"
Sep 16 02:31:02.004870 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:02.004454    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-619550-openshift-default-7787b6c55b-5jrjc"
Sep 16 02:31:02.162154 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:02.162119    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-60674e-openshift-default-6c56877b8b-qc47f"]
Sep 16 02:31:02.164555 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:31:02.164530    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d81e13b_62c5_4fe2_a392_c5bac92f0ba5.slice/crio-140a797fc3861f3a70c91df8260b42966dda0ec979da682eac9a8ec8753e2cf8 WatchSource:0}: Error finding container 140a797fc3861f3a70c91df8260b42966dda0ec979da682eac9a8ec8753e2cf8: Status 404 returned error can't find the container with id 140a797fc3861f3a70c91df8260b42966dda0ec979da682eac9a8ec8753e2cf8
Sep 16 02:31:02.166781 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:02.166744    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 16 02:31:02.166851 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:02.166814    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 16 02:31:02.166851 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:02.166845    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 16 02:31:02.220340 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:02.220309    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-542ed5-openshift-default-7dc67cb65b-gcjgl"]
Sep 16 02:31:02.222356 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:31:02.222327    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod556c7c3b_f739_4660_95e6_3f4348276d47.slice/crio-5add03532bfcb78d3ff5cc3859df08070e476794ec176a9b0186fafdfbc54cb9 WatchSource:0}: Error finding container 5add03532bfcb78d3ff5cc3859df08070e476794ec176a9b0186fafdfbc54cb9: Status 404 returned error can't find the container with id 5add03532bfcb78d3ff5cc3859df08070e476794ec176a9b0186fafdfbc54cb9
Sep 16 02:31:02.224587 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:02.224554    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 16 02:31:02.224664 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:02.224620    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 16 02:31:02.224664 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:02.224646    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 16 02:31:02.281597 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:02.281560    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-619550-openshift-default-7787b6c55b-5jrjc"]
Sep 16 02:31:02.283976 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:31:02.283941    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b5f2556_ebb9_4089_b0f5_9cc37357b31f.slice/crio-0ea2ddd137bad541944ee77f109130007bed7f1947f39f52a5d431e4d66d8173 WatchSource:0}: Error finding container 0ea2ddd137bad541944ee77f109130007bed7f1947f39f52a5d431e4d66d8173: Status 404 returned error can't find the container with id 0ea2ddd137bad541944ee77f109130007bed7f1947f39f52a5d431e4d66d8173
Sep 16 02:31:02.287035 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:02.286989    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 16 02:31:02.287141 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:02.287086    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 16 02:31:02.287141 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:02.287132    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 16 02:31:02.995245 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:02.995212    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-619550-openshift-default-7787b6c55b-5jrjc" event={"ID":"8b5f2556-ebb9-4089-b0f5-9cc37357b31f","Type":"ContainerStarted","Data":"be7f09b2d6a914f953c5af311e8fcae270069eb0e4847cc070c2409eae8a274c"}
Sep 16 02:31:02.995758 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:02.995268    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-619550-openshift-default-7787b6c55b-5jrjc" event={"ID":"8b5f2556-ebb9-4089-b0f5-9cc37357b31f","Type":"ContainerStarted","Data":"0ea2ddd137bad541944ee77f109130007bed7f1947f39f52a5d431e4d66d8173"}
Sep 16 02:31:02.997058 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:02.997029    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-60674e-openshift-default-6c56877b8b-qc47f" event={"ID":"6d81e13b-62c5-4fe2-a392-c5bac92f0ba5","Type":"ContainerStarted","Data":"0ac6c1c28f7f6b211391cee13d996781b9d16a1382fb2b1a550c05c66ef27839"}
Sep 16 02:31:02.997183 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:02.997085    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-60674e-openshift-default-6c56877b8b-qc47f" event={"ID":"6d81e13b-62c5-4fe2-a392-c5bac92f0ba5","Type":"ContainerStarted","Data":"140a797fc3861f3a70c91df8260b42966dda0ec979da682eac9a8ec8753e2cf8"}
Sep 16 02:31:02.998671 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:02.998647    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-542ed5-openshift-default-7dc67cb65b-gcjgl" event={"ID":"556c7c3b-f739-4660-95e6-3f4348276d47","Type":"ContainerStarted","Data":"39aea0fac2e04c00fa9cf4c3bc63d72d737ffa22269698207df720fb0a2de7c2"}
Sep 16 02:31:02.998765 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:02.998693    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-542ed5-openshift-default-7dc67cb65b-gcjgl" event={"ID":"556c7c3b-f739-4660-95e6-3f4348276d47","Type":"ContainerStarted","Data":"5add03532bfcb78d3ff5cc3859df08070e476794ec176a9b0186fafdfbc54cb9"}
Sep 16 02:31:03.005176 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:03.005146    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-619550-openshift-default-7787b6c55b-5jrjc"
Sep 16 02:31:03.011499 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:03.011470    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-619550-openshift-default-7787b6c55b-5jrjc"
Sep 16 02:31:03.016207 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:03.016152    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-619550-openshift-default-7787b6c55b-5jrjc" podStartSLOduration=2.016133312 podStartE2EDuration="2.016133312s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 02:31:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:31:03.015319099 +0000 UTC m=+1075.933808375" watchObservedRunningTime="2026-09-16 02:31:03.016133312 +0000 UTC m=+1075.934622563"
Sep 16 02:31:03.033308 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:03.033242    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-542ed5-openshift-default-7dc67cb65b-gcjgl" podStartSLOduration=2.033219384 podStartE2EDuration="2.033219384s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 02:31:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:31:03.031405572 +0000 UTC m=+1075.949894824" watchObservedRunningTime="2026-09-16 02:31:03.033219384 +0000 UTC m=+1075.951708637"
Sep 16 02:31:03.050629 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:03.050573    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-60674e-openshift-default-6c56877b8b-qc47f" podStartSLOduration=2.050557004 podStartE2EDuration="2.050557004s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 02:31:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:31:03.049292081 +0000 UTC m=+1075.967781331" watchObservedRunningTime="2026-09-16 02:31:03.050557004 +0000 UTC m=+1075.969046288"
Sep 16 02:31:03.881194 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:03.881148    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-60674e-openshift-default-6c56877b8b-qc47f"
Sep 16 02:31:03.886393 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:03.886363    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-60674e-openshift-default-6c56877b8b-qc47f"
Sep 16 02:31:03.937888 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:03.937843    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-542ed5-openshift-default-7dc67cb65b-gcjgl"
Sep 16 02:31:03.943228 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:03.943199    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-542ed5-openshift-default-7dc67cb65b-gcjgl"
Sep 16 02:31:04.003500 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:04.003463    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-60674e-openshift-default-6c56877b8b-qc47f"
Sep 16 02:31:04.003961 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:04.003831    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-542ed5-openshift-default-7dc67cb65b-gcjgl"
Sep 16 02:31:04.003961 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:04.003882    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-619550-openshift-default-7787b6c55b-5jrjc"
Sep 16 02:31:04.004528 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:04.004504    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-542ed5-openshift-default-7dc67cb65b-gcjgl"
Sep 16 02:31:04.004673 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:04.004653    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-60674e-openshift-default-6c56877b8b-qc47f"
Sep 16 02:31:04.004774 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:04.004733    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-619550-openshift-default-7787b6c55b-5jrjc"
Sep 16 02:31:09.013531 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:09.013486    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-542ed5-6bc6cbdb75-dlvtl"]
Sep 16 02:31:09.018438 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:09.018410    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-542ed5-6bc6cbdb75-dlvtl"
Sep 16 02:31:09.020794 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:09.020769    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-542ed5-dockercfg-xknsv\""
Sep 16 02:31:09.020911 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:09.020769    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-542ed5\""
Sep 16 02:31:09.025376 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:09.025156    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-542ed5-6bc6cbdb75-dlvtl"]
Sep 16 02:31:09.094099 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:09.094062    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c56k7\" (UniqueName: \"kubernetes.io/projected/6833535b-d82b-40b4-8f53-ebf02c5214be-kube-api-access-c56k7\") pod \"payload-processing-e2e-worker-w3-542ed5-6bc6cbdb75-dlvtl\" (UID: \"6833535b-d82b-40b4-8f53-ebf02c5214be\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-542ed5-6bc6cbdb75-dlvtl"
Sep 16 02:31:09.094287 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:09.094126    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6833535b-d82b-40b4-8f53-ebf02c5214be-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-542ed5-6bc6cbdb75-dlvtl\" (UID: \"6833535b-d82b-40b4-8f53-ebf02c5214be\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-542ed5-6bc6cbdb75-dlvtl"
Sep 16 02:31:09.195738 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:09.195664    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6833535b-d82b-40b4-8f53-ebf02c5214be-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-542ed5-6bc6cbdb75-dlvtl\" (UID: \"6833535b-d82b-40b4-8f53-ebf02c5214be\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-542ed5-6bc6cbdb75-dlvtl"
Sep 16 02:31:09.195921 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:09.195833    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c56k7\" (UniqueName: \"kubernetes.io/projected/6833535b-d82b-40b4-8f53-ebf02c5214be-kube-api-access-c56k7\") pod \"payload-processing-e2e-worker-w3-542ed5-6bc6cbdb75-dlvtl\" (UID: \"6833535b-d82b-40b4-8f53-ebf02c5214be\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-542ed5-6bc6cbdb75-dlvtl"
Sep 16 02:31:09.196469 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:09.196447    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6833535b-d82b-40b4-8f53-ebf02c5214be-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-542ed5-6bc6cbdb75-dlvtl\" (UID: \"6833535b-d82b-40b4-8f53-ebf02c5214be\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-542ed5-6bc6cbdb75-dlvtl"
Sep 16 02:31:09.203331 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:09.203305    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c56k7\" (UniqueName: \"kubernetes.io/projected/6833535b-d82b-40b4-8f53-ebf02c5214be-kube-api-access-c56k7\") pod \"payload-processing-e2e-worker-w3-542ed5-6bc6cbdb75-dlvtl\" (UID: \"6833535b-d82b-40b4-8f53-ebf02c5214be\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-542ed5-6bc6cbdb75-dlvtl"
Sep 16 02:31:09.332244 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:09.332155    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-542ed5-6bc6cbdb75-dlvtl"
Sep 16 02:31:09.603118 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:09.603090    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-542ed5-6bc6cbdb75-dlvtl"]
Sep 16 02:31:09.605986 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:31:09.605960    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6833535b_d82b_40b4_8f53_ebf02c5214be.slice/crio-eff1e74ed7f9bdc653a5ffe90536c107b30caba7a660f3079c1b41974358ec5b WatchSource:0}: Error finding container eff1e74ed7f9bdc653a5ffe90536c107b30caba7a660f3079c1b41974358ec5b: Status 404 returned error can't find the container with id eff1e74ed7f9bdc653a5ffe90536c107b30caba7a660f3079c1b41974358ec5b
Sep 16 02:31:10.036507 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:10.036469    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-542ed5-6bc6cbdb75-dlvtl" event={"ID":"6833535b-d82b-40b4-8f53-ebf02c5214be","Type":"ContainerStarted","Data":"0d9c8ad9ad972d871297062de3f7e828b0d238bfa83c4e0d6c3cfaff9f290b15"}
Sep 16 02:31:10.036938 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:10.036530    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-542ed5-6bc6cbdb75-dlvtl" event={"ID":"6833535b-d82b-40b4-8f53-ebf02c5214be","Type":"ContainerStarted","Data":"eff1e74ed7f9bdc653a5ffe90536c107b30caba7a660f3079c1b41974358ec5b"}
Sep 16 02:31:10.036938 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:10.036585    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-542ed5-6bc6cbdb75-dlvtl"
Sep 16 02:31:10.053353 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:10.053304    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w3-542ed5-6bc6cbdb75-dlvtl" podStartSLOduration=2.053290586 podStartE2EDuration="2.053290586s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 02:31:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:31:10.052796838 +0000 UTC m=+1082.971286088" watchObservedRunningTime="2026-09-16 02:31:10.053290586 +0000 UTC m=+1082.971779834"
Sep 16 02:31:20.045949 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:20.045902    2558 patch_prober.go:28] interesting pod/e2e-inf-a-60674e-openshift-default-6c56877b8b-qc47f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 16 02:31:20.046674 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:20.045903    2558 patch_prober.go:28] interesting pod/e2e-worker-w0-619550-openshift-default-7787b6c55b-5jrjc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 16 02:31:20.046674 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:20.046528    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-60674e-openshift-default-6c56877b8b-qc47f" podUID="6d81e13b-62c5-4fe2-a392-c5bac92f0ba5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 16 02:31:20.046674 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:20.046608    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-619550-openshift-default-7787b6c55b-5jrjc" podUID="8b5f2556-ebb9-4089-b0f5-9cc37357b31f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 16 02:31:21.043944 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:21.043784    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-542ed5-6bc6cbdb75-dlvtl"
Sep 16 02:31:21.130198 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:21.130165    2558 generic.go:356] "Generic (PLEG): container finished" podID="8b5f2556-ebb9-4089-b0f5-9cc37357b31f" containerID="be7f09b2d6a914f953c5af311e8fcae270069eb0e4847cc070c2409eae8a274c" exitCode=0
Sep 16 02:31:21.130620 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:21.130235    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-619550-openshift-default-7787b6c55b-5jrjc" event={"ID":"8b5f2556-ebb9-4089-b0f5-9cc37357b31f","Type":"ContainerDied","Data":"be7f09b2d6a914f953c5af311e8fcae270069eb0e4847cc070c2409eae8a274c"}
Sep 16 02:31:21.130676 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:21.130644    2558 scope.go:117] "RemoveContainer" containerID="be7f09b2d6a914f953c5af311e8fcae270069eb0e4847cc070c2409eae8a274c"
Sep 16 02:31:21.134405 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:21.134368    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 16 02:31:21.134522 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:21.134474    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 16 02:31:21.134522 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:21.134501    2558 generic.go:356] "Generic (PLEG): container finished" podID="6d81e13b-62c5-4fe2-a392-c5bac92f0ba5" containerID="0ac6c1c28f7f6b211391cee13d996781b9d16a1382fb2b1a550c05c66ef27839" exitCode=0
Sep 16 02:31:21.134630 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:21.134522    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 16 02:31:21.134630 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:21.134586    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-60674e-openshift-default-6c56877b8b-qc47f" event={"ID":"6d81e13b-62c5-4fe2-a392-c5bac92f0ba5","Type":"ContainerDied","Data":"0ac6c1c28f7f6b211391cee13d996781b9d16a1382fb2b1a550c05c66ef27839"}
Sep 16 02:31:21.135002 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:21.134986    2558 scope.go:117] "RemoveContainer" containerID="0ac6c1c28f7f6b211391cee13d996781b9d16a1382fb2b1a550c05c66ef27839"
Sep 16 02:31:21.136108 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:21.136084    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 16 02:31:21.136178 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:21.136154    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 16 02:31:21.136283 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:21.136211    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 16 02:31:22.141262 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:22.141230    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-619550-openshift-default-7787b6c55b-5jrjc" event={"ID":"8b5f2556-ebb9-4089-b0f5-9cc37357b31f","Type":"ContainerStarted","Data":"0a54bd28fb3fdf28b0c5cee0ccceb41d0cbbe48235a2a4e790ea0d674b23170b"}
Sep 16 02:31:22.143154 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:22.143134    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-60674e-openshift-default-6c56877b8b-qc47f" event={"ID":"6d81e13b-62c5-4fe2-a392-c5bac92f0ba5","Type":"ContainerStarted","Data":"40e50b8af904fbd01b62dc8a1f9dfc60fb2c3c2091d72faa3c43a599a41d3dc1"}
Sep 16 02:31:22.881824 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:22.881781    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-60674e-openshift-default-6c56877b8b-qc47f"
Sep 16 02:31:22.882025 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:22.881909    2558 patch_prober.go:28] interesting pod/e2e-inf-a-60674e-openshift-default-6c56877b8b-qc47f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body=
Sep 16 02:31:22.882025 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:22.881967    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-60674e-openshift-default-6c56877b8b-qc47f" podUID="6d81e13b-62c5-4fe2-a392-c5bac92f0ba5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused"
Sep 16 02:31:23.005229 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:23.005186    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-619550-openshift-default-7787b6c55b-5jrjc"
Sep 16 02:31:23.005596 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:23.005568    2558 patch_prober.go:28] interesting pod/e2e-worker-w0-619550-openshift-default-7787b6c55b-5jrjc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body=
Sep 16 02:31:23.005656 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:23.005629    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-619550-openshift-default-7787b6c55b-5jrjc" podUID="8b5f2556-ebb9-4089-b0f5-9cc37357b31f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused"
Sep 16 02:31:23.881805 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:23.881760    2558 patch_prober.go:28] interesting pod/e2e-inf-a-60674e-openshift-default-6c56877b8b-qc47f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body=
Sep 16 02:31:23.882307 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:23.881835    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-60674e-openshift-default-6c56877b8b-qc47f" podUID="6d81e13b-62c5-4fe2-a392-c5bac92f0ba5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused"
Sep 16 02:31:24.005348 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:24.005305    2558 patch_prober.go:28] interesting pod/e2e-worker-w0-619550-openshift-default-7787b6c55b-5jrjc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body=
Sep 16 02:31:24.005507 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:24.005366    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-619550-openshift-default-7787b6c55b-5jrjc" podUID="8b5f2556-ebb9-4089-b0f5-9cc37357b31f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused"
Sep 16 02:31:24.882719 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:24.881886    2558 patch_prober.go:28] interesting pod/e2e-inf-a-60674e-openshift-default-6c56877b8b-qc47f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body=
Sep 16 02:31:24.882719 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:24.881955    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-60674e-openshift-default-6c56877b8b-qc47f" podUID="6d81e13b-62c5-4fe2-a392-c5bac92f0ba5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused"
Sep 16 02:31:25.006006 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:25.005966    2558 patch_prober.go:28] interesting pod/e2e-worker-w0-619550-openshift-default-7787b6c55b-5jrjc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body=
Sep 16 02:31:25.006209 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:25.006023    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-619550-openshift-default-7787b6c55b-5jrjc" podUID="8b5f2556-ebb9-4089-b0f5-9cc37357b31f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused"
Sep 16 02:31:25.881092 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:25.881043    2558 patch_prober.go:28] interesting pod/e2e-inf-a-60674e-openshift-default-6c56877b8b-qc47f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body=
Sep 16 02:31:25.881299 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:25.881117    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-60674e-openshift-default-6c56877b8b-qc47f" podUID="6d81e13b-62c5-4fe2-a392-c5bac92f0ba5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused"
Sep 16 02:31:26.005624 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:26.005583    2558 patch_prober.go:28] interesting pod/e2e-worker-w0-619550-openshift-default-7787b6c55b-5jrjc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body=
Sep 16 02:31:26.006026 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:26.005642    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-619550-openshift-default-7787b6c55b-5jrjc" podUID="8b5f2556-ebb9-4089-b0f5-9cc37357b31f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused"
Sep 16 02:31:26.881511 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:26.881467    2558 patch_prober.go:28] interesting pod/e2e-inf-a-60674e-openshift-default-6c56877b8b-qc47f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body=
Sep 16 02:31:26.882020 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:26.881556    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-60674e-openshift-default-6c56877b8b-qc47f" podUID="6d81e13b-62c5-4fe2-a392-c5bac92f0ba5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused"
Sep 16 02:31:27.005888 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:27.005796    2558 patch_prober.go:28] interesting pod/e2e-worker-w0-619550-openshift-default-7787b6c55b-5jrjc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body=
Sep 16 02:31:27.005888 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:27.005851    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-619550-openshift-default-7787b6c55b-5jrjc" podUID="8b5f2556-ebb9-4089-b0f5-9cc37357b31f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused"
Sep 16 02:31:27.188768 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:27.187864    2558 generic.go:356] "Generic (PLEG): container finished" podID="556c7c3b-f739-4660-95e6-3f4348276d47" containerID="39aea0fac2e04c00fa9cf4c3bc63d72d737ffa22269698207df720fb0a2de7c2" exitCode=0
Sep 16 02:31:27.188768 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:27.187937    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-542ed5-openshift-default-7dc67cb65b-gcjgl" event={"ID":"556c7c3b-f739-4660-95e6-3f4348276d47","Type":"ContainerDied","Data":"39aea0fac2e04c00fa9cf4c3bc63d72d737ffa22269698207df720fb0a2de7c2"}
Sep 16 02:31:27.188768 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:27.188434    2558 scope.go:117] "RemoveContainer" containerID="39aea0fac2e04c00fa9cf4c3bc63d72d737ffa22269698207df720fb0a2de7c2"
Sep 16 02:31:27.191272 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:27.190494    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 16 02:31:27.191272 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:27.190579    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 16 02:31:27.191272 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:27.190626    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 16 02:31:27.884610 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:27.884572    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-60674e-openshift-default-6c56877b8b-qc47f"
Sep 16 02:31:27.884860 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:27.884829    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-60674e-openshift-default-6c56877b8b-qc47f"
Sep 16 02:31:27.885637 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:27.885617    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-60674e-openshift-default-6c56877b8b-qc47f"
Sep 16 02:31:28.009227 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:28.009202    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-619550-openshift-default-7787b6c55b-5jrjc"
Sep 16 02:31:28.009592 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:28.009552    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-619550-openshift-default-7787b6c55b-5jrjc"
Sep 16 02:31:28.010160 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:28.010138    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-619550-openshift-default-7787b6c55b-5jrjc"
Sep 16 02:31:28.197909 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:28.197868    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-542ed5-openshift-default-7dc67cb65b-gcjgl" event={"ID":"556c7c3b-f739-4660-95e6-3f4348276d47","Type":"ContainerStarted","Data":"c6a5d86755d56a2090e7af9115b4541a5d4f6aa58533695334e8f077ad59d9a7"}
Sep 16 02:31:28.938126 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:28.938084    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-542ed5-openshift-default-7dc67cb65b-gcjgl"
Sep 16 02:31:28.938355 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:28.938329    2558 patch_prober.go:28] interesting pod/e2e-worker-w3-542ed5-openshift-default-7dc67cb65b-gcjgl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" start-of-body=
Sep 16 02:31:28.938408 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:28.938393    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-542ed5-openshift-default-7dc67cb65b-gcjgl" podUID="556c7c3b-f739-4660-95e6-3f4348276d47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused"
Sep 16 02:31:29.938207 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:29.938165    2558 patch_prober.go:28] interesting pod/e2e-worker-w3-542ed5-openshift-default-7dc67cb65b-gcjgl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" start-of-body=
Sep 16 02:31:29.938594 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:29.938229    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-542ed5-openshift-default-7dc67cb65b-gcjgl" podUID="556c7c3b-f739-4660-95e6-3f4348276d47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused"
Sep 16 02:31:30.523126 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:30.523094    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-1a323c-openshift-default-665cc4674b-v4ff5"]
Sep 16 02:31:30.526725 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:30.526686    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-1a323c-openshift-default-665cc4674b-v4ff5"
Sep 16 02:31:30.529668 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:30.529559    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-1a323c-openshift-default-dockercfg-zcg7h\""
Sep 16 02:31:30.541004 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:30.540978    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-1a323c-openshift-default-665cc4674b-v4ff5"]
Sep 16 02:31:30.659163 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:30.659101    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ebb7007e-e1de-4d70-b0dc-2b5f8de79a7f-istiod-ca-cert\") pod \"e2e-inf-b-1a323c-openshift-default-665cc4674b-v4ff5\" (UID: \"ebb7007e-e1de-4d70-b0dc-2b5f8de79a7f\") " pod="openshift-ingress/e2e-inf-b-1a323c-openshift-default-665cc4674b-v4ff5"
Sep 16 02:31:30.659363 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:30.659192    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ebb7007e-e1de-4d70-b0dc-2b5f8de79a7f-workload-certs\") pod \"e2e-inf-b-1a323c-openshift-default-665cc4674b-v4ff5\" (UID: \"ebb7007e-e1de-4d70-b0dc-2b5f8de79a7f\") " pod="openshift-ingress/e2e-inf-b-1a323c-openshift-default-665cc4674b-v4ff5"
Sep 16 02:31:30.659363 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:30.659251    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ebb7007e-e1de-4d70-b0dc-2b5f8de79a7f-istio-token\") pod \"e2e-inf-b-1a323c-openshift-default-665cc4674b-v4ff5\" (UID: \"ebb7007e-e1de-4d70-b0dc-2b5f8de79a7f\") " pod="openshift-ingress/e2e-inf-b-1a323c-openshift-default-665cc4674b-v4ff5"
Sep 16 02:31:30.659363 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:30.659303    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ebb7007e-e1de-4d70-b0dc-2b5f8de79a7f-istio-data\") pod \"e2e-inf-b-1a323c-openshift-default-665cc4674b-v4ff5\" (UID: \"ebb7007e-e1de-4d70-b0dc-2b5f8de79a7f\") " pod="openshift-ingress/e2e-inf-b-1a323c-openshift-default-665cc4674b-v4ff5"
Sep 16 02:31:30.659363 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:30.659328    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebb7007e-e1de-4d70-b0dc-2b5f8de79a7f-credential-socket\") pod \"e2e-inf-b-1a323c-openshift-default-665cc4674b-v4ff5\" (UID: \"ebb7007e-e1de-4d70-b0dc-2b5f8de79a7f\") " pod="openshift-ingress/e2e-inf-b-1a323c-openshift-default-665cc4674b-v4ff5"
Sep 16 02:31:30.659363 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:30.659358    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ebb7007e-e1de-4d70-b0dc-2b5f8de79a7f-istio-podinfo\") pod \"e2e-inf-b-1a323c-openshift-default-665cc4674b-v4ff5\" (UID: \"ebb7007e-e1de-4d70-b0dc-2b5f8de79a7f\") " pod="openshift-ingress/e2e-inf-b-1a323c-openshift-default-665cc4674b-v4ff5"
Sep 16 02:31:30.659658 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:30.659407    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebb7007e-e1de-4d70-b0dc-2b5f8de79a7f-workload-socket\") pod \"e2e-inf-b-1a323c-openshift-default-665cc4674b-v4ff5\" (UID: \"ebb7007e-e1de-4d70-b0dc-2b5f8de79a7f\") " pod="openshift-ingress/e2e-inf-b-1a323c-openshift-default-665cc4674b-v4ff5"
Sep 16 02:31:30.659658 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:30.659458    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v797g\" (UniqueName: \"kubernetes.io/projected/ebb7007e-e1de-4d70-b0dc-2b5f8de79a7f-kube-api-access-v797g\") pod \"e2e-inf-b-1a323c-openshift-default-665cc4674b-v4ff5\" (UID: \"ebb7007e-e1de-4d70-b0dc-2b5f8de79a7f\") " pod="openshift-ingress/e2e-inf-b-1a323c-openshift-default-665cc4674b-v4ff5"
Sep 16 02:31:30.659658 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:30.659483    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ebb7007e-e1de-4d70-b0dc-2b5f8de79a7f-istio-envoy\") pod \"e2e-inf-b-1a323c-openshift-default-665cc4674b-v4ff5\" (UID: \"ebb7007e-e1de-4d70-b0dc-2b5f8de79a7f\") " pod="openshift-ingress/e2e-inf-b-1a323c-openshift-default-665cc4674b-v4ff5"
Sep 16 02:31:30.763831 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:30.760210    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebb7007e-e1de-4d70-b0dc-2b5f8de79a7f-workload-socket\") pod \"e2e-inf-b-1a323c-openshift-default-665cc4674b-v4ff5\" (UID: \"ebb7007e-e1de-4d70-b0dc-2b5f8de79a7f\") " pod="openshift-ingress/e2e-inf-b-1a323c-openshift-default-665cc4674b-v4ff5"
Sep 16 02:31:30.763831 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:30.760283    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v797g\" (UniqueName: \"kubernetes.io/projected/ebb7007e-e1de-4d70-b0dc-2b5f8de79a7f-kube-api-access-v797g\") pod \"e2e-inf-b-1a323c-openshift-default-665cc4674b-v4ff5\" (UID: \"ebb7007e-e1de-4d70-b0dc-2b5f8de79a7f\") " pod="openshift-ingress/e2e-inf-b-1a323c-openshift-default-665cc4674b-v4ff5"
Sep 16 02:31:30.763831 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:30.760316    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ebb7007e-e1de-4d70-b0dc-2b5f8de79a7f-istio-envoy\") pod \"e2e-inf-b-1a323c-openshift-default-665cc4674b-v4ff5\" (UID: \"ebb7007e-e1de-4d70-b0dc-2b5f8de79a7f\") " pod="openshift-ingress/e2e-inf-b-1a323c-openshift-default-665cc4674b-v4ff5"
Sep 16 02:31:30.763831 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:30.760363    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ebb7007e-e1de-4d70-b0dc-2b5f8de79a7f-istiod-ca-cert\") pod \"e2e-inf-b-1a323c-openshift-default-665cc4674b-v4ff5\" (UID: \"ebb7007e-e1de-4d70-b0dc-2b5f8de79a7f\") " pod="openshift-ingress/e2e-inf-b-1a323c-openshift-default-665cc4674b-v4ff5"
Sep 16 02:31:30.763831 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:30.760429    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ebb7007e-e1de-4d70-b0dc-2b5f8de79a7f-workload-certs\") pod \"e2e-inf-b-1a323c-openshift-default-665cc4674b-v4ff5\" (UID: \"ebb7007e-e1de-4d70-b0dc-2b5f8de79a7f\") " pod="openshift-ingress/e2e-inf-b-1a323c-openshift-default-665cc4674b-v4ff5"
Sep 16 02:31:30.763831 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:30.760489    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ebb7007e-e1de-4d70-b0dc-2b5f8de79a7f-istio-token\") pod \"e2e-inf-b-1a323c-openshift-default-665cc4674b-v4ff5\" (UID: \"ebb7007e-e1de-4d70-b0dc-2b5f8de79a7f\") " pod="openshift-ingress/e2e-inf-b-1a323c-openshift-default-665cc4674b-v4ff5"
Sep 16 02:31:30.763831 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:30.760541    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ebb7007e-e1de-4d70-b0dc-2b5f8de79a7f-istio-data\") pod \"e2e-inf-b-1a323c-openshift-default-665cc4674b-v4ff5\" (UID: \"ebb7007e-e1de-4d70-b0dc-2b5f8de79a7f\") " pod="openshift-ingress/e2e-inf-b-1a323c-openshift-default-665cc4674b-v4ff5"
Sep 16 02:31:30.763831 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:30.760568    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebb7007e-e1de-4d70-b0dc-2b5f8de79a7f-credential-socket\") pod \"e2e-inf-b-1a323c-openshift-default-665cc4674b-v4ff5\" (UID: \"ebb7007e-e1de-4d70-b0dc-2b5f8de79a7f\") " pod="openshift-ingress/e2e-inf-b-1a323c-openshift-default-665cc4674b-v4ff5"
Sep 16 02:31:30.763831 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:30.760597    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ebb7007e-e1de-4d70-b0dc-2b5f8de79a7f-istio-podinfo\") pod \"e2e-inf-b-1a323c-openshift-default-665cc4674b-v4ff5\" (UID: \"ebb7007e-e1de-4d70-b0dc-2b5f8de79a7f\") " pod="openshift-ingress/e2e-inf-b-1a323c-openshift-default-665cc4674b-v4ff5"
Sep 16 02:31:30.763831 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:30.761871    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ebb7007e-e1de-4d70-b0dc-2b5f8de79a7f-istiod-ca-cert\") pod \"e2e-inf-b-1a323c-openshift-default-665cc4674b-v4ff5\" (UID: \"ebb7007e-e1de-4d70-b0dc-2b5f8de79a7f\") " pod="openshift-ingress/e2e-inf-b-1a323c-openshift-default-665cc4674b-v4ff5"
Sep 16 02:31:30.763831 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:30.762159    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebb7007e-e1de-4d70-b0dc-2b5f8de79a7f-workload-socket\") pod \"e2e-inf-b-1a323c-openshift-default-665cc4674b-v4ff5\" (UID: \"ebb7007e-e1de-4d70-b0dc-2b5f8de79a7f\") " pod="openshift-ingress/e2e-inf-b-1a323c-openshift-default-665cc4674b-v4ff5"
Sep 16 02:31:30.763831 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:30.763237    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ebb7007e-e1de-4d70-b0dc-2b5f8de79a7f-workload-certs\") pod \"e2e-inf-b-1a323c-openshift-default-665cc4674b-v4ff5\" (UID: \"ebb7007e-e1de-4d70-b0dc-2b5f8de79a7f\") " pod="openshift-ingress/e2e-inf-b-1a323c-openshift-default-665cc4674b-v4ff5"
Sep 16 02:31:30.763831 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:30.763523    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ebb7007e-e1de-4d70-b0dc-2b5f8de79a7f-istio-data\") pod \"e2e-inf-b-1a323c-openshift-default-665cc4674b-v4ff5\" (UID: \"ebb7007e-e1de-4d70-b0dc-2b5f8de79a7f\") " pod="openshift-ingress/e2e-inf-b-1a323c-openshift-default-665cc4674b-v4ff5"
Sep 16 02:31:30.763831 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:30.763782    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebb7007e-e1de-4d70-b0dc-2b5f8de79a7f-credential-socket\") pod \"e2e-inf-b-1a323c-openshift-default-665cc4674b-v4ff5\" (UID: \"ebb7007e-e1de-4d70-b0dc-2b5f8de79a7f\") " pod="openshift-ingress/e2e-inf-b-1a323c-openshift-default-665cc4674b-v4ff5"
Sep 16 02:31:30.765293 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:30.765245    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ebb7007e-e1de-4d70-b0dc-2b5f8de79a7f-istio-podinfo\") pod \"e2e-inf-b-1a323c-openshift-default-665cc4674b-v4ff5\" (UID: \"ebb7007e-e1de-4d70-b0dc-2b5f8de79a7f\") " pod="openshift-ingress/e2e-inf-b-1a323c-openshift-default-665cc4674b-v4ff5"
Sep 16 02:31:30.769895 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:30.769796    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ebb7007e-e1de-4d70-b0dc-2b5f8de79a7f-istio-envoy\") pod \"e2e-inf-b-1a323c-openshift-default-665cc4674b-v4ff5\" (UID: \"ebb7007e-e1de-4d70-b0dc-2b5f8de79a7f\") " pod="openshift-ingress/e2e-inf-b-1a323c-openshift-default-665cc4674b-v4ff5"
Sep 16 02:31:30.772468 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:30.772034    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v797g\" (UniqueName: \"kubernetes.io/projected/ebb7007e-e1de-4d70-b0dc-2b5f8de79a7f-kube-api-access-v797g\") pod \"e2e-inf-b-1a323c-openshift-default-665cc4674b-v4ff5\" (UID: \"ebb7007e-e1de-4d70-b0dc-2b5f8de79a7f\") " pod="openshift-ingress/e2e-inf-b-1a323c-openshift-default-665cc4674b-v4ff5"
Sep 16 02:31:30.772468 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:30.772427    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ebb7007e-e1de-4d70-b0dc-2b5f8de79a7f-istio-token\") pod \"e2e-inf-b-1a323c-openshift-default-665cc4674b-v4ff5\" (UID: \"ebb7007e-e1de-4d70-b0dc-2b5f8de79a7f\") " pod="openshift-ingress/e2e-inf-b-1a323c-openshift-default-665cc4674b-v4ff5"
Sep 16 02:31:30.839145 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:30.839049    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-1a323c-openshift-default-665cc4674b-v4ff5"
Sep 16 02:31:30.938294 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:30.938217    2558 patch_prober.go:28] interesting pod/e2e-worker-w3-542ed5-openshift-default-7dc67cb65b-gcjgl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" start-of-body=
Sep 16 02:31:30.938780 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:30.938292    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-542ed5-openshift-default-7dc67cb65b-gcjgl" podUID="556c7c3b-f739-4660-95e6-3f4348276d47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused"
Sep 16 02:31:31.132663 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:31.132641    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-1a323c-openshift-default-665cc4674b-v4ff5"]
Sep 16 02:31:31.134608 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:31:31.134584    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebb7007e_e1de_4d70_b0dc_2b5f8de79a7f.slice/crio-98ad9c83084a63b59ae04fd3fb9ceab274997c6ebdb0fa993d1f5b69a4cf4f58 WatchSource:0}: Error finding container 98ad9c83084a63b59ae04fd3fb9ceab274997c6ebdb0fa993d1f5b69a4cf4f58: Status 404 returned error can't find the container with id 98ad9c83084a63b59ae04fd3fb9ceab274997c6ebdb0fa993d1f5b69a4cf4f58
Sep 16 02:31:31.136894 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:31.136856    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 16 02:31:31.136977 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:31.136957    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 16 02:31:31.137030 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:31.137014    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 16 02:31:31.214127 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:31.214090    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-1a323c-openshift-default-665cc4674b-v4ff5" event={"ID":"ebb7007e-e1de-4d70-b0dc-2b5f8de79a7f","Type":"ContainerStarted","Data":"98ad9c83084a63b59ae04fd3fb9ceab274997c6ebdb0fa993d1f5b69a4cf4f58"}
Sep 16 02:31:31.938216 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:31.938181    2558 patch_prober.go:28] interesting pod/e2e-worker-w3-542ed5-openshift-default-7dc67cb65b-gcjgl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" start-of-body=
Sep 16 02:31:31.938423 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:31.938240    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-542ed5-openshift-default-7dc67cb65b-gcjgl" podUID="556c7c3b-f739-4660-95e6-3f4348276d47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused"
Sep 16 02:31:32.220224 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:32.220129    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-1a323c-openshift-default-665cc4674b-v4ff5" event={"ID":"ebb7007e-e1de-4d70-b0dc-2b5f8de79a7f","Type":"ContainerStarted","Data":"234c95d72ff6a47852a289fd5955be8627dc4e8921348dfab401d9aed020d744"}
Sep 16 02:31:32.243176 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:32.243121    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-1a323c-openshift-default-665cc4674b-v4ff5" podStartSLOduration=2.243105396 podStartE2EDuration="2.243105396s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 02:31:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:31:32.24133744 +0000 UTC m=+1105.159826691" watchObservedRunningTime="2026-09-16 02:31:32.243105396 +0000 UTC m=+1105.161594645"
Sep 16 02:31:32.839965 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:32.839921    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-1a323c-openshift-default-665cc4674b-v4ff5"
Sep 16 02:31:32.848258 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:32.848216    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-1a323c-openshift-default-665cc4674b-v4ff5"
Sep 16 02:31:32.937688 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:32.937656    2558 patch_prober.go:28] interesting pod/e2e-worker-w3-542ed5-openshift-default-7dc67cb65b-gcjgl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" start-of-body=
Sep 16 02:31:32.937876 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:32.937728    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-542ed5-openshift-default-7dc67cb65b-gcjgl" podUID="556c7c3b-f739-4660-95e6-3f4348276d47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused"
Sep 16 02:31:33.225589 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:33.225547    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-1a323c-openshift-default-665cc4674b-v4ff5"
Sep 16 02:31:33.226611 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:33.226590    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-1a323c-openshift-default-665cc4674b-v4ff5"
Sep 16 02:31:33.942407 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:33.942375    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-542ed5-openshift-default-7dc67cb65b-gcjgl"
Sep 16 02:31:33.942692 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:33.942670    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-542ed5-openshift-default-7dc67cb65b-gcjgl"
Sep 16 02:31:33.943533 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:33.943514    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-542ed5-openshift-default-7dc67cb65b-gcjgl"
Sep 16 02:31:52.323134 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:52.323099    2558 generic.go:356] "Generic (PLEG): container finished" podID="ebb7007e-e1de-4d70-b0dc-2b5f8de79a7f" containerID="234c95d72ff6a47852a289fd5955be8627dc4e8921348dfab401d9aed020d744" exitCode=0
Sep 16 02:31:52.323565 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:52.323150    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-1a323c-openshift-default-665cc4674b-v4ff5" event={"ID":"ebb7007e-e1de-4d70-b0dc-2b5f8de79a7f","Type":"ContainerDied","Data":"234c95d72ff6a47852a289fd5955be8627dc4e8921348dfab401d9aed020d744"}
Sep 16 02:31:52.323650 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:52.323632    2558 scope.go:117] "RemoveContainer" containerID="234c95d72ff6a47852a289fd5955be8627dc4e8921348dfab401d9aed020d744"
Sep 16 02:31:52.324784 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:52.324751    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 16 02:31:52.324884 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:52.324827    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 16 02:31:52.324884 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:52.324863    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 16 02:31:53.332203 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:53.332169    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-1a323c-openshift-default-665cc4674b-v4ff5" event={"ID":"ebb7007e-e1de-4d70-b0dc-2b5f8de79a7f","Type":"ContainerStarted","Data":"bac18692cd45e135c7049ed420845e7a0ecfd8de6a82c236e48e568099b9d34e"}
Sep 16 02:31:53.840103 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:53.840061    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-1a323c-openshift-default-665cc4674b-v4ff5"
Sep 16 02:31:53.840302 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:53.840278    2558 patch_prober.go:28] interesting pod/e2e-inf-b-1a323c-openshift-default-665cc4674b-v4ff5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body=
Sep 16 02:31:53.840346 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:53.840329    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-1a323c-openshift-default-665cc4674b-v4ff5" podUID="ebb7007e-e1de-4d70-b0dc-2b5f8de79a7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused"
Sep 16 02:31:54.840377 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:54.840338    2558 patch_prober.go:28] interesting pod/e2e-inf-b-1a323c-openshift-default-665cc4674b-v4ff5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body=
Sep 16 02:31:54.840882 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:54.840403    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-1a323c-openshift-default-665cc4674b-v4ff5" podUID="ebb7007e-e1de-4d70-b0dc-2b5f8de79a7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused"
Sep 16 02:31:55.840103 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:55.840068    2558 patch_prober.go:28] interesting pod/e2e-inf-b-1a323c-openshift-default-665cc4674b-v4ff5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body=
Sep 16 02:31:55.840273 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:55.840124    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-1a323c-openshift-default-665cc4674b-v4ff5" podUID="ebb7007e-e1de-4d70-b0dc-2b5f8de79a7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused"
Sep 16 02:31:56.840211 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:56.840176    2558 patch_prober.go:28] interesting pod/e2e-inf-b-1a323c-openshift-default-665cc4674b-v4ff5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body=
Sep 16 02:31:56.840575 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:56.840232    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-1a323c-openshift-default-665cc4674b-v4ff5" podUID="ebb7007e-e1de-4d70-b0dc-2b5f8de79a7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused"
Sep 16 02:31:57.840320 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:57.840286    2558 patch_prober.go:28] interesting pod/e2e-inf-b-1a323c-openshift-default-665cc4674b-v4ff5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body=
Sep 16 02:31:57.840686 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:57.840337    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-1a323c-openshift-default-665cc4674b-v4ff5" podUID="ebb7007e-e1de-4d70-b0dc-2b5f8de79a7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused"
Sep 16 02:31:58.856132 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:58.856093    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-1a323c-openshift-default-665cc4674b-v4ff5"
Sep 16 02:31:58.856675 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:58.856653    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-1a323c-openshift-default-665cc4674b-v4ff5"
Sep 16 02:31:58.857365 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:31:58.857346    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-1a323c-openshift-default-665cc4674b-v4ff5"
Sep 16 02:32:00.544279 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:32:00.544251    2558 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-processing-5d7dbbd56b-pltkr" secret="" err="secret \"payload-processing-dockercfg-zzx6j\" not found"
Sep 16 02:33:27.546580 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:33:27.546547    2558 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-processing-5d7dbbd56b-pltkr" secret="" err="secret \"payload-processing-dockercfg-zzx6j\" not found"
Sep 16 02:33:44.076826 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:33:44.076792    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-1a323c/test-model-1a323c-kserve-664b4bd4dc-4dcn8"]
Sep 16 02:33:44.080121 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:33:44.080100    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-1a323c/test-model-1a323c-kserve-664b4bd4dc-4dcn8"
Sep 16 02:33:44.083282 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:33:44.083244    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-1a323c\"/\"openshift-service-ca.crt\""
Sep 16 02:33:44.083422 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:33:44.083247    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-1a323c\"/\"kube-root-ca.crt\""
Sep 16 02:33:44.083422 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:33:44.083329    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-1a323c\"/\"default-dockercfg-qxsbp\""
Sep 16 02:33:44.083422 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:33:44.083353    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-1a323c\"/\"test-model-1a323c-kserve-self-signed-certs\""
Sep 16 02:33:44.089249 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:33:44.089224    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-1a323c/test-model-1a323c-kserve-664b4bd4dc-4dcn8"]
Sep 16 02:33:44.178046 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:33:44.178012    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d0489db-6d2a-4427-8c9f-34993be5e3eb-model-cache\") pod \"test-model-1a323c-kserve-664b4bd4dc-4dcn8\" (UID: \"8d0489db-6d2a-4427-8c9f-34993be5e3eb\") " pod="ai-tenant-e2e-inf-b-1a323c/test-model-1a323c-kserve-664b4bd4dc-4dcn8"
Sep 16 02:33:44.178046 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:33:44.178054    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gb4v6\" (UniqueName: \"kubernetes.io/projected/8d0489db-6d2a-4427-8c9f-34993be5e3eb-kube-api-access-gb4v6\") pod \"test-model-1a323c-kserve-664b4bd4dc-4dcn8\" (UID: \"8d0489db-6d2a-4427-8c9f-34993be5e3eb\") " pod="ai-tenant-e2e-inf-b-1a323c/test-model-1a323c-kserve-664b4bd4dc-4dcn8"
Sep 16 02:33:44.178269 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:33:44.178130    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d0489db-6d2a-4427-8c9f-34993be5e3eb-tls-certs\") pod \"test-model-1a323c-kserve-664b4bd4dc-4dcn8\" (UID: \"8d0489db-6d2a-4427-8c9f-34993be5e3eb\") " pod="ai-tenant-e2e-inf-b-1a323c/test-model-1a323c-kserve-664b4bd4dc-4dcn8"
Sep 16 02:33:44.178269 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:33:44.178156    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d0489db-6d2a-4427-8c9f-34993be5e3eb-tmp-dir\") pod \"test-model-1a323c-kserve-664b4bd4dc-4dcn8\" (UID: \"8d0489db-6d2a-4427-8c9f-34993be5e3eb\") " pod="ai-tenant-e2e-inf-b-1a323c/test-model-1a323c-kserve-664b4bd4dc-4dcn8"
Sep 16 02:33:44.178269 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:33:44.178223    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8d0489db-6d2a-4427-8c9f-34993be5e3eb-home\") pod \"test-model-1a323c-kserve-664b4bd4dc-4dcn8\" (UID: \"8d0489db-6d2a-4427-8c9f-34993be5e3eb\") " pod="ai-tenant-e2e-inf-b-1a323c/test-model-1a323c-kserve-664b4bd4dc-4dcn8"
Sep 16 02:33:44.178269 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:33:44.178251    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8d0489db-6d2a-4427-8c9f-34993be5e3eb-dshm\") pod \"test-model-1a323c-kserve-664b4bd4dc-4dcn8\" (UID: \"8d0489db-6d2a-4427-8c9f-34993be5e3eb\") " pod="ai-tenant-e2e-inf-b-1a323c/test-model-1a323c-kserve-664b4bd4dc-4dcn8"
Sep 16 02:33:44.279555 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:33:44.279515    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8d0489db-6d2a-4427-8c9f-34993be5e3eb-home\") pod \"test-model-1a323c-kserve-664b4bd4dc-4dcn8\" (UID: \"8d0489db-6d2a-4427-8c9f-34993be5e3eb\") " pod="ai-tenant-e2e-inf-b-1a323c/test-model-1a323c-kserve-664b4bd4dc-4dcn8"
Sep 16 02:33:44.279555 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:33:44.279554    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8d0489db-6d2a-4427-8c9f-34993be5e3eb-dshm\") pod \"test-model-1a323c-kserve-664b4bd4dc-4dcn8\" (UID: \"8d0489db-6d2a-4427-8c9f-34993be5e3eb\") " pod="ai-tenant-e2e-inf-b-1a323c/test-model-1a323c-kserve-664b4bd4dc-4dcn8"
Sep 16 02:33:44.279843 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:33:44.279579    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d0489db-6d2a-4427-8c9f-34993be5e3eb-model-cache\") pod \"test-model-1a323c-kserve-664b4bd4dc-4dcn8\" (UID: \"8d0489db-6d2a-4427-8c9f-34993be5e3eb\") " pod="ai-tenant-e2e-inf-b-1a323c/test-model-1a323c-kserve-664b4bd4dc-4dcn8"
Sep 16 02:33:44.279843 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:33:44.279602    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gb4v6\" (UniqueName: \"kubernetes.io/projected/8d0489db-6d2a-4427-8c9f-34993be5e3eb-kube-api-access-gb4v6\") pod \"test-model-1a323c-kserve-664b4bd4dc-4dcn8\" (UID: \"8d0489db-6d2a-4427-8c9f-34993be5e3eb\") " pod="ai-tenant-e2e-inf-b-1a323c/test-model-1a323c-kserve-664b4bd4dc-4dcn8"
Sep 16 02:33:44.279843 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:33:44.279660    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d0489db-6d2a-4427-8c9f-34993be5e3eb-tls-certs\") pod \"test-model-1a323c-kserve-664b4bd4dc-4dcn8\" (UID: \"8d0489db-6d2a-4427-8c9f-34993be5e3eb\") " pod="ai-tenant-e2e-inf-b-1a323c/test-model-1a323c-kserve-664b4bd4dc-4dcn8"
Sep 16 02:33:44.279843 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:33:44.279688    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d0489db-6d2a-4427-8c9f-34993be5e3eb-tmp-dir\") pod \"test-model-1a323c-kserve-664b4bd4dc-4dcn8\" (UID: \"8d0489db-6d2a-4427-8c9f-34993be5e3eb\") " pod="ai-tenant-e2e-inf-b-1a323c/test-model-1a323c-kserve-664b4bd4dc-4dcn8"
Sep 16 02:33:44.280112 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:33:44.280081    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8d0489db-6d2a-4427-8c9f-34993be5e3eb-home\") pod \"test-model-1a323c-kserve-664b4bd4dc-4dcn8\" (UID: \"8d0489db-6d2a-4427-8c9f-34993be5e3eb\") " pod="ai-tenant-e2e-inf-b-1a323c/test-model-1a323c-kserve-664b4bd4dc-4dcn8"
Sep 16 02:33:44.280112 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:33:44.280099    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d0489db-6d2a-4427-8c9f-34993be5e3eb-model-cache\") pod \"test-model-1a323c-kserve-664b4bd4dc-4dcn8\" (UID: \"8d0489db-6d2a-4427-8c9f-34993be5e3eb\") " pod="ai-tenant-e2e-inf-b-1a323c/test-model-1a323c-kserve-664b4bd4dc-4dcn8"
Sep 16 02:33:44.280315 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:33:44.280142    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d0489db-6d2a-4427-8c9f-34993be5e3eb-tmp-dir\") pod \"test-model-1a323c-kserve-664b4bd4dc-4dcn8\" (UID: \"8d0489db-6d2a-4427-8c9f-34993be5e3eb\") " pod="ai-tenant-e2e-inf-b-1a323c/test-model-1a323c-kserve-664b4bd4dc-4dcn8"
Sep 16 02:33:44.281853 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:33:44.281833    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8d0489db-6d2a-4427-8c9f-34993be5e3eb-dshm\") pod \"test-model-1a323c-kserve-664b4bd4dc-4dcn8\" (UID: \"8d0489db-6d2a-4427-8c9f-34993be5e3eb\") " pod="ai-tenant-e2e-inf-b-1a323c/test-model-1a323c-kserve-664b4bd4dc-4dcn8"
Sep 16 02:33:44.282116 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:33:44.282101    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d0489db-6d2a-4427-8c9f-34993be5e3eb-tls-certs\") pod \"test-model-1a323c-kserve-664b4bd4dc-4dcn8\" (UID: \"8d0489db-6d2a-4427-8c9f-34993be5e3eb\") " pod="ai-tenant-e2e-inf-b-1a323c/test-model-1a323c-kserve-664b4bd4dc-4dcn8"
Sep 16 02:33:44.286599 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:33:44.286580    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gb4v6\" (UniqueName: \"kubernetes.io/projected/8d0489db-6d2a-4427-8c9f-34993be5e3eb-kube-api-access-gb4v6\") pod \"test-model-1a323c-kserve-664b4bd4dc-4dcn8\" (UID: \"8d0489db-6d2a-4427-8c9f-34993be5e3eb\") " pod="ai-tenant-e2e-inf-b-1a323c/test-model-1a323c-kserve-664b4bd4dc-4dcn8"
Sep 16 02:33:44.390415 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:33:44.390322    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-1a323c/test-model-1a323c-kserve-664b4bd4dc-4dcn8"
Sep 16 02:33:44.662316 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:33:44.662291    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-1a323c/test-model-1a323c-kserve-664b4bd4dc-4dcn8"]
Sep 16 02:33:44.664608 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:33:44.664584    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8d0489db_6d2a_4427_8c9f_34993be5e3eb.slice/crio-7769f584a16196aa987cf97e5586426f80b99d330003a66afdf18c521b459dae WatchSource:0}: Error finding container 7769f584a16196aa987cf97e5586426f80b99d330003a66afdf18c521b459dae: Status 404 returned error can't find the container with id 7769f584a16196aa987cf97e5586426f80b99d330003a66afdf18c521b459dae
Sep 16 02:33:44.881602 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:33:44.881561    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-1a323c/test-model-1a323c-kserve-664b4bd4dc-4dcn8" event={"ID":"8d0489db-6d2a-4427-8c9f-34993be5e3eb","Type":"ContainerStarted","Data":"65de22837bf72e0fd262250e51438886167154653728742c7bcf92df65222da9"}
Sep 16 02:33:44.881602 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:33:44.881608    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-1a323c/test-model-1a323c-kserve-664b4bd4dc-4dcn8" event={"ID":"8d0489db-6d2a-4427-8c9f-34993be5e3eb","Type":"ContainerStarted","Data":"7769f584a16196aa987cf97e5586426f80b99d330003a66afdf18c521b459dae"}
Sep 16 02:33:44.897889 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:33:44.897844    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-1a323c/test-model-1a323c-kserve-664b4bd4dc-4dcn8" podStartSLOduration=0.897828402 podStartE2EDuration="897.828402ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 02:33:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:33:44.89633891 +0000 UTC m=+1237.814828364" watchObservedRunningTime="2026-09-16 02:33:44.897828402 +0000 UTC m=+1237.816317670"
Sep 16 02:33:54.390655 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:33:54.390609    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-1a323c/test-model-1a323c-kserve-664b4bd4dc-4dcn8"
Sep 16 02:33:54.390655 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:33:54.390664    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-1a323c/test-model-1a323c-kserve-664b4bd4dc-4dcn8"
Sep 16 02:33:54.403452 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:33:54.403418    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-1a323c/test-model-1a323c-kserve-664b4bd4dc-4dcn8"
Sep 16 02:33:54.941093 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:33:54.941066    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-1a323c/test-model-1a323c-kserve-664b4bd4dc-4dcn8"
Sep 16 02:34:44.544061 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:34:44.544025    2558 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-processing-5d7dbbd56b-pltkr" secret="" err="secret \"payload-processing-dockercfg-zzx6j\" not found"
Sep 16 02:35:08.519855 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:08.519816    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-e725a02e-openshift-default-98b5bd466-mjznc"]
Sep 16 02:35:08.523320 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:08.523301    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-e725a02e-openshift-default-98b5bd466-mjznc"
Sep 16 02:35:08.525690 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:08.525670    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-e725a02e-openshift-default-dockercfg-2g6j8\""
Sep 16 02:35:08.534045 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:08.534013    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-e725a02e-openshift-default-98b5bd466-mjznc"]
Sep 16 02:35:08.619546 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:08.619502    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gr89d\" (UniqueName: \"kubernetes.io/projected/54c61f6e-cab0-494d-8ce9-93132feb0228-kube-api-access-gr89d\") pod \"e2e-ait-e725a02e-openshift-default-98b5bd466-mjznc\" (UID: \"54c61f6e-cab0-494d-8ce9-93132feb0228\") " pod="openshift-ingress/e2e-ait-e725a02e-openshift-default-98b5bd466-mjznc"
Sep 16 02:35:08.619926 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:08.619898    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54c61f6e-cab0-494d-8ce9-93132feb0228-istiod-ca-cert\") pod \"e2e-ait-e725a02e-openshift-default-98b5bd466-mjznc\" (UID: \"54c61f6e-cab0-494d-8ce9-93132feb0228\") " pod="openshift-ingress/e2e-ait-e725a02e-openshift-default-98b5bd466-mjznc"
Sep 16 02:35:08.620105 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:08.620088    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54c61f6e-cab0-494d-8ce9-93132feb0228-istio-podinfo\") pod \"e2e-ait-e725a02e-openshift-default-98b5bd466-mjznc\" (UID: \"54c61f6e-cab0-494d-8ce9-93132feb0228\") " pod="openshift-ingress/e2e-ait-e725a02e-openshift-default-98b5bd466-mjznc"
Sep 16 02:35:08.620223 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:08.620207    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54c61f6e-cab0-494d-8ce9-93132feb0228-istio-envoy\") pod \"e2e-ait-e725a02e-openshift-default-98b5bd466-mjznc\" (UID: \"54c61f6e-cab0-494d-8ce9-93132feb0228\") " pod="openshift-ingress/e2e-ait-e725a02e-openshift-default-98b5bd466-mjznc"
Sep 16 02:35:08.620345 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:08.620330    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54c61f6e-cab0-494d-8ce9-93132feb0228-istio-token\") pod \"e2e-ait-e725a02e-openshift-default-98b5bd466-mjznc\" (UID: \"54c61f6e-cab0-494d-8ce9-93132feb0228\") " pod="openshift-ingress/e2e-ait-e725a02e-openshift-default-98b5bd466-mjznc"
Sep 16 02:35:08.620473 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:08.620451    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54c61f6e-cab0-494d-8ce9-93132feb0228-workload-socket\") pod \"e2e-ait-e725a02e-openshift-default-98b5bd466-mjznc\" (UID: \"54c61f6e-cab0-494d-8ce9-93132feb0228\") " pod="openshift-ingress/e2e-ait-e725a02e-openshift-default-98b5bd466-mjznc"
Sep 16 02:35:08.620613 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:08.620501    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54c61f6e-cab0-494d-8ce9-93132feb0228-credential-socket\") pod \"e2e-ait-e725a02e-openshift-default-98b5bd466-mjznc\" (UID: \"54c61f6e-cab0-494d-8ce9-93132feb0228\") " pod="openshift-ingress/e2e-ait-e725a02e-openshift-default-98b5bd466-mjznc"
Sep 16 02:35:08.620613 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:08.620547    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54c61f6e-cab0-494d-8ce9-93132feb0228-istio-data\") pod \"e2e-ait-e725a02e-openshift-default-98b5bd466-mjznc\" (UID: \"54c61f6e-cab0-494d-8ce9-93132feb0228\") " pod="openshift-ingress/e2e-ait-e725a02e-openshift-default-98b5bd466-mjznc"
Sep 16 02:35:08.620613 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:08.620590    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54c61f6e-cab0-494d-8ce9-93132feb0228-workload-certs\") pod \"e2e-ait-e725a02e-openshift-default-98b5bd466-mjznc\" (UID: \"54c61f6e-cab0-494d-8ce9-93132feb0228\") " pod="openshift-ingress/e2e-ait-e725a02e-openshift-default-98b5bd466-mjznc"
Sep 16 02:35:08.721817 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:08.721777    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54c61f6e-cab0-494d-8ce9-93132feb0228-istio-token\") pod \"e2e-ait-e725a02e-openshift-default-98b5bd466-mjznc\" (UID: \"54c61f6e-cab0-494d-8ce9-93132feb0228\") " pod="openshift-ingress/e2e-ait-e725a02e-openshift-default-98b5bd466-mjznc"
Sep 16 02:35:08.721817 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:08.721822    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54c61f6e-cab0-494d-8ce9-93132feb0228-workload-socket\") pod \"e2e-ait-e725a02e-openshift-default-98b5bd466-mjznc\" (UID: \"54c61f6e-cab0-494d-8ce9-93132feb0228\") " pod="openshift-ingress/e2e-ait-e725a02e-openshift-default-98b5bd466-mjznc"
Sep 16 02:35:08.722144 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:08.721848    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54c61f6e-cab0-494d-8ce9-93132feb0228-credential-socket\") pod \"e2e-ait-e725a02e-openshift-default-98b5bd466-mjznc\" (UID: \"54c61f6e-cab0-494d-8ce9-93132feb0228\") " pod="openshift-ingress/e2e-ait-e725a02e-openshift-default-98b5bd466-mjznc"
Sep 16 02:35:08.722144 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:08.721873    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54c61f6e-cab0-494d-8ce9-93132feb0228-istio-data\") pod \"e2e-ait-e725a02e-openshift-default-98b5bd466-mjznc\" (UID: \"54c61f6e-cab0-494d-8ce9-93132feb0228\") " pod="openshift-ingress/e2e-ait-e725a02e-openshift-default-98b5bd466-mjznc"
Sep 16 02:35:08.722144 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:08.721897    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54c61f6e-cab0-494d-8ce9-93132feb0228-workload-certs\") pod \"e2e-ait-e725a02e-openshift-default-98b5bd466-mjznc\" (UID: \"54c61f6e-cab0-494d-8ce9-93132feb0228\") " pod="openshift-ingress/e2e-ait-e725a02e-openshift-default-98b5bd466-mjznc"
Sep 16 02:35:08.722144 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:08.721929    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gr89d\" (UniqueName: \"kubernetes.io/projected/54c61f6e-cab0-494d-8ce9-93132feb0228-kube-api-access-gr89d\") pod \"e2e-ait-e725a02e-openshift-default-98b5bd466-mjznc\" (UID: \"54c61f6e-cab0-494d-8ce9-93132feb0228\") " pod="openshift-ingress/e2e-ait-e725a02e-openshift-default-98b5bd466-mjznc"
Sep 16 02:35:08.722144 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:08.722012    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54c61f6e-cab0-494d-8ce9-93132feb0228-istiod-ca-cert\") pod \"e2e-ait-e725a02e-openshift-default-98b5bd466-mjznc\" (UID: \"54c61f6e-cab0-494d-8ce9-93132feb0228\") " pod="openshift-ingress/e2e-ait-e725a02e-openshift-default-98b5bd466-mjznc"
Sep 16 02:35:08.722144 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:08.722053    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54c61f6e-cab0-494d-8ce9-93132feb0228-istio-podinfo\") pod \"e2e-ait-e725a02e-openshift-default-98b5bd466-mjznc\" (UID: \"54c61f6e-cab0-494d-8ce9-93132feb0228\") " pod="openshift-ingress/e2e-ait-e725a02e-openshift-default-98b5bd466-mjznc"
Sep 16 02:35:08.722144 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:08.722071    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54c61f6e-cab0-494d-8ce9-93132feb0228-istio-envoy\") pod \"e2e-ait-e725a02e-openshift-default-98b5bd466-mjznc\" (UID: \"54c61f6e-cab0-494d-8ce9-93132feb0228\") " pod="openshift-ingress/e2e-ait-e725a02e-openshift-default-98b5bd466-mjznc"
Sep 16 02:35:08.722489 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:08.722324    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54c61f6e-cab0-494d-8ce9-93132feb0228-workload-socket\") pod \"e2e-ait-e725a02e-openshift-default-98b5bd466-mjznc\" (UID: \"54c61f6e-cab0-494d-8ce9-93132feb0228\") " pod="openshift-ingress/e2e-ait-e725a02e-openshift-default-98b5bd466-mjznc"
Sep 16 02:35:08.722489 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:08.722371    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54c61f6e-cab0-494d-8ce9-93132feb0228-istio-data\") pod \"e2e-ait-e725a02e-openshift-default-98b5bd466-mjznc\" (UID: \"54c61f6e-cab0-494d-8ce9-93132feb0228\") " pod="openshift-ingress/e2e-ait-e725a02e-openshift-default-98b5bd466-mjznc"
Sep 16 02:35:08.722489 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:08.722420    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54c61f6e-cab0-494d-8ce9-93132feb0228-credential-socket\") pod \"e2e-ait-e725a02e-openshift-default-98b5bd466-mjznc\" (UID: \"54c61f6e-cab0-494d-8ce9-93132feb0228\") " pod="openshift-ingress/e2e-ait-e725a02e-openshift-default-98b5bd466-mjznc"
Sep 16 02:35:08.722489 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:08.722470    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54c61f6e-cab0-494d-8ce9-93132feb0228-workload-certs\") pod \"e2e-ait-e725a02e-openshift-default-98b5bd466-mjznc\" (UID: \"54c61f6e-cab0-494d-8ce9-93132feb0228\") " pod="openshift-ingress/e2e-ait-e725a02e-openshift-default-98b5bd466-mjznc"
Sep 16 02:35:08.722908 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:08.722889    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54c61f6e-cab0-494d-8ce9-93132feb0228-istiod-ca-cert\") pod \"e2e-ait-e725a02e-openshift-default-98b5bd466-mjznc\" (UID: \"54c61f6e-cab0-494d-8ce9-93132feb0228\") " pod="openshift-ingress/e2e-ait-e725a02e-openshift-default-98b5bd466-mjznc"
Sep 16 02:35:08.724279 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:08.724254    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54c61f6e-cab0-494d-8ce9-93132feb0228-istio-envoy\") pod \"e2e-ait-e725a02e-openshift-default-98b5bd466-mjznc\" (UID: \"54c61f6e-cab0-494d-8ce9-93132feb0228\") " pod="openshift-ingress/e2e-ait-e725a02e-openshift-default-98b5bd466-mjznc"
Sep 16 02:35:08.724542 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:08.724523    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54c61f6e-cab0-494d-8ce9-93132feb0228-istio-podinfo\") pod \"e2e-ait-e725a02e-openshift-default-98b5bd466-mjznc\" (UID: \"54c61f6e-cab0-494d-8ce9-93132feb0228\") " pod="openshift-ingress/e2e-ait-e725a02e-openshift-default-98b5bd466-mjznc"
Sep 16 02:35:08.730794 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:08.730766    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54c61f6e-cab0-494d-8ce9-93132feb0228-istio-token\") pod \"e2e-ait-e725a02e-openshift-default-98b5bd466-mjznc\" (UID: \"54c61f6e-cab0-494d-8ce9-93132feb0228\") " pod="openshift-ingress/e2e-ait-e725a02e-openshift-default-98b5bd466-mjznc"
Sep 16 02:35:08.730904 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:08.730890    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gr89d\" (UniqueName: \"kubernetes.io/projected/54c61f6e-cab0-494d-8ce9-93132feb0228-kube-api-access-gr89d\") pod \"e2e-ait-e725a02e-openshift-default-98b5bd466-mjznc\" (UID: \"54c61f6e-cab0-494d-8ce9-93132feb0228\") " pod="openshift-ingress/e2e-ait-e725a02e-openshift-default-98b5bd466-mjznc"
Sep 16 02:35:08.836217 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:08.836126    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-e725a02e-openshift-default-98b5bd466-mjznc"
Sep 16 02:35:09.108787 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:09.108680    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-e725a02e-openshift-default-98b5bd466-mjznc"]
Sep 16 02:35:09.111956 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:35:09.111926    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54c61f6e_cab0_494d_8ce9_93132feb0228.slice/crio-84b5a66813d331674b5346f50d28b885e02c38f3b88be5ba3c9da774fa99e10b WatchSource:0}: Error finding container 84b5a66813d331674b5346f50d28b885e02c38f3b88be5ba3c9da774fa99e10b: Status 404 returned error can't find the container with id 84b5a66813d331674b5346f50d28b885e02c38f3b88be5ba3c9da774fa99e10b
Sep 16 02:35:09.114059 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:09.114023    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 16 02:35:09.114214 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:09.114090    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 16 02:35:09.114214 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:09.114119    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 16 02:35:09.325940 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:09.325900    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e725a02e-openshift-default-98b5bd466-mjznc" event={"ID":"54c61f6e-cab0-494d-8ce9-93132feb0228","Type":"ContainerStarted","Data":"36eabbe9fbe28d8a787de5a6f8717b62ccb7ccdd89026ee300ac156dc8923b58"}
Sep 16 02:35:09.326106 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:09.325963    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e725a02e-openshift-default-98b5bd466-mjznc" event={"ID":"54c61f6e-cab0-494d-8ce9-93132feb0228","Type":"ContainerStarted","Data":"84b5a66813d331674b5346f50d28b885e02c38f3b88be5ba3c9da774fa99e10b"}
Sep 16 02:35:09.346039 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:09.345994    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-e725a02e-openshift-default-98b5bd466-mjznc" podStartSLOduration=1.345978325 podStartE2EDuration="1.345978325s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 02:35:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:35:09.344971115 +0000 UTC m=+1322.263460401" watchObservedRunningTime="2026-09-16 02:35:09.345978325 +0000 UTC m=+1322.264467575"
Sep 16 02:35:09.836758 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:09.836721    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-e725a02e-openshift-default-98b5bd466-mjznc"
Sep 16 02:35:10.841920 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:10.841887    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-e725a02e-openshift-default-98b5bd466-mjznc"
Sep 16 02:35:11.336770 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:11.336738    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-e725a02e-openshift-default-98b5bd466-mjznc"
Sep 16 02:35:11.337599 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:11.337582    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-e725a02e-openshift-default-98b5bd466-mjznc"
Sep 16 02:35:13.978811 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:13.978774    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e725a02e-d8d8c6978-rd5kt"]
Sep 16 02:35:13.982388 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:13.982368    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-e725a02e-d8d8c6978-rd5kt"
Sep 16 02:35:13.984592 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:13.984566    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-e725a02e\""
Sep 16 02:35:13.984721 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:13.984569    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-e725a02e-dockercfg-fftbs\""
Sep 16 02:35:13.993198 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:13.993170    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e725a02e-d8d8c6978-rd5kt"]
Sep 16 02:35:14.015371 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:14.015340    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e725a02e-7748b98d45-lrhfb"]
Sep 16 02:35:14.018656 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:14.018640    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-e725a02e-7748b98d45-lrhfb"
Sep 16 02:35:14.031121 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:14.031092    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e725a02e-7748b98d45-lrhfb"]
Sep 16 02:35:14.082359 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:14.082316    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ndjd\" (UniqueName: \"kubernetes.io/projected/3d031e4a-9163-4d80-b23d-d13c51b92e0f-kube-api-access-6ndjd\") pod \"payload-pre-processing-e2e-ait-e725a02e-d8d8c6978-rd5kt\" (UID: \"3d031e4a-9163-4d80-b23d-d13c51b92e0f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e725a02e-d8d8c6978-rd5kt"
Sep 16 02:35:14.082726 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:14.082683    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d031e4a-9163-4d80-b23d-d13c51b92e0f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-e725a02e-d8d8c6978-rd5kt\" (UID: \"3d031e4a-9163-4d80-b23d-d13c51b92e0f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e725a02e-d8d8c6978-rd5kt"
Sep 16 02:35:14.184276 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:14.184237    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vpjzs\" (UniqueName: \"kubernetes.io/projected/b7523e7b-3054-4098-a6bd-fb882489c146-kube-api-access-vpjzs\") pod \"payload-processing-e2e-ait-e725a02e-7748b98d45-lrhfb\" (UID: \"b7523e7b-3054-4098-a6bd-fb882489c146\") " pod="openshift-ingress/payload-processing-e2e-ait-e725a02e-7748b98d45-lrhfb"
Sep 16 02:35:14.184459 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:14.184318    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ndjd\" (UniqueName: \"kubernetes.io/projected/3d031e4a-9163-4d80-b23d-d13c51b92e0f-kube-api-access-6ndjd\") pod \"payload-pre-processing-e2e-ait-e725a02e-d8d8c6978-rd5kt\" (UID: \"3d031e4a-9163-4d80-b23d-d13c51b92e0f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e725a02e-d8d8c6978-rd5kt"
Sep 16 02:35:14.184459 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:14.184377    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7523e7b-3054-4098-a6bd-fb882489c146-plugins-config-volume\") pod \"payload-processing-e2e-ait-e725a02e-7748b98d45-lrhfb\" (UID: \"b7523e7b-3054-4098-a6bd-fb882489c146\") " pod="openshift-ingress/payload-processing-e2e-ait-e725a02e-7748b98d45-lrhfb"
Sep 16 02:35:14.184459 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:14.184422    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d031e4a-9163-4d80-b23d-d13c51b92e0f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-e725a02e-d8d8c6978-rd5kt\" (UID: \"3d031e4a-9163-4d80-b23d-d13c51b92e0f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e725a02e-d8d8c6978-rd5kt"
Sep 16 02:35:14.185173 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:14.185152    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d031e4a-9163-4d80-b23d-d13c51b92e0f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-e725a02e-d8d8c6978-rd5kt\" (UID: \"3d031e4a-9163-4d80-b23d-d13c51b92e0f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e725a02e-d8d8c6978-rd5kt"
Sep 16 02:35:14.191726 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:14.191691    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ndjd\" (UniqueName: \"kubernetes.io/projected/3d031e4a-9163-4d80-b23d-d13c51b92e0f-kube-api-access-6ndjd\") pod \"payload-pre-processing-e2e-ait-e725a02e-d8d8c6978-rd5kt\" (UID: \"3d031e4a-9163-4d80-b23d-d13c51b92e0f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e725a02e-d8d8c6978-rd5kt"
Sep 16 02:35:14.285298 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:14.285203    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7523e7b-3054-4098-a6bd-fb882489c146-plugins-config-volume\") pod \"payload-processing-e2e-ait-e725a02e-7748b98d45-lrhfb\" (UID: \"b7523e7b-3054-4098-a6bd-fb882489c146\") " pod="openshift-ingress/payload-processing-e2e-ait-e725a02e-7748b98d45-lrhfb"
Sep 16 02:35:14.285502 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:14.285308    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpjzs\" (UniqueName: \"kubernetes.io/projected/b7523e7b-3054-4098-a6bd-fb882489c146-kube-api-access-vpjzs\") pod \"payload-processing-e2e-ait-e725a02e-7748b98d45-lrhfb\" (UID: \"b7523e7b-3054-4098-a6bd-fb882489c146\") " pod="openshift-ingress/payload-processing-e2e-ait-e725a02e-7748b98d45-lrhfb"
Sep 16 02:35:14.286093 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:14.286073    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7523e7b-3054-4098-a6bd-fb882489c146-plugins-config-volume\") pod \"payload-processing-e2e-ait-e725a02e-7748b98d45-lrhfb\" (UID: \"b7523e7b-3054-4098-a6bd-fb882489c146\") " pod="openshift-ingress/payload-processing-e2e-ait-e725a02e-7748b98d45-lrhfb"
Sep 16 02:35:14.293071 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:14.293051    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpjzs\" (UniqueName: \"kubernetes.io/projected/b7523e7b-3054-4098-a6bd-fb882489c146-kube-api-access-vpjzs\") pod \"payload-processing-e2e-ait-e725a02e-7748b98d45-lrhfb\" (UID: \"b7523e7b-3054-4098-a6bd-fb882489c146\") " pod="openshift-ingress/payload-processing-e2e-ait-e725a02e-7748b98d45-lrhfb"
Sep 16 02:35:14.293817 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:14.293796    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-e725a02e-d8d8c6978-rd5kt"
Sep 16 02:35:14.329436 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:14.329404    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-e725a02e-7748b98d45-lrhfb"
Sep 16 02:35:14.579854 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:14.579826    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e725a02e-d8d8c6978-rd5kt"]
Sep 16 02:35:14.582253 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:35:14.582226    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d031e4a_9163_4d80_b23d_d13c51b92e0f.slice/crio-3a7670074cecbb071fce3e5d86f2fc8a4a6606c47cacc1da1c5b0c5e9d4aa934 WatchSource:0}: Error finding container 3a7670074cecbb071fce3e5d86f2fc8a4a6606c47cacc1da1c5b0c5e9d4aa934: Status 404 returned error can't find the container with id 3a7670074cecbb071fce3e5d86f2fc8a4a6606c47cacc1da1c5b0c5e9d4aa934
Sep 16 02:35:14.620963 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:14.620932    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e725a02e-7748b98d45-lrhfb"]
Sep 16 02:35:14.623054 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:35:14.623028    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7523e7b_3054_4098_a6bd_fb882489c146.slice/crio-8fccf70fdcc827506a50f9772f3eb7d0c95053c7e2570e4ec781d19f6618e32a WatchSource:0}: Error finding container 8fccf70fdcc827506a50f9772f3eb7d0c95053c7e2570e4ec781d19f6618e32a: Status 404 returned error can't find the container with id 8fccf70fdcc827506a50f9772f3eb7d0c95053c7e2570e4ec781d19f6618e32a
Sep 16 02:35:15.370072 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:15.370031    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-e725a02e-7748b98d45-lrhfb" event={"ID":"b7523e7b-3054-4098-a6bd-fb882489c146","Type":"ContainerStarted","Data":"a35f968b65c615f12beaf3579fb4434d769086581ea4e4fa4bb66e7d31dd2c67"}
Sep 16 02:35:15.370563 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:15.370102    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-e725a02e-7748b98d45-lrhfb" event={"ID":"b7523e7b-3054-4098-a6bd-fb882489c146","Type":"ContainerStarted","Data":"8fccf70fdcc827506a50f9772f3eb7d0c95053c7e2570e4ec781d19f6618e32a"}
Sep 16 02:35:15.370563 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:15.370150    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-e725a02e-7748b98d45-lrhfb"
Sep 16 02:35:15.371617 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:15.371593    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-e725a02e-d8d8c6978-rd5kt" event={"ID":"3d031e4a-9163-4d80-b23d-d13c51b92e0f","Type":"ContainerStarted","Data":"ad85fbdf95464fa2d05a5583c1397bd28196e90dd6c5464ff97797899f01466e"}
Sep 16 02:35:15.371768 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:15.371638    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-e725a02e-d8d8c6978-rd5kt" event={"ID":"3d031e4a-9163-4d80-b23d-d13c51b92e0f","Type":"ContainerStarted","Data":"3a7670074cecbb071fce3e5d86f2fc8a4a6606c47cacc1da1c5b0c5e9d4aa934"}
Sep 16 02:35:15.371768 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:15.371659    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-e725a02e-d8d8c6978-rd5kt"
Sep 16 02:35:15.387458 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:15.387410    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-e725a02e-7748b98d45-lrhfb" podStartSLOduration=2.387395885 podStartE2EDuration="2.387395885s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 02:35:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:35:15.384264579 +0000 UTC m=+1328.302753840" watchObservedRunningTime="2026-09-16 02:35:15.387395885 +0000 UTC m=+1328.305885162"
Sep 16 02:35:15.400004 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:15.399955    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-e725a02e-d8d8c6978-rd5kt" podStartSLOduration=2.399938481 podStartE2EDuration="2.399938481s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 02:35:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:35:15.398074256 +0000 UTC m=+1328.316563506" watchObservedRunningTime="2026-09-16 02:35:15.399938481 +0000 UTC m=+1328.318427750"
Sep 16 02:35:26.378389 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:26.378353    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-e725a02e-7748b98d45-lrhfb"
Sep 16 02:35:26.378801 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:26.378533    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-e725a02e-d8d8c6978-rd5kt"
Sep 16 02:35:36.129465 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:36.129382    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e725a02e-6s59l"]
Sep 16 02:35:36.133627 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:36.133607    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e725a02e-6s59l"
Sep 16 02:35:36.135789 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:36.135762    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 16 02:35:36.135906 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:36.135838    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jpd4f\""
Sep 16 02:35:36.140272 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:36.140240    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e725a02e-6s59l"]
Sep 16 02:35:36.213643 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:36.213607    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1d50dd7-561a-46d0-b7fa-8d006e37c4ae-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e725a02e-6s59l\" (UID: \"c1d50dd7-561a-46d0-b7fa-8d006e37c4ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e725a02e-6s59l"
Sep 16 02:35:36.315180 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:36.315142    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1d50dd7-561a-46d0-b7fa-8d006e37c4ae-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e725a02e-6s59l\" (UID: \"c1d50dd7-561a-46d0-b7fa-8d006e37c4ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e725a02e-6s59l"
Sep 16 02:35:36.315829 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:36.315803    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1d50dd7-561a-46d0-b7fa-8d006e37c4ae-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e725a02e-6s59l\" (UID: \"c1d50dd7-561a-46d0-b7fa-8d006e37c4ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e725a02e-6s59l"
Sep 16 02:35:36.447334 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:36.447035    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e725a02e-6s59l"
Sep 16 02:35:36.728776 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:36.728732    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e725a02e-6s59l"]
Sep 16 02:35:36.731222 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:35:36.731197    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1d50dd7_561a_46d0_b7fa_8d006e37c4ae.slice/crio-3732ea080c12af92c4227ceb8b7948ac263209e04902c28b6766127aa992dfc1 WatchSource:0}: Error finding container 3732ea080c12af92c4227ceb8b7948ac263209e04902c28b6766127aa992dfc1: Status 404 returned error can't find the container with id 3732ea080c12af92c4227ceb8b7948ac263209e04902c28b6766127aa992dfc1
Sep 16 02:35:37.493154 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:37.493112    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e725a02e-6s59l" event={"ID":"c1d50dd7-561a-46d0-b7fa-8d006e37c4ae","Type":"ContainerStarted","Data":"6ba257f9fa9a6db8acfcb229fac839b9c2c095ad70a4e84d5544ac3325429347"}
Sep 16 02:35:37.493555 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:37.493169    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e725a02e-6s59l" event={"ID":"c1d50dd7-561a-46d0-b7fa-8d006e37c4ae","Type":"ContainerStarted","Data":"3732ea080c12af92c4227ceb8b7948ac263209e04902c28b6766127aa992dfc1"}
Sep 16 02:35:38.502063 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:38.502029    2558 generic.go:356] "Generic (PLEG): container finished" podID="c1d50dd7-561a-46d0-b7fa-8d006e37c4ae" containerID="6ba257f9fa9a6db8acfcb229fac839b9c2c095ad70a4e84d5544ac3325429347" exitCode=0
Sep 16 02:35:38.502463 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:38.502111    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e725a02e-6s59l" event={"ID":"c1d50dd7-561a-46d0-b7fa-8d006e37c4ae","Type":"ContainerDied","Data":"6ba257f9fa9a6db8acfcb229fac839b9c2c095ad70a4e84d5544ac3325429347"}
Sep 16 02:35:39.646005 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:39.645975    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e725a02e-6s59l"
Sep 16 02:35:39.752572 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:39.752539    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1d50dd7-561a-46d0-b7fa-8d006e37c4ae-maas-api-service-ca\") pod \"c1d50dd7-561a-46d0-b7fa-8d006e37c4ae\" (UID: \"c1d50dd7-561a-46d0-b7fa-8d006e37c4ae\") "
Sep 16 02:35:39.752933 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:39.752906    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1d50dd7-561a-46d0-b7fa-8d006e37c4ae-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c1d50dd7-561a-46d0-b7fa-8d006e37c4ae" (UID: "c1d50dd7-561a-46d0-b7fa-8d006e37c4ae"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 02:35:39.854171 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:39.854080    2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1d50dd7-561a-46d0-b7fa-8d006e37c4ae-maas-api-service-ca\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:35:40.515760 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:40.515721    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e725a02e-6s59l" event={"ID":"c1d50dd7-561a-46d0-b7fa-8d006e37c4ae","Type":"ContainerDied","Data":"3732ea080c12af92c4227ceb8b7948ac263209e04902c28b6766127aa992dfc1"}
Sep 16 02:35:40.515929 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:40.515765    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e725a02e-6s59l"
Sep 16 02:35:40.515929 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:40.515784    2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3732ea080c12af92c4227ceb8b7948ac263209e04902c28b6766127aa992dfc1"
Sep 16 02:35:41.338065 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:41.337988    2558 patch_prober.go:28] interesting pod/e2e-ait-e725a02e-openshift-default-98b5bd466-mjznc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body=
Sep 16 02:35:41.338065 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:41.338043    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-e725a02e-openshift-default-98b5bd466-mjznc" podUID="54c61f6e-cab0-494d-8ce9-93132feb0228" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused"
Sep 16 02:35:41.522270 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:41.522237    2558 generic.go:356] "Generic (PLEG): container finished" podID="54c61f6e-cab0-494d-8ce9-93132feb0228" containerID="36eabbe9fbe28d8a787de5a6f8717b62ccb7ccdd89026ee300ac156dc8923b58" exitCode=0
Sep 16 02:35:41.522433 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:41.522308    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e725a02e-openshift-default-98b5bd466-mjznc" event={"ID":"54c61f6e-cab0-494d-8ce9-93132feb0228","Type":"ContainerDied","Data":"36eabbe9fbe28d8a787de5a6f8717b62ccb7ccdd89026ee300ac156dc8923b58"}
Sep 16 02:35:41.522735 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:41.522721    2558 scope.go:117] "RemoveContainer" containerID="36eabbe9fbe28d8a787de5a6f8717b62ccb7ccdd89026ee300ac156dc8923b58"
Sep 16 02:35:41.523938 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:41.523910    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 16 02:35:41.524007 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:41.523978    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 16 02:35:41.524045 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:41.524008    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 16 02:35:42.529533 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:42.529501    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e725a02e-openshift-default-98b5bd466-mjznc" event={"ID":"54c61f6e-cab0-494d-8ce9-93132feb0228","Type":"ContainerStarted","Data":"d28a4e2e720c9eef514c796e7bfdfcbf410d5b1a023a13d7a0e9d098bd329364"}
Sep 16 02:35:42.683481 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:42.683443    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e725a02e-7748b98d45-lrhfb"]
Sep 16 02:35:42.683780 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:42.683748    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-e725a02e-7748b98d45-lrhfb" podUID="b7523e7b-3054-4098-a6bd-fb882489c146" containerName="payload-processing" containerID="cri-o://a35f968b65c615f12beaf3579fb4434d769086581ea4e4fa4bb66e7d31dd2c67" gracePeriod=30
Sep 16 02:35:42.688726 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:42.688405    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e725a02e-d8d8c6978-rd5kt"]
Sep 16 02:35:42.688726 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:42.688601    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-e725a02e-d8d8c6978-rd5kt" podUID="3d031e4a-9163-4d80-b23d-d13c51b92e0f" containerName="payload-pre-processing" containerID="cri-o://ad85fbdf95464fa2d05a5583c1397bd28196e90dd6c5464ff97797899f01466e" gracePeriod=30
Sep 16 02:35:42.836891 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:42.836669    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-e725a02e-openshift-default-98b5bd466-mjznc"
Sep 16 02:35:42.836891 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:42.836880    2558 patch_prober.go:28] interesting pod/e2e-ait-e725a02e-openshift-default-98b5bd466-mjznc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body=
Sep 16 02:35:42.837147 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:42.836940    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-e725a02e-openshift-default-98b5bd466-mjznc" podUID="54c61f6e-cab0-494d-8ce9-93132feb0228" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused"
Sep 16 02:35:42.837147 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:42.837021    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e725a02e-6s59l"]
Sep 16 02:35:42.839357 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:42.839330    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e725a02e-6s59l"]
Sep 16 02:35:42.982733 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:42.969688    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-e725a02e-openshift-default-98b5bd466-mjznc"]
Sep 16 02:35:42.988499 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:42.988277    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-e725a02e-7748b98d45-lrhfb"
Sep 16 02:35:43.028283 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:43.028256    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-e725a02e-d8d8c6978-rd5kt"
Sep 16 02:35:43.091777 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:43.091092    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7523e7b-3054-4098-a6bd-fb882489c146-plugins-config-volume\") pod \"b7523e7b-3054-4098-a6bd-fb882489c146\" (UID: \"b7523e7b-3054-4098-a6bd-fb882489c146\") "
Sep 16 02:35:43.091777 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:43.091250    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vpjzs\" (UniqueName: \"kubernetes.io/projected/b7523e7b-3054-4098-a6bd-fb882489c146-kube-api-access-vpjzs\") pod \"b7523e7b-3054-4098-a6bd-fb882489c146\" (UID: \"b7523e7b-3054-4098-a6bd-fb882489c146\") "
Sep 16 02:35:43.092629 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:43.092603    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7523e7b-3054-4098-a6bd-fb882489c146-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b7523e7b-3054-4098-a6bd-fb882489c146" (UID: "b7523e7b-3054-4098-a6bd-fb882489c146"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 02:35:43.095489 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:43.095445    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7523e7b-3054-4098-a6bd-fb882489c146-kube-api-access-vpjzs" (OuterVolumeSpecName: "kube-api-access-vpjzs") pod "b7523e7b-3054-4098-a6bd-fb882489c146" (UID: "b7523e7b-3054-4098-a6bd-fb882489c146"). InnerVolumeSpecName "kube-api-access-vpjzs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 02:35:43.192917 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:43.192883    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d031e4a-9163-4d80-b23d-d13c51b92e0f-plugins-config-volume\") pod \"3d031e4a-9163-4d80-b23d-d13c51b92e0f\" (UID: \"3d031e4a-9163-4d80-b23d-d13c51b92e0f\") "
Sep 16 02:35:43.193104 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:43.192991    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6ndjd\" (UniqueName: \"kubernetes.io/projected/3d031e4a-9163-4d80-b23d-d13c51b92e0f-kube-api-access-6ndjd\") pod \"3d031e4a-9163-4d80-b23d-d13c51b92e0f\" (UID: \"3d031e4a-9163-4d80-b23d-d13c51b92e0f\") "
Sep 16 02:35:43.193242 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:43.193228    2558 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7523e7b-3054-4098-a6bd-fb882489c146-plugins-config-volume\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:35:43.193301 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:43.193243    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vpjzs\" (UniqueName: \"kubernetes.io/projected/b7523e7b-3054-4098-a6bd-fb882489c146-kube-api-access-vpjzs\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:35:43.193360 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:43.193332    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d031e4a-9163-4d80-b23d-d13c51b92e0f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3d031e4a-9163-4d80-b23d-d13c51b92e0f" (UID: "3d031e4a-9163-4d80-b23d-d13c51b92e0f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 02:35:43.195102 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:43.195067    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d031e4a-9163-4d80-b23d-d13c51b92e0f-kube-api-access-6ndjd" (OuterVolumeSpecName: "kube-api-access-6ndjd") pod "3d031e4a-9163-4d80-b23d-d13c51b92e0f" (UID: "3d031e4a-9163-4d80-b23d-d13c51b92e0f"). InnerVolumeSpecName "kube-api-access-6ndjd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 02:35:43.294369 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:43.294325    2558 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d031e4a-9163-4d80-b23d-d13c51b92e0f-plugins-config-volume\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:35:43.294620 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:43.294600    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6ndjd\" (UniqueName: \"kubernetes.io/projected/3d031e4a-9163-4d80-b23d-d13c51b92e0f-kube-api-access-6ndjd\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:35:43.537635 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:43.537601    2558 generic.go:356] "Generic (PLEG): container finished" podID="3d031e4a-9163-4d80-b23d-d13c51b92e0f" containerID="ad85fbdf95464fa2d05a5583c1397bd28196e90dd6c5464ff97797899f01466e" exitCode=0
Sep 16 02:35:43.538146 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:43.537668    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-e725a02e-d8d8c6978-rd5kt"
Sep 16 02:35:43.538146 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:43.537706    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-e725a02e-d8d8c6978-rd5kt" event={"ID":"3d031e4a-9163-4d80-b23d-d13c51b92e0f","Type":"ContainerDied","Data":"ad85fbdf95464fa2d05a5583c1397bd28196e90dd6c5464ff97797899f01466e"}
Sep 16 02:35:43.538146 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:43.537794    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-e725a02e-d8d8c6978-rd5kt" event={"ID":"3d031e4a-9163-4d80-b23d-d13c51b92e0f","Type":"ContainerDied","Data":"3a7670074cecbb071fce3e5d86f2fc8a4a6606c47cacc1da1c5b0c5e9d4aa934"}
Sep 16 02:35:43.538146 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:43.537825    2558 scope.go:117] "RemoveContainer" containerID="ad85fbdf95464fa2d05a5583c1397bd28196e90dd6c5464ff97797899f01466e"
Sep 16 02:35:43.539042 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:43.539022    2558 generic.go:356] "Generic (PLEG): container finished" podID="b7523e7b-3054-4098-a6bd-fb882489c146" containerID="a35f968b65c615f12beaf3579fb4434d769086581ea4e4fa4bb66e7d31dd2c67" exitCode=0
Sep 16 02:35:43.539133 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:43.539071    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-e725a02e-7748b98d45-lrhfb"
Sep 16 02:35:43.539133 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:43.539112    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-e725a02e-7748b98d45-lrhfb" event={"ID":"b7523e7b-3054-4098-a6bd-fb882489c146","Type":"ContainerDied","Data":"a35f968b65c615f12beaf3579fb4434d769086581ea4e4fa4bb66e7d31dd2c67"}
Sep 16 02:35:43.539248 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:43.539143    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-e725a02e-7748b98d45-lrhfb" event={"ID":"b7523e7b-3054-4098-a6bd-fb882489c146","Type":"ContainerDied","Data":"8fccf70fdcc827506a50f9772f3eb7d0c95053c7e2570e4ec781d19f6618e32a"}
Sep 16 02:35:43.539578 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:43.539544    2558 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/e2e-ait-e725a02e-openshift-default-98b5bd466-mjznc" secret="" err="secret \"e2e-ait-e725a02e-openshift-default-dockercfg-2g6j8\" not found"
Sep 16 02:35:43.561531 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:43.561502    2558 scope.go:117] "RemoveContainer" containerID="ad85fbdf95464fa2d05a5583c1397bd28196e90dd6c5464ff97797899f01466e"
Sep 16 02:35:43.571846 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:35:43.571785    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad85fbdf95464fa2d05a5583c1397bd28196e90dd6c5464ff97797899f01466e\": container with ID starting with ad85fbdf95464fa2d05a5583c1397bd28196e90dd6c5464ff97797899f01466e not found: ID does not exist" containerID="ad85fbdf95464fa2d05a5583c1397bd28196e90dd6c5464ff97797899f01466e"
Sep 16 02:35:43.572000 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:43.571846    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad85fbdf95464fa2d05a5583c1397bd28196e90dd6c5464ff97797899f01466e"} err="failed to get container status \"ad85fbdf95464fa2d05a5583c1397bd28196e90dd6c5464ff97797899f01466e\": rpc error: code = NotFound desc = could not find container \"ad85fbdf95464fa2d05a5583c1397bd28196e90dd6c5464ff97797899f01466e\": container with ID starting with ad85fbdf95464fa2d05a5583c1397bd28196e90dd6c5464ff97797899f01466e not found: ID does not exist"
Sep 16 02:35:43.572000 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:43.571872    2558 scope.go:117] "RemoveContainer" containerID="a35f968b65c615f12beaf3579fb4434d769086581ea4e4fa4bb66e7d31dd2c67"
Sep 16 02:35:43.573684 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:43.573656    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c1d50dd7-561a-46d0-b7fa-8d006e37c4ae" path="/var/lib/kubelet/pods/c1d50dd7-561a-46d0-b7fa-8d006e37c4ae/volumes"
Sep 16 02:35:43.582428 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:43.582409    2558 scope.go:117] "RemoveContainer" containerID="a35f968b65c615f12beaf3579fb4434d769086581ea4e4fa4bb66e7d31dd2c67"
Sep 16 02:35:43.582777 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:35:43.582759    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a35f968b65c615f12beaf3579fb4434d769086581ea4e4fa4bb66e7d31dd2c67\": container with ID starting with a35f968b65c615f12beaf3579fb4434d769086581ea4e4fa4bb66e7d31dd2c67 not found: ID does not exist" containerID="a35f968b65c615f12beaf3579fb4434d769086581ea4e4fa4bb66e7d31dd2c67"
Sep 16 02:35:43.582851 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:43.582786    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a35f968b65c615f12beaf3579fb4434d769086581ea4e4fa4bb66e7d31dd2c67"} err="failed to get container status \"a35f968b65c615f12beaf3579fb4434d769086581ea4e4fa4bb66e7d31dd2c67\": rpc error: code = NotFound desc = could not find container \"a35f968b65c615f12beaf3579fb4434d769086581ea4e4fa4bb66e7d31dd2c67\": container with ID starting with a35f968b65c615f12beaf3579fb4434d769086581ea4e4fa4bb66e7d31dd2c67 not found: ID does not exist"
Sep 16 02:35:43.592236 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:43.592212    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e725a02e-7748b98d45-lrhfb"]
Sep 16 02:35:43.595834 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:43.595813    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e725a02e-7748b98d45-lrhfb"]
Sep 16 02:35:43.605446 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:43.605423    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e725a02e-d8d8c6978-rd5kt"]
Sep 16 02:35:43.607077 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:43.607059    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e725a02e-d8d8c6978-rd5kt"]
Sep 16 02:35:43.836896 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:43.836811    2558 patch_prober.go:28] interesting pod/e2e-ait-e725a02e-openshift-default-98b5bd466-mjznc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body=
Sep 16 02:35:43.836896 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:43.836876    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-e725a02e-openshift-default-98b5bd466-mjznc" podUID="54c61f6e-cab0-494d-8ce9-93132feb0228" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused"
Sep 16 02:35:43.914142 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:43.914109    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-0bdfbb54-gw-openshift-default-5ccb9467b4-4j8g7"]
Sep 16 02:35:43.914720 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:43.914675    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c1d50dd7-561a-46d0-b7fa-8d006e37c4ae" containerName="revoke-keys"
Sep 16 02:35:43.914720 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:43.914716    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="c1d50dd7-561a-46d0-b7fa-8d006e37c4ae" containerName="revoke-keys"
Sep 16 02:35:43.914829 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:43.914738    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b7523e7b-3054-4098-a6bd-fb882489c146" containerName="payload-processing"
Sep 16 02:35:43.914829 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:43.914748    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="b7523e7b-3054-4098-a6bd-fb882489c146" containerName="payload-processing"
Sep 16 02:35:43.914829 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:43.914762    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d031e4a-9163-4d80-b23d-d13c51b92e0f" containerName="payload-pre-processing"
Sep 16 02:35:43.914829 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:43.914772    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d031e4a-9163-4d80-b23d-d13c51b92e0f" containerName="payload-pre-processing"
Sep 16 02:35:43.914955 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:43.914882    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="c1d50dd7-561a-46d0-b7fa-8d006e37c4ae" containerName="revoke-keys"
Sep 16 02:35:43.914955 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:43.914894    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d031e4a-9163-4d80-b23d-d13c51b92e0f" containerName="payload-pre-processing"
Sep 16 02:35:43.914955 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:43.914901    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="b7523e7b-3054-4098-a6bd-fb882489c146" containerName="payload-processing"
Sep 16 02:35:43.919321 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:43.919301    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-0bdfbb54-gw-openshift-default-5ccb9467b4-4j8g7"
Sep 16 02:35:43.921766 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:43.921743    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-0bdfbb54-gw-openshift-default-dockercfg-bkzr4\""
Sep 16 02:35:43.927183 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:43.926742    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-0bdfbb54-gw-openshift-default-5ccb9467b4-4j8g7"]
Sep 16 02:35:44.002779 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:44.002729    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e0742d4-fc6e-4bcd-8d28-ee56bcd52b83-istio-data\") pod \"e2e-migrate-0bdfbb54-gw-openshift-default-5ccb9467b4-4j8g7\" (UID: \"9e0742d4-fc6e-4bcd-8d28-ee56bcd52b83\") " pod="openshift-ingress/e2e-migrate-0bdfbb54-gw-openshift-default-5ccb9467b4-4j8g7"
Sep 16 02:35:44.003179 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:44.003153    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e0742d4-fc6e-4bcd-8d28-ee56bcd52b83-istio-podinfo\") pod \"e2e-migrate-0bdfbb54-gw-openshift-default-5ccb9467b4-4j8g7\" (UID: \"9e0742d4-fc6e-4bcd-8d28-ee56bcd52b83\") " pod="openshift-ingress/e2e-migrate-0bdfbb54-gw-openshift-default-5ccb9467b4-4j8g7"
Sep 16 02:35:44.003347 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:44.003331    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e0742d4-fc6e-4bcd-8d28-ee56bcd52b83-istio-token\") pod \"e2e-migrate-0bdfbb54-gw-openshift-default-5ccb9467b4-4j8g7\" (UID: \"9e0742d4-fc6e-4bcd-8d28-ee56bcd52b83\") " pod="openshift-ingress/e2e-migrate-0bdfbb54-gw-openshift-default-5ccb9467b4-4j8g7"
Sep 16 02:35:44.003502 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:44.003487    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e0742d4-fc6e-4bcd-8d28-ee56bcd52b83-workload-socket\") pod \"e2e-migrate-0bdfbb54-gw-openshift-default-5ccb9467b4-4j8g7\" (UID: \"9e0742d4-fc6e-4bcd-8d28-ee56bcd52b83\") " pod="openshift-ingress/e2e-migrate-0bdfbb54-gw-openshift-default-5ccb9467b4-4j8g7"
Sep 16 02:35:44.003692 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:44.003677    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e0742d4-fc6e-4bcd-8d28-ee56bcd52b83-credential-socket\") pod \"e2e-migrate-0bdfbb54-gw-openshift-default-5ccb9467b4-4j8g7\" (UID: \"9e0742d4-fc6e-4bcd-8d28-ee56bcd52b83\") " pod="openshift-ingress/e2e-migrate-0bdfbb54-gw-openshift-default-5ccb9467b4-4j8g7"
Sep 16 02:35:44.003883 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:44.003856    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e0742d4-fc6e-4bcd-8d28-ee56bcd52b83-istiod-ca-cert\") pod \"e2e-migrate-0bdfbb54-gw-openshift-default-5ccb9467b4-4j8g7\" (UID: \"9e0742d4-fc6e-4bcd-8d28-ee56bcd52b83\") " pod="openshift-ingress/e2e-migrate-0bdfbb54-gw-openshift-default-5ccb9467b4-4j8g7"
Sep 16 02:35:44.004145 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:44.004108    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cv65k\" (UniqueName: \"kubernetes.io/projected/9e0742d4-fc6e-4bcd-8d28-ee56bcd52b83-kube-api-access-cv65k\") pod \"e2e-migrate-0bdfbb54-gw-openshift-default-5ccb9467b4-4j8g7\" (UID: \"9e0742d4-fc6e-4bcd-8d28-ee56bcd52b83\") " pod="openshift-ingress/e2e-migrate-0bdfbb54-gw-openshift-default-5ccb9467b4-4j8g7"
Sep 16 02:35:44.004331 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:44.004296    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e0742d4-fc6e-4bcd-8d28-ee56bcd52b83-istio-envoy\") pod \"e2e-migrate-0bdfbb54-gw-openshift-default-5ccb9467b4-4j8g7\" (UID: \"9e0742d4-fc6e-4bcd-8d28-ee56bcd52b83\") " pod="openshift-ingress/e2e-migrate-0bdfbb54-gw-openshift-default-5ccb9467b4-4j8g7"
Sep 16 02:35:44.004481 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:44.004465    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e0742d4-fc6e-4bcd-8d28-ee56bcd52b83-workload-certs\") pod \"e2e-migrate-0bdfbb54-gw-openshift-default-5ccb9467b4-4j8g7\" (UID: \"9e0742d4-fc6e-4bcd-8d28-ee56bcd52b83\") " pod="openshift-ingress/e2e-migrate-0bdfbb54-gw-openshift-default-5ccb9467b4-4j8g7"
Sep 16 02:35:44.105417 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:44.105324    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e0742d4-fc6e-4bcd-8d28-ee56bcd52b83-credential-socket\") pod \"e2e-migrate-0bdfbb54-gw-openshift-default-5ccb9467b4-4j8g7\" (UID: \"9e0742d4-fc6e-4bcd-8d28-ee56bcd52b83\") " pod="openshift-ingress/e2e-migrate-0bdfbb54-gw-openshift-default-5ccb9467b4-4j8g7"
Sep 16 02:35:44.105417 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:44.105371    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e0742d4-fc6e-4bcd-8d28-ee56bcd52b83-istiod-ca-cert\") pod \"e2e-migrate-0bdfbb54-gw-openshift-default-5ccb9467b4-4j8g7\" (UID: \"9e0742d4-fc6e-4bcd-8d28-ee56bcd52b83\") " pod="openshift-ingress/e2e-migrate-0bdfbb54-gw-openshift-default-5ccb9467b4-4j8g7"
Sep 16 02:35:44.105620 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:44.105425    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cv65k\" (UniqueName: \"kubernetes.io/projected/9e0742d4-fc6e-4bcd-8d28-ee56bcd52b83-kube-api-access-cv65k\") pod \"e2e-migrate-0bdfbb54-gw-openshift-default-5ccb9467b4-4j8g7\" (UID: \"9e0742d4-fc6e-4bcd-8d28-ee56bcd52b83\") " pod="openshift-ingress/e2e-migrate-0bdfbb54-gw-openshift-default-5ccb9467b4-4j8g7"
Sep 16 02:35:44.105620 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:44.105549    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e0742d4-fc6e-4bcd-8d28-ee56bcd52b83-istio-envoy\") pod \"e2e-migrate-0bdfbb54-gw-openshift-default-5ccb9467b4-4j8g7\" (UID: \"9e0742d4-fc6e-4bcd-8d28-ee56bcd52b83\") " pod="openshift-ingress/e2e-migrate-0bdfbb54-gw-openshift-default-5ccb9467b4-4j8g7"
Sep 16 02:35:44.105620 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:44.105588    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e0742d4-fc6e-4bcd-8d28-ee56bcd52b83-workload-certs\") pod \"e2e-migrate-0bdfbb54-gw-openshift-default-5ccb9467b4-4j8g7\" (UID: \"9e0742d4-fc6e-4bcd-8d28-ee56bcd52b83\") " pod="openshift-ingress/e2e-migrate-0bdfbb54-gw-openshift-default-5ccb9467b4-4j8g7"
Sep 16 02:35:44.105779 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:44.105632    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e0742d4-fc6e-4bcd-8d28-ee56bcd52b83-istio-data\") pod \"e2e-migrate-0bdfbb54-gw-openshift-default-5ccb9467b4-4j8g7\" (UID: \"9e0742d4-fc6e-4bcd-8d28-ee56bcd52b83\") " pod="openshift-ingress/e2e-migrate-0bdfbb54-gw-openshift-default-5ccb9467b4-4j8g7"
Sep 16 02:35:44.105779 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:44.105724    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e0742d4-fc6e-4bcd-8d28-ee56bcd52b83-credential-socket\") pod \"e2e-migrate-0bdfbb54-gw-openshift-default-5ccb9467b4-4j8g7\" (UID: \"9e0742d4-fc6e-4bcd-8d28-ee56bcd52b83\") " pod="openshift-ingress/e2e-migrate-0bdfbb54-gw-openshift-default-5ccb9467b4-4j8g7"
Sep 16 02:35:44.105779 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:44.105765    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e0742d4-fc6e-4bcd-8d28-ee56bcd52b83-istio-podinfo\") pod \"e2e-migrate-0bdfbb54-gw-openshift-default-5ccb9467b4-4j8g7\" (UID: \"9e0742d4-fc6e-4bcd-8d28-ee56bcd52b83\") " pod="openshift-ingress/e2e-migrate-0bdfbb54-gw-openshift-default-5ccb9467b4-4j8g7"
Sep 16 02:35:44.105941 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:44.105804    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e0742d4-fc6e-4bcd-8d28-ee56bcd52b83-istio-token\") pod \"e2e-migrate-0bdfbb54-gw-openshift-default-5ccb9467b4-4j8g7\" (UID: \"9e0742d4-fc6e-4bcd-8d28-ee56bcd52b83\") " pod="openshift-ingress/e2e-migrate-0bdfbb54-gw-openshift-default-5ccb9467b4-4j8g7"
Sep 16 02:35:44.105941 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:44.105853    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e0742d4-fc6e-4bcd-8d28-ee56bcd52b83-workload-socket\") pod \"e2e-migrate-0bdfbb54-gw-openshift-default-5ccb9467b4-4j8g7\" (UID: \"9e0742d4-fc6e-4bcd-8d28-ee56bcd52b83\") " pod="openshift-ingress/e2e-migrate-0bdfbb54-gw-openshift-default-5ccb9467b4-4j8g7"
Sep 16 02:35:44.106043 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:44.105993    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e0742d4-fc6e-4bcd-8d28-ee56bcd52b83-workload-certs\") pod \"e2e-migrate-0bdfbb54-gw-openshift-default-5ccb9467b4-4j8g7\" (UID: \"9e0742d4-fc6e-4bcd-8d28-ee56bcd52b83\") " pod="openshift-ingress/e2e-migrate-0bdfbb54-gw-openshift-default-5ccb9467b4-4j8g7"
Sep 16 02:35:44.106096 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:44.106072    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e0742d4-fc6e-4bcd-8d28-ee56bcd52b83-istio-data\") pod \"e2e-migrate-0bdfbb54-gw-openshift-default-5ccb9467b4-4j8g7\" (UID: \"9e0742d4-fc6e-4bcd-8d28-ee56bcd52b83\") " pod="openshift-ingress/e2e-migrate-0bdfbb54-gw-openshift-default-5ccb9467b4-4j8g7"
Sep 16 02:35:44.106206 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:44.106186    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e0742d4-fc6e-4bcd-8d28-ee56bcd52b83-istiod-ca-cert\") pod \"e2e-migrate-0bdfbb54-gw-openshift-default-5ccb9467b4-4j8g7\" (UID: \"9e0742d4-fc6e-4bcd-8d28-ee56bcd52b83\") " pod="openshift-ingress/e2e-migrate-0bdfbb54-gw-openshift-default-5ccb9467b4-4j8g7"
Sep 16 02:35:44.106262 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:44.106188    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e0742d4-fc6e-4bcd-8d28-ee56bcd52b83-workload-socket\") pod \"e2e-migrate-0bdfbb54-gw-openshift-default-5ccb9467b4-4j8g7\" (UID: \"9e0742d4-fc6e-4bcd-8d28-ee56bcd52b83\") " pod="openshift-ingress/e2e-migrate-0bdfbb54-gw-openshift-default-5ccb9467b4-4j8g7"
Sep 16 02:35:44.108002 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:44.107981    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e0742d4-fc6e-4bcd-8d28-ee56bcd52b83-istio-envoy\") pod \"e2e-migrate-0bdfbb54-gw-openshift-default-5ccb9467b4-4j8g7\" (UID: \"9e0742d4-fc6e-4bcd-8d28-ee56bcd52b83\") " pod="openshift-ingress/e2e-migrate-0bdfbb54-gw-openshift-default-5ccb9467b4-4j8g7"
Sep 16 02:35:44.108268 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:44.108251    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e0742d4-fc6e-4bcd-8d28-ee56bcd52b83-istio-podinfo\") pod \"e2e-migrate-0bdfbb54-gw-openshift-default-5ccb9467b4-4j8g7\" (UID: \"9e0742d4-fc6e-4bcd-8d28-ee56bcd52b83\") " pod="openshift-ingress/e2e-migrate-0bdfbb54-gw-openshift-default-5ccb9467b4-4j8g7"
Sep 16 02:35:44.112813 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:44.112784    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e0742d4-fc6e-4bcd-8d28-ee56bcd52b83-istio-token\") pod \"e2e-migrate-0bdfbb54-gw-openshift-default-5ccb9467b4-4j8g7\" (UID: \"9e0742d4-fc6e-4bcd-8d28-ee56bcd52b83\") " pod="openshift-ingress/e2e-migrate-0bdfbb54-gw-openshift-default-5ccb9467b4-4j8g7"
Sep 16 02:35:44.112910 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:44.112847    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cv65k\" (UniqueName: \"kubernetes.io/projected/9e0742d4-fc6e-4bcd-8d28-ee56bcd52b83-kube-api-access-cv65k\") pod \"e2e-migrate-0bdfbb54-gw-openshift-default-5ccb9467b4-4j8g7\" (UID: \"9e0742d4-fc6e-4bcd-8d28-ee56bcd52b83\") " pod="openshift-ingress/e2e-migrate-0bdfbb54-gw-openshift-default-5ccb9467b4-4j8g7"
Sep 16 02:35:44.231401 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:44.231365    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-0bdfbb54-gw-openshift-default-5ccb9467b4-4j8g7"
Sep 16 02:35:44.524922 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:44.524893    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-0bdfbb54-gw-openshift-default-5ccb9467b4-4j8g7"]
Sep 16 02:35:44.526727 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:35:44.526692    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e0742d4_fc6e_4bcd_8d28_ee56bcd52b83.slice/crio-4ebd747875fe9e24d92c2b4b07f6dad758d441e3510b2051ba7ec4d3bc78da26 WatchSource:0}: Error finding container 4ebd747875fe9e24d92c2b4b07f6dad758d441e3510b2051ba7ec4d3bc78da26: Status 404 returned error can't find the container with id 4ebd747875fe9e24d92c2b4b07f6dad758d441e3510b2051ba7ec4d3bc78da26
Sep 16 02:35:44.528931 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:44.528898    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 16 02:35:44.529009 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:44.528964    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 16 02:35:44.529009 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:44.528994    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 16 02:35:44.558369 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:44.558280    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-0bdfbb54-gw-openshift-default-5ccb9467b4-4j8g7" event={"ID":"9e0742d4-fc6e-4bcd-8d28-ee56bcd52b83","Type":"ContainerStarted","Data":"4ebd747875fe9e24d92c2b4b07f6dad758d441e3510b2051ba7ec4d3bc78da26"}
Sep 16 02:35:44.561801 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:44.561264    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-e725a02e-openshift-default-98b5bd466-mjznc" podUID="54c61f6e-cab0-494d-8ce9-93132feb0228" containerName="istio-proxy" containerID="cri-o://d28a4e2e720c9eef514c796e7bfdfcbf410d5b1a023a13d7a0e9d098bd329364" gracePeriod=30
Sep 16 02:35:45.554531 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:45.554492    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3d031e4a-9163-4d80-b23d-d13c51b92e0f" path="/var/lib/kubelet/pods/3d031e4a-9163-4d80-b23d-d13c51b92e0f/volumes"
Sep 16 02:35:45.555592 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:45.555567    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b7523e7b-3054-4098-a6bd-fb882489c146" path="/var/lib/kubelet/pods/b7523e7b-3054-4098-a6bd-fb882489c146/volumes"
Sep 16 02:35:45.572182 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:45.572142    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-0bdfbb54-gw-openshift-default-5ccb9467b4-4j8g7" event={"ID":"9e0742d4-fc6e-4bcd-8d28-ee56bcd52b83","Type":"ContainerStarted","Data":"44a78496d708a283f79b657a7c02630dfe18dc95324467bf08a92f1c4428664d"}
Sep 16 02:35:45.595069 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:45.595014    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-0bdfbb54-gw-openshift-default-5ccb9467b4-4j8g7" podStartSLOduration=2.594997512 podStartE2EDuration="2.594997512s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 02:35:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:35:45.593075991 +0000 UTC m=+1358.511565242" watchObservedRunningTime="2026-09-16 02:35:45.594997512 +0000 UTC m=+1358.513486761"
Sep 16 02:35:46.231883 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:46.231845    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-0bdfbb54-gw-openshift-default-5ccb9467b4-4j8g7"
Sep 16 02:35:46.238495 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:46.238468    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-0bdfbb54-gw-openshift-default-5ccb9467b4-4j8g7"
Sep 16 02:35:46.577894 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:46.577806    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-0bdfbb54-gw-openshift-default-5ccb9467b4-4j8g7"
Sep 16 02:35:46.578984 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:46.578963    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-0bdfbb54-gw-openshift-default-5ccb9467b4-4j8g7"
Sep 16 02:35:50.104240 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:50.104202    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-0bdfbb54-5557cf5966-mrnsd"]
Sep 16 02:35:50.108911 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:50.108891    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-0bdfbb54-5557cf5966-mrnsd"
Sep 16 02:35:50.111238 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:50.111221    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-0bdfbb54-dockercfg-c5d9n\""
Sep 16 02:35:50.111362 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:50.111342    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-0bdfbb54\""
Sep 16 02:35:50.123877 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:50.123848    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-0bdfbb54-5557cf5966-mrnsd"]
Sep 16 02:35:50.276413 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:50.276378    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7g6rp\" (UniqueName: \"kubernetes.io/projected/2f1860b3-bb58-40c5-a590-9c665f7a7fda-kube-api-access-7g6rp\") pod \"payload-pre-processing-e2e-migrate-0bdfbb54-5557cf5966-mrnsd\" (UID: \"2f1860b3-bb58-40c5-a590-9c665f7a7fda\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-0bdfbb54-5557cf5966-mrnsd"
Sep 16 02:35:50.276596 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:50.276456    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f1860b3-bb58-40c5-a590-9c665f7a7fda-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-0bdfbb54-5557cf5966-mrnsd\" (UID: \"2f1860b3-bb58-40c5-a590-9c665f7a7fda\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-0bdfbb54-5557cf5966-mrnsd"
Sep 16 02:35:50.377415 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:50.377333    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7g6rp\" (UniqueName: \"kubernetes.io/projected/2f1860b3-bb58-40c5-a590-9c665f7a7fda-kube-api-access-7g6rp\") pod \"payload-pre-processing-e2e-migrate-0bdfbb54-5557cf5966-mrnsd\" (UID: \"2f1860b3-bb58-40c5-a590-9c665f7a7fda\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-0bdfbb54-5557cf5966-mrnsd"
Sep 16 02:35:50.377554 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:50.377411    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f1860b3-bb58-40c5-a590-9c665f7a7fda-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-0bdfbb54-5557cf5966-mrnsd\" (UID: \"2f1860b3-bb58-40c5-a590-9c665f7a7fda\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-0bdfbb54-5557cf5966-mrnsd"
Sep 16 02:35:50.378177 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:50.378152    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f1860b3-bb58-40c5-a590-9c665f7a7fda-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-0bdfbb54-5557cf5966-mrnsd\" (UID: \"2f1860b3-bb58-40c5-a590-9c665f7a7fda\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-0bdfbb54-5557cf5966-mrnsd"
Sep 16 02:35:50.384856 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:50.384827    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7g6rp\" (UniqueName: \"kubernetes.io/projected/2f1860b3-bb58-40c5-a590-9c665f7a7fda-kube-api-access-7g6rp\") pod \"payload-pre-processing-e2e-migrate-0bdfbb54-5557cf5966-mrnsd\" (UID: \"2f1860b3-bb58-40c5-a590-9c665f7a7fda\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-0bdfbb54-5557cf5966-mrnsd"
Sep 16 02:35:50.419706 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:50.419677    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-0bdfbb54-5557cf5966-mrnsd"
Sep 16 02:35:50.689002 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:50.688978    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-0bdfbb54-5557cf5966-mrnsd"]
Sep 16 02:35:50.691030 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:35:50.690998    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f1860b3_bb58_40c5_a590_9c665f7a7fda.slice/crio-7c179ed2fae57f1dcbf040b8471336e8d224f617d000ffaa5e4b714eddbddd30 WatchSource:0}: Error finding container 7c179ed2fae57f1dcbf040b8471336e8d224f617d000ffaa5e4b714eddbddd30: Status 404 returned error can't find the container with id 7c179ed2fae57f1dcbf040b8471336e8d224f617d000ffaa5e4b714eddbddd30
Sep 16 02:35:51.608363 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:51.608325    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-0bdfbb54-5557cf5966-mrnsd" event={"ID":"2f1860b3-bb58-40c5-a590-9c665f7a7fda","Type":"ContainerStarted","Data":"85a5b1220f3d9052738b2f754ee328345820f4b16f442a3d44534afe6388c7b0"}
Sep 16 02:35:51.608845 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:51.608382    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-0bdfbb54-5557cf5966-mrnsd" event={"ID":"2f1860b3-bb58-40c5-a590-9c665f7a7fda","Type":"ContainerStarted","Data":"7c179ed2fae57f1dcbf040b8471336e8d224f617d000ffaa5e4b714eddbddd30"}
Sep 16 02:35:51.608845 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:51.608448    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-0bdfbb54-5557cf5966-mrnsd"
Sep 16 02:35:51.626726 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:51.626629    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-0bdfbb54-5557cf5966-mrnsd" podStartSLOduration=1.6266142970000002 podStartE2EDuration="1.626614297s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 02:35:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:35:51.625445258 +0000 UTC m=+1364.543934508" watchObservedRunningTime="2026-09-16 02:35:51.626614297 +0000 UTC m=+1364.545103545"
Sep 16 02:35:53.012758 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:53.012734    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-e725a02e-openshift-default-98b5bd466-mjznc"
Sep 16 02:35:53.207819 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:53.207781    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54c61f6e-cab0-494d-8ce9-93132feb0228-istio-data\") pod \"54c61f6e-cab0-494d-8ce9-93132feb0228\" (UID: \"54c61f6e-cab0-494d-8ce9-93132feb0228\") "
Sep 16 02:35:53.207819 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:53.207823    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54c61f6e-cab0-494d-8ce9-93132feb0228-istiod-ca-cert\") pod \"54c61f6e-cab0-494d-8ce9-93132feb0228\" (UID: \"54c61f6e-cab0-494d-8ce9-93132feb0228\") "
Sep 16 02:35:53.208084 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:53.207864    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54c61f6e-cab0-494d-8ce9-93132feb0228-istio-envoy\") pod \"54c61f6e-cab0-494d-8ce9-93132feb0228\" (UID: \"54c61f6e-cab0-494d-8ce9-93132feb0228\") "
Sep 16 02:35:53.208084 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:53.207894    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gr89d\" (UniqueName: \"kubernetes.io/projected/54c61f6e-cab0-494d-8ce9-93132feb0228-kube-api-access-gr89d\") pod \"54c61f6e-cab0-494d-8ce9-93132feb0228\" (UID: \"54c61f6e-cab0-494d-8ce9-93132feb0228\") "
Sep 16 02:35:53.208084 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:53.207919    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54c61f6e-cab0-494d-8ce9-93132feb0228-credential-socket\") pod \"54c61f6e-cab0-494d-8ce9-93132feb0228\" (UID: \"54c61f6e-cab0-494d-8ce9-93132feb0228\") "
Sep 16 02:35:53.208084 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:53.207976    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54c61f6e-cab0-494d-8ce9-93132feb0228-istio-podinfo\") pod \"54c61f6e-cab0-494d-8ce9-93132feb0228\" (UID: \"54c61f6e-cab0-494d-8ce9-93132feb0228\") "
Sep 16 02:35:53.208084 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:53.208036    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54c61f6e-cab0-494d-8ce9-93132feb0228-istio-token\") pod \"54c61f6e-cab0-494d-8ce9-93132feb0228\" (UID: \"54c61f6e-cab0-494d-8ce9-93132feb0228\") "
Sep 16 02:35:53.208411 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:53.208088    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54c61f6e-cab0-494d-8ce9-93132feb0228-workload-socket\") pod \"54c61f6e-cab0-494d-8ce9-93132feb0228\" (UID: \"54c61f6e-cab0-494d-8ce9-93132feb0228\") "
Sep 16 02:35:53.208411 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:53.208117    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54c61f6e-cab0-494d-8ce9-93132feb0228-istio-data" (OuterVolumeSpecName: "istio-data") pod "54c61f6e-cab0-494d-8ce9-93132feb0228" (UID: "54c61f6e-cab0-494d-8ce9-93132feb0228"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:35:53.208411 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:53.208153    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54c61f6e-cab0-494d-8ce9-93132feb0228-workload-certs\") pod \"54c61f6e-cab0-494d-8ce9-93132feb0228\" (UID: \"54c61f6e-cab0-494d-8ce9-93132feb0228\") "
Sep 16 02:35:53.208411 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:53.208271    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54c61f6e-cab0-494d-8ce9-93132feb0228-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "54c61f6e-cab0-494d-8ce9-93132feb0228" (UID: "54c61f6e-cab0-494d-8ce9-93132feb0228"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:35:53.208411 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:53.208396    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54c61f6e-cab0-494d-8ce9-93132feb0228-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "54c61f6e-cab0-494d-8ce9-93132feb0228" (UID: "54c61f6e-cab0-494d-8ce9-93132feb0228"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:35:53.208624 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:53.208430    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54c61f6e-cab0-494d-8ce9-93132feb0228-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "54c61f6e-cab0-494d-8ce9-93132feb0228" (UID: "54c61f6e-cab0-494d-8ce9-93132feb0228"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:35:53.208624 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:53.208612    2558 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54c61f6e-cab0-494d-8ce9-93132feb0228-workload-socket\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:35:53.208624 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:53.208614    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54c61f6e-cab0-494d-8ce9-93132feb0228-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "54c61f6e-cab0-494d-8ce9-93132feb0228" (UID: "54c61f6e-cab0-494d-8ce9-93132feb0228"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 02:35:53.208781 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:53.208632    2558 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54c61f6e-cab0-494d-8ce9-93132feb0228-workload-certs\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:35:53.208781 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:53.208645    2558 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54c61f6e-cab0-494d-8ce9-93132feb0228-istio-data\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:35:53.208781 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:53.208657    2558 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54c61f6e-cab0-494d-8ce9-93132feb0228-credential-socket\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:35:53.210771 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:53.210725    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54c61f6e-cab0-494d-8ce9-93132feb0228-istio-token" (OuterVolumeSpecName: "istio-token") pod "54c61f6e-cab0-494d-8ce9-93132feb0228" (UID: "54c61f6e-cab0-494d-8ce9-93132feb0228"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 02:35:53.210771 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:53.210740    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54c61f6e-cab0-494d-8ce9-93132feb0228-kube-api-access-gr89d" (OuterVolumeSpecName: "kube-api-access-gr89d") pod "54c61f6e-cab0-494d-8ce9-93132feb0228" (UID: "54c61f6e-cab0-494d-8ce9-93132feb0228"). InnerVolumeSpecName "kube-api-access-gr89d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 02:35:53.210771 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:53.210748    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54c61f6e-cab0-494d-8ce9-93132feb0228-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "54c61f6e-cab0-494d-8ce9-93132feb0228" (UID: "54c61f6e-cab0-494d-8ce9-93132feb0228"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:35:53.210771 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:53.210733    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/54c61f6e-cab0-494d-8ce9-93132feb0228-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "54c61f6e-cab0-494d-8ce9-93132feb0228" (UID: "54c61f6e-cab0-494d-8ce9-93132feb0228"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 02:35:53.309528 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:53.309472    2558 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54c61f6e-cab0-494d-8ce9-93132feb0228-istiod-ca-cert\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:35:53.309528 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:53.309521    2558 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54c61f6e-cab0-494d-8ce9-93132feb0228-istio-envoy\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:35:53.309528 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:53.309530    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gr89d\" (UniqueName: \"kubernetes.io/projected/54c61f6e-cab0-494d-8ce9-93132feb0228-kube-api-access-gr89d\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:35:53.309528 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:53.309539    2558 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54c61f6e-cab0-494d-8ce9-93132feb0228-istio-podinfo\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:35:53.309528 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:53.309548    2558 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54c61f6e-cab0-494d-8ce9-93132feb0228-istio-token\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:35:53.621243 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:53.621211    2558 generic.go:356] "Generic (PLEG): container finished" podID="54c61f6e-cab0-494d-8ce9-93132feb0228" containerID="d28a4e2e720c9eef514c796e7bfdfcbf410d5b1a023a13d7a0e9d098bd329364" exitCode=0
Sep 16 02:35:53.621384 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:53.621256    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-e725a02e-openshift-default-98b5bd466-mjznc"
Sep 16 02:35:53.621384 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:53.621271    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e725a02e-openshift-default-98b5bd466-mjznc" event={"ID":"54c61f6e-cab0-494d-8ce9-93132feb0228","Type":"ContainerDied","Data":"d28a4e2e720c9eef514c796e7bfdfcbf410d5b1a023a13d7a0e9d098bd329364"}
Sep 16 02:35:53.621384 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:53.621317    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e725a02e-openshift-default-98b5bd466-mjznc" event={"ID":"54c61f6e-cab0-494d-8ce9-93132feb0228","Type":"ContainerDied","Data":"84b5a66813d331674b5346f50d28b885e02c38f3b88be5ba3c9da774fa99e10b"}
Sep 16 02:35:53.621384 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:53.621340    2558 scope.go:117] "RemoveContainer" containerID="d28a4e2e720c9eef514c796e7bfdfcbf410d5b1a023a13d7a0e9d098bd329364"
Sep 16 02:35:53.630068 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:53.630047    2558 scope.go:117] "RemoveContainer" containerID="36eabbe9fbe28d8a787de5a6f8717b62ccb7ccdd89026ee300ac156dc8923b58"
Sep 16 02:35:53.638659 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:53.638638    2558 scope.go:117] "RemoveContainer" containerID="d28a4e2e720c9eef514c796e7bfdfcbf410d5b1a023a13d7a0e9d098bd329364"
Sep 16 02:35:53.639363 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:53.639340    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-e725a02e-openshift-default-98b5bd466-mjznc"]
Sep 16 02:35:53.639474 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:35:53.639397    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d28a4e2e720c9eef514c796e7bfdfcbf410d5b1a023a13d7a0e9d098bd329364\": container with ID starting with d28a4e2e720c9eef514c796e7bfdfcbf410d5b1a023a13d7a0e9d098bd329364 not found: ID does not exist" containerID="d28a4e2e720c9eef514c796e7bfdfcbf410d5b1a023a13d7a0e9d098bd329364"
Sep 16 02:35:53.639474 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:53.639429    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d28a4e2e720c9eef514c796e7bfdfcbf410d5b1a023a13d7a0e9d098bd329364"} err="failed to get container status \"d28a4e2e720c9eef514c796e7bfdfcbf410d5b1a023a13d7a0e9d098bd329364\": rpc error: code = NotFound desc = could not find container \"d28a4e2e720c9eef514c796e7bfdfcbf410d5b1a023a13d7a0e9d098bd329364\": container with ID starting with d28a4e2e720c9eef514c796e7bfdfcbf410d5b1a023a13d7a0e9d098bd329364 not found: ID does not exist"
Sep 16 02:35:53.639474 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:53.639453    2558 scope.go:117] "RemoveContainer" containerID="36eabbe9fbe28d8a787de5a6f8717b62ccb7ccdd89026ee300ac156dc8923b58"
Sep 16 02:35:53.639862 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:35:53.639835    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36eabbe9fbe28d8a787de5a6f8717b62ccb7ccdd89026ee300ac156dc8923b58\": container with ID starting with 36eabbe9fbe28d8a787de5a6f8717b62ccb7ccdd89026ee300ac156dc8923b58 not found: ID does not exist" containerID="36eabbe9fbe28d8a787de5a6f8717b62ccb7ccdd89026ee300ac156dc8923b58"
Sep 16 02:35:53.639954 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:53.639871    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36eabbe9fbe28d8a787de5a6f8717b62ccb7ccdd89026ee300ac156dc8923b58"} err="failed to get container status \"36eabbe9fbe28d8a787de5a6f8717b62ccb7ccdd89026ee300ac156dc8923b58\": rpc error: code = NotFound desc = could not find container \"36eabbe9fbe28d8a787de5a6f8717b62ccb7ccdd89026ee300ac156dc8923b58\": container with ID starting with 36eabbe9fbe28d8a787de5a6f8717b62ccb7ccdd89026ee300ac156dc8923b58 not found: ID does not exist"
Sep 16 02:35:53.642650 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:53.642628    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-e725a02e-openshift-default-98b5bd466-mjznc"]
Sep 16 02:35:55.549990 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:35:55.549952    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="54c61f6e-cab0-494d-8ce9-93132feb0228" path="/var/lib/kubelet/pods/54c61f6e-cab0-494d-8ce9-93132feb0228/volumes"
Sep 16 02:36:02.616564 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:02.616529    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-0bdfbb54-5557cf5966-mrnsd"
Sep 16 02:36:08.386275 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:08.386241    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-542ed5/distinct-2-w3-542ed5-kserve-57dc75646d-x4qrb"]
Sep 16 02:36:08.386754 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:08.386643    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="54c61f6e-cab0-494d-8ce9-93132feb0228" containerName="istio-proxy"
Sep 16 02:36:08.386754 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:08.386654    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="54c61f6e-cab0-494d-8ce9-93132feb0228" containerName="istio-proxy"
Sep 16 02:36:08.386754 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:08.386668    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="54c61f6e-cab0-494d-8ce9-93132feb0228" containerName="istio-proxy"
Sep 16 02:36:08.386754 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:08.386674    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="54c61f6e-cab0-494d-8ce9-93132feb0228" containerName="istio-proxy"
Sep 16 02:36:08.387003 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:08.386758    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="54c61f6e-cab0-494d-8ce9-93132feb0228" containerName="istio-proxy"
Sep 16 02:36:08.387003 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:08.386769    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="54c61f6e-cab0-494d-8ce9-93132feb0228" containerName="istio-proxy"
Sep 16 02:36:08.391065 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:08.391045    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-542ed5/distinct-2-w3-542ed5-kserve-57dc75646d-x4qrb"
Sep 16 02:36:08.393338 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:08.393316    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-542ed5\"/\"openshift-service-ca.crt\""
Sep 16 02:36:08.394181 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:08.394159    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-542ed5\"/\"kube-root-ca.crt\""
Sep 16 02:36:08.394296 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:08.394229    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-542ed5\"/\"default-dockercfg-vtfl8\""
Sep 16 02:36:08.394296 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:08.394281    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-542ed5\"/\"distinct-2-w3-542ed5-kserve-self-signed-certs\""
Sep 16 02:36:08.398007 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:08.397983    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-542ed5/distinct-2-w3-542ed5-kserve-57dc75646d-x4qrb"]
Sep 16 02:36:08.458896 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:08.458855    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ff6a30ad-287e-40df-aab7-455812130778-home\") pod \"distinct-2-w3-542ed5-kserve-57dc75646d-x4qrb\" (UID: \"ff6a30ad-287e-40df-aab7-455812130778\") " pod="e2e-models-e2e-worker-w3-542ed5/distinct-2-w3-542ed5-kserve-57dc75646d-x4qrb"
Sep 16 02:36:08.459097 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:08.458915    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ngl5\" (UniqueName: \"kubernetes.io/projected/ff6a30ad-287e-40df-aab7-455812130778-kube-api-access-8ngl5\") pod \"distinct-2-w3-542ed5-kserve-57dc75646d-x4qrb\" (UID: \"ff6a30ad-287e-40df-aab7-455812130778\") " pod="e2e-models-e2e-worker-w3-542ed5/distinct-2-w3-542ed5-kserve-57dc75646d-x4qrb"
Sep 16 02:36:08.459097 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:08.459009    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff6a30ad-287e-40df-aab7-455812130778-model-cache\") pod \"distinct-2-w3-542ed5-kserve-57dc75646d-x4qrb\" (UID: \"ff6a30ad-287e-40df-aab7-455812130778\") " pod="e2e-models-e2e-worker-w3-542ed5/distinct-2-w3-542ed5-kserve-57dc75646d-x4qrb"
Sep 16 02:36:08.459097 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:08.459081    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff6a30ad-287e-40df-aab7-455812130778-tmp-dir\") pod \"distinct-2-w3-542ed5-kserve-57dc75646d-x4qrb\" (UID: \"ff6a30ad-287e-40df-aab7-455812130778\") " pod="e2e-models-e2e-worker-w3-542ed5/distinct-2-w3-542ed5-kserve-57dc75646d-x4qrb"
Sep 16 02:36:08.459255 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:08.459117    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff6a30ad-287e-40df-aab7-455812130778-tls-certs\") pod \"distinct-2-w3-542ed5-kserve-57dc75646d-x4qrb\" (UID: \"ff6a30ad-287e-40df-aab7-455812130778\") " pod="e2e-models-e2e-worker-w3-542ed5/distinct-2-w3-542ed5-kserve-57dc75646d-x4qrb"
Sep 16 02:36:08.459255 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:08.459174    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff6a30ad-287e-40df-aab7-455812130778-dshm\") pod \"distinct-2-w3-542ed5-kserve-57dc75646d-x4qrb\" (UID: \"ff6a30ad-287e-40df-aab7-455812130778\") " pod="e2e-models-e2e-worker-w3-542ed5/distinct-2-w3-542ed5-kserve-57dc75646d-x4qrb"
Sep 16 02:36:08.559976 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:08.559896    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff6a30ad-287e-40df-aab7-455812130778-tmp-dir\") pod \"distinct-2-w3-542ed5-kserve-57dc75646d-x4qrb\" (UID: \"ff6a30ad-287e-40df-aab7-455812130778\") " pod="e2e-models-e2e-worker-w3-542ed5/distinct-2-w3-542ed5-kserve-57dc75646d-x4qrb"
Sep 16 02:36:08.559976 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:08.559950    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff6a30ad-287e-40df-aab7-455812130778-tls-certs\") pod \"distinct-2-w3-542ed5-kserve-57dc75646d-x4qrb\" (UID: \"ff6a30ad-287e-40df-aab7-455812130778\") " pod="e2e-models-e2e-worker-w3-542ed5/distinct-2-w3-542ed5-kserve-57dc75646d-x4qrb"
Sep 16 02:36:08.560981 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:08.560004    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff6a30ad-287e-40df-aab7-455812130778-dshm\") pod \"distinct-2-w3-542ed5-kserve-57dc75646d-x4qrb\" (UID: \"ff6a30ad-287e-40df-aab7-455812130778\") " pod="e2e-models-e2e-worker-w3-542ed5/distinct-2-w3-542ed5-kserve-57dc75646d-x4qrb"
Sep 16 02:36:08.560981 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:08.560064    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ff6a30ad-287e-40df-aab7-455812130778-home\") pod \"distinct-2-w3-542ed5-kserve-57dc75646d-x4qrb\" (UID: \"ff6a30ad-287e-40df-aab7-455812130778\") " pod="e2e-models-e2e-worker-w3-542ed5/distinct-2-w3-542ed5-kserve-57dc75646d-x4qrb"
Sep 16 02:36:08.560981 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:08.560119    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ngl5\" (UniqueName: \"kubernetes.io/projected/ff6a30ad-287e-40df-aab7-455812130778-kube-api-access-8ngl5\") pod \"distinct-2-w3-542ed5-kserve-57dc75646d-x4qrb\" (UID: \"ff6a30ad-287e-40df-aab7-455812130778\") " pod="e2e-models-e2e-worker-w3-542ed5/distinct-2-w3-542ed5-kserve-57dc75646d-x4qrb"
Sep 16 02:36:08.560981 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:08.560174    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff6a30ad-287e-40df-aab7-455812130778-model-cache\") pod \"distinct-2-w3-542ed5-kserve-57dc75646d-x4qrb\" (UID: \"ff6a30ad-287e-40df-aab7-455812130778\") " pod="e2e-models-e2e-worker-w3-542ed5/distinct-2-w3-542ed5-kserve-57dc75646d-x4qrb"
Sep 16 02:36:08.560981 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:08.560384    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff6a30ad-287e-40df-aab7-455812130778-tmp-dir\") pod \"distinct-2-w3-542ed5-kserve-57dc75646d-x4qrb\" (UID: \"ff6a30ad-287e-40df-aab7-455812130778\") " pod="e2e-models-e2e-worker-w3-542ed5/distinct-2-w3-542ed5-kserve-57dc75646d-x4qrb"
Sep 16 02:36:08.560981 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:08.560442    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff6a30ad-287e-40df-aab7-455812130778-model-cache\") pod \"distinct-2-w3-542ed5-kserve-57dc75646d-x4qrb\" (UID: \"ff6a30ad-287e-40df-aab7-455812130778\") " pod="e2e-models-e2e-worker-w3-542ed5/distinct-2-w3-542ed5-kserve-57dc75646d-x4qrb"
Sep 16 02:36:08.560981 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:08.560630    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ff6a30ad-287e-40df-aab7-455812130778-home\") pod \"distinct-2-w3-542ed5-kserve-57dc75646d-x4qrb\" (UID: \"ff6a30ad-287e-40df-aab7-455812130778\") " pod="e2e-models-e2e-worker-w3-542ed5/distinct-2-w3-542ed5-kserve-57dc75646d-x4qrb"
Sep 16 02:36:08.570557 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:08.568112    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff6a30ad-287e-40df-aab7-455812130778-dshm\") pod \"distinct-2-w3-542ed5-kserve-57dc75646d-x4qrb\" (UID: \"ff6a30ad-287e-40df-aab7-455812130778\") " pod="e2e-models-e2e-worker-w3-542ed5/distinct-2-w3-542ed5-kserve-57dc75646d-x4qrb"
Sep 16 02:36:08.570557 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:08.568394    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff6a30ad-287e-40df-aab7-455812130778-tls-certs\") pod \"distinct-2-w3-542ed5-kserve-57dc75646d-x4qrb\" (UID: \"ff6a30ad-287e-40df-aab7-455812130778\") " pod="e2e-models-e2e-worker-w3-542ed5/distinct-2-w3-542ed5-kserve-57dc75646d-x4qrb"
Sep 16 02:36:08.574888 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:08.574831    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ngl5\" (UniqueName: \"kubernetes.io/projected/ff6a30ad-287e-40df-aab7-455812130778-kube-api-access-8ngl5\") pod \"distinct-2-w3-542ed5-kserve-57dc75646d-x4qrb\" (UID: \"ff6a30ad-287e-40df-aab7-455812130778\") " pod="e2e-models-e2e-worker-w3-542ed5/distinct-2-w3-542ed5-kserve-57dc75646d-x4qrb"
Sep 16 02:36:08.703346 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:08.703315    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-542ed5/distinct-2-w3-542ed5-kserve-57dc75646d-x4qrb"
Sep 16 02:36:08.975727 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:08.975684    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-542ed5/distinct-2-w3-542ed5-kserve-57dc75646d-x4qrb"]
Sep 16 02:36:08.977187 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:36:08.977162    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podff6a30ad_287e_40df_aab7_455812130778.slice/crio-2be640db84bd7689f1f8c19feb1e9db2aeb7cd2214d866bed98176e04b6a19c1 WatchSource:0}: Error finding container 2be640db84bd7689f1f8c19feb1e9db2aeb7cd2214d866bed98176e04b6a19c1: Status 404 returned error can't find the container with id 2be640db84bd7689f1f8c19feb1e9db2aeb7cd2214d866bed98176e04b6a19c1
Sep 16 02:36:09.544728 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:09.544672    2558 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-processing-5d7dbbd56b-pltkr" secret="" err="secret \"payload-processing-dockercfg-zzx6j\" not found"
Sep 16 02:36:09.705412 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:09.705378    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-542ed5/distinct-2-w3-542ed5-kserve-57dc75646d-x4qrb" event={"ID":"ff6a30ad-287e-40df-aab7-455812130778","Type":"ContainerStarted","Data":"e38a085d80dd4fb1b8cae4d9398b5031570c400cce6c254591c9c5106c3e958d"}
Sep 16 02:36:09.705580 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:09.705420    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-542ed5/distinct-2-w3-542ed5-kserve-57dc75646d-x4qrb" event={"ID":"ff6a30ad-287e-40df-aab7-455812130778","Type":"ContainerStarted","Data":"2be640db84bd7689f1f8c19feb1e9db2aeb7cd2214d866bed98176e04b6a19c1"}
Sep 16 02:36:09.723061 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:09.723013    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-542ed5/distinct-2-w3-542ed5-kserve-57dc75646d-x4qrb" podStartSLOduration=1.722998716 podStartE2EDuration="1.722998716s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 02:36:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:36:09.722171696 +0000 UTC m=+1382.640660949" watchObservedRunningTime="2026-09-16 02:36:09.722998716 +0000 UTC m=+1382.641487965"
Sep 16 02:36:11.467628 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:11.465935    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-1a323c/test-model-1a323c-kserve-664b4bd4dc-4dcn8"]
Sep 16 02:36:11.619276 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:11.619235    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-60674e-cqjxh"]
Sep 16 02:36:11.624075 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:11.624046    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-60674e-cqjxh"
Sep 16 02:36:11.627012 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:11.626988    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 16 02:36:11.627222 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:11.627200    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jpd4f\""
Sep 16 02:36:11.628457 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:11.628434    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-60674e-cqjxh"]
Sep 16 02:36:11.692853 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:11.692818    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f64097fa-d692-4f8b-9d7d-af4bce63dd46-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-60674e-cqjxh\" (UID: \"f64097fa-d692-4f8b-9d7d-af4bce63dd46\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-60674e-cqjxh"
Sep 16 02:36:11.794418 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:11.794336    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f64097fa-d692-4f8b-9d7d-af4bce63dd46-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-60674e-cqjxh\" (UID: \"f64097fa-d692-4f8b-9d7d-af4bce63dd46\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-60674e-cqjxh"
Sep 16 02:36:11.795002 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:11.794980    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f64097fa-d692-4f8b-9d7d-af4bce63dd46-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-60674e-cqjxh\" (UID: \"f64097fa-d692-4f8b-9d7d-af4bce63dd46\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-60674e-cqjxh"
Sep 16 02:36:11.936363 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:11.936329    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-60674e-cqjxh"
Sep 16 02:36:12.227830 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:12.227762    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-60674e-cqjxh"]
Sep 16 02:36:12.230048 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:36:12.230019    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf64097fa_d692_4f8b_9d7d_af4bce63dd46.slice/crio-43ba835a02a6de89a5109e36f81da2fee24a4033e12df1761da22af430b410b3 WatchSource:0}: Error finding container 43ba835a02a6de89a5109e36f81da2fee24a4033e12df1761da22af430b410b3: Status 404 returned error can't find the container with id 43ba835a02a6de89a5109e36f81da2fee24a4033e12df1761da22af430b410b3
Sep 16 02:36:12.724357 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:12.724324    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-60674e-cqjxh" event={"ID":"f64097fa-d692-4f8b-9d7d-af4bce63dd46","Type":"ContainerStarted","Data":"2d2afcaf99d50ae1fff28690713ba0e4e1a351feeb0ba3d1a6d62e8b31343f3c"}
Sep 16 02:36:12.724773 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:12.724375    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-60674e-cqjxh" event={"ID":"f64097fa-d692-4f8b-9d7d-af4bce63dd46","Type":"ContainerStarted","Data":"43ba835a02a6de89a5109e36f81da2fee24a4033e12df1761da22af430b410b3"}
Sep 16 02:36:12.740558 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:12.740508    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-60674e-cqjxh" podStartSLOduration=1.74049199 podStartE2EDuration="1.74049199s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 02:36:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:36:12.73793338 +0000 UTC m=+1385.656422641" watchObservedRunningTime="2026-09-16 02:36:12.74049199 +0000 UTC m=+1385.658981238"
Sep 16 02:36:13.731136 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:13.731096    2558 generic.go:356] "Generic (PLEG): container finished" podID="f64097fa-d692-4f8b-9d7d-af4bce63dd46" containerID="2d2afcaf99d50ae1fff28690713ba0e4e1a351feeb0ba3d1a6d62e8b31343f3c" exitCode=0
Sep 16 02:36:13.731518 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:13.731162    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-60674e-cqjxh" event={"ID":"f64097fa-d692-4f8b-9d7d-af4bce63dd46","Type":"ContainerDied","Data":"2d2afcaf99d50ae1fff28690713ba0e4e1a351feeb0ba3d1a6d62e8b31343f3c"}
Sep 16 02:36:14.905801 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:14.905777    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-60674e-cqjxh"
Sep 16 02:36:15.027767 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:15.027646    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f64097fa-d692-4f8b-9d7d-af4bce63dd46-maas-api-service-ca\") pod \"f64097fa-d692-4f8b-9d7d-af4bce63dd46\" (UID: \"f64097fa-d692-4f8b-9d7d-af4bce63dd46\") "
Sep 16 02:36:15.027979 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:15.027955    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f64097fa-d692-4f8b-9d7d-af4bce63dd46-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f64097fa-d692-4f8b-9d7d-af4bce63dd46" (UID: "f64097fa-d692-4f8b-9d7d-af4bce63dd46"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 02:36:15.129220 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:15.129182    2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f64097fa-d692-4f8b-9d7d-af4bce63dd46-maas-api-service-ca\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:36:15.742587 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:15.742552    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-60674e-cqjxh" event={"ID":"f64097fa-d692-4f8b-9d7d-af4bce63dd46","Type":"ContainerDied","Data":"43ba835a02a6de89a5109e36f81da2fee24a4033e12df1761da22af430b410b3"}
Sep 16 02:36:15.742587 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:15.742572    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-60674e-cqjxh"
Sep 16 02:36:15.742797 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:15.742601    2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="43ba835a02a6de89a5109e36f81da2fee24a4033e12df1761da22af430b410b3"
Sep 16 02:36:17.619884 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:17.619837    2558 patch_prober.go:28] interesting pod/e2e-migrate-0bdfbb54-gw-openshift-default-5ccb9467b4-4j8g7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 16 02:36:17.620260 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:17.619932    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-0bdfbb54-gw-openshift-default-5ccb9467b4-4j8g7" podUID="9e0742d4-fc6e-4bcd-8d28-ee56bcd52b83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 16 02:36:18.704423 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:18.704383    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-542ed5/distinct-2-w3-542ed5-kserve-57dc75646d-x4qrb"
Sep 16 02:36:18.704928 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:18.704461    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-542ed5/distinct-2-w3-542ed5-kserve-57dc75646d-x4qrb"
Sep 16 02:36:18.717120 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:18.717090    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-542ed5/distinct-2-w3-542ed5-kserve-57dc75646d-x4qrb"
Sep 16 02:36:18.772872 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:18.772839    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-542ed5/distinct-2-w3-542ed5-kserve-57dc75646d-x4qrb"
Sep 16 02:36:19.766996 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:19.766961    2558 generic.go:356] "Generic (PLEG): container finished" podID="9e0742d4-fc6e-4bcd-8d28-ee56bcd52b83" containerID="44a78496d708a283f79b657a7c02630dfe18dc95324467bf08a92f1c4428664d" exitCode=0
Sep 16 02:36:19.767513 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:19.767032    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-0bdfbb54-gw-openshift-default-5ccb9467b4-4j8g7" event={"ID":"9e0742d4-fc6e-4bcd-8d28-ee56bcd52b83","Type":"ContainerDied","Data":"44a78496d708a283f79b657a7c02630dfe18dc95324467bf08a92f1c4428664d"}
Sep 16 02:36:19.767596 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:19.767579    2558 scope.go:117] "RemoveContainer" containerID="44a78496d708a283f79b657a7c02630dfe18dc95324467bf08a92f1c4428664d"
Sep 16 02:36:19.770044 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:19.769165    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 16 02:36:19.770044 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:19.769247    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 16 02:36:19.770044 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:19.769291    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 16 02:36:20.775532 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:20.775493    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-0bdfbb54-gw-openshift-default-5ccb9467b4-4j8g7" event={"ID":"9e0742d4-fc6e-4bcd-8d28-ee56bcd52b83","Type":"ContainerStarted","Data":"81cbb3a8bef74a3221c78ab2d1beaa0a99256fc447c6afcf41e482a9ced54e3e"}
Sep 16 02:36:21.232026 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:21.231985    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-0bdfbb54-gw-openshift-default-5ccb9467b4-4j8g7"
Sep 16 02:36:21.232219 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:21.232195    2558 patch_prober.go:28] interesting pod/e2e-migrate-0bdfbb54-gw-openshift-default-5ccb9467b4-4j8g7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body=
Sep 16 02:36:21.232278 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:21.232258    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-0bdfbb54-gw-openshift-default-5ccb9467b4-4j8g7" podUID="9e0742d4-fc6e-4bcd-8d28-ee56bcd52b83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused"
Sep 16 02:36:22.232716 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:22.232656    2558 patch_prober.go:28] interesting pod/e2e-migrate-0bdfbb54-gw-openshift-default-5ccb9467b4-4j8g7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body=
Sep 16 02:36:22.233119 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:22.232750    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-0bdfbb54-gw-openshift-default-5ccb9467b4-4j8g7" podUID="9e0742d4-fc6e-4bcd-8d28-ee56bcd52b83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused"
Sep 16 02:36:22.529883 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:22.529797    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-60674e-cqjxh"]
Sep 16 02:36:22.531763 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:22.531744    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-60674e-cqjxh"]
Sep 16 02:36:23.232111 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:23.232072    2558 patch_prober.go:28] interesting pod/e2e-migrate-0bdfbb54-gw-openshift-default-5ccb9467b4-4j8g7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body=
Sep 16 02:36:23.232318 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:23.232135    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-0bdfbb54-gw-openshift-default-5ccb9467b4-4j8g7" podUID="9e0742d4-fc6e-4bcd-8d28-ee56bcd52b83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused"
Sep 16 02:36:23.555046 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:23.554946    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f64097fa-d692-4f8b-9d7d-af4bce63dd46" path="/var/lib/kubelet/pods/f64097fa-d692-4f8b-9d7d-af4bce63dd46/volumes"
Sep 16 02:36:24.232612 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:24.232575    2558 patch_prober.go:28] interesting pod/e2e-migrate-0bdfbb54-gw-openshift-default-5ccb9467b4-4j8g7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body=
Sep 16 02:36:24.232803 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:24.232652    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-0bdfbb54-gw-openshift-default-5ccb9467b4-4j8g7" podUID="9e0742d4-fc6e-4bcd-8d28-ee56bcd52b83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused"
Sep 16 02:36:25.232146 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:25.232109    2558 patch_prober.go:28] interesting pod/e2e-migrate-0bdfbb54-gw-openshift-default-5ccb9467b4-4j8g7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body=
Sep 16 02:36:25.232525 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:25.232167    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-0bdfbb54-gw-openshift-default-5ccb9467b4-4j8g7" podUID="9e0742d4-fc6e-4bcd-8d28-ee56bcd52b83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused"
Sep 16 02:36:26.235543 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:26.235512    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-0bdfbb54-gw-openshift-default-5ccb9467b4-4j8g7"
Sep 16 02:36:26.235917 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:26.235830    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-0bdfbb54-gw-openshift-default-5ccb9467b4-4j8g7"
Sep 16 02:36:26.236501 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:26.236483    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-0bdfbb54-gw-openshift-default-5ccb9467b4-4j8g7"
Sep 16 02:36:26.495311 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:26.495219    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-1a323c/test-model-1a323c-kserve-664b4bd4dc-4dcn8" podUID="8d0489db-6d2a-4427-8c9f-34993be5e3eb" containerName="main" containerID="cri-o://65de22837bf72e0fd262250e51438886167154653728742c7bcf92df65222da9" gracePeriod=45
Sep 16 02:36:26.747589 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:26.747532    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-1a323c/test-model-1a323c-kserve-664b4bd4dc-4dcn8"
Sep 16 02:36:26.813103 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:26.813065    2558 generic.go:356] "Generic (PLEG): container finished" podID="8d0489db-6d2a-4427-8c9f-34993be5e3eb" containerID="65de22837bf72e0fd262250e51438886167154653728742c7bcf92df65222da9" exitCode=0
Sep 16 02:36:26.813262 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:26.813108    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-1a323c/test-model-1a323c-kserve-664b4bd4dc-4dcn8"
Sep 16 02:36:26.813262 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:26.813147    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-1a323c/test-model-1a323c-kserve-664b4bd4dc-4dcn8" event={"ID":"8d0489db-6d2a-4427-8c9f-34993be5e3eb","Type":"ContainerDied","Data":"65de22837bf72e0fd262250e51438886167154653728742c7bcf92df65222da9"}
Sep 16 02:36:26.813262 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:26.813196    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-1a323c/test-model-1a323c-kserve-664b4bd4dc-4dcn8" event={"ID":"8d0489db-6d2a-4427-8c9f-34993be5e3eb","Type":"ContainerDied","Data":"7769f584a16196aa987cf97e5586426f80b99d330003a66afdf18c521b459dae"}
Sep 16 02:36:26.813262 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:26.813215    2558 scope.go:117] "RemoveContainer" containerID="65de22837bf72e0fd262250e51438886167154653728742c7bcf92df65222da9"
Sep 16 02:36:26.821367 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:26.821350    2558 scope.go:117] "RemoveContainer" containerID="65de22837bf72e0fd262250e51438886167154653728742c7bcf92df65222da9"
Sep 16 02:36:26.821786 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:36:26.821765    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65de22837bf72e0fd262250e51438886167154653728742c7bcf92df65222da9\": container with ID starting with 65de22837bf72e0fd262250e51438886167154653728742c7bcf92df65222da9 not found: ID does not exist" containerID="65de22837bf72e0fd262250e51438886167154653728742c7bcf92df65222da9"
Sep 16 02:36:26.821845 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:26.821812    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65de22837bf72e0fd262250e51438886167154653728742c7bcf92df65222da9"} err="failed to get container status \"65de22837bf72e0fd262250e51438886167154653728742c7bcf92df65222da9\": rpc error: code = NotFound desc = could not find container \"65de22837bf72e0fd262250e51438886167154653728742c7bcf92df65222da9\": container with ID starting with 65de22837bf72e0fd262250e51438886167154653728742c7bcf92df65222da9 not found: ID does not exist"
Sep 16 02:36:26.848174 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:26.848154    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8d0489db-6d2a-4427-8c9f-34993be5e3eb-home\") pod \"8d0489db-6d2a-4427-8c9f-34993be5e3eb\" (UID: \"8d0489db-6d2a-4427-8c9f-34993be5e3eb\") "
Sep 16 02:36:26.848267 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:26.848197    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gb4v6\" (UniqueName: \"kubernetes.io/projected/8d0489db-6d2a-4427-8c9f-34993be5e3eb-kube-api-access-gb4v6\") pod \"8d0489db-6d2a-4427-8c9f-34993be5e3eb\" (UID: \"8d0489db-6d2a-4427-8c9f-34993be5e3eb\") "
Sep 16 02:36:26.848267 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:26.848215    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d0489db-6d2a-4427-8c9f-34993be5e3eb-tmp-dir\") pod \"8d0489db-6d2a-4427-8c9f-34993be5e3eb\" (UID: \"8d0489db-6d2a-4427-8c9f-34993be5e3eb\") "
Sep 16 02:36:26.848374 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:26.848275    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d0489db-6d2a-4427-8c9f-34993be5e3eb-model-cache\") pod \"8d0489db-6d2a-4427-8c9f-34993be5e3eb\" (UID: \"8d0489db-6d2a-4427-8c9f-34993be5e3eb\") "
Sep 16 02:36:26.848374 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:26.848290    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d0489db-6d2a-4427-8c9f-34993be5e3eb-tls-certs\") pod \"8d0489db-6d2a-4427-8c9f-34993be5e3eb\" (UID: \"8d0489db-6d2a-4427-8c9f-34993be5e3eb\") "
Sep 16 02:36:26.848374 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:26.848322    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8d0489db-6d2a-4427-8c9f-34993be5e3eb-dshm\") pod \"8d0489db-6d2a-4427-8c9f-34993be5e3eb\" (UID: \"8d0489db-6d2a-4427-8c9f-34993be5e3eb\") "
Sep 16 02:36:26.848525 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:26.848435    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d0489db-6d2a-4427-8c9f-34993be5e3eb-home" (OuterVolumeSpecName: "home") pod "8d0489db-6d2a-4427-8c9f-34993be5e3eb" (UID: "8d0489db-6d2a-4427-8c9f-34993be5e3eb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:36:26.848583 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:26.848534    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d0489db-6d2a-4427-8c9f-34993be5e3eb-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8d0489db-6d2a-4427-8c9f-34993be5e3eb" (UID: "8d0489db-6d2a-4427-8c9f-34993be5e3eb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:36:26.848583 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:26.848547    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d0489db-6d2a-4427-8c9f-34993be5e3eb-model-cache" (OuterVolumeSpecName: "model-cache") pod "8d0489db-6d2a-4427-8c9f-34993be5e3eb" (UID: "8d0489db-6d2a-4427-8c9f-34993be5e3eb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:36:26.848726 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:26.848693    2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8d0489db-6d2a-4427-8c9f-34993be5e3eb-home\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:36:26.848772 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:26.848732    2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d0489db-6d2a-4427-8c9f-34993be5e3eb-tmp-dir\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:36:26.848772 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:26.848746    2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d0489db-6d2a-4427-8c9f-34993be5e3eb-model-cache\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:36:26.850358 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:26.850332    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d0489db-6d2a-4427-8c9f-34993be5e3eb-kube-api-access-gb4v6" (OuterVolumeSpecName: "kube-api-access-gb4v6") pod "8d0489db-6d2a-4427-8c9f-34993be5e3eb" (UID: "8d0489db-6d2a-4427-8c9f-34993be5e3eb"). InnerVolumeSpecName "kube-api-access-gb4v6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 02:36:26.850358 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:26.850338    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d0489db-6d2a-4427-8c9f-34993be5e3eb-dshm" (OuterVolumeSpecName: "dshm") pod "8d0489db-6d2a-4427-8c9f-34993be5e3eb" (UID: "8d0489db-6d2a-4427-8c9f-34993be5e3eb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:36:26.850484 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:26.850368    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d0489db-6d2a-4427-8c9f-34993be5e3eb-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8d0489db-6d2a-4427-8c9f-34993be5e3eb" (UID: "8d0489db-6d2a-4427-8c9f-34993be5e3eb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 02:36:26.949682 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:26.949646    2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d0489db-6d2a-4427-8c9f-34993be5e3eb-tls-certs\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:36:26.949682 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:26.949676    2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8d0489db-6d2a-4427-8c9f-34993be5e3eb-dshm\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:36:26.949682 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:26.949687    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gb4v6\" (UniqueName: \"kubernetes.io/projected/8d0489db-6d2a-4427-8c9f-34993be5e3eb-kube-api-access-gb4v6\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:36:27.134921 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:27.134888    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-1a323c/test-model-1a323c-kserve-664b4bd4dc-4dcn8"]
Sep 16 02:36:27.138400 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:27.138372    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-1a323c/test-model-1a323c-kserve-664b4bd4dc-4dcn8"]
Sep 16 02:36:27.551200 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:27.551166    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8d0489db-6d2a-4427-8c9f-34993be5e3eb" path="/var/lib/kubelet/pods/8d0489db-6d2a-4427-8c9f-34993be5e3eb/volumes"
Sep 16 02:36:28.842963 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:28.842929    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-0bdfbb54-5557cf5966-mrnsd"]
Sep 16 02:36:28.843456 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:28.843151    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-0bdfbb54-5557cf5966-mrnsd" podUID="2f1860b3-bb58-40c5-a590-9c665f7a7fda" containerName="payload-pre-processing" containerID="cri-o://85a5b1220f3d9052738b2f754ee328345820f4b16f442a3d44534afe6388c7b0" gracePeriod=30
Sep 16 02:36:29.138798 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:29.129458    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-0bdfbb54-gw-openshift-default-5ccb9467b4-4j8g7"]
Sep 16 02:36:29.139085 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:29.129684    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-0bdfbb54-gw-openshift-default-5ccb9467b4-4j8g7" podUID="9e0742d4-fc6e-4bcd-8d28-ee56bcd52b83" containerName="istio-proxy" containerID="cri-o://81cbb3a8bef74a3221c78ab2d1beaa0a99256fc447c6afcf41e482a9ced54e3e" gracePeriod=30
Sep 16 02:36:29.174258 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:29.174227    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-0bdfbb54-5557cf5966-mrnsd"
Sep 16 02:36:29.277088 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:29.277042    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7g6rp\" (UniqueName: \"kubernetes.io/projected/2f1860b3-bb58-40c5-a590-9c665f7a7fda-kube-api-access-7g6rp\") pod \"2f1860b3-bb58-40c5-a590-9c665f7a7fda\" (UID: \"2f1860b3-bb58-40c5-a590-9c665f7a7fda\") "
Sep 16 02:36:29.277280 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:29.277208    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f1860b3-bb58-40c5-a590-9c665f7a7fda-plugins-config-volume\") pod \"2f1860b3-bb58-40c5-a590-9c665f7a7fda\" (UID: \"2f1860b3-bb58-40c5-a590-9c665f7a7fda\") "
Sep 16 02:36:29.277572 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:29.277544    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f1860b3-bb58-40c5-a590-9c665f7a7fda-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2f1860b3-bb58-40c5-a590-9c665f7a7fda" (UID: "2f1860b3-bb58-40c5-a590-9c665f7a7fda"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 02:36:29.277688 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:29.277624    2558 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f1860b3-bb58-40c5-a590-9c665f7a7fda-plugins-config-volume\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:36:29.279210 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:29.279190    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f1860b3-bb58-40c5-a590-9c665f7a7fda-kube-api-access-7g6rp" (OuterVolumeSpecName: "kube-api-access-7g6rp") pod "2f1860b3-bb58-40c5-a590-9c665f7a7fda" (UID: "2f1860b3-bb58-40c5-a590-9c665f7a7fda"). InnerVolumeSpecName "kube-api-access-7g6rp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 02:36:29.378651 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:29.378561    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7g6rp\" (UniqueName: \"kubernetes.io/projected/2f1860b3-bb58-40c5-a590-9c665f7a7fda-kube-api-access-7g6rp\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:36:29.466432 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:29.466400    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-60674e-openshift-default-6c56877b8b-qc47f"]
Sep 16 02:36:29.466645 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:29.466613    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-60674e-openshift-default-6c56877b8b-qc47f" podUID="6d81e13b-62c5-4fe2-a392-c5bac92f0ba5" containerName="istio-proxy" containerID="cri-o://40e50b8af904fbd01b62dc8a1f9dfc60fb2c3c2091d72faa3c43a599a41d3dc1" gracePeriod=30
Sep 16 02:36:29.830372 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:29.830329    2558 generic.go:356] "Generic (PLEG): container finished" podID="2f1860b3-bb58-40c5-a590-9c665f7a7fda" containerID="85a5b1220f3d9052738b2f754ee328345820f4b16f442a3d44534afe6388c7b0" exitCode=0
Sep 16 02:36:29.830518 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:29.830389    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-0bdfbb54-5557cf5966-mrnsd"
Sep 16 02:36:29.830518 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:29.830412    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-0bdfbb54-5557cf5966-mrnsd" event={"ID":"2f1860b3-bb58-40c5-a590-9c665f7a7fda","Type":"ContainerDied","Data":"85a5b1220f3d9052738b2f754ee328345820f4b16f442a3d44534afe6388c7b0"}
Sep 16 02:36:29.830518 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:29.830467    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-0bdfbb54-5557cf5966-mrnsd" event={"ID":"2f1860b3-bb58-40c5-a590-9c665f7a7fda","Type":"ContainerDied","Data":"7c179ed2fae57f1dcbf040b8471336e8d224f617d000ffaa5e4b714eddbddd30"}
Sep 16 02:36:29.830518 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:29.830489    2558 scope.go:117] "RemoveContainer" containerID="85a5b1220f3d9052738b2f754ee328345820f4b16f442a3d44534afe6388c7b0"
Sep 16 02:36:29.839227 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:29.839197    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1a323c-h2l24"]
Sep 16 02:36:29.839801 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:29.839774    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d0489db-6d2a-4427-8c9f-34993be5e3eb" containerName="main"
Sep 16 02:36:29.839801 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:29.839800    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d0489db-6d2a-4427-8c9f-34993be5e3eb" containerName="main"
Sep 16 02:36:29.839997 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:29.839824    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f1860b3-bb58-40c5-a590-9c665f7a7fda" containerName="payload-pre-processing"
Sep 16 02:36:29.839997 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:29.839834    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f1860b3-bb58-40c5-a590-9c665f7a7fda" containerName="payload-pre-processing"
Sep 16 02:36:29.839997 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:29.839851    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f64097fa-d692-4f8b-9d7d-af4bce63dd46" containerName="revoke-keys"
Sep 16 02:36:29.839997 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:29.839860    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="f64097fa-d692-4f8b-9d7d-af4bce63dd46" containerName="revoke-keys"
Sep 16 02:36:29.839997 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:29.839983    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="f64097fa-d692-4f8b-9d7d-af4bce63dd46" containerName="revoke-keys"
Sep 16 02:36:29.839997 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:29.839996    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="2f1860b3-bb58-40c5-a590-9c665f7a7fda" containerName="payload-pre-processing"
Sep 16 02:36:29.840287 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:29.840007    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="8d0489db-6d2a-4427-8c9f-34993be5e3eb" containerName="main"
Sep 16 02:36:29.840686 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:29.840666    2558 scope.go:117] "RemoveContainer" containerID="85a5b1220f3d9052738b2f754ee328345820f4b16f442a3d44534afe6388c7b0"
Sep 16 02:36:29.841081 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:36:29.841057    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85a5b1220f3d9052738b2f754ee328345820f4b16f442a3d44534afe6388c7b0\": container with ID starting with 85a5b1220f3d9052738b2f754ee328345820f4b16f442a3d44534afe6388c7b0 not found: ID does not exist" containerID="85a5b1220f3d9052738b2f754ee328345820f4b16f442a3d44534afe6388c7b0"
Sep 16 02:36:29.841153 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:29.841093    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85a5b1220f3d9052738b2f754ee328345820f4b16f442a3d44534afe6388c7b0"} err="failed to get container status \"85a5b1220f3d9052738b2f754ee328345820f4b16f442a3d44534afe6388c7b0\": rpc error: code = NotFound desc = could not find container \"85a5b1220f3d9052738b2f754ee328345820f4b16f442a3d44534afe6388c7b0\": container with ID starting with 85a5b1220f3d9052738b2f754ee328345820f4b16f442a3d44534afe6388c7b0 not found: ID does not exist"
Sep 16 02:36:29.844820 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:29.844801    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1a323c-h2l24"
Sep 16 02:36:29.846833 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:29.846814    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jpd4f\""
Sep 16 02:36:29.846941 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:29.846817    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 16 02:36:29.855169 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:29.855147    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1a323c-h2l24"]
Sep 16 02:36:29.862780 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:29.862757    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-0bdfbb54-5557cf5966-mrnsd"]
Sep 16 02:36:29.866479 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:29.866451    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-0bdfbb54-5557cf5966-mrnsd"]
Sep 16 02:36:29.937721 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:29.937668    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-4108a843-openshift-default-7dd58f44f4-q5j6n"]
Sep 16 02:36:29.941186 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:29.941170    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-4108a843-openshift-default-7dd58f44f4-q5j6n"
Sep 16 02:36:29.943251 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:29.943231    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-4108a843-openshift-default-dockercfg-9kz74\""
Sep 16 02:36:29.953935 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:29.953914    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-4108a843-openshift-default-7dd58f44f4-q5j6n"]
Sep 16 02:36:29.985986 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:29.985956    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7247ce0-45c5-44dc-be9b-0b9fd9c3893a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-1a323c-h2l24\" (UID: \"d7247ce0-45c5-44dc-be9b-0b9fd9c3893a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1a323c-h2l24"
Sep 16 02:36:30.087806 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:30.087708    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62dc3fab-3a25-4d2a-bfb1-0b951833ef2d-istio-envoy\") pod \"e2e-ait-4108a843-openshift-default-7dd58f44f4-q5j6n\" (UID: \"62dc3fab-3a25-4d2a-bfb1-0b951833ef2d\") " pod="openshift-ingress/e2e-ait-4108a843-openshift-default-7dd58f44f4-q5j6n"
Sep 16 02:36:30.087806 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:30.087784    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62dc3fab-3a25-4d2a-bfb1-0b951833ef2d-istio-token\") pod \"e2e-ait-4108a843-openshift-default-7dd58f44f4-q5j6n\" (UID: \"62dc3fab-3a25-4d2a-bfb1-0b951833ef2d\") " pod="openshift-ingress/e2e-ait-4108a843-openshift-default-7dd58f44f4-q5j6n"
Sep 16 02:36:30.088035 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:30.087817    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62dc3fab-3a25-4d2a-bfb1-0b951833ef2d-credential-socket\") pod \"e2e-ait-4108a843-openshift-default-7dd58f44f4-q5j6n\" (UID: \"62dc3fab-3a25-4d2a-bfb1-0b951833ef2d\") " pod="openshift-ingress/e2e-ait-4108a843-openshift-default-7dd58f44f4-q5j6n"
Sep 16 02:36:30.088035 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:30.087864    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvlbp\" (UniqueName: \"kubernetes.io/projected/62dc3fab-3a25-4d2a-bfb1-0b951833ef2d-kube-api-access-cvlbp\") pod \"e2e-ait-4108a843-openshift-default-7dd58f44f4-q5j6n\" (UID: \"62dc3fab-3a25-4d2a-bfb1-0b951833ef2d\") " pod="openshift-ingress/e2e-ait-4108a843-openshift-default-7dd58f44f4-q5j6n"
Sep 16 02:36:30.088035 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:30.087911    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62dc3fab-3a25-4d2a-bfb1-0b951833ef2d-workload-certs\") pod \"e2e-ait-4108a843-openshift-default-7dd58f44f4-q5j6n\" (UID: \"62dc3fab-3a25-4d2a-bfb1-0b951833ef2d\") " pod="openshift-ingress/e2e-ait-4108a843-openshift-default-7dd58f44f4-q5j6n"
Sep 16 02:36:30.088035 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:30.087962    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62dc3fab-3a25-4d2a-bfb1-0b951833ef2d-istio-data\") pod \"e2e-ait-4108a843-openshift-default-7dd58f44f4-q5j6n\" (UID: \"62dc3fab-3a25-4d2a-bfb1-0b951833ef2d\") " pod="openshift-ingress/e2e-ait-4108a843-openshift-default-7dd58f44f4-q5j6n"
Sep 16 02:36:30.088035 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:30.088020    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62dc3fab-3a25-4d2a-bfb1-0b951833ef2d-istio-podinfo\") pod \"e2e-ait-4108a843-openshift-default-7dd58f44f4-q5j6n\" (UID: \"62dc3fab-3a25-4d2a-bfb1-0b951833ef2d\") " pod="openshift-ingress/e2e-ait-4108a843-openshift-default-7dd58f44f4-q5j6n"
Sep 16 02:36:30.088285 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:30.088060    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62dc3fab-3a25-4d2a-bfb1-0b951833ef2d-istiod-ca-cert\") pod \"e2e-ait-4108a843-openshift-default-7dd58f44f4-q5j6n\" (UID: \"62dc3fab-3a25-4d2a-bfb1-0b951833ef2d\") " pod="openshift-ingress/e2e-ait-4108a843-openshift-default-7dd58f44f4-q5j6n"
Sep 16 02:36:30.088285 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:30.088142    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7247ce0-45c5-44dc-be9b-0b9fd9c3893a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-1a323c-h2l24\" (UID: \"d7247ce0-45c5-44dc-be9b-0b9fd9c3893a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1a323c-h2l24"
Sep 16 02:36:30.088285 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:30.088180    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62dc3fab-3a25-4d2a-bfb1-0b951833ef2d-workload-socket\") pod \"e2e-ait-4108a843-openshift-default-7dd58f44f4-q5j6n\" (UID: \"62dc3fab-3a25-4d2a-bfb1-0b951833ef2d\") " pod="openshift-ingress/e2e-ait-4108a843-openshift-default-7dd58f44f4-q5j6n"
Sep 16 02:36:30.088948 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:30.088918    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7247ce0-45c5-44dc-be9b-0b9fd9c3893a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-1a323c-h2l24\" (UID: \"d7247ce0-45c5-44dc-be9b-0b9fd9c3893a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1a323c-h2l24"
Sep 16 02:36:30.157717 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:30.157669    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1a323c-h2l24"
Sep 16 02:36:30.189134 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:30.189107    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62dc3fab-3a25-4d2a-bfb1-0b951833ef2d-istio-envoy\") pod \"e2e-ait-4108a843-openshift-default-7dd58f44f4-q5j6n\" (UID: \"62dc3fab-3a25-4d2a-bfb1-0b951833ef2d\") " pod="openshift-ingress/e2e-ait-4108a843-openshift-default-7dd58f44f4-q5j6n"
Sep 16 02:36:30.189304 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:30.189146    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62dc3fab-3a25-4d2a-bfb1-0b951833ef2d-istio-token\") pod \"e2e-ait-4108a843-openshift-default-7dd58f44f4-q5j6n\" (UID: \"62dc3fab-3a25-4d2a-bfb1-0b951833ef2d\") " pod="openshift-ingress/e2e-ait-4108a843-openshift-default-7dd58f44f4-q5j6n"
Sep 16 02:36:30.189304 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:30.189168    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62dc3fab-3a25-4d2a-bfb1-0b951833ef2d-credential-socket\") pod \"e2e-ait-4108a843-openshift-default-7dd58f44f4-q5j6n\" (UID: \"62dc3fab-3a25-4d2a-bfb1-0b951833ef2d\") " pod="openshift-ingress/e2e-ait-4108a843-openshift-default-7dd58f44f4-q5j6n"
Sep 16 02:36:30.189304 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:30.189196    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvlbp\" (UniqueName: \"kubernetes.io/projected/62dc3fab-3a25-4d2a-bfb1-0b951833ef2d-kube-api-access-cvlbp\") pod \"e2e-ait-4108a843-openshift-default-7dd58f44f4-q5j6n\" (UID: \"62dc3fab-3a25-4d2a-bfb1-0b951833ef2d\") " pod="openshift-ingress/e2e-ait-4108a843-openshift-default-7dd58f44f4-q5j6n"
Sep 16 02:36:30.189304 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:30.189242    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62dc3fab-3a25-4d2a-bfb1-0b951833ef2d-workload-certs\") pod \"e2e-ait-4108a843-openshift-default-7dd58f44f4-q5j6n\" (UID: \"62dc3fab-3a25-4d2a-bfb1-0b951833ef2d\") " pod="openshift-ingress/e2e-ait-4108a843-openshift-default-7dd58f44f4-q5j6n"
Sep 16 02:36:30.189304 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:30.189266    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62dc3fab-3a25-4d2a-bfb1-0b951833ef2d-istio-data\") pod \"e2e-ait-4108a843-openshift-default-7dd58f44f4-q5j6n\" (UID: \"62dc3fab-3a25-4d2a-bfb1-0b951833ef2d\") " pod="openshift-ingress/e2e-ait-4108a843-openshift-default-7dd58f44f4-q5j6n"
Sep 16 02:36:30.189304 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:30.189296    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62dc3fab-3a25-4d2a-bfb1-0b951833ef2d-istio-podinfo\") pod \"e2e-ait-4108a843-openshift-default-7dd58f44f4-q5j6n\" (UID: \"62dc3fab-3a25-4d2a-bfb1-0b951833ef2d\") " pod="openshift-ingress/e2e-ait-4108a843-openshift-default-7dd58f44f4-q5j6n"
Sep 16 02:36:30.189626 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:30.189330    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62dc3fab-3a25-4d2a-bfb1-0b951833ef2d-istiod-ca-cert\") pod \"e2e-ait-4108a843-openshift-default-7dd58f44f4-q5j6n\" (UID: \"62dc3fab-3a25-4d2a-bfb1-0b951833ef2d\") " pod="openshift-ingress/e2e-ait-4108a843-openshift-default-7dd58f44f4-q5j6n"
Sep 16 02:36:30.189626 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:30.189387    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62dc3fab-3a25-4d2a-bfb1-0b951833ef2d-workload-socket\") pod \"e2e-ait-4108a843-openshift-default-7dd58f44f4-q5j6n\" (UID: \"62dc3fab-3a25-4d2a-bfb1-0b951833ef2d\") " pod="openshift-ingress/e2e-ait-4108a843-openshift-default-7dd58f44f4-q5j6n"
Sep 16 02:36:30.189626 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:30.189570    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62dc3fab-3a25-4d2a-bfb1-0b951833ef2d-credential-socket\") pod \"e2e-ait-4108a843-openshift-default-7dd58f44f4-q5j6n\" (UID: \"62dc3fab-3a25-4d2a-bfb1-0b951833ef2d\") " pod="openshift-ingress/e2e-ait-4108a843-openshift-default-7dd58f44f4-q5j6n"
Sep 16 02:36:30.189818 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:30.189767    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62dc3fab-3a25-4d2a-bfb1-0b951833ef2d-istio-data\") pod \"e2e-ait-4108a843-openshift-default-7dd58f44f4-q5j6n\" (UID: \"62dc3fab-3a25-4d2a-bfb1-0b951833ef2d\") " pod="openshift-ingress/e2e-ait-4108a843-openshift-default-7dd58f44f4-q5j6n"
Sep 16 02:36:30.189818 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:30.189808    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62dc3fab-3a25-4d2a-bfb1-0b951833ef2d-workload-socket\") pod \"e2e-ait-4108a843-openshift-default-7dd58f44f4-q5j6n\" (UID: \"62dc3fab-3a25-4d2a-bfb1-0b951833ef2d\") " pod="openshift-ingress/e2e-ait-4108a843-openshift-default-7dd58f44f4-q5j6n"
Sep 16 02:36:30.189936 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:30.189896    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62dc3fab-3a25-4d2a-bfb1-0b951833ef2d-workload-certs\") pod \"e2e-ait-4108a843-openshift-default-7dd58f44f4-q5j6n\" (UID: \"62dc3fab-3a25-4d2a-bfb1-0b951833ef2d\") " pod="openshift-ingress/e2e-ait-4108a843-openshift-default-7dd58f44f4-q5j6n"
Sep 16 02:36:30.190246 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:30.190215    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62dc3fab-3a25-4d2a-bfb1-0b951833ef2d-istiod-ca-cert\") pod \"e2e-ait-4108a843-openshift-default-7dd58f44f4-q5j6n\" (UID: \"62dc3fab-3a25-4d2a-bfb1-0b951833ef2d\") " pod="openshift-ingress/e2e-ait-4108a843-openshift-default-7dd58f44f4-q5j6n"
Sep 16 02:36:30.192153 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:30.192126    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62dc3fab-3a25-4d2a-bfb1-0b951833ef2d-istio-envoy\") pod \"e2e-ait-4108a843-openshift-default-7dd58f44f4-q5j6n\" (UID: \"62dc3fab-3a25-4d2a-bfb1-0b951833ef2d\") " pod="openshift-ingress/e2e-ait-4108a843-openshift-default-7dd58f44f4-q5j6n"
Sep 16 02:36:30.192289 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:30.192273    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62dc3fab-3a25-4d2a-bfb1-0b951833ef2d-istio-podinfo\") pod \"e2e-ait-4108a843-openshift-default-7dd58f44f4-q5j6n\" (UID: \"62dc3fab-3a25-4d2a-bfb1-0b951833ef2d\") " pod="openshift-ingress/e2e-ait-4108a843-openshift-default-7dd58f44f4-q5j6n"
Sep 16 02:36:30.203947 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:30.203817    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62dc3fab-3a25-4d2a-bfb1-0b951833ef2d-istio-token\") pod \"e2e-ait-4108a843-openshift-default-7dd58f44f4-q5j6n\" (UID: \"62dc3fab-3a25-4d2a-bfb1-0b951833ef2d\") " pod="openshift-ingress/e2e-ait-4108a843-openshift-default-7dd58f44f4-q5j6n"
Sep 16 02:36:30.204070 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:30.204026    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvlbp\" (UniqueName: \"kubernetes.io/projected/62dc3fab-3a25-4d2a-bfb1-0b951833ef2d-kube-api-access-cvlbp\") pod \"e2e-ait-4108a843-openshift-default-7dd58f44f4-q5j6n\" (UID: \"62dc3fab-3a25-4d2a-bfb1-0b951833ef2d\") " pod="openshift-ingress/e2e-ait-4108a843-openshift-default-7dd58f44f4-q5j6n"
Sep 16 02:36:30.256918 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:30.256877    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-4108a843-openshift-default-7dd58f44f4-q5j6n"
Sep 16 02:36:30.430655 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:30.430628    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1a323c-h2l24"]
Sep 16 02:36:30.432184 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:36:30.432162    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7247ce0_45c5_44dc_be9b_0b9fd9c3893a.slice/crio-f612b281f92c49d7a0f21570423a09c7159de493856b5f507f7ab20736a1b6af WatchSource:0}: Error finding container f612b281f92c49d7a0f21570423a09c7159de493856b5f507f7ab20736a1b6af: Status 404 returned error can't find the container with id f612b281f92c49d7a0f21570423a09c7159de493856b5f507f7ab20736a1b6af
Sep 16 02:36:30.535412 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:30.535384    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-4108a843-openshift-default-7dd58f44f4-q5j6n"]
Sep 16 02:36:30.536792 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:36:30.536757    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62dc3fab_3a25_4d2a_bfb1_0b951833ef2d.slice/crio-d524b12c9a6d4c2c19420556a1fbd8e07c81cd6dc434d8a3b52502e413330b9c WatchSource:0}: Error finding container d524b12c9a6d4c2c19420556a1fbd8e07c81cd6dc434d8a3b52502e413330b9c: Status 404 returned error can't find the container with id d524b12c9a6d4c2c19420556a1fbd8e07c81cd6dc434d8a3b52502e413330b9c
Sep 16 02:36:30.539023 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:30.538991    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 16 02:36:30.539088 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:30.539053    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 16 02:36:30.539131 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:30.539085    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 16 02:36:30.836628 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:30.836534    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-4108a843-openshift-default-7dd58f44f4-q5j6n" event={"ID":"62dc3fab-3a25-4d2a-bfb1-0b951833ef2d","Type":"ContainerStarted","Data":"dad703198e12fcc83a6786f16b9260cddec9b5d6874d435139156e7b815b0d43"}
Sep 16 02:36:30.836628 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:30.836597    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-4108a843-openshift-default-7dd58f44f4-q5j6n" event={"ID":"62dc3fab-3a25-4d2a-bfb1-0b951833ef2d","Type":"ContainerStarted","Data":"d524b12c9a6d4c2c19420556a1fbd8e07c81cd6dc434d8a3b52502e413330b9c"}
Sep 16 02:36:30.839236 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:30.839204    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1a323c-h2l24" event={"ID":"d7247ce0-45c5-44dc-be9b-0b9fd9c3893a","Type":"ContainerStarted","Data":"aefe1b4a34baee9387c9397e9d0bf71bea316c74b02625fbb6502fb30451f122"}
Sep 16 02:36:30.839355 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:30.839260    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1a323c-h2l24" event={"ID":"d7247ce0-45c5-44dc-be9b-0b9fd9c3893a","Type":"ContainerStarted","Data":"f612b281f92c49d7a0f21570423a09c7159de493856b5f507f7ab20736a1b6af"}
Sep 16 02:36:30.854720 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:30.854647    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-4108a843-openshift-default-7dd58f44f4-q5j6n" podStartSLOduration=1.854631528 podStartE2EDuration="1.854631528s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 02:36:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:36:30.853870149 +0000 UTC m=+1403.772359402" watchObservedRunningTime="2026-09-16 02:36:30.854631528 +0000 UTC m=+1403.773120776"
Sep 16 02:36:30.867429 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:30.867375    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1a323c-h2l24" podStartSLOduration=1.867355753 podStartE2EDuration="1.867355753s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 02:36:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:36:30.866715572 +0000 UTC m=+1403.785204820" watchObservedRunningTime="2026-09-16 02:36:30.867355753 +0000 UTC m=+1403.785845002"
Sep 16 02:36:31.257538 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:31.257505    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-4108a843-openshift-default-7dd58f44f4-q5j6n"
Sep 16 02:36:31.262645 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:31.262624    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-4108a843-openshift-default-7dd58f44f4-q5j6n"
Sep 16 02:36:31.550368 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:31.550331    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2f1860b3-bb58-40c5-a590-9c665f7a7fda" path="/var/lib/kubelet/pods/2f1860b3-bb58-40c5-a590-9c665f7a7fda/volumes"
Sep 16 02:36:31.845924 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:31.845843    2558 generic.go:356] "Generic (PLEG): container finished" podID="d7247ce0-45c5-44dc-be9b-0b9fd9c3893a" containerID="aefe1b4a34baee9387c9397e9d0bf71bea316c74b02625fbb6502fb30451f122" exitCode=0
Sep 16 02:36:31.846062 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:31.845930    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1a323c-h2l24" event={"ID":"d7247ce0-45c5-44dc-be9b-0b9fd9c3893a","Type":"ContainerDied","Data":"aefe1b4a34baee9387c9397e9d0bf71bea316c74b02625fbb6502fb30451f122"}
Sep 16 02:36:31.846259 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:31.846236    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-4108a843-openshift-default-7dd58f44f4-q5j6n"
Sep 16 02:36:31.847051 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:31.847035    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-4108a843-openshift-default-7dd58f44f4-q5j6n"
Sep 16 02:36:32.987718 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:32.987672    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1a323c-h2l24"
Sep 16 02:36:33.119955 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:33.119876    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7247ce0-45c5-44dc-be9b-0b9fd9c3893a-maas-api-service-ca\") pod \"d7247ce0-45c5-44dc-be9b-0b9fd9c3893a\" (UID: \"d7247ce0-45c5-44dc-be9b-0b9fd9c3893a\") "
Sep 16 02:36:33.120216 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:33.120192    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7247ce0-45c5-44dc-be9b-0b9fd9c3893a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d7247ce0-45c5-44dc-be9b-0b9fd9c3893a" (UID: "d7247ce0-45c5-44dc-be9b-0b9fd9c3893a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 02:36:33.221241 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:33.221210    2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7247ce0-45c5-44dc-be9b-0b9fd9c3893a-maas-api-service-ca\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:36:33.858558 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:33.858462    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1a323c-h2l24"
Sep 16 02:36:33.858558 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:33.858498    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1a323c-h2l24" event={"ID":"d7247ce0-45c5-44dc-be9b-0b9fd9c3893a","Type":"ContainerDied","Data":"f612b281f92c49d7a0f21570423a09c7159de493856b5f507f7ab20736a1b6af"}
Sep 16 02:36:33.858558 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:33.858549    2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f612b281f92c49d7a0f21570423a09c7159de493856b5f507f7ab20736a1b6af"
Sep 16 02:36:34.593468 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:34.593444    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-0bdfbb54-gw-openshift-default-5ccb9467b4-4j8g7"
Sep 16 02:36:34.713161 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:34.713137    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-60674e-openshift-default-6c56877b8b-qc47f"
Sep 16 02:36:34.736778 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:34.736745    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e0742d4-fc6e-4bcd-8d28-ee56bcd52b83-istio-podinfo\") pod \"9e0742d4-fc6e-4bcd-8d28-ee56bcd52b83\" (UID: \"9e0742d4-fc6e-4bcd-8d28-ee56bcd52b83\") "
Sep 16 02:36:34.736940 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:34.736796    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e0742d4-fc6e-4bcd-8d28-ee56bcd52b83-istiod-ca-cert\") pod \"9e0742d4-fc6e-4bcd-8d28-ee56bcd52b83\" (UID: \"9e0742d4-fc6e-4bcd-8d28-ee56bcd52b83\") "
Sep 16 02:36:34.736940 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:34.736843    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e0742d4-fc6e-4bcd-8d28-ee56bcd52b83-istio-data\") pod \"9e0742d4-fc6e-4bcd-8d28-ee56bcd52b83\" (UID: \"9e0742d4-fc6e-4bcd-8d28-ee56bcd52b83\") "
Sep 16 02:36:34.736940 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:34.736874    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cv65k\" (UniqueName: \"kubernetes.io/projected/9e0742d4-fc6e-4bcd-8d28-ee56bcd52b83-kube-api-access-cv65k\") pod \"9e0742d4-fc6e-4bcd-8d28-ee56bcd52b83\" (UID: \"9e0742d4-fc6e-4bcd-8d28-ee56bcd52b83\") "
Sep 16 02:36:34.736940 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:34.736908    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e0742d4-fc6e-4bcd-8d28-ee56bcd52b83-workload-socket\") pod \"9e0742d4-fc6e-4bcd-8d28-ee56bcd52b83\" (UID: \"9e0742d4-fc6e-4bcd-8d28-ee56bcd52b83\") "
Sep 16 02:36:34.736940 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:34.736930    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e0742d4-fc6e-4bcd-8d28-ee56bcd52b83-istio-envoy\") pod \"9e0742d4-fc6e-4bcd-8d28-ee56bcd52b83\" (UID: \"9e0742d4-fc6e-4bcd-8d28-ee56bcd52b83\") "
Sep 16 02:36:34.737207 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:34.736968    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e0742d4-fc6e-4bcd-8d28-ee56bcd52b83-istio-token\") pod \"9e0742d4-fc6e-4bcd-8d28-ee56bcd52b83\" (UID: \"9e0742d4-fc6e-4bcd-8d28-ee56bcd52b83\") "
Sep 16 02:36:34.737207 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:34.737062    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e0742d4-fc6e-4bcd-8d28-ee56bcd52b83-credential-socket\") pod \"9e0742d4-fc6e-4bcd-8d28-ee56bcd52b83\" (UID: \"9e0742d4-fc6e-4bcd-8d28-ee56bcd52b83\") "
Sep 16 02:36:34.737207 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:34.737090    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e0742d4-fc6e-4bcd-8d28-ee56bcd52b83-workload-certs\") pod \"9e0742d4-fc6e-4bcd-8d28-ee56bcd52b83\" (UID: \"9e0742d4-fc6e-4bcd-8d28-ee56bcd52b83\") "
Sep 16 02:36:34.737369 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:34.737267    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e0742d4-fc6e-4bcd-8d28-ee56bcd52b83-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "9e0742d4-fc6e-4bcd-8d28-ee56bcd52b83" (UID: "9e0742d4-fc6e-4bcd-8d28-ee56bcd52b83"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 02:36:34.738731 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:34.737625    2558 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e0742d4-fc6e-4bcd-8d28-ee56bcd52b83-istiod-ca-cert\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:36:34.738731 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:34.737832    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e0742d4-fc6e-4bcd-8d28-ee56bcd52b83-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "9e0742d4-fc6e-4bcd-8d28-ee56bcd52b83" (UID: "9e0742d4-fc6e-4bcd-8d28-ee56bcd52b83"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:36:34.738731 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:34.737831    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e0742d4-fc6e-4bcd-8d28-ee56bcd52b83-istio-data" (OuterVolumeSpecName: "istio-data") pod "9e0742d4-fc6e-4bcd-8d28-ee56bcd52b83" (UID: "9e0742d4-fc6e-4bcd-8d28-ee56bcd52b83"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:36:34.738731 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:34.738020    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e0742d4-fc6e-4bcd-8d28-ee56bcd52b83-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "9e0742d4-fc6e-4bcd-8d28-ee56bcd52b83" (UID: "9e0742d4-fc6e-4bcd-8d28-ee56bcd52b83"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:36:34.738731 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:34.738124    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e0742d4-fc6e-4bcd-8d28-ee56bcd52b83-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "9e0742d4-fc6e-4bcd-8d28-ee56bcd52b83" (UID: "9e0742d4-fc6e-4bcd-8d28-ee56bcd52b83"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:36:34.739659 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:34.739634    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9e0742d4-fc6e-4bcd-8d28-ee56bcd52b83-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "9e0742d4-fc6e-4bcd-8d28-ee56bcd52b83" (UID: "9e0742d4-fc6e-4bcd-8d28-ee56bcd52b83"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 02:36:34.739659 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:34.739648    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e0742d4-fc6e-4bcd-8d28-ee56bcd52b83-kube-api-access-cv65k" (OuterVolumeSpecName: "kube-api-access-cv65k") pod "9e0742d4-fc6e-4bcd-8d28-ee56bcd52b83" (UID: "9e0742d4-fc6e-4bcd-8d28-ee56bcd52b83"). InnerVolumeSpecName "kube-api-access-cv65k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 02:36:34.739837 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:34.739661    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e0742d4-fc6e-4bcd-8d28-ee56bcd52b83-istio-token" (OuterVolumeSpecName: "istio-token") pod "9e0742d4-fc6e-4bcd-8d28-ee56bcd52b83" (UID: "9e0742d4-fc6e-4bcd-8d28-ee56bcd52b83"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 02:36:34.740022 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:34.739997    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e0742d4-fc6e-4bcd-8d28-ee56bcd52b83-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "9e0742d4-fc6e-4bcd-8d28-ee56bcd52b83" (UID: "9e0742d4-fc6e-4bcd-8d28-ee56bcd52b83"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:36:34.838550 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:34.838474    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d81e13b-62c5-4fe2-a392-c5bac92f0ba5-workload-socket\") pod \"6d81e13b-62c5-4fe2-a392-c5bac92f0ba5\" (UID: \"6d81e13b-62c5-4fe2-a392-c5bac92f0ba5\") "
Sep 16 02:36:34.838692 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:34.838573    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d81e13b-62c5-4fe2-a392-c5bac92f0ba5-istio-podinfo\") pod \"6d81e13b-62c5-4fe2-a392-c5bac92f0ba5\" (UID: \"6d81e13b-62c5-4fe2-a392-c5bac92f0ba5\") "
Sep 16 02:36:34.838692 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:34.838633    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d81e13b-62c5-4fe2-a392-c5bac92f0ba5-istio-token\") pod \"6d81e13b-62c5-4fe2-a392-c5bac92f0ba5\" (UID: \"6d81e13b-62c5-4fe2-a392-c5bac92f0ba5\") "
Sep 16 02:36:34.838692 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:34.838666    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ssrw2\" (UniqueName: \"kubernetes.io/projected/6d81e13b-62c5-4fe2-a392-c5bac92f0ba5-kube-api-access-ssrw2\") pod \"6d81e13b-62c5-4fe2-a392-c5bac92f0ba5\" (UID: \"6d81e13b-62c5-4fe2-a392-c5bac92f0ba5\") "
Sep 16 02:36:34.838692 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:34.838689    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d81e13b-62c5-4fe2-a392-c5bac92f0ba5-credential-socket\") pod \"6d81e13b-62c5-4fe2-a392-c5bac92f0ba5\" (UID: \"6d81e13b-62c5-4fe2-a392-c5bac92f0ba5\") "
Sep 16 02:36:34.838971 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:34.838727    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d81e13b-62c5-4fe2-a392-c5bac92f0ba5-istiod-ca-cert\") pod \"6d81e13b-62c5-4fe2-a392-c5bac92f0ba5\" (UID: \"6d81e13b-62c5-4fe2-a392-c5bac92f0ba5\") "
Sep 16 02:36:34.838971 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:34.838742    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d81e13b-62c5-4fe2-a392-c5bac92f0ba5-istio-envoy\") pod \"6d81e13b-62c5-4fe2-a392-c5bac92f0ba5\" (UID: \"6d81e13b-62c5-4fe2-a392-c5bac92f0ba5\") "
Sep 16 02:36:34.838971 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:34.838811    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d81e13b-62c5-4fe2-a392-c5bac92f0ba5-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "6d81e13b-62c5-4fe2-a392-c5bac92f0ba5" (UID: "6d81e13b-62c5-4fe2-a392-c5bac92f0ba5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:36:34.838971 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:34.838872    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d81e13b-62c5-4fe2-a392-c5bac92f0ba5-workload-certs\") pod \"6d81e13b-62c5-4fe2-a392-c5bac92f0ba5\" (UID: \"6d81e13b-62c5-4fe2-a392-c5bac92f0ba5\") "
Sep 16 02:36:34.838971 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:34.838909    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d81e13b-62c5-4fe2-a392-c5bac92f0ba5-istio-data\") pod \"6d81e13b-62c5-4fe2-a392-c5bac92f0ba5\" (UID: \"6d81e13b-62c5-4fe2-a392-c5bac92f0ba5\") "
Sep 16 02:36:34.839236 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:34.839053    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d81e13b-62c5-4fe2-a392-c5bac92f0ba5-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "6d81e13b-62c5-4fe2-a392-c5bac92f0ba5" (UID: "6d81e13b-62c5-4fe2-a392-c5bac92f0ba5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:36:34.839236 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:34.839186    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d81e13b-62c5-4fe2-a392-c5bac92f0ba5-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "6d81e13b-62c5-4fe2-a392-c5bac92f0ba5" (UID: "6d81e13b-62c5-4fe2-a392-c5bac92f0ba5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:36:34.839348 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:34.839325    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d81e13b-62c5-4fe2-a392-c5bac92f0ba5-istio-data" (OuterVolumeSpecName: "istio-data") pod "6d81e13b-62c5-4fe2-a392-c5bac92f0ba5" (UID: "6d81e13b-62c5-4fe2-a392-c5bac92f0ba5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:36:34.839401 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:34.839343    2558 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e0742d4-fc6e-4bcd-8d28-ee56bcd52b83-istio-podinfo\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:36:34.839401 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:34.839363    2558 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d81e13b-62c5-4fe2-a392-c5bac92f0ba5-credential-socket\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:36:34.839401 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:34.839380    2558 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d81e13b-62c5-4fe2-a392-c5bac92f0ba5-workload-certs\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:36:34.839401 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:34.839393    2558 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e0742d4-fc6e-4bcd-8d28-ee56bcd52b83-istio-data\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:36:34.839602 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:34.839406    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cv65k\" (UniqueName: \"kubernetes.io/projected/9e0742d4-fc6e-4bcd-8d28-ee56bcd52b83-kube-api-access-cv65k\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:36:34.839602 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:34.839420    2558 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d81e13b-62c5-4fe2-a392-c5bac92f0ba5-workload-socket\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:36:34.839602 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:34.839433    2558 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e0742d4-fc6e-4bcd-8d28-ee56bcd52b83-workload-socket\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:36:34.839602 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:34.839445    2558 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e0742d4-fc6e-4bcd-8d28-ee56bcd52b83-istio-envoy\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:36:34.839602 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:34.839378    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d81e13b-62c5-4fe2-a392-c5bac92f0ba5-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "6d81e13b-62c5-4fe2-a392-c5bac92f0ba5" (UID: "6d81e13b-62c5-4fe2-a392-c5bac92f0ba5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 02:36:34.839602 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:34.839458    2558 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e0742d4-fc6e-4bcd-8d28-ee56bcd52b83-istio-token\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:36:34.839602 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:34.839471    2558 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e0742d4-fc6e-4bcd-8d28-ee56bcd52b83-credential-socket\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:36:34.839602 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:34.839486    2558 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e0742d4-fc6e-4bcd-8d28-ee56bcd52b83-workload-certs\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:36:34.840848 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:34.840826    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6d81e13b-62c5-4fe2-a392-c5bac92f0ba5-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "6d81e13b-62c5-4fe2-a392-c5bac92f0ba5" (UID: "6d81e13b-62c5-4fe2-a392-c5bac92f0ba5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 02:36:34.840848 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:34.840840    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d81e13b-62c5-4fe2-a392-c5bac92f0ba5-istio-token" (OuterVolumeSpecName: "istio-token") pod "6d81e13b-62c5-4fe2-a392-c5bac92f0ba5" (UID: "6d81e13b-62c5-4fe2-a392-c5bac92f0ba5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 02:36:34.840975 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:34.840862    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d81e13b-62c5-4fe2-a392-c5bac92f0ba5-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "6d81e13b-62c5-4fe2-a392-c5bac92f0ba5" (UID: "6d81e13b-62c5-4fe2-a392-c5bac92f0ba5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:36:34.840975 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:34.840911    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d81e13b-62c5-4fe2-a392-c5bac92f0ba5-kube-api-access-ssrw2" (OuterVolumeSpecName: "kube-api-access-ssrw2") pod "6d81e13b-62c5-4fe2-a392-c5bac92f0ba5" (UID: "6d81e13b-62c5-4fe2-a392-c5bac92f0ba5"). InnerVolumeSpecName "kube-api-access-ssrw2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 02:36:34.866020 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:34.865993    2558 generic.go:356] "Generic (PLEG): container finished" podID="9e0742d4-fc6e-4bcd-8d28-ee56bcd52b83" containerID="81cbb3a8bef74a3221c78ab2d1beaa0a99256fc447c6afcf41e482a9ced54e3e" exitCode=0
Sep 16 02:36:34.866113 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:34.866038    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-0bdfbb54-gw-openshift-default-5ccb9467b4-4j8g7"
Sep 16 02:36:34.866113 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:34.866077    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-0bdfbb54-gw-openshift-default-5ccb9467b4-4j8g7" event={"ID":"9e0742d4-fc6e-4bcd-8d28-ee56bcd52b83","Type":"ContainerDied","Data":"81cbb3a8bef74a3221c78ab2d1beaa0a99256fc447c6afcf41e482a9ced54e3e"}
Sep 16 02:36:34.866193 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:34.866129    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-0bdfbb54-gw-openshift-default-5ccb9467b4-4j8g7" event={"ID":"9e0742d4-fc6e-4bcd-8d28-ee56bcd52b83","Type":"ContainerDied","Data":"4ebd747875fe9e24d92c2b4b07f6dad758d441e3510b2051ba7ec4d3bc78da26"}
Sep 16 02:36:34.866193 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:34.866149    2558 scope.go:117] "RemoveContainer" containerID="81cbb3a8bef74a3221c78ab2d1beaa0a99256fc447c6afcf41e482a9ced54e3e"
Sep 16 02:36:34.867954 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:34.867934    2558 generic.go:356] "Generic (PLEG): container finished" podID="6d81e13b-62c5-4fe2-a392-c5bac92f0ba5" containerID="40e50b8af904fbd01b62dc8a1f9dfc60fb2c3c2091d72faa3c43a599a41d3dc1" exitCode=0
Sep 16 02:36:34.868071 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:34.867964    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-60674e-openshift-default-6c56877b8b-qc47f" event={"ID":"6d81e13b-62c5-4fe2-a392-c5bac92f0ba5","Type":"ContainerDied","Data":"40e50b8af904fbd01b62dc8a1f9dfc60fb2c3c2091d72faa3c43a599a41d3dc1"}
Sep 16 02:36:34.868071 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:34.867973    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-60674e-openshift-default-6c56877b8b-qc47f"
Sep 16 02:36:34.868071 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:34.867995    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-60674e-openshift-default-6c56877b8b-qc47f" event={"ID":"6d81e13b-62c5-4fe2-a392-c5bac92f0ba5","Type":"ContainerDied","Data":"140a797fc3861f3a70c91df8260b42966dda0ec979da682eac9a8ec8753e2cf8"}
Sep 16 02:36:34.874991 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:34.874955    2558 scope.go:117] "RemoveContainer" containerID="44a78496d708a283f79b657a7c02630dfe18dc95324467bf08a92f1c4428664d"
Sep 16 02:36:34.883435 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:34.883414    2558 scope.go:117] "RemoveContainer" containerID="81cbb3a8bef74a3221c78ab2d1beaa0a99256fc447c6afcf41e482a9ced54e3e"
Sep 16 02:36:34.883746 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:36:34.883722    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81cbb3a8bef74a3221c78ab2d1beaa0a99256fc447c6afcf41e482a9ced54e3e\": container with ID starting with 81cbb3a8bef74a3221c78ab2d1beaa0a99256fc447c6afcf41e482a9ced54e3e not found: ID does not exist" containerID="81cbb3a8bef74a3221c78ab2d1beaa0a99256fc447c6afcf41e482a9ced54e3e"
Sep 16 02:36:34.883811 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:34.883759    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81cbb3a8bef74a3221c78ab2d1beaa0a99256fc447c6afcf41e482a9ced54e3e"} err="failed to get container status \"81cbb3a8bef74a3221c78ab2d1beaa0a99256fc447c6afcf41e482a9ced54e3e\": rpc error: code = NotFound desc = could not find container \"81cbb3a8bef74a3221c78ab2d1beaa0a99256fc447c6afcf41e482a9ced54e3e\": container with ID starting with 81cbb3a8bef74a3221c78ab2d1beaa0a99256fc447c6afcf41e482a9ced54e3e not found: ID does not exist"
Sep 16 02:36:34.883811 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:34.883783    2558 scope.go:117] "RemoveContainer" containerID="44a78496d708a283f79b657a7c02630dfe18dc95324467bf08a92f1c4428664d"
Sep 16 02:36:34.884044 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:36:34.884026    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44a78496d708a283f79b657a7c02630dfe18dc95324467bf08a92f1c4428664d\": container with ID starting with 44a78496d708a283f79b657a7c02630dfe18dc95324467bf08a92f1c4428664d not found: ID does not exist" containerID="44a78496d708a283f79b657a7c02630dfe18dc95324467bf08a92f1c4428664d"
Sep 16 02:36:34.884104 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:34.884053    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44a78496d708a283f79b657a7c02630dfe18dc95324467bf08a92f1c4428664d"} err="failed to get container status \"44a78496d708a283f79b657a7c02630dfe18dc95324467bf08a92f1c4428664d\": rpc error: code = NotFound desc = could not find container \"44a78496d708a283f79b657a7c02630dfe18dc95324467bf08a92f1c4428664d\": container with ID starting with 44a78496d708a283f79b657a7c02630dfe18dc95324467bf08a92f1c4428664d not found: ID does not exist"
Sep 16 02:36:34.884104 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:34.884075    2558 scope.go:117] "RemoveContainer" containerID="40e50b8af904fbd01b62dc8a1f9dfc60fb2c3c2091d72faa3c43a599a41d3dc1"
Sep 16 02:36:34.889972 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:34.889948    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-0bdfbb54-gw-openshift-default-5ccb9467b4-4j8g7"]
Sep 16 02:36:34.892611 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:34.892592    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-0bdfbb54-gw-openshift-default-5ccb9467b4-4j8g7"]
Sep 16 02:36:34.893510 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:34.893490    2558 scope.go:117] "RemoveContainer" containerID="0ac6c1c28f7f6b211391cee13d996781b9d16a1382fb2b1a550c05c66ef27839"
Sep 16 02:36:34.901517 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:34.901497    2558 scope.go:117] "RemoveContainer" containerID="40e50b8af904fbd01b62dc8a1f9dfc60fb2c3c2091d72faa3c43a599a41d3dc1"
Sep 16 02:36:34.901794 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:36:34.901769    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40e50b8af904fbd01b62dc8a1f9dfc60fb2c3c2091d72faa3c43a599a41d3dc1\": container with ID starting with 40e50b8af904fbd01b62dc8a1f9dfc60fb2c3c2091d72faa3c43a599a41d3dc1 not found: ID does not exist" containerID="40e50b8af904fbd01b62dc8a1f9dfc60fb2c3c2091d72faa3c43a599a41d3dc1"
Sep 16 02:36:34.901887 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:34.901795    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40e50b8af904fbd01b62dc8a1f9dfc60fb2c3c2091d72faa3c43a599a41d3dc1"} err="failed to get container status \"40e50b8af904fbd01b62dc8a1f9dfc60fb2c3c2091d72faa3c43a599a41d3dc1\": rpc error: code = NotFound desc = could not find container \"40e50b8af904fbd01b62dc8a1f9dfc60fb2c3c2091d72faa3c43a599a41d3dc1\": container with ID starting with 40e50b8af904fbd01b62dc8a1f9dfc60fb2c3c2091d72faa3c43a599a41d3dc1 not found: ID does not exist"
Sep 16 02:36:34.901887 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:34.901813    2558 scope.go:117] "RemoveContainer" containerID="0ac6c1c28f7f6b211391cee13d996781b9d16a1382fb2b1a550c05c66ef27839"
Sep 16 02:36:34.902101 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:36:34.902085    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ac6c1c28f7f6b211391cee13d996781b9d16a1382fb2b1a550c05c66ef27839\": container with ID starting with 0ac6c1c28f7f6b211391cee13d996781b9d16a1382fb2b1a550c05c66ef27839 not found: ID does not exist" containerID="0ac6c1c28f7f6b211391cee13d996781b9d16a1382fb2b1a550c05c66ef27839"
Sep 16 02:36:34.902148 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:34.902105    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ac6c1c28f7f6b211391cee13d996781b9d16a1382fb2b1a550c05c66ef27839"} err="failed to get container status \"0ac6c1c28f7f6b211391cee13d996781b9d16a1382fb2b1a550c05c66ef27839\": rpc error: code = NotFound desc = could not find container \"0ac6c1c28f7f6b211391cee13d996781b9d16a1382fb2b1a550c05c66ef27839\": container with ID starting with 0ac6c1c28f7f6b211391cee13d996781b9d16a1382fb2b1a550c05c66ef27839 not found: ID does not exist"
Sep 16 02:36:34.903236 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:34.903218    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-60674e-openshift-default-6c56877b8b-qc47f"]
Sep 16 02:36:34.910328 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:34.910304    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-60674e-openshift-default-6c56877b8b-qc47f"]
Sep 16 02:36:34.940968 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:34.940945    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ssrw2\" (UniqueName: \"kubernetes.io/projected/6d81e13b-62c5-4fe2-a392-c5bac92f0ba5-kube-api-access-ssrw2\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:36:34.941057 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:34.940968    2558 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d81e13b-62c5-4fe2-a392-c5bac92f0ba5-istiod-ca-cert\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:36:34.941057 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:34.940979    2558 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d81e13b-62c5-4fe2-a392-c5bac92f0ba5-istio-envoy\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:36:34.941057 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:34.940986    2558 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d81e13b-62c5-4fe2-a392-c5bac92f0ba5-istio-data\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:36:34.941057 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:34.940993    2558 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d81e13b-62c5-4fe2-a392-c5bac92f0ba5-istio-podinfo\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:36:34.941057 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:34.941000    2558 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d81e13b-62c5-4fe2-a392-c5bac92f0ba5-istio-token\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:36:35.550988 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:35.550956    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6d81e13b-62c5-4fe2-a392-c5bac92f0ba5" path="/var/lib/kubelet/pods/6d81e13b-62c5-4fe2-a392-c5bac92f0ba5/volumes"
Sep 16 02:36:35.551399 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:35.551385    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9e0742d4-fc6e-4bcd-8d28-ee56bcd52b83" path="/var/lib/kubelet/pods/9e0742d4-fc6e-4bcd-8d28-ee56bcd52b83/volumes"
Sep 16 02:36:35.924931 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:35.924899    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-4108a843-6bdfd44f58-qs4r5"]
Sep 16 02:36:35.925309 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:35.925276    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9e0742d4-fc6e-4bcd-8d28-ee56bcd52b83" containerName="istio-proxy"
Sep 16 02:36:35.925309 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:35.925288    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="9e0742d4-fc6e-4bcd-8d28-ee56bcd52b83" containerName="istio-proxy"
Sep 16 02:36:35.925309 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:35.925296    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d7247ce0-45c5-44dc-be9b-0b9fd9c3893a" containerName="revoke-keys"
Sep 16 02:36:35.925309 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:35.925302    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="d7247ce0-45c5-44dc-be9b-0b9fd9c3893a" containerName="revoke-keys"
Sep 16 02:36:35.925436 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:35.925318    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d81e13b-62c5-4fe2-a392-c5bac92f0ba5" containerName="istio-proxy"
Sep 16 02:36:35.925436 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:35.925324    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d81e13b-62c5-4fe2-a392-c5bac92f0ba5" containerName="istio-proxy"
Sep 16 02:36:35.925436 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:35.925389    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="6d81e13b-62c5-4fe2-a392-c5bac92f0ba5" containerName="istio-proxy"
Sep 16 02:36:35.925436 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:35.925395    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="9e0742d4-fc6e-4bcd-8d28-ee56bcd52b83" containerName="istio-proxy"
Sep 16 02:36:35.925436 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:35.925401    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="6d81e13b-62c5-4fe2-a392-c5bac92f0ba5" containerName="istio-proxy"
Sep 16 02:36:35.925436 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:35.925410    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="9e0742d4-fc6e-4bcd-8d28-ee56bcd52b83" containerName="istio-proxy"
Sep 16 02:36:35.925436 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:35.925415    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="d7247ce0-45c5-44dc-be9b-0b9fd9c3893a" containerName="revoke-keys"
Sep 16 02:36:35.930211 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:35.930189    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-4108a843-6bdfd44f58-qs4r5"
Sep 16 02:36:35.932662 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:35.932641    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-4108a843\""
Sep 16 02:36:35.932748 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:35.932730    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-4108a843-dockercfg-r2n92\""
Sep 16 02:36:35.941969 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:35.941946    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-4108a843-6bdfd44f58-qs4r5"]
Sep 16 02:36:35.964653 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:35.964613    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4108a843-75cd969bb9-pprlr"]
Sep 16 02:36:35.966138 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:35.966109    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9e0742d4-fc6e-4bcd-8d28-ee56bcd52b83" containerName="istio-proxy"
Sep 16 02:36:35.966366 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:35.966133    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="9e0742d4-fc6e-4bcd-8d28-ee56bcd52b83" containerName="istio-proxy"
Sep 16 02:36:35.966481 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:35.966373    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d81e13b-62c5-4fe2-a392-c5bac92f0ba5" containerName="istio-proxy"
Sep 16 02:36:35.966481 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:35.966381    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d81e13b-62c5-4fe2-a392-c5bac92f0ba5" containerName="istio-proxy"
Sep 16 02:36:35.969935 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:35.969909    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-4108a843-75cd969bb9-pprlr"
Sep 16 02:36:35.970937 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:35.970909    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4108a843-75cd969bb9-pprlr"]
Sep 16 02:36:36.052591 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:36.052551    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqxxd\" (UniqueName: \"kubernetes.io/projected/85748850-ba20-401c-8865-1042a8303491-kube-api-access-cqxxd\") pod \"payload-pre-processing-e2e-ait-4108a843-6bdfd44f58-qs4r5\" (UID: \"85748850-ba20-401c-8865-1042a8303491\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-4108a843-6bdfd44f58-qs4r5"
Sep 16 02:36:36.052944 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:36.052920    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85748850-ba20-401c-8865-1042a8303491-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-4108a843-6bdfd44f58-qs4r5\" (UID: \"85748850-ba20-401c-8865-1042a8303491\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-4108a843-6bdfd44f58-qs4r5"
Sep 16 02:36:36.154461 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:36.154424    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/971d8189-0b6b-4504-bcd3-a4335290edb5-plugins-config-volume\") pod \"payload-processing-e2e-ait-4108a843-75cd969bb9-pprlr\" (UID: \"971d8189-0b6b-4504-bcd3-a4335290edb5\") " pod="openshift-ingress/payload-processing-e2e-ait-4108a843-75cd969bb9-pprlr"
Sep 16 02:36:36.154634 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:36.154479    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85748850-ba20-401c-8865-1042a8303491-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-4108a843-6bdfd44f58-qs4r5\" (UID: \"85748850-ba20-401c-8865-1042a8303491\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-4108a843-6bdfd44f58-qs4r5"
Sep 16 02:36:36.154634 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:36.154604    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6pwg\" (UniqueName: \"kubernetes.io/projected/971d8189-0b6b-4504-bcd3-a4335290edb5-kube-api-access-w6pwg\") pod \"payload-processing-e2e-ait-4108a843-75cd969bb9-pprlr\" (UID: \"971d8189-0b6b-4504-bcd3-a4335290edb5\") " pod="openshift-ingress/payload-processing-e2e-ait-4108a843-75cd969bb9-pprlr"
Sep 16 02:36:36.154832 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:36.154648    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqxxd\" (UniqueName: \"kubernetes.io/projected/85748850-ba20-401c-8865-1042a8303491-kube-api-access-cqxxd\") pod \"payload-pre-processing-e2e-ait-4108a843-6bdfd44f58-qs4r5\" (UID: \"85748850-ba20-401c-8865-1042a8303491\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-4108a843-6bdfd44f58-qs4r5"
Sep 16 02:36:36.155250 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:36.155226    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85748850-ba20-401c-8865-1042a8303491-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-4108a843-6bdfd44f58-qs4r5\" (UID: \"85748850-ba20-401c-8865-1042a8303491\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-4108a843-6bdfd44f58-qs4r5"
Sep 16 02:36:36.162326 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:36.162299    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqxxd\" (UniqueName: \"kubernetes.io/projected/85748850-ba20-401c-8865-1042a8303491-kube-api-access-cqxxd\") pod \"payload-pre-processing-e2e-ait-4108a843-6bdfd44f58-qs4r5\" (UID: \"85748850-ba20-401c-8865-1042a8303491\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-4108a843-6bdfd44f58-qs4r5"
Sep 16 02:36:36.241261 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:36.241165    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-4108a843-6bdfd44f58-qs4r5"
Sep 16 02:36:36.256001 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:36.255958    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6pwg\" (UniqueName: \"kubernetes.io/projected/971d8189-0b6b-4504-bcd3-a4335290edb5-kube-api-access-w6pwg\") pod \"payload-processing-e2e-ait-4108a843-75cd969bb9-pprlr\" (UID: \"971d8189-0b6b-4504-bcd3-a4335290edb5\") " pod="openshift-ingress/payload-processing-e2e-ait-4108a843-75cd969bb9-pprlr"
Sep 16 02:36:36.256121 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:36.256075    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/971d8189-0b6b-4504-bcd3-a4335290edb5-plugins-config-volume\") pod \"payload-processing-e2e-ait-4108a843-75cd969bb9-pprlr\" (UID: \"971d8189-0b6b-4504-bcd3-a4335290edb5\") " pod="openshift-ingress/payload-processing-e2e-ait-4108a843-75cd969bb9-pprlr"
Sep 16 02:36:36.256781 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:36.256758    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/971d8189-0b6b-4504-bcd3-a4335290edb5-plugins-config-volume\") pod \"payload-processing-e2e-ait-4108a843-75cd969bb9-pprlr\" (UID: \"971d8189-0b6b-4504-bcd3-a4335290edb5\") " pod="openshift-ingress/payload-processing-e2e-ait-4108a843-75cd969bb9-pprlr"
Sep 16 02:36:36.263111 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:36.263080    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6pwg\" (UniqueName: \"kubernetes.io/projected/971d8189-0b6b-4504-bcd3-a4335290edb5-kube-api-access-w6pwg\") pod \"payload-processing-e2e-ait-4108a843-75cd969bb9-pprlr\" (UID: \"971d8189-0b6b-4504-bcd3-a4335290edb5\") " pod="openshift-ingress/payload-processing-e2e-ait-4108a843-75cd969bb9-pprlr"
Sep 16 02:36:36.283148 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:36.283108    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-4108a843-75cd969bb9-pprlr"
Sep 16 02:36:36.516595 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:36.516563    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-4108a843-6bdfd44f58-qs4r5"]
Sep 16 02:36:36.518169 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:36:36.518134    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85748850_ba20_401c_8865_1042a8303491.slice/crio-4b383422d358b20d92c03e5d6dcd0ab5978e7d6b43b5c94556c3936c50467a2a WatchSource:0}: Error finding container 4b383422d358b20d92c03e5d6dcd0ab5978e7d6b43b5c94556c3936c50467a2a: Status 404 returned error can't find the container with id 4b383422d358b20d92c03e5d6dcd0ab5978e7d6b43b5c94556c3936c50467a2a
Sep 16 02:36:36.563093 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:36.563066    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4108a843-75cd969bb9-pprlr"]
Sep 16 02:36:36.564842 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:36:36.564812    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod971d8189_0b6b_4504_bcd3_a4335290edb5.slice/crio-96fe2b4f0f3dbb897c2a7625e7bdd6319e6d26f8d989bea8092425b8bc363d6f WatchSource:0}: Error finding container 96fe2b4f0f3dbb897c2a7625e7bdd6319e6d26f8d989bea8092425b8bc363d6f: Status 404 returned error can't find the container with id 96fe2b4f0f3dbb897c2a7625e7bdd6319e6d26f8d989bea8092425b8bc363d6f
Sep 16 02:36:36.883292 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:36.883184    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-4108a843-6bdfd44f58-qs4r5" event={"ID":"85748850-ba20-401c-8865-1042a8303491","Type":"ContainerStarted","Data":"abc14286b2cdeca5df6a06523206ffa34dd66e48f655e70e28c674edcf12383d"}
Sep 16 02:36:36.883446 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:36.883330    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-4108a843-6bdfd44f58-qs4r5"
Sep 16 02:36:36.883446 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:36.883392    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-4108a843-6bdfd44f58-qs4r5" event={"ID":"85748850-ba20-401c-8865-1042a8303491","Type":"ContainerStarted","Data":"4b383422d358b20d92c03e5d6dcd0ab5978e7d6b43b5c94556c3936c50467a2a"}
Sep 16 02:36:36.884909 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:36.884887    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-4108a843-75cd969bb9-pprlr" event={"ID":"971d8189-0b6b-4504-bcd3-a4335290edb5","Type":"ContainerStarted","Data":"e82548b2bf10bcffe825681c3eab383fda1f2ae4a8c34033af06b8d00dd40292"}
Sep 16 02:36:36.885019 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:36.884928    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-4108a843-75cd969bb9-pprlr" event={"ID":"971d8189-0b6b-4504-bcd3-a4335290edb5","Type":"ContainerStarted","Data":"96fe2b4f0f3dbb897c2a7625e7bdd6319e6d26f8d989bea8092425b8bc363d6f"}
Sep 16 02:36:36.885019 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:36.884969    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-4108a843-75cd969bb9-pprlr"
Sep 16 02:36:36.899371 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:36.899325    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-4108a843-6bdfd44f58-qs4r5" podStartSLOduration=1.899311393 podStartE2EDuration="1.899311393s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 02:36:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:36:36.89792371 +0000 UTC m=+1409.816412962" watchObservedRunningTime="2026-09-16 02:36:36.899311393 +0000 UTC m=+1409.817800642"
Sep 16 02:36:36.913336 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:36.913287    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-4108a843-75cd969bb9-pprlr" podStartSLOduration=1.913272125 podStartE2EDuration="1.913272125s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 02:36:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:36:36.912187837 +0000 UTC m=+1409.830677088" watchObservedRunningTime="2026-09-16 02:36:36.913272125 +0000 UTC m=+1409.831761373"
Sep 16 02:36:40.861637 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:40.861599    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1a323c-h2l24"]
Sep 16 02:36:40.864041 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:40.864020    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1a323c-h2l24"]
Sep 16 02:36:41.550074 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:41.550036    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d7247ce0-45c5-44dc-be9b-0b9fd9c3893a" path="/var/lib/kubelet/pods/d7247ce0-45c5-44dc-be9b-0b9fd9c3893a/volumes"
Sep 16 02:36:47.346718 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:47.346656    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-1a323c-openshift-default-665cc4674b-v4ff5"]
Sep 16 02:36:47.347134 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:47.347027    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-1a323c-openshift-default-665cc4674b-v4ff5" podUID="ebb7007e-e1de-4d70-b0dc-2b5f8de79a7f" containerName="istio-proxy" containerID="cri-o://bac18692cd45e135c7049ed420845e7a0ecfd8de6a82c236e48e568099b9d34e" gracePeriod=30
Sep 16 02:36:47.893899 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:47.893868    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-4108a843-6bdfd44f58-qs4r5"
Sep 16 02:36:47.894114 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:47.894085    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-4108a843-75cd969bb9-pprlr"
Sep 16 02:36:52.603083 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:52.603056    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-1a323c-openshift-default-665cc4674b-v4ff5"
Sep 16 02:36:52.711684 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:52.711645    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebb7007e-e1de-4d70-b0dc-2b5f8de79a7f-credential-socket\") pod \"ebb7007e-e1de-4d70-b0dc-2b5f8de79a7f\" (UID: \"ebb7007e-e1de-4d70-b0dc-2b5f8de79a7f\") "
Sep 16 02:36:52.711857 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:52.711736    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ebb7007e-e1de-4d70-b0dc-2b5f8de79a7f-istio-envoy\") pod \"ebb7007e-e1de-4d70-b0dc-2b5f8de79a7f\" (UID: \"ebb7007e-e1de-4d70-b0dc-2b5f8de79a7f\") "
Sep 16 02:36:52.711857 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:52.711785    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebb7007e-e1de-4d70-b0dc-2b5f8de79a7f-workload-socket\") pod \"ebb7007e-e1de-4d70-b0dc-2b5f8de79a7f\" (UID: \"ebb7007e-e1de-4d70-b0dc-2b5f8de79a7f\") "
Sep 16 02:36:52.711857 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:52.711844    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ebb7007e-e1de-4d70-b0dc-2b5f8de79a7f-istio-token\") pod \"ebb7007e-e1de-4d70-b0dc-2b5f8de79a7f\" (UID: \"ebb7007e-e1de-4d70-b0dc-2b5f8de79a7f\") "
Sep 16 02:36:52.712048 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:52.711900    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ebb7007e-e1de-4d70-b0dc-2b5f8de79a7f-istiod-ca-cert\") pod \"ebb7007e-e1de-4d70-b0dc-2b5f8de79a7f\" (UID: \"ebb7007e-e1de-4d70-b0dc-2b5f8de79a7f\") "
Sep 16 02:36:52.712048 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:52.711927    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ebb7007e-e1de-4d70-b0dc-2b5f8de79a7f-istio-podinfo\") pod \"ebb7007e-e1de-4d70-b0dc-2b5f8de79a7f\" (UID: \"ebb7007e-e1de-4d70-b0dc-2b5f8de79a7f\") "
Sep 16 02:36:52.712048 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:52.711962    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v797g\" (UniqueName: \"kubernetes.io/projected/ebb7007e-e1de-4d70-b0dc-2b5f8de79a7f-kube-api-access-v797g\") pod \"ebb7007e-e1de-4d70-b0dc-2b5f8de79a7f\" (UID: \"ebb7007e-e1de-4d70-b0dc-2b5f8de79a7f\") "
Sep 16 02:36:52.712048 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:52.711984    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebb7007e-e1de-4d70-b0dc-2b5f8de79a7f-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "ebb7007e-e1de-4d70-b0dc-2b5f8de79a7f" (UID: "ebb7007e-e1de-4d70-b0dc-2b5f8de79a7f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:36:52.712048 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:52.712007    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ebb7007e-e1de-4d70-b0dc-2b5f8de79a7f-workload-certs\") pod \"ebb7007e-e1de-4d70-b0dc-2b5f8de79a7f\" (UID: \"ebb7007e-e1de-4d70-b0dc-2b5f8de79a7f\") "
Sep 16 02:36:52.712378 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:52.712092    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ebb7007e-e1de-4d70-b0dc-2b5f8de79a7f-istio-data\") pod \"ebb7007e-e1de-4d70-b0dc-2b5f8de79a7f\" (UID: \"ebb7007e-e1de-4d70-b0dc-2b5f8de79a7f\") "
Sep 16 02:36:52.712378 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:52.712110    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebb7007e-e1de-4d70-b0dc-2b5f8de79a7f-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "ebb7007e-e1de-4d70-b0dc-2b5f8de79a7f" (UID: "ebb7007e-e1de-4d70-b0dc-2b5f8de79a7f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:36:52.712378 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:52.712306    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebb7007e-e1de-4d70-b0dc-2b5f8de79a7f-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "ebb7007e-e1de-4d70-b0dc-2b5f8de79a7f" (UID: "ebb7007e-e1de-4d70-b0dc-2b5f8de79a7f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:36:52.712543 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:52.712411    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebb7007e-e1de-4d70-b0dc-2b5f8de79a7f-istio-data" (OuterVolumeSpecName: "istio-data") pod "ebb7007e-e1de-4d70-b0dc-2b5f8de79a7f" (UID: "ebb7007e-e1de-4d70-b0dc-2b5f8de79a7f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:36:52.712543 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:52.712423    2558 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ebb7007e-e1de-4d70-b0dc-2b5f8de79a7f-workload-certs\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:36:52.712543 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:52.712443    2558 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebb7007e-e1de-4d70-b0dc-2b5f8de79a7f-credential-socket\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:36:52.712543 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:52.712456    2558 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebb7007e-e1de-4d70-b0dc-2b5f8de79a7f-workload-socket\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:36:52.712543 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:52.712499    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ebb7007e-e1de-4d70-b0dc-2b5f8de79a7f-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "ebb7007e-e1de-4d70-b0dc-2b5f8de79a7f" (UID: "ebb7007e-e1de-4d70-b0dc-2b5f8de79a7f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 02:36:52.714056 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:52.714029    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebb7007e-e1de-4d70-b0dc-2b5f8de79a7f-istio-token" (OuterVolumeSpecName: "istio-token") pod "ebb7007e-e1de-4d70-b0dc-2b5f8de79a7f" (UID: "ebb7007e-e1de-4d70-b0dc-2b5f8de79a7f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 02:36:52.714202 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:52.714168    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebb7007e-e1de-4d70-b0dc-2b5f8de79a7f-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "ebb7007e-e1de-4d70-b0dc-2b5f8de79a7f" (UID: "ebb7007e-e1de-4d70-b0dc-2b5f8de79a7f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:36:52.714454 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:52.714431    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ebb7007e-e1de-4d70-b0dc-2b5f8de79a7f-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "ebb7007e-e1de-4d70-b0dc-2b5f8de79a7f" (UID: "ebb7007e-e1de-4d70-b0dc-2b5f8de79a7f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 02:36:52.714546 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:52.714450    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebb7007e-e1de-4d70-b0dc-2b5f8de79a7f-kube-api-access-v797g" (OuterVolumeSpecName: "kube-api-access-v797g") pod "ebb7007e-e1de-4d70-b0dc-2b5f8de79a7f" (UID: "ebb7007e-e1de-4d70-b0dc-2b5f8de79a7f"). InnerVolumeSpecName "kube-api-access-v797g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 02:36:52.813974 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:52.813937    2558 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ebb7007e-e1de-4d70-b0dc-2b5f8de79a7f-istio-data\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:36:52.813974 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:52.813969    2558 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ebb7007e-e1de-4d70-b0dc-2b5f8de79a7f-istio-envoy\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:36:52.813974 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:52.813984    2558 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ebb7007e-e1de-4d70-b0dc-2b5f8de79a7f-istio-token\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:36:52.814238 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:52.813996    2558 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ebb7007e-e1de-4d70-b0dc-2b5f8de79a7f-istiod-ca-cert\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:36:52.814238 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:52.814007    2558 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ebb7007e-e1de-4d70-b0dc-2b5f8de79a7f-istio-podinfo\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:36:52.814238 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:52.814018    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v797g\" (UniqueName: \"kubernetes.io/projected/ebb7007e-e1de-4d70-b0dc-2b5f8de79a7f-kube-api-access-v797g\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:36:52.977344 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:52.977262    2558 generic.go:356] "Generic (PLEG): container finished" podID="ebb7007e-e1de-4d70-b0dc-2b5f8de79a7f" containerID="bac18692cd45e135c7049ed420845e7a0ecfd8de6a82c236e48e568099b9d34e" exitCode=0
Sep 16 02:36:52.977344 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:52.977304    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-1a323c-openshift-default-665cc4674b-v4ff5" event={"ID":"ebb7007e-e1de-4d70-b0dc-2b5f8de79a7f","Type":"ContainerDied","Data":"bac18692cd45e135c7049ed420845e7a0ecfd8de6a82c236e48e568099b9d34e"}
Sep 16 02:36:52.977344 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:52.977317    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-1a323c-openshift-default-665cc4674b-v4ff5"
Sep 16 02:36:52.977344 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:52.977346    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-1a323c-openshift-default-665cc4674b-v4ff5" event={"ID":"ebb7007e-e1de-4d70-b0dc-2b5f8de79a7f","Type":"ContainerDied","Data":"98ad9c83084a63b59ae04fd3fb9ceab274997c6ebdb0fa993d1f5b69a4cf4f58"}
Sep 16 02:36:52.977616 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:52.977368    2558 scope.go:117] "RemoveContainer" containerID="bac18692cd45e135c7049ed420845e7a0ecfd8de6a82c236e48e568099b9d34e"
Sep 16 02:36:52.986939 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:52.986919    2558 scope.go:117] "RemoveContainer" containerID="234c95d72ff6a47852a289fd5955be8627dc4e8921348dfab401d9aed020d744"
Sep 16 02:36:52.995187 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:52.995170    2558 scope.go:117] "RemoveContainer" containerID="bac18692cd45e135c7049ed420845e7a0ecfd8de6a82c236e48e568099b9d34e"
Sep 16 02:36:52.995469 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:36:52.995448    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bac18692cd45e135c7049ed420845e7a0ecfd8de6a82c236e48e568099b9d34e\": container with ID starting with bac18692cd45e135c7049ed420845e7a0ecfd8de6a82c236e48e568099b9d34e not found: ID does not exist" containerID="bac18692cd45e135c7049ed420845e7a0ecfd8de6a82c236e48e568099b9d34e"
Sep 16 02:36:52.995518 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:52.995478    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bac18692cd45e135c7049ed420845e7a0ecfd8de6a82c236e48e568099b9d34e"} err="failed to get container status \"bac18692cd45e135c7049ed420845e7a0ecfd8de6a82c236e48e568099b9d34e\": rpc error: code = NotFound desc = could not find container \"bac18692cd45e135c7049ed420845e7a0ecfd8de6a82c236e48e568099b9d34e\": container with ID starting with bac18692cd45e135c7049ed420845e7a0ecfd8de6a82c236e48e568099b9d34e not found: ID does not exist"
Sep 16 02:36:52.995518 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:52.995492    2558 scope.go:117] "RemoveContainer" containerID="234c95d72ff6a47852a289fd5955be8627dc4e8921348dfab401d9aed020d744"
Sep 16 02:36:52.995749 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:36:52.995728    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"234c95d72ff6a47852a289fd5955be8627dc4e8921348dfab401d9aed020d744\": container with ID starting with 234c95d72ff6a47852a289fd5955be8627dc4e8921348dfab401d9aed020d744 not found: ID does not exist" containerID="234c95d72ff6a47852a289fd5955be8627dc4e8921348dfab401d9aed020d744"
Sep 16 02:36:52.995815 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:52.995758    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"234c95d72ff6a47852a289fd5955be8627dc4e8921348dfab401d9aed020d744"} err="failed to get container status \"234c95d72ff6a47852a289fd5955be8627dc4e8921348dfab401d9aed020d744\": rpc error: code = NotFound desc = could not find container \"234c95d72ff6a47852a289fd5955be8627dc4e8921348dfab401d9aed020d744\": container with ID starting with 234c95d72ff6a47852a289fd5955be8627dc4e8921348dfab401d9aed020d744 not found: ID does not exist"
Sep 16 02:36:53.000067 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:53.000030    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-1a323c-openshift-default-665cc4674b-v4ff5"]
Sep 16 02:36:53.003249 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:53.003227    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-1a323c-openshift-default-665cc4674b-v4ff5"]
Sep 16 02:36:53.550030 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:53.549987    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ebb7007e-e1de-4d70-b0dc-2b5f8de79a7f" path="/var/lib/kubelet/pods/ebb7007e-e1de-4d70-b0dc-2b5f8de79a7f/volumes"
Sep 16 02:36:57.116761 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:57.116723    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-24aabe-7bc4694f6b-k4j76"]
Sep 16 02:36:57.117172 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:57.117126    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ebb7007e-e1de-4d70-b0dc-2b5f8de79a7f" containerName="istio-proxy"
Sep 16 02:36:57.117172 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:57.117141    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="ebb7007e-e1de-4d70-b0dc-2b5f8de79a7f" containerName="istio-proxy"
Sep 16 02:36:57.117172 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:57.117157    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ebb7007e-e1de-4d70-b0dc-2b5f8de79a7f" containerName="istio-proxy"
Sep 16 02:36:57.117172 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:57.117164    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="ebb7007e-e1de-4d70-b0dc-2b5f8de79a7f" containerName="istio-proxy"
Sep 16 02:36:57.117378 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:57.117298    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="ebb7007e-e1de-4d70-b0dc-2b5f8de79a7f" containerName="istio-proxy"
Sep 16 02:36:57.117378 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:57.117314    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="ebb7007e-e1de-4d70-b0dc-2b5f8de79a7f" containerName="istio-proxy"
Sep 16 02:36:57.121430 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:57.121409    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-24aabe-7bc4694f6b-k4j76"
Sep 16 02:36:57.124104 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:57.124080    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-24aabe\""
Sep 16 02:36:57.124220 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:57.124201    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-24aabe-dockercfg-dcwtf\""
Sep 16 02:36:57.137412 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:57.137389    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-24aabe-7bc4694f6b-k4j76"]
Sep 16 02:36:57.156291 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:57.156261    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e7244d4-fbe0-4519-805f-3eb191354cfa-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-24aabe-7bc4694f6b-k4j76\" (UID: \"0e7244d4-fbe0-4519-805f-3eb191354cfa\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-24aabe-7bc4694f6b-k4j76"
Sep 16 02:36:57.156408 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:57.156330    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7j88\" (UniqueName: \"kubernetes.io/projected/0e7244d4-fbe0-4519-805f-3eb191354cfa-kube-api-access-l7j88\") pod \"payload-pre-processing-e2e-shared-a-w2-24aabe-7bc4694f6b-k4j76\" (UID: \"0e7244d4-fbe0-4519-805f-3eb191354cfa\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-24aabe-7bc4694f6b-k4j76"
Sep 16 02:36:57.257727 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:57.257681    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e7244d4-fbe0-4519-805f-3eb191354cfa-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-24aabe-7bc4694f6b-k4j76\" (UID: \"0e7244d4-fbe0-4519-805f-3eb191354cfa\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-24aabe-7bc4694f6b-k4j76"
Sep 16 02:36:57.257899 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:57.257773    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7j88\" (UniqueName: \"kubernetes.io/projected/0e7244d4-fbe0-4519-805f-3eb191354cfa-kube-api-access-l7j88\") pod \"payload-pre-processing-e2e-shared-a-w2-24aabe-7bc4694f6b-k4j76\" (UID: \"0e7244d4-fbe0-4519-805f-3eb191354cfa\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-24aabe-7bc4694f6b-k4j76"
Sep 16 02:36:57.258448 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:57.258424    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e7244d4-fbe0-4519-805f-3eb191354cfa-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-24aabe-7bc4694f6b-k4j76\" (UID: \"0e7244d4-fbe0-4519-805f-3eb191354cfa\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-24aabe-7bc4694f6b-k4j76"
Sep 16 02:36:57.265021 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:57.265001    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7j88\" (UniqueName: \"kubernetes.io/projected/0e7244d4-fbe0-4519-805f-3eb191354cfa-kube-api-access-l7j88\") pod \"payload-pre-processing-e2e-shared-a-w2-24aabe-7bc4694f6b-k4j76\" (UID: \"0e7244d4-fbe0-4519-805f-3eb191354cfa\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-24aabe-7bc4694f6b-k4j76"
Sep 16 02:36:57.431845 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:57.431808    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-24aabe-7bc4694f6b-k4j76"
Sep 16 02:36:57.701182 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:57.701150    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-24aabe-7bc4694f6b-k4j76"]
Sep 16 02:36:57.703663 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:36:57.703625    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e7244d4_fbe0_4519_805f_3eb191354cfa.slice/crio-68190c736cec01421ea48b267c799d096e2fe9c872e49e56642bc6b6163b3f70 WatchSource:0}: Error finding container 68190c736cec01421ea48b267c799d096e2fe9c872e49e56642bc6b6163b3f70: Status 404 returned error can't find the container with id 68190c736cec01421ea48b267c799d096e2fe9c872e49e56642bc6b6163b3f70
Sep 16 02:36:58.006440 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:58.006343    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-24aabe-7bc4694f6b-k4j76" event={"ID":"0e7244d4-fbe0-4519-805f-3eb191354cfa","Type":"ContainerStarted","Data":"93e52de39ec76e9288a404c12d16eb6d4315b31e58285c6fc2b1df4f3d51b271"}
Sep 16 02:36:58.006440 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:58.006401    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-24aabe-7bc4694f6b-k4j76" event={"ID":"0e7244d4-fbe0-4519-805f-3eb191354cfa","Type":"ContainerStarted","Data":"68190c736cec01421ea48b267c799d096e2fe9c872e49e56642bc6b6163b3f70"}
Sep 16 02:36:58.006660 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:58.006520    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-24aabe-7bc4694f6b-k4j76"
Sep 16 02:36:58.020137 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:36:58.020083    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-24aabe-7bc4694f6b-k4j76" podStartSLOduration=1.02006349 podStartE2EDuration="1.02006349s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 02:36:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:36:58.019834183 +0000 UTC m=+1430.938323432" watchObservedRunningTime="2026-09-16 02:36:58.02006349 +0000 UTC m=+1430.938552741"
Sep 16 02:37:04.155538 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:04.154734    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4108a843-75cd969bb9-pprlr"]
Sep 16 02:37:04.155538 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:04.154989    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-4108a843-75cd969bb9-pprlr" podUID="971d8189-0b6b-4504-bcd3-a4335290edb5" containerName="payload-processing" containerID="cri-o://e82548b2bf10bcffe825681c3eab383fda1f2ae4a8c34033af06b8d00dd40292" gracePeriod=30
Sep 16 02:37:04.157267 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:04.157239    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-4108a843-6bdfd44f58-qs4r5"]
Sep 16 02:37:04.157647 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:04.157614    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-4108a843-6bdfd44f58-qs4r5" podUID="85748850-ba20-401c-8865-1042a8303491" containerName="payload-pre-processing" containerID="cri-o://abc14286b2cdeca5df6a06523206ffa34dd66e48f655e70e28c674edcf12383d" gracePeriod=30
Sep 16 02:37:04.489621 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:04.489580    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-4108a843-6bdfd44f58-qs4r5"
Sep 16 02:37:04.493014 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:04.492985    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-4108a843-75cd969bb9-pprlr"
Sep 16 02:37:04.540681 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:04.540640    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85748850-ba20-401c-8865-1042a8303491-plugins-config-volume\") pod \"85748850-ba20-401c-8865-1042a8303491\" (UID: \"85748850-ba20-401c-8865-1042a8303491\") "
Sep 16 02:37:04.540884 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:04.540716    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w6pwg\" (UniqueName: \"kubernetes.io/projected/971d8189-0b6b-4504-bcd3-a4335290edb5-kube-api-access-w6pwg\") pod \"971d8189-0b6b-4504-bcd3-a4335290edb5\" (UID: \"971d8189-0b6b-4504-bcd3-a4335290edb5\") "
Sep 16 02:37:04.540884 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:04.540864    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cqxxd\" (UniqueName: \"kubernetes.io/projected/85748850-ba20-401c-8865-1042a8303491-kube-api-access-cqxxd\") pod \"85748850-ba20-401c-8865-1042a8303491\" (UID: \"85748850-ba20-401c-8865-1042a8303491\") "
Sep 16 02:37:04.541142 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:04.541114    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85748850-ba20-401c-8865-1042a8303491-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "85748850-ba20-401c-8865-1042a8303491" (UID: "85748850-ba20-401c-8865-1042a8303491"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 02:37:04.541316 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:04.541295    2558 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85748850-ba20-401c-8865-1042a8303491-plugins-config-volume\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:37:04.543015 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:04.542990    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/971d8189-0b6b-4504-bcd3-a4335290edb5-kube-api-access-w6pwg" (OuterVolumeSpecName: "kube-api-access-w6pwg") pod "971d8189-0b6b-4504-bcd3-a4335290edb5" (UID: "971d8189-0b6b-4504-bcd3-a4335290edb5"). InnerVolumeSpecName "kube-api-access-w6pwg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 02:37:04.543378 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:04.543356    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85748850-ba20-401c-8865-1042a8303491-kube-api-access-cqxxd" (OuterVolumeSpecName: "kube-api-access-cqxxd") pod "85748850-ba20-401c-8865-1042a8303491" (UID: "85748850-ba20-401c-8865-1042a8303491"). InnerVolumeSpecName "kube-api-access-cqxxd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 02:37:04.642240 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:04.642201    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/971d8189-0b6b-4504-bcd3-a4335290edb5-plugins-config-volume\") pod \"971d8189-0b6b-4504-bcd3-a4335290edb5\" (UID: \"971d8189-0b6b-4504-bcd3-a4335290edb5\") "
Sep 16 02:37:04.642465 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:04.642435    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cqxxd\" (UniqueName: \"kubernetes.io/projected/85748850-ba20-401c-8865-1042a8303491-kube-api-access-cqxxd\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:37:04.642465 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:04.642448    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w6pwg\" (UniqueName: \"kubernetes.io/projected/971d8189-0b6b-4504-bcd3-a4335290edb5-kube-api-access-w6pwg\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:37:04.642572 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:04.642547    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/971d8189-0b6b-4504-bcd3-a4335290edb5-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "971d8189-0b6b-4504-bcd3-a4335290edb5" (UID: "971d8189-0b6b-4504-bcd3-a4335290edb5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 02:37:04.743903 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:04.743810    2558 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/971d8189-0b6b-4504-bcd3-a4335290edb5-plugins-config-volume\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:37:05.045722 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:05.045623    2558 generic.go:356] "Generic (PLEG): container finished" podID="971d8189-0b6b-4504-bcd3-a4335290edb5" containerID="e82548b2bf10bcffe825681c3eab383fda1f2ae4a8c34033af06b8d00dd40292" exitCode=0
Sep 16 02:37:05.045896 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:05.045746    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-4108a843-75cd969bb9-pprlr"
Sep 16 02:37:05.045896 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:05.045762    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-4108a843-75cd969bb9-pprlr" event={"ID":"971d8189-0b6b-4504-bcd3-a4335290edb5","Type":"ContainerDied","Data":"e82548b2bf10bcffe825681c3eab383fda1f2ae4a8c34033af06b8d00dd40292"}
Sep 16 02:37:05.045896 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:05.045823    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-4108a843-75cd969bb9-pprlr" event={"ID":"971d8189-0b6b-4504-bcd3-a4335290edb5","Type":"ContainerDied","Data":"96fe2b4f0f3dbb897c2a7625e7bdd6319e6d26f8d989bea8092425b8bc363d6f"}
Sep 16 02:37:05.045896 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:05.045853    2558 scope.go:117] "RemoveContainer" containerID="e82548b2bf10bcffe825681c3eab383fda1f2ae4a8c34033af06b8d00dd40292"
Sep 16 02:37:05.047258 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:05.047234    2558 generic.go:356] "Generic (PLEG): container finished" podID="85748850-ba20-401c-8865-1042a8303491" containerID="abc14286b2cdeca5df6a06523206ffa34dd66e48f655e70e28c674edcf12383d" exitCode=0
Sep 16 02:37:05.047370 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:05.047279    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-4108a843-6bdfd44f58-qs4r5"
Sep 16 02:37:05.047370 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:05.047328    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-4108a843-6bdfd44f58-qs4r5" event={"ID":"85748850-ba20-401c-8865-1042a8303491","Type":"ContainerDied","Data":"abc14286b2cdeca5df6a06523206ffa34dd66e48f655e70e28c674edcf12383d"}
Sep 16 02:37:05.047479 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:05.047395    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-4108a843-6bdfd44f58-qs4r5" event={"ID":"85748850-ba20-401c-8865-1042a8303491","Type":"ContainerDied","Data":"4b383422d358b20d92c03e5d6dcd0ab5978e7d6b43b5c94556c3936c50467a2a"}
Sep 16 02:37:05.057079 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:05.057036    2558 scope.go:117] "RemoveContainer" containerID="e82548b2bf10bcffe825681c3eab383fda1f2ae4a8c34033af06b8d00dd40292"
Sep 16 02:37:05.057576 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:37:05.057547    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e82548b2bf10bcffe825681c3eab383fda1f2ae4a8c34033af06b8d00dd40292\": container with ID starting with e82548b2bf10bcffe825681c3eab383fda1f2ae4a8c34033af06b8d00dd40292 not found: ID does not exist" containerID="e82548b2bf10bcffe825681c3eab383fda1f2ae4a8c34033af06b8d00dd40292"
Sep 16 02:37:05.057646 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:05.057590    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e82548b2bf10bcffe825681c3eab383fda1f2ae4a8c34033af06b8d00dd40292"} err="failed to get container status \"e82548b2bf10bcffe825681c3eab383fda1f2ae4a8c34033af06b8d00dd40292\": rpc error: code = NotFound desc = could not find container \"e82548b2bf10bcffe825681c3eab383fda1f2ae4a8c34033af06b8d00dd40292\": container with ID starting with e82548b2bf10bcffe825681c3eab383fda1f2ae4a8c34033af06b8d00dd40292 not found: ID does not exist"
Sep 16 02:37:05.057646 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:05.057615    2558 scope.go:117] "RemoveContainer" containerID="abc14286b2cdeca5df6a06523206ffa34dd66e48f655e70e28c674edcf12383d"
Sep 16 02:37:05.068397 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:05.068370    2558 scope.go:117] "RemoveContainer" containerID="abc14286b2cdeca5df6a06523206ffa34dd66e48f655e70e28c674edcf12383d"
Sep 16 02:37:05.068769 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:05.068749    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4108a843-75cd969bb9-pprlr"]
Sep 16 02:37:05.068769 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:37:05.068757    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"abc14286b2cdeca5df6a06523206ffa34dd66e48f655e70e28c674edcf12383d\": container with ID starting with abc14286b2cdeca5df6a06523206ffa34dd66e48f655e70e28c674edcf12383d not found: ID does not exist" containerID="abc14286b2cdeca5df6a06523206ffa34dd66e48f655e70e28c674edcf12383d"
Sep 16 02:37:05.068919 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:05.068784    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"abc14286b2cdeca5df6a06523206ffa34dd66e48f655e70e28c674edcf12383d"} err="failed to get container status \"abc14286b2cdeca5df6a06523206ffa34dd66e48f655e70e28c674edcf12383d\": rpc error: code = NotFound desc = could not find container \"abc14286b2cdeca5df6a06523206ffa34dd66e48f655e70e28c674edcf12383d\": container with ID starting with abc14286b2cdeca5df6a06523206ffa34dd66e48f655e70e28c674edcf12383d not found: ID does not exist"
Sep 16 02:37:05.072096 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:05.072068    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4108a843-75cd969bb9-pprlr"]
Sep 16 02:37:05.081308 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:05.081277    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-4108a843-6bdfd44f58-qs4r5"]
Sep 16 02:37:05.084875 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:05.084845    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-4108a843-6bdfd44f58-qs4r5"]
Sep 16 02:37:05.375780 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:05.375683    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-4108a843-openshift-default-7dd58f44f4-q5j6n"]
Sep 16 02:37:05.376175 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:05.375952    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-4108a843-openshift-default-7dd58f44f4-q5j6n" podUID="62dc3fab-3a25-4d2a-bfb1-0b951833ef2d" containerName="istio-proxy" containerID="cri-o://dad703198e12fcc83a6786f16b9260cddec9b5d6874d435139156e7b815b0d43" gracePeriod=30
Sep 16 02:37:05.550979 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:05.550945    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="85748850-ba20-401c-8865-1042a8303491" path="/var/lib/kubelet/pods/85748850-ba20-401c-8865-1042a8303491/volumes"
Sep 16 02:37:05.551363 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:05.551346    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="971d8189-0b6b-4504-bcd3-a4335290edb5" path="/var/lib/kubelet/pods/971d8189-0b6b-4504-bcd3-a4335290edb5/volumes"
Sep 16 02:37:06.058397 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:06.058365    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-2e69f8df-openshift-default-6b45f469f6-v2kbk"]
Sep 16 02:37:06.059060 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:06.059039    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="85748850-ba20-401c-8865-1042a8303491" containerName="payload-pre-processing"
Sep 16 02:37:06.059109 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:06.059065    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="85748850-ba20-401c-8865-1042a8303491" containerName="payload-pre-processing"
Sep 16 02:37:06.059109 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:06.059088    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="971d8189-0b6b-4504-bcd3-a4335290edb5" containerName="payload-processing"
Sep 16 02:37:06.059109 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:06.059097    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="971d8189-0b6b-4504-bcd3-a4335290edb5" containerName="payload-processing"
Sep 16 02:37:06.059262 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:06.059249    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="85748850-ba20-401c-8865-1042a8303491" containerName="payload-pre-processing"
Sep 16 02:37:06.059298 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:06.059268    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="971d8189-0b6b-4504-bcd3-a4335290edb5" containerName="payload-processing"
Sep 16 02:37:06.064317 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:06.064292    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-2e69f8df-openshift-default-6b45f469f6-v2kbk"
Sep 16 02:37:06.067486 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:06.067229    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-2e69f8df-openshift-default-dockercfg-js6zg\""
Sep 16 02:37:06.070365 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:06.070342    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-2e69f8df-openshift-default-6b45f469f6-v2kbk"]
Sep 16 02:37:06.156863 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:06.156134    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a9173c4-9b43-4203-8287-1b34dfd261b1-workload-certs\") pod \"e2e-derive-2e69f8df-openshift-default-6b45f469f6-v2kbk\" (UID: \"5a9173c4-9b43-4203-8287-1b34dfd261b1\") " pod="openshift-ingress/e2e-derive-2e69f8df-openshift-default-6b45f469f6-v2kbk"
Sep 16 02:37:06.156863 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:06.156238    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a9173c4-9b43-4203-8287-1b34dfd261b1-istio-envoy\") pod \"e2e-derive-2e69f8df-openshift-default-6b45f469f6-v2kbk\" (UID: \"5a9173c4-9b43-4203-8287-1b34dfd261b1\") " pod="openshift-ingress/e2e-derive-2e69f8df-openshift-default-6b45f469f6-v2kbk"
Sep 16 02:37:06.156863 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:06.156267    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a9173c4-9b43-4203-8287-1b34dfd261b1-istio-podinfo\") pod \"e2e-derive-2e69f8df-openshift-default-6b45f469f6-v2kbk\" (UID: \"5a9173c4-9b43-4203-8287-1b34dfd261b1\") " pod="openshift-ingress/e2e-derive-2e69f8df-openshift-default-6b45f469f6-v2kbk"
Sep 16 02:37:06.156863 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:06.156307    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a9173c4-9b43-4203-8287-1b34dfd261b1-istio-data\") pod \"e2e-derive-2e69f8df-openshift-default-6b45f469f6-v2kbk\" (UID: \"5a9173c4-9b43-4203-8287-1b34dfd261b1\") " pod="openshift-ingress/e2e-derive-2e69f8df-openshift-default-6b45f469f6-v2kbk"
Sep 16 02:37:06.156863 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:06.156333    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a9173c4-9b43-4203-8287-1b34dfd261b1-credential-socket\") pod \"e2e-derive-2e69f8df-openshift-default-6b45f469f6-v2kbk\" (UID: \"5a9173c4-9b43-4203-8287-1b34dfd261b1\") " pod="openshift-ingress/e2e-derive-2e69f8df-openshift-default-6b45f469f6-v2kbk"
Sep 16 02:37:06.156863 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:06.156395    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a9173c4-9b43-4203-8287-1b34dfd261b1-workload-socket\") pod \"e2e-derive-2e69f8df-openshift-default-6b45f469f6-v2kbk\" (UID: \"5a9173c4-9b43-4203-8287-1b34dfd261b1\") " pod="openshift-ingress/e2e-derive-2e69f8df-openshift-default-6b45f469f6-v2kbk"
Sep 16 02:37:06.156863 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:06.156434    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a9173c4-9b43-4203-8287-1b34dfd261b1-istio-token\") pod \"e2e-derive-2e69f8df-openshift-default-6b45f469f6-v2kbk\" (UID: \"5a9173c4-9b43-4203-8287-1b34dfd261b1\") " pod="openshift-ingress/e2e-derive-2e69f8df-openshift-default-6b45f469f6-v2kbk"
Sep 16 02:37:06.156863 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:06.156481    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jstg\" (UniqueName: \"kubernetes.io/projected/5a9173c4-9b43-4203-8287-1b34dfd261b1-kube-api-access-4jstg\") pod \"e2e-derive-2e69f8df-openshift-default-6b45f469f6-v2kbk\" (UID: \"5a9173c4-9b43-4203-8287-1b34dfd261b1\") " pod="openshift-ingress/e2e-derive-2e69f8df-openshift-default-6b45f469f6-v2kbk"
Sep 16 02:37:06.156863 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:06.156511    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a9173c4-9b43-4203-8287-1b34dfd261b1-istiod-ca-cert\") pod \"e2e-derive-2e69f8df-openshift-default-6b45f469f6-v2kbk\" (UID: \"5a9173c4-9b43-4203-8287-1b34dfd261b1\") " pod="openshift-ingress/e2e-derive-2e69f8df-openshift-default-6b45f469f6-v2kbk"
Sep 16 02:37:06.257198 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:06.257150    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jstg\" (UniqueName: \"kubernetes.io/projected/5a9173c4-9b43-4203-8287-1b34dfd261b1-kube-api-access-4jstg\") pod \"e2e-derive-2e69f8df-openshift-default-6b45f469f6-v2kbk\" (UID: \"5a9173c4-9b43-4203-8287-1b34dfd261b1\") " pod="openshift-ingress/e2e-derive-2e69f8df-openshift-default-6b45f469f6-v2kbk"
Sep 16 02:37:06.257198 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:06.257200    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a9173c4-9b43-4203-8287-1b34dfd261b1-istiod-ca-cert\") pod \"e2e-derive-2e69f8df-openshift-default-6b45f469f6-v2kbk\" (UID: \"5a9173c4-9b43-4203-8287-1b34dfd261b1\") " pod="openshift-ingress/e2e-derive-2e69f8df-openshift-default-6b45f469f6-v2kbk"
Sep 16 02:37:06.257437 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:06.257233    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a9173c4-9b43-4203-8287-1b34dfd261b1-workload-certs\") pod \"e2e-derive-2e69f8df-openshift-default-6b45f469f6-v2kbk\" (UID: \"5a9173c4-9b43-4203-8287-1b34dfd261b1\") " pod="openshift-ingress/e2e-derive-2e69f8df-openshift-default-6b45f469f6-v2kbk"
Sep 16 02:37:06.257437 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:06.257286    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a9173c4-9b43-4203-8287-1b34dfd261b1-istio-envoy\") pod \"e2e-derive-2e69f8df-openshift-default-6b45f469f6-v2kbk\" (UID: \"5a9173c4-9b43-4203-8287-1b34dfd261b1\") " pod="openshift-ingress/e2e-derive-2e69f8df-openshift-default-6b45f469f6-v2kbk"
Sep 16 02:37:06.257437 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:06.257303    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a9173c4-9b43-4203-8287-1b34dfd261b1-istio-podinfo\") pod \"e2e-derive-2e69f8df-openshift-default-6b45f469f6-v2kbk\" (UID: \"5a9173c4-9b43-4203-8287-1b34dfd261b1\") " pod="openshift-ingress/e2e-derive-2e69f8df-openshift-default-6b45f469f6-v2kbk"
Sep 16 02:37:06.257437 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:06.257328    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a9173c4-9b43-4203-8287-1b34dfd261b1-istio-data\") pod \"e2e-derive-2e69f8df-openshift-default-6b45f469f6-v2kbk\" (UID: \"5a9173c4-9b43-4203-8287-1b34dfd261b1\") " pod="openshift-ingress/e2e-derive-2e69f8df-openshift-default-6b45f469f6-v2kbk"
Sep 16 02:37:06.257437 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:06.257351    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a9173c4-9b43-4203-8287-1b34dfd261b1-credential-socket\") pod \"e2e-derive-2e69f8df-openshift-default-6b45f469f6-v2kbk\" (UID: \"5a9173c4-9b43-4203-8287-1b34dfd261b1\") " pod="openshift-ingress/e2e-derive-2e69f8df-openshift-default-6b45f469f6-v2kbk"
Sep 16 02:37:06.257437 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:06.257410    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a9173c4-9b43-4203-8287-1b34dfd261b1-workload-socket\") pod \"e2e-derive-2e69f8df-openshift-default-6b45f469f6-v2kbk\" (UID: \"5a9173c4-9b43-4203-8287-1b34dfd261b1\") " pod="openshift-ingress/e2e-derive-2e69f8df-openshift-default-6b45f469f6-v2kbk"
Sep 16 02:37:06.257768 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:06.257451    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a9173c4-9b43-4203-8287-1b34dfd261b1-istio-token\") pod \"e2e-derive-2e69f8df-openshift-default-6b45f469f6-v2kbk\" (UID: \"5a9173c4-9b43-4203-8287-1b34dfd261b1\") " pod="openshift-ingress/e2e-derive-2e69f8df-openshift-default-6b45f469f6-v2kbk"
Sep 16 02:37:06.257768 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:06.257758    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a9173c4-9b43-4203-8287-1b34dfd261b1-workload-certs\") pod \"e2e-derive-2e69f8df-openshift-default-6b45f469f6-v2kbk\" (UID: \"5a9173c4-9b43-4203-8287-1b34dfd261b1\") " pod="openshift-ingress/e2e-derive-2e69f8df-openshift-default-6b45f469f6-v2kbk"
Sep 16 02:37:06.257948 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:06.257818    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a9173c4-9b43-4203-8287-1b34dfd261b1-credential-socket\") pod \"e2e-derive-2e69f8df-openshift-default-6b45f469f6-v2kbk\" (UID: \"5a9173c4-9b43-4203-8287-1b34dfd261b1\") " pod="openshift-ingress/e2e-derive-2e69f8df-openshift-default-6b45f469f6-v2kbk"
Sep 16 02:37:06.258037 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:06.258011    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a9173c4-9b43-4203-8287-1b34dfd261b1-workload-socket\") pod \"e2e-derive-2e69f8df-openshift-default-6b45f469f6-v2kbk\" (UID: \"5a9173c4-9b43-4203-8287-1b34dfd261b1\") " pod="openshift-ingress/e2e-derive-2e69f8df-openshift-default-6b45f469f6-v2kbk"
Sep 16 02:37:06.258104 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:06.258060    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a9173c4-9b43-4203-8287-1b34dfd261b1-istio-data\") pod \"e2e-derive-2e69f8df-openshift-default-6b45f469f6-v2kbk\" (UID: \"5a9173c4-9b43-4203-8287-1b34dfd261b1\") " pod="openshift-ingress/e2e-derive-2e69f8df-openshift-default-6b45f469f6-v2kbk"
Sep 16 02:37:06.258104 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:06.258060    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a9173c4-9b43-4203-8287-1b34dfd261b1-istiod-ca-cert\") pod \"e2e-derive-2e69f8df-openshift-default-6b45f469f6-v2kbk\" (UID: \"5a9173c4-9b43-4203-8287-1b34dfd261b1\") " pod="openshift-ingress/e2e-derive-2e69f8df-openshift-default-6b45f469f6-v2kbk"
Sep 16 02:37:06.259997 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:06.259973    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a9173c4-9b43-4203-8287-1b34dfd261b1-istio-podinfo\") pod \"e2e-derive-2e69f8df-openshift-default-6b45f469f6-v2kbk\" (UID: \"5a9173c4-9b43-4203-8287-1b34dfd261b1\") " pod="openshift-ingress/e2e-derive-2e69f8df-openshift-default-6b45f469f6-v2kbk"
Sep 16 02:37:06.260145 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:06.260128    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a9173c4-9b43-4203-8287-1b34dfd261b1-istio-envoy\") pod \"e2e-derive-2e69f8df-openshift-default-6b45f469f6-v2kbk\" (UID: \"5a9173c4-9b43-4203-8287-1b34dfd261b1\") " pod="openshift-ingress/e2e-derive-2e69f8df-openshift-default-6b45f469f6-v2kbk"
Sep 16 02:37:06.264401 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:06.264373    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a9173c4-9b43-4203-8287-1b34dfd261b1-istio-token\") pod \"e2e-derive-2e69f8df-openshift-default-6b45f469f6-v2kbk\" (UID: \"5a9173c4-9b43-4203-8287-1b34dfd261b1\") " pod="openshift-ingress/e2e-derive-2e69f8df-openshift-default-6b45f469f6-v2kbk"
Sep 16 02:37:06.265289 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:06.265265    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jstg\" (UniqueName: \"kubernetes.io/projected/5a9173c4-9b43-4203-8287-1b34dfd261b1-kube-api-access-4jstg\") pod \"e2e-derive-2e69f8df-openshift-default-6b45f469f6-v2kbk\" (UID: \"5a9173c4-9b43-4203-8287-1b34dfd261b1\") " pod="openshift-ingress/e2e-derive-2e69f8df-openshift-default-6b45f469f6-v2kbk"
Sep 16 02:37:06.376830 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:06.376750    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-2e69f8df-openshift-default-6b45f469f6-v2kbk"
Sep 16 02:37:06.660848 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:06.660030    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-2e69f8df-openshift-default-6b45f469f6-v2kbk"]
Sep 16 02:37:06.661925 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:37:06.661898    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a9173c4_9b43_4203_8287_1b34dfd261b1.slice/crio-24c825891f0fdec70a30fa3c580fbbf886bb3dfa49547b96c3615cd70c1724b0 WatchSource:0}: Error finding container 24c825891f0fdec70a30fa3c580fbbf886bb3dfa49547b96c3615cd70c1724b0: Status 404 returned error can't find the container with id 24c825891f0fdec70a30fa3c580fbbf886bb3dfa49547b96c3615cd70c1724b0
Sep 16 02:37:06.664231 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:06.664191    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 16 02:37:06.664328 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:06.664258    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 16 02:37:06.664328 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:06.664287    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 16 02:37:06.918052 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:06.918024    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-4108a843-openshift-default-7dd58f44f4-q5j6n"
Sep 16 02:37:07.065349 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:07.065320    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62dc3fab-3a25-4d2a-bfb1-0b951833ef2d-istio-podinfo\") pod \"62dc3fab-3a25-4d2a-bfb1-0b951833ef2d\" (UID: \"62dc3fab-3a25-4d2a-bfb1-0b951833ef2d\") "
Sep 16 02:37:07.065546 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:07.065418    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62dc3fab-3a25-4d2a-bfb1-0b951833ef2d-istiod-ca-cert\") pod \"62dc3fab-3a25-4d2a-bfb1-0b951833ef2d\" (UID: \"62dc3fab-3a25-4d2a-bfb1-0b951833ef2d\") "
Sep 16 02:37:07.065546 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:07.065460    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62dc3fab-3a25-4d2a-bfb1-0b951833ef2d-credential-socket\") pod \"62dc3fab-3a25-4d2a-bfb1-0b951833ef2d\" (UID: \"62dc3fab-3a25-4d2a-bfb1-0b951833ef2d\") "
Sep 16 02:37:07.065546 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:07.065500    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62dc3fab-3a25-4d2a-bfb1-0b951833ef2d-istio-token\") pod \"62dc3fab-3a25-4d2a-bfb1-0b951833ef2d\" (UID: \"62dc3fab-3a25-4d2a-bfb1-0b951833ef2d\") "
Sep 16 02:37:07.065546 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:07.065524    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62dc3fab-3a25-4d2a-bfb1-0b951833ef2d-istio-data\") pod \"62dc3fab-3a25-4d2a-bfb1-0b951833ef2d\" (UID: \"62dc3fab-3a25-4d2a-bfb1-0b951833ef2d\") "
Sep 16 02:37:07.065791 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:07.065547    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62dc3fab-3a25-4d2a-bfb1-0b951833ef2d-workload-certs\") pod \"62dc3fab-3a25-4d2a-bfb1-0b951833ef2d\" (UID: \"62dc3fab-3a25-4d2a-bfb1-0b951833ef2d\") "
Sep 16 02:37:07.065791 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:07.065575    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62dc3fab-3a25-4d2a-bfb1-0b951833ef2d-workload-socket\") pod \"62dc3fab-3a25-4d2a-bfb1-0b951833ef2d\" (UID: \"62dc3fab-3a25-4d2a-bfb1-0b951833ef2d\") "
Sep 16 02:37:07.065791 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:07.065608    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cvlbp\" (UniqueName: \"kubernetes.io/projected/62dc3fab-3a25-4d2a-bfb1-0b951833ef2d-kube-api-access-cvlbp\") pod \"62dc3fab-3a25-4d2a-bfb1-0b951833ef2d\" (UID: \"62dc3fab-3a25-4d2a-bfb1-0b951833ef2d\") "
Sep 16 02:37:07.065791 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:07.065645    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62dc3fab-3a25-4d2a-bfb1-0b951833ef2d-istio-envoy\") pod \"62dc3fab-3a25-4d2a-bfb1-0b951833ef2d\" (UID: \"62dc3fab-3a25-4d2a-bfb1-0b951833ef2d\") "
Sep 16 02:37:07.066004 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:07.065802    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62dc3fab-3a25-4d2a-bfb1-0b951833ef2d-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "62dc3fab-3a25-4d2a-bfb1-0b951833ef2d" (UID: "62dc3fab-3a25-4d2a-bfb1-0b951833ef2d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:37:07.066004 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:07.065829    2558 generic.go:356] "Generic (PLEG): container finished" podID="62dc3fab-3a25-4d2a-bfb1-0b951833ef2d" containerID="dad703198e12fcc83a6786f16b9260cddec9b5d6874d435139156e7b815b0d43" exitCode=0
Sep 16 02:37:07.066004 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:07.065983    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62dc3fab-3a25-4d2a-bfb1-0b951833ef2d-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "62dc3fab-3a25-4d2a-bfb1-0b951833ef2d" (UID: "62dc3fab-3a25-4d2a-bfb1-0b951833ef2d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 02:37:07.066183 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:07.066035    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-4108a843-openshift-default-7dd58f44f4-q5j6n" event={"ID":"62dc3fab-3a25-4d2a-bfb1-0b951833ef2d","Type":"ContainerDied","Data":"dad703198e12fcc83a6786f16b9260cddec9b5d6874d435139156e7b815b0d43"}
Sep 16 02:37:07.066183 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:07.066041    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-4108a843-openshift-default-7dd58f44f4-q5j6n"
Sep 16 02:37:07.066183 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:07.066086    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-4108a843-openshift-default-7dd58f44f4-q5j6n" event={"ID":"62dc3fab-3a25-4d2a-bfb1-0b951833ef2d","Type":"ContainerDied","Data":"d524b12c9a6d4c2c19420556a1fbd8e07c81cd6dc434d8a3b52502e413330b9c"}
Sep 16 02:37:07.066183 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:07.066116    2558 scope.go:117] "RemoveContainer" containerID="dad703198e12fcc83a6786f16b9260cddec9b5d6874d435139156e7b815b0d43"
Sep 16 02:37:07.066183 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:07.066134    2558 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62dc3fab-3a25-4d2a-bfb1-0b951833ef2d-istiod-ca-cert\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:37:07.066183 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:07.066151    2558 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62dc3fab-3a25-4d2a-bfb1-0b951833ef2d-credential-socket\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:37:07.066490 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:07.066180    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62dc3fab-3a25-4d2a-bfb1-0b951833ef2d-istio-data" (OuterVolumeSpecName: "istio-data") pod "62dc3fab-3a25-4d2a-bfb1-0b951833ef2d" (UID: "62dc3fab-3a25-4d2a-bfb1-0b951833ef2d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:37:07.066490 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:07.066369    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62dc3fab-3a25-4d2a-bfb1-0b951833ef2d-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "62dc3fab-3a25-4d2a-bfb1-0b951833ef2d" (UID: "62dc3fab-3a25-4d2a-bfb1-0b951833ef2d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:37:07.066490 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:07.066444    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62dc3fab-3a25-4d2a-bfb1-0b951833ef2d-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "62dc3fab-3a25-4d2a-bfb1-0b951833ef2d" (UID: "62dc3fab-3a25-4d2a-bfb1-0b951833ef2d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:37:07.068080 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:07.068054    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62dc3fab-3a25-4d2a-bfb1-0b951833ef2d-istio-token" (OuterVolumeSpecName: "istio-token") pod "62dc3fab-3a25-4d2a-bfb1-0b951833ef2d" (UID: "62dc3fab-3a25-4d2a-bfb1-0b951833ef2d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 02:37:07.068281 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:07.068228    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62dc3fab-3a25-4d2a-bfb1-0b951833ef2d-kube-api-access-cvlbp" (OuterVolumeSpecName: "kube-api-access-cvlbp") pod "62dc3fab-3a25-4d2a-bfb1-0b951833ef2d" (UID: "62dc3fab-3a25-4d2a-bfb1-0b951833ef2d"). InnerVolumeSpecName "kube-api-access-cvlbp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 02:37:07.068475 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:07.068447    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/62dc3fab-3a25-4d2a-bfb1-0b951833ef2d-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "62dc3fab-3a25-4d2a-bfb1-0b951833ef2d" (UID: "62dc3fab-3a25-4d2a-bfb1-0b951833ef2d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 02:37:07.068587 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:07.068510    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62dc3fab-3a25-4d2a-bfb1-0b951833ef2d-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "62dc3fab-3a25-4d2a-bfb1-0b951833ef2d" (UID: "62dc3fab-3a25-4d2a-bfb1-0b951833ef2d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:37:07.069211 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:07.069189    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-2e69f8df-openshift-default-6b45f469f6-v2kbk" event={"ID":"5a9173c4-9b43-4203-8287-1b34dfd261b1","Type":"ContainerStarted","Data":"324046f3600b748f7c3359c523e456400a772984af0c2afcaa162ffb9e3e84e8"}
Sep 16 02:37:07.069299 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:07.069236    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-2e69f8df-openshift-default-6b45f469f6-v2kbk" event={"ID":"5a9173c4-9b43-4203-8287-1b34dfd261b1","Type":"ContainerStarted","Data":"24c825891f0fdec70a30fa3c580fbbf886bb3dfa49547b96c3615cd70c1724b0"}
Sep 16 02:37:07.076160 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:07.076142    2558 scope.go:117] "RemoveContainer" containerID="dad703198e12fcc83a6786f16b9260cddec9b5d6874d435139156e7b815b0d43"
Sep 16 02:37:07.076426 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:37:07.076405    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dad703198e12fcc83a6786f16b9260cddec9b5d6874d435139156e7b815b0d43\": container with ID starting with dad703198e12fcc83a6786f16b9260cddec9b5d6874d435139156e7b815b0d43 not found: ID does not exist" containerID="dad703198e12fcc83a6786f16b9260cddec9b5d6874d435139156e7b815b0d43"
Sep 16 02:37:07.076507 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:07.076431    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dad703198e12fcc83a6786f16b9260cddec9b5d6874d435139156e7b815b0d43"} err="failed to get container status \"dad703198e12fcc83a6786f16b9260cddec9b5d6874d435139156e7b815b0d43\": rpc error: code = NotFound desc = could not find container \"dad703198e12fcc83a6786f16b9260cddec9b5d6874d435139156e7b815b0d43\": container with ID starting with dad703198e12fcc83a6786f16b9260cddec9b5d6874d435139156e7b815b0d43 not found: ID does not exist"
Sep 16 02:37:07.088899 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:07.088850    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-2e69f8df-openshift-default-6b45f469f6-v2kbk" podStartSLOduration=1.0888374299999999 podStartE2EDuration="1.08883743s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 02:37:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:37:07.08721161 +0000 UTC m=+1440.005700885" watchObservedRunningTime="2026-09-16 02:37:07.08883743 +0000 UTC m=+1440.007326679"
Sep 16 02:37:07.167169 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:07.167077    2558 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62dc3fab-3a25-4d2a-bfb1-0b951833ef2d-istio-token\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:37:07.167169 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:07.167112    2558 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62dc3fab-3a25-4d2a-bfb1-0b951833ef2d-istio-data\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:37:07.167169 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:07.167126    2558 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62dc3fab-3a25-4d2a-bfb1-0b951833ef2d-workload-certs\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:37:07.167169 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:07.167140    2558 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62dc3fab-3a25-4d2a-bfb1-0b951833ef2d-workload-socket\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:37:07.167169 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:07.167155    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cvlbp\" (UniqueName: \"kubernetes.io/projected/62dc3fab-3a25-4d2a-bfb1-0b951833ef2d-kube-api-access-cvlbp\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:37:07.167169 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:07.167170    2558 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62dc3fab-3a25-4d2a-bfb1-0b951833ef2d-istio-envoy\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:37:07.167521 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:07.167184    2558 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62dc3fab-3a25-4d2a-bfb1-0b951833ef2d-istio-podinfo\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:37:07.377080 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:07.377053    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-2e69f8df-openshift-default-6b45f469f6-v2kbk"
Sep 16 02:37:07.382234 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:07.382210    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-2e69f8df-openshift-default-6b45f469f6-v2kbk"
Sep 16 02:37:07.387641 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:07.387617    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-4108a843-openshift-default-7dd58f44f4-q5j6n"]
Sep 16 02:37:07.392505 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:07.392484    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-4108a843-openshift-default-7dd58f44f4-q5j6n"]
Sep 16 02:37:07.550840 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:07.550750    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="62dc3fab-3a25-4d2a-bfb1-0b951833ef2d" path="/var/lib/kubelet/pods/62dc3fab-3a25-4d2a-bfb1-0b951833ef2d/volumes"
Sep 16 02:37:08.074684 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:08.074655    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-2e69f8df-openshift-default-6b45f469f6-v2kbk"
Sep 16 02:37:08.076228 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:08.076209    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-2e69f8df-openshift-default-6b45f469f6-v2kbk"
Sep 16 02:37:09.014227 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:09.014195    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-24aabe-7bc4694f6b-k4j76"
Sep 16 02:37:09.550018 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:09.549984    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-a53a44-openshift-default-75f7864f96-gh2ct"]
Sep 16 02:37:09.550440 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:09.550423    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62dc3fab-3a25-4d2a-bfb1-0b951833ef2d" containerName="istio-proxy"
Sep 16 02:37:09.550538 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:09.550441    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="62dc3fab-3a25-4d2a-bfb1-0b951833ef2d" containerName="istio-proxy"
Sep 16 02:37:09.550538 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:09.550526    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="62dc3fab-3a25-4d2a-bfb1-0b951833ef2d" containerName="istio-proxy"
Sep 16 02:37:09.554932 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:09.554909    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-a53a44-openshift-default-75f7864f96-gh2ct"
Sep 16 02:37:09.557150 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:09.557090    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-a53a44-openshift-default-dockercfg-lq8n8\""
Sep 16 02:37:09.560436 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:09.560413    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-a53a44-openshift-default-75f7864f96-gh2ct"]
Sep 16 02:37:09.694375 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:09.694337    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e8ae5f7-3d2a-4725-81c3-a2ae2fdaaaa2-workload-socket\") pod \"e2e-shared-b-w2-a53a44-openshift-default-75f7864f96-gh2ct\" (UID: \"1e8ae5f7-3d2a-4725-81c3-a2ae2fdaaaa2\") " pod="openshift-ingress/e2e-shared-b-w2-a53a44-openshift-default-75f7864f96-gh2ct"
Sep 16 02:37:09.694566 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:09.694417    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e8ae5f7-3d2a-4725-81c3-a2ae2fdaaaa2-istio-envoy\") pod \"e2e-shared-b-w2-a53a44-openshift-default-75f7864f96-gh2ct\" (UID: \"1e8ae5f7-3d2a-4725-81c3-a2ae2fdaaaa2\") " pod="openshift-ingress/e2e-shared-b-w2-a53a44-openshift-default-75f7864f96-gh2ct"
Sep 16 02:37:09.694566 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:09.694498    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e8ae5f7-3d2a-4725-81c3-a2ae2fdaaaa2-istio-data\") pod \"e2e-shared-b-w2-a53a44-openshift-default-75f7864f96-gh2ct\" (UID: \"1e8ae5f7-3d2a-4725-81c3-a2ae2fdaaaa2\") " pod="openshift-ingress/e2e-shared-b-w2-a53a44-openshift-default-75f7864f96-gh2ct"
Sep 16 02:37:09.694566 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:09.694559    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4qgqj\" (UniqueName: \"kubernetes.io/projected/1e8ae5f7-3d2a-4725-81c3-a2ae2fdaaaa2-kube-api-access-4qgqj\") pod \"e2e-shared-b-w2-a53a44-openshift-default-75f7864f96-gh2ct\" (UID: \"1e8ae5f7-3d2a-4725-81c3-a2ae2fdaaaa2\") " pod="openshift-ingress/e2e-shared-b-w2-a53a44-openshift-default-75f7864f96-gh2ct"
Sep 16 02:37:09.694741 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:09.694601    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e8ae5f7-3d2a-4725-81c3-a2ae2fdaaaa2-istiod-ca-cert\") pod \"e2e-shared-b-w2-a53a44-openshift-default-75f7864f96-gh2ct\" (UID: \"1e8ae5f7-3d2a-4725-81c3-a2ae2fdaaaa2\") " pod="openshift-ingress/e2e-shared-b-w2-a53a44-openshift-default-75f7864f96-gh2ct"
Sep 16 02:37:09.694741 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:09.694636    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e8ae5f7-3d2a-4725-81c3-a2ae2fdaaaa2-istio-podinfo\") pod \"e2e-shared-b-w2-a53a44-openshift-default-75f7864f96-gh2ct\" (UID: \"1e8ae5f7-3d2a-4725-81c3-a2ae2fdaaaa2\") " pod="openshift-ingress/e2e-shared-b-w2-a53a44-openshift-default-75f7864f96-gh2ct"
Sep 16 02:37:09.694741 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:09.694665    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e8ae5f7-3d2a-4725-81c3-a2ae2fdaaaa2-workload-certs\") pod \"e2e-shared-b-w2-a53a44-openshift-default-75f7864f96-gh2ct\" (UID: \"1e8ae5f7-3d2a-4725-81c3-a2ae2fdaaaa2\") " pod="openshift-ingress/e2e-shared-b-w2-a53a44-openshift-default-75f7864f96-gh2ct"
Sep 16 02:37:09.694741 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:09.694691    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e8ae5f7-3d2a-4725-81c3-a2ae2fdaaaa2-credential-socket\") pod \"e2e-shared-b-w2-a53a44-openshift-default-75f7864f96-gh2ct\" (UID: \"1e8ae5f7-3d2a-4725-81c3-a2ae2fdaaaa2\") " pod="openshift-ingress/e2e-shared-b-w2-a53a44-openshift-default-75f7864f96-gh2ct"
Sep 16 02:37:09.694914 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:09.694779    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e8ae5f7-3d2a-4725-81c3-a2ae2fdaaaa2-istio-token\") pod \"e2e-shared-b-w2-a53a44-openshift-default-75f7864f96-gh2ct\" (UID: \"1e8ae5f7-3d2a-4725-81c3-a2ae2fdaaaa2\") " pod="openshift-ingress/e2e-shared-b-w2-a53a44-openshift-default-75f7864f96-gh2ct"
Sep 16 02:37:09.795593 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:09.795557    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e8ae5f7-3d2a-4725-81c3-a2ae2fdaaaa2-workload-certs\") pod \"e2e-shared-b-w2-a53a44-openshift-default-75f7864f96-gh2ct\" (UID: \"1e8ae5f7-3d2a-4725-81c3-a2ae2fdaaaa2\") " pod="openshift-ingress/e2e-shared-b-w2-a53a44-openshift-default-75f7864f96-gh2ct"
Sep 16 02:37:09.795593 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:09.795596    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e8ae5f7-3d2a-4725-81c3-a2ae2fdaaaa2-credential-socket\") pod \"e2e-shared-b-w2-a53a44-openshift-default-75f7864f96-gh2ct\" (UID: \"1e8ae5f7-3d2a-4725-81c3-a2ae2fdaaaa2\") " pod="openshift-ingress/e2e-shared-b-w2-a53a44-openshift-default-75f7864f96-gh2ct"
Sep 16 02:37:09.795883 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:09.795636    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e8ae5f7-3d2a-4725-81c3-a2ae2fdaaaa2-istio-token\") pod \"e2e-shared-b-w2-a53a44-openshift-default-75f7864f96-gh2ct\" (UID: \"1e8ae5f7-3d2a-4725-81c3-a2ae2fdaaaa2\") " pod="openshift-ingress/e2e-shared-b-w2-a53a44-openshift-default-75f7864f96-gh2ct"
Sep 16 02:37:09.795883 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:09.795665    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e8ae5f7-3d2a-4725-81c3-a2ae2fdaaaa2-workload-socket\") pod \"e2e-shared-b-w2-a53a44-openshift-default-75f7864f96-gh2ct\" (UID: \"1e8ae5f7-3d2a-4725-81c3-a2ae2fdaaaa2\") " pod="openshift-ingress/e2e-shared-b-w2-a53a44-openshift-default-75f7864f96-gh2ct"
Sep 16 02:37:09.795883 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:09.795743    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e8ae5f7-3d2a-4725-81c3-a2ae2fdaaaa2-istio-envoy\") pod \"e2e-shared-b-w2-a53a44-openshift-default-75f7864f96-gh2ct\" (UID: \"1e8ae5f7-3d2a-4725-81c3-a2ae2fdaaaa2\") " pod="openshift-ingress/e2e-shared-b-w2-a53a44-openshift-default-75f7864f96-gh2ct"
Sep 16 02:37:09.795883 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:09.795784    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e8ae5f7-3d2a-4725-81c3-a2ae2fdaaaa2-istio-data\") pod \"e2e-shared-b-w2-a53a44-openshift-default-75f7864f96-gh2ct\" (UID: \"1e8ae5f7-3d2a-4725-81c3-a2ae2fdaaaa2\") " pod="openshift-ingress/e2e-shared-b-w2-a53a44-openshift-default-75f7864f96-gh2ct"
Sep 16 02:37:09.795883 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:09.795824    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qgqj\" (UniqueName: \"kubernetes.io/projected/1e8ae5f7-3d2a-4725-81c3-a2ae2fdaaaa2-kube-api-access-4qgqj\") pod \"e2e-shared-b-w2-a53a44-openshift-default-75f7864f96-gh2ct\" (UID: \"1e8ae5f7-3d2a-4725-81c3-a2ae2fdaaaa2\") " pod="openshift-ingress/e2e-shared-b-w2-a53a44-openshift-default-75f7864f96-gh2ct"
Sep 16 02:37:09.795883 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:09.795874    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e8ae5f7-3d2a-4725-81c3-a2ae2fdaaaa2-istiod-ca-cert\") pod \"e2e-shared-b-w2-a53a44-openshift-default-75f7864f96-gh2ct\" (UID: \"1e8ae5f7-3d2a-4725-81c3-a2ae2fdaaaa2\") " pod="openshift-ingress/e2e-shared-b-w2-a53a44-openshift-default-75f7864f96-gh2ct"
Sep 16 02:37:09.796207 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:09.796043    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e8ae5f7-3d2a-4725-81c3-a2ae2fdaaaa2-workload-certs\") pod \"e2e-shared-b-w2-a53a44-openshift-default-75f7864f96-gh2ct\" (UID: \"1e8ae5f7-3d2a-4725-81c3-a2ae2fdaaaa2\") " pod="openshift-ingress/e2e-shared-b-w2-a53a44-openshift-default-75f7864f96-gh2ct"
Sep 16 02:37:09.796207 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:09.796080    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e8ae5f7-3d2a-4725-81c3-a2ae2fdaaaa2-istio-podinfo\") pod \"e2e-shared-b-w2-a53a44-openshift-default-75f7864f96-gh2ct\" (UID: \"1e8ae5f7-3d2a-4725-81c3-a2ae2fdaaaa2\") " pod="openshift-ingress/e2e-shared-b-w2-a53a44-openshift-default-75f7864f96-gh2ct"
Sep 16 02:37:09.796207 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:09.796112    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e8ae5f7-3d2a-4725-81c3-a2ae2fdaaaa2-credential-socket\") pod \"e2e-shared-b-w2-a53a44-openshift-default-75f7864f96-gh2ct\" (UID: \"1e8ae5f7-3d2a-4725-81c3-a2ae2fdaaaa2\") " pod="openshift-ingress/e2e-shared-b-w2-a53a44-openshift-default-75f7864f96-gh2ct"
Sep 16 02:37:09.796207 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:09.796141    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e8ae5f7-3d2a-4725-81c3-a2ae2fdaaaa2-workload-socket\") pod \"e2e-shared-b-w2-a53a44-openshift-default-75f7864f96-gh2ct\" (UID: \"1e8ae5f7-3d2a-4725-81c3-a2ae2fdaaaa2\") " pod="openshift-ingress/e2e-shared-b-w2-a53a44-openshift-default-75f7864f96-gh2ct"
Sep 16 02:37:09.796207 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:09.796183    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e8ae5f7-3d2a-4725-81c3-a2ae2fdaaaa2-istio-data\") pod \"e2e-shared-b-w2-a53a44-openshift-default-75f7864f96-gh2ct\" (UID: \"1e8ae5f7-3d2a-4725-81c3-a2ae2fdaaaa2\") " pod="openshift-ingress/e2e-shared-b-w2-a53a44-openshift-default-75f7864f96-gh2ct"
Sep 16 02:37:09.796640 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:09.796618    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e8ae5f7-3d2a-4725-81c3-a2ae2fdaaaa2-istiod-ca-cert\") pod \"e2e-shared-b-w2-a53a44-openshift-default-75f7864f96-gh2ct\" (UID: \"1e8ae5f7-3d2a-4725-81c3-a2ae2fdaaaa2\") " pod="openshift-ingress/e2e-shared-b-w2-a53a44-openshift-default-75f7864f96-gh2ct"
Sep 16 02:37:09.798027 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:09.797996    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e8ae5f7-3d2a-4725-81c3-a2ae2fdaaaa2-istio-envoy\") pod \"e2e-shared-b-w2-a53a44-openshift-default-75f7864f96-gh2ct\" (UID: \"1e8ae5f7-3d2a-4725-81c3-a2ae2fdaaaa2\") " pod="openshift-ingress/e2e-shared-b-w2-a53a44-openshift-default-75f7864f96-gh2ct"
Sep 16 02:37:09.798352 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:09.798333    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e8ae5f7-3d2a-4725-81c3-a2ae2fdaaaa2-istio-podinfo\") pod \"e2e-shared-b-w2-a53a44-openshift-default-75f7864f96-gh2ct\" (UID: \"1e8ae5f7-3d2a-4725-81c3-a2ae2fdaaaa2\") " pod="openshift-ingress/e2e-shared-b-w2-a53a44-openshift-default-75f7864f96-gh2ct"
Sep 16 02:37:09.803044 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:09.802984    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qgqj\" (UniqueName: \"kubernetes.io/projected/1e8ae5f7-3d2a-4725-81c3-a2ae2fdaaaa2-kube-api-access-4qgqj\") pod \"e2e-shared-b-w2-a53a44-openshift-default-75f7864f96-gh2ct\" (UID: \"1e8ae5f7-3d2a-4725-81c3-a2ae2fdaaaa2\") " pod="openshift-ingress/e2e-shared-b-w2-a53a44-openshift-default-75f7864f96-gh2ct"
Sep 16 02:37:09.803044 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:09.803033    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e8ae5f7-3d2a-4725-81c3-a2ae2fdaaaa2-istio-token\") pod \"e2e-shared-b-w2-a53a44-openshift-default-75f7864f96-gh2ct\" (UID: \"1e8ae5f7-3d2a-4725-81c3-a2ae2fdaaaa2\") " pod="openshift-ingress/e2e-shared-b-w2-a53a44-openshift-default-75f7864f96-gh2ct"
Sep 16 02:37:09.866472 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:09.866441    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-a53a44-openshift-default-75f7864f96-gh2ct"
Sep 16 02:37:10.145400 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:10.145372    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-a53a44-openshift-default-75f7864f96-gh2ct"]
Sep 16 02:37:10.147351 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:37:10.147327    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e8ae5f7_3d2a_4725_81c3_a2ae2fdaaaa2.slice/crio-1ad9a48c5fbc142a3522c8896d94c17c72be7badb6f68b57845e1f734b4f9192 WatchSource:0}: Error finding container 1ad9a48c5fbc142a3522c8896d94c17c72be7badb6f68b57845e1f734b4f9192: Status 404 returned error can't find the container with id 1ad9a48c5fbc142a3522c8896d94c17c72be7badb6f68b57845e1f734b4f9192
Sep 16 02:37:10.149665 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:10.149633    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 16 02:37:10.149785 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:10.149714    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 16 02:37:10.149785 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:10.149749    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 16 02:37:10.692667 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:10.692623    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-2e69f8df-667fdbfbf8-bbj87"]
Sep 16 02:37:10.696293 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:10.696269    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-2e69f8df-667fdbfbf8-bbj87"
Sep 16 02:37:10.698574 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:10.698550    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-2e69f8df-dockercfg-89qvm\""
Sep 16 02:37:10.698825 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:10.698803    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-2e69f8df\""
Sep 16 02:37:10.707381 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:10.707355    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-2e69f8df-667fdbfbf8-bbj87"]
Sep 16 02:37:10.736444 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:10.736411    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-2e69f8df-65c4c95d57-sltg4"]
Sep 16 02:37:10.739810 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:10.739790    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-2e69f8df-65c4c95d57-sltg4"
Sep 16 02:37:10.751621 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:10.751597    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-2e69f8df-65c4c95d57-sltg4"]
Sep 16 02:37:10.808239 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:10.807583    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkkg5\" (UniqueName: \"kubernetes.io/projected/827a8b2e-8691-4577-a2b5-6f6b0f84aea9-kube-api-access-zkkg5\") pod \"payload-pre-processing-e2e-derive-2e69f8df-667fdbfbf8-bbj87\" (UID: \"827a8b2e-8691-4577-a2b5-6f6b0f84aea9\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-2e69f8df-667fdbfbf8-bbj87"
Sep 16 02:37:10.808239 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:10.807745    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/827a8b2e-8691-4577-a2b5-6f6b0f84aea9-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-2e69f8df-667fdbfbf8-bbj87\" (UID: \"827a8b2e-8691-4577-a2b5-6f6b0f84aea9\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-2e69f8df-667fdbfbf8-bbj87"
Sep 16 02:37:10.908861 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:10.908826    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ks4zc\" (UniqueName: \"kubernetes.io/projected/89da9e44-caa5-4acf-9abb-90a220879254-kube-api-access-ks4zc\") pod \"payload-processing-e2e-derive-2e69f8df-65c4c95d57-sltg4\" (UID: \"89da9e44-caa5-4acf-9abb-90a220879254\") " pod="openshift-ingress/payload-processing-e2e-derive-2e69f8df-65c4c95d57-sltg4"
Sep 16 02:37:10.909041 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:10.908907    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/827a8b2e-8691-4577-a2b5-6f6b0f84aea9-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-2e69f8df-667fdbfbf8-bbj87\" (UID: \"827a8b2e-8691-4577-a2b5-6f6b0f84aea9\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-2e69f8df-667fdbfbf8-bbj87"
Sep 16 02:37:10.909041 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:10.908952    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89da9e44-caa5-4acf-9abb-90a220879254-plugins-config-volume\") pod \"payload-processing-e2e-derive-2e69f8df-65c4c95d57-sltg4\" (UID: \"89da9e44-caa5-4acf-9abb-90a220879254\") " pod="openshift-ingress/payload-processing-e2e-derive-2e69f8df-65c4c95d57-sltg4"
Sep 16 02:37:10.909041 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:10.908976    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkkg5\" (UniqueName: \"kubernetes.io/projected/827a8b2e-8691-4577-a2b5-6f6b0f84aea9-kube-api-access-zkkg5\") pod \"payload-pre-processing-e2e-derive-2e69f8df-667fdbfbf8-bbj87\" (UID: \"827a8b2e-8691-4577-a2b5-6f6b0f84aea9\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-2e69f8df-667fdbfbf8-bbj87"
Sep 16 02:37:10.909585 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:10.909565    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/827a8b2e-8691-4577-a2b5-6f6b0f84aea9-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-2e69f8df-667fdbfbf8-bbj87\" (UID: \"827a8b2e-8691-4577-a2b5-6f6b0f84aea9\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-2e69f8df-667fdbfbf8-bbj87"
Sep 16 02:37:10.916301 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:10.916280    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkkg5\" (UniqueName: \"kubernetes.io/projected/827a8b2e-8691-4577-a2b5-6f6b0f84aea9-kube-api-access-zkkg5\") pod \"payload-pre-processing-e2e-derive-2e69f8df-667fdbfbf8-bbj87\" (UID: \"827a8b2e-8691-4577-a2b5-6f6b0f84aea9\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-2e69f8df-667fdbfbf8-bbj87"
Sep 16 02:37:11.008857 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:11.008768    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-2e69f8df-667fdbfbf8-bbj87"
Sep 16 02:37:11.010238 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:11.010208    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ks4zc\" (UniqueName: \"kubernetes.io/projected/89da9e44-caa5-4acf-9abb-90a220879254-kube-api-access-ks4zc\") pod \"payload-processing-e2e-derive-2e69f8df-65c4c95d57-sltg4\" (UID: \"89da9e44-caa5-4acf-9abb-90a220879254\") " pod="openshift-ingress/payload-processing-e2e-derive-2e69f8df-65c4c95d57-sltg4"
Sep 16 02:37:11.010389 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:11.010371    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89da9e44-caa5-4acf-9abb-90a220879254-plugins-config-volume\") pod \"payload-processing-e2e-derive-2e69f8df-65c4c95d57-sltg4\" (UID: \"89da9e44-caa5-4acf-9abb-90a220879254\") " pod="openshift-ingress/payload-processing-e2e-derive-2e69f8df-65c4c95d57-sltg4"
Sep 16 02:37:11.010996 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:11.010977    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89da9e44-caa5-4acf-9abb-90a220879254-plugins-config-volume\") pod \"payload-processing-e2e-derive-2e69f8df-65c4c95d57-sltg4\" (UID: \"89da9e44-caa5-4acf-9abb-90a220879254\") " pod="openshift-ingress/payload-processing-e2e-derive-2e69f8df-65c4c95d57-sltg4"
Sep 16 02:37:11.018325 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:11.018303    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ks4zc\" (UniqueName: \"kubernetes.io/projected/89da9e44-caa5-4acf-9abb-90a220879254-kube-api-access-ks4zc\") pod \"payload-processing-e2e-derive-2e69f8df-65c4c95d57-sltg4\" (UID: \"89da9e44-caa5-4acf-9abb-90a220879254\") " pod="openshift-ingress/payload-processing-e2e-derive-2e69f8df-65c4c95d57-sltg4"
Sep 16 02:37:11.049609 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:11.049572    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-2e69f8df-65c4c95d57-sltg4"
Sep 16 02:37:11.100003 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:11.099965    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-a53a44-openshift-default-75f7864f96-gh2ct" event={"ID":"1e8ae5f7-3d2a-4725-81c3-a2ae2fdaaaa2","Type":"ContainerStarted","Data":"2abbcfbc772a22c012be5618e2976bf43fc84c1176e7305eb471f6b44ca9298a"}
Sep 16 02:37:11.100003 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:11.100022    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-a53a44-openshift-default-75f7864f96-gh2ct" event={"ID":"1e8ae5f7-3d2a-4725-81c3-a2ae2fdaaaa2","Type":"ContainerStarted","Data":"1ad9a48c5fbc142a3522c8896d94c17c72be7badb6f68b57845e1f734b4f9192"}
Sep 16 02:37:11.119377 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:11.118610    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-a53a44-openshift-default-75f7864f96-gh2ct" podStartSLOduration=2.118459845 podStartE2EDuration="2.118459845s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 02:37:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:37:11.117959143 +0000 UTC m=+1444.036448405" watchObservedRunningTime="2026-09-16 02:37:11.118459845 +0000 UTC m=+1444.036949096"
Sep 16 02:37:11.278834 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:11.278808    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-2e69f8df-667fdbfbf8-bbj87"]
Sep 16 02:37:11.280145 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:37:11.280113    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod827a8b2e_8691_4577_a2b5_6f6b0f84aea9.slice/crio-2d580c21a6eceabbeadd1b08b07c1c9228a8822c5136ddd0ad8c0d23977c1c6a WatchSource:0}: Error finding container 2d580c21a6eceabbeadd1b08b07c1c9228a8822c5136ddd0ad8c0d23977c1c6a: Status 404 returned error can't find the container with id 2d580c21a6eceabbeadd1b08b07c1c9228a8822c5136ddd0ad8c0d23977c1c6a
Sep 16 02:37:11.343954 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:11.343922    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-2e69f8df-65c4c95d57-sltg4"]
Sep 16 02:37:11.345680 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:37:11.345653    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89da9e44_caa5_4acf_9abb_90a220879254.slice/crio-cde1d1df9a034810b33703b5226142e7b8d62f52941fb71da245ab8c95abcd39 WatchSource:0}: Error finding container cde1d1df9a034810b33703b5226142e7b8d62f52941fb71da245ab8c95abcd39: Status 404 returned error can't find the container with id cde1d1df9a034810b33703b5226142e7b8d62f52941fb71da245ab8c95abcd39
Sep 16 02:37:11.867252 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:11.867209    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-a53a44-openshift-default-75f7864f96-gh2ct"
Sep 16 02:37:11.872499 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:11.872468    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-a53a44-openshift-default-75f7864f96-gh2ct"
Sep 16 02:37:12.112296 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:12.112258    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-2e69f8df-65c4c95d57-sltg4" event={"ID":"89da9e44-caa5-4acf-9abb-90a220879254","Type":"ContainerStarted","Data":"41270fc6d0b0c503355adcdc03446be4db3860248c217a6798ab32cb8458f326"}
Sep 16 02:37:12.112601 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:12.112574    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-2e69f8df-65c4c95d57-sltg4" event={"ID":"89da9e44-caa5-4acf-9abb-90a220879254","Type":"ContainerStarted","Data":"cde1d1df9a034810b33703b5226142e7b8d62f52941fb71da245ab8c95abcd39"}
Sep 16 02:37:12.113675 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:12.113651    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-2e69f8df-65c4c95d57-sltg4"
Sep 16 02:37:12.118675 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:12.118593    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-2e69f8df-667fdbfbf8-bbj87" event={"ID":"827a8b2e-8691-4577-a2b5-6f6b0f84aea9","Type":"ContainerStarted","Data":"481af599442c48332f7ab62868b9910a4dd4a318792762186c7a0d3612a072e2"}
Sep 16 02:37:12.119212 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:12.119189    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-2e69f8df-667fdbfbf8-bbj87" event={"ID":"827a8b2e-8691-4577-a2b5-6f6b0f84aea9","Type":"ContainerStarted","Data":"2d580c21a6eceabbeadd1b08b07c1c9228a8822c5136ddd0ad8c0d23977c1c6a"}
Sep 16 02:37:12.119400 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:12.119387    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-a53a44-openshift-default-75f7864f96-gh2ct"
Sep 16 02:37:12.119508 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:12.119499    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-2e69f8df-667fdbfbf8-bbj87"
Sep 16 02:37:12.120501 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:12.120478    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-a53a44-openshift-default-75f7864f96-gh2ct"
Sep 16 02:37:12.131284 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:12.131219    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-2e69f8df-65c4c95d57-sltg4" podStartSLOduration=2.131201632 podStartE2EDuration="2.131201632s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 02:37:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:37:12.129555521 +0000 UTC m=+1445.048044770" watchObservedRunningTime="2026-09-16 02:37:12.131201632 +0000 UTC m=+1445.049690880"
Sep 16 02:37:12.166489 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:12.166425    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-2e69f8df-667fdbfbf8-bbj87" podStartSLOduration=2.166400345 podStartE2EDuration="2.166400345s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 02:37:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:37:12.163488428 +0000 UTC m=+1445.081977679" watchObservedRunningTime="2026-09-16 02:37:12.166400345 +0000 UTC m=+1445.084889595"
Sep 16 02:37:12.544222 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:12.544184    2558 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-processing-5d7dbbd56b-pltkr" secret="" err="secret \"payload-processing-dockercfg-zzx6j\" not found"
Sep 16 02:37:24.129307 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:24.129270    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-2e69f8df-667fdbfbf8-bbj87"
Sep 16 02:37:24.129821 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:24.129435    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-2e69f8df-65c4c95d57-sltg4"
Sep 16 02:37:33.303337 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:33.303294    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2e69f8df-8lr6p"]
Sep 16 02:37:33.308441 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:33.308417    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2e69f8df-8lr6p"
Sep 16 02:37:33.311404 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:33.311375    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jpd4f\""
Sep 16 02:37:33.311556 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:33.311375    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 16 02:37:33.313457 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:33.313423    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2e69f8df-8lr6p"]
Sep 16 02:37:33.318751 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:33.318723    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc3ab2d4-becc-4aa9-95ed-6f5cf8aec1e7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-2e69f8df-8lr6p\" (UID: \"fc3ab2d4-becc-4aa9-95ed-6f5cf8aec1e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2e69f8df-8lr6p"
Sep 16 02:37:33.419706 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:33.419665    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc3ab2d4-becc-4aa9-95ed-6f5cf8aec1e7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-2e69f8df-8lr6p\" (UID: \"fc3ab2d4-becc-4aa9-95ed-6f5cf8aec1e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2e69f8df-8lr6p"
Sep 16 02:37:33.420308 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:33.420279    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc3ab2d4-becc-4aa9-95ed-6f5cf8aec1e7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-2e69f8df-8lr6p\" (UID: \"fc3ab2d4-becc-4aa9-95ed-6f5cf8aec1e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2e69f8df-8lr6p"
Sep 16 02:37:33.621074 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:33.620981    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2e69f8df-8lr6p"
Sep 16 02:37:33.907752 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:33.907723    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2e69f8df-8lr6p"]
Sep 16 02:37:33.910156 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:37:33.910112    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc3ab2d4_becc_4aa9_95ed_6f5cf8aec1e7.slice/crio-16f8e50d778494049dc08a0be77054aefc320c262ccbcff4a9e9ca8d72573913 WatchSource:0}: Error finding container 16f8e50d778494049dc08a0be77054aefc320c262ccbcff4a9e9ca8d72573913: Status 404 returned error can't find the container with id 16f8e50d778494049dc08a0be77054aefc320c262ccbcff4a9e9ca8d72573913
Sep 16 02:37:34.235457 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:34.235411    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2e69f8df-8lr6p" event={"ID":"fc3ab2d4-becc-4aa9-95ed-6f5cf8aec1e7","Type":"ContainerStarted","Data":"16f8e50d778494049dc08a0be77054aefc320c262ccbcff4a9e9ca8d72573913"}
Sep 16 02:37:35.242911 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:35.242875    2558 generic.go:356] "Generic (PLEG): container finished" podID="fc3ab2d4-becc-4aa9-95ed-6f5cf8aec1e7" containerID="f94d0d0a7b37b8afba7d2431f9143811533648bc0c7d28ab63e380adcecf1f22" exitCode=0
Sep 16 02:37:35.243301 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:35.242961    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2e69f8df-8lr6p" event={"ID":"fc3ab2d4-becc-4aa9-95ed-6f5cf8aec1e7","Type":"ContainerDied","Data":"f94d0d0a7b37b8afba7d2431f9143811533648bc0c7d28ab63e380adcecf1f22"}
Sep 16 02:37:36.523665 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:36.523637    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2e69f8df-8lr6p"
Sep 16 02:37:36.559857 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:36.559043    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc3ab2d4-becc-4aa9-95ed-6f5cf8aec1e7-maas-api-service-ca\") pod \"fc3ab2d4-becc-4aa9-95ed-6f5cf8aec1e7\" (UID: \"fc3ab2d4-becc-4aa9-95ed-6f5cf8aec1e7\") "
Sep 16 02:37:36.560236 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:36.560206    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc3ab2d4-becc-4aa9-95ed-6f5cf8aec1e7-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "fc3ab2d4-becc-4aa9-95ed-6f5cf8aec1e7" (UID: "fc3ab2d4-becc-4aa9-95ed-6f5cf8aec1e7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 02:37:36.663723 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:36.663489    2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc3ab2d4-becc-4aa9-95ed-6f5cf8aec1e7-maas-api-service-ca\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:37:37.262382 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:37.262350    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2e69f8df-8lr6p"
Sep 16 02:37:37.265798 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:37.262434    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2e69f8df-8lr6p" event={"ID":"fc3ab2d4-becc-4aa9-95ed-6f5cf8aec1e7","Type":"ContainerDied","Data":"16f8e50d778494049dc08a0be77054aefc320c262ccbcff4a9e9ca8d72573913"}
Sep 16 02:37:37.265984 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:37.265826    2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="16f8e50d778494049dc08a0be77054aefc320c262ccbcff4a9e9ca8d72573913"
Sep 16 02:37:40.698956 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:40.697613    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-2e69f8df-65c4c95d57-sltg4"]
Sep 16 02:37:40.698956 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:40.697884    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-2e69f8df-65c4c95d57-sltg4" podUID="89da9e44-caa5-4acf-9abb-90a220879254" containerName="payload-processing" containerID="cri-o://41270fc6d0b0c503355adcdc03446be4db3860248c217a6798ab32cb8458f326" gracePeriod=30
Sep 16 02:37:40.709754 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:40.708570    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-2e69f8df-667fdbfbf8-bbj87"]
Sep 16 02:37:40.709754 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:40.708829    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-2e69f8df-667fdbfbf8-bbj87" podUID="827a8b2e-8691-4577-a2b5-6f6b0f84aea9" containerName="payload-pre-processing" containerID="cri-o://481af599442c48332f7ab62868b9910a4dd4a318792762186c7a0d3612a072e2" gracePeriod=30
Sep 16 02:37:40.855270 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:40.854856    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2e69f8df-8lr6p"]
Sep 16 02:37:40.858628 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:40.858578    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2e69f8df-8lr6p"]
Sep 16 02:37:40.987994 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:40.987964    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-2e69f8df-openshift-default-6b45f469f6-v2kbk"]
Sep 16 02:37:40.989338 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:40.989256    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-2e69f8df-65c4c95d57-sltg4"
Sep 16 02:37:40.989530 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:40.989501    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-2e69f8df-openshift-default-6b45f469f6-v2kbk" podUID="5a9173c4-9b43-4203-8287-1b34dfd261b1" containerName="istio-proxy" containerID="cri-o://324046f3600b748f7c3359c523e456400a772984af0c2afcaa162ffb9e3e84e8" gracePeriod=30
Sep 16 02:37:41.035952 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:41.035921    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-2e69f8df-667fdbfbf8-bbj87"
Sep 16 02:37:41.119619 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:41.117101    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89da9e44-caa5-4acf-9abb-90a220879254-plugins-config-volume\") pod \"89da9e44-caa5-4acf-9abb-90a220879254\" (UID: \"89da9e44-caa5-4acf-9abb-90a220879254\") "
Sep 16 02:37:41.119619 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:41.117182    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zkkg5\" (UniqueName: \"kubernetes.io/projected/827a8b2e-8691-4577-a2b5-6f6b0f84aea9-kube-api-access-zkkg5\") pod \"827a8b2e-8691-4577-a2b5-6f6b0f84aea9\" (UID: \"827a8b2e-8691-4577-a2b5-6f6b0f84aea9\") "
Sep 16 02:37:41.119619 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:41.117249    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ks4zc\" (UniqueName: \"kubernetes.io/projected/89da9e44-caa5-4acf-9abb-90a220879254-kube-api-access-ks4zc\") pod \"89da9e44-caa5-4acf-9abb-90a220879254\" (UID: \"89da9e44-caa5-4acf-9abb-90a220879254\") "
Sep 16 02:37:41.119619 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:41.117363    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/827a8b2e-8691-4577-a2b5-6f6b0f84aea9-plugins-config-volume\") pod \"827a8b2e-8691-4577-a2b5-6f6b0f84aea9\" (UID: \"827a8b2e-8691-4577-a2b5-6f6b0f84aea9\") "
Sep 16 02:37:41.119619 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:41.118982    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89da9e44-caa5-4acf-9abb-90a220879254-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "89da9e44-caa5-4acf-9abb-90a220879254" (UID: "89da9e44-caa5-4acf-9abb-90a220879254"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 02:37:41.119619 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:41.119182    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/827a8b2e-8691-4577-a2b5-6f6b0f84aea9-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "827a8b2e-8691-4577-a2b5-6f6b0f84aea9" (UID: "827a8b2e-8691-4577-a2b5-6f6b0f84aea9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 02:37:41.122845 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:41.122798    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/827a8b2e-8691-4577-a2b5-6f6b0f84aea9-kube-api-access-zkkg5" (OuterVolumeSpecName: "kube-api-access-zkkg5") pod "827a8b2e-8691-4577-a2b5-6f6b0f84aea9" (UID: "827a8b2e-8691-4577-a2b5-6f6b0f84aea9"). InnerVolumeSpecName "kube-api-access-zkkg5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 02:37:41.125073 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:41.125025    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89da9e44-caa5-4acf-9abb-90a220879254-kube-api-access-ks4zc" (OuterVolumeSpecName: "kube-api-access-ks4zc") pod "89da9e44-caa5-4acf-9abb-90a220879254" (UID: "89da9e44-caa5-4acf-9abb-90a220879254"). InnerVolumeSpecName "kube-api-access-ks4zc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 02:37:41.219223 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:41.219117    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zkkg5\" (UniqueName: \"kubernetes.io/projected/827a8b2e-8691-4577-a2b5-6f6b0f84aea9-kube-api-access-zkkg5\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:37:41.219223 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:41.219155    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ks4zc\" (UniqueName: \"kubernetes.io/projected/89da9e44-caa5-4acf-9abb-90a220879254-kube-api-access-ks4zc\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:37:41.219223 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:41.219171    2558 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/827a8b2e-8691-4577-a2b5-6f6b0f84aea9-plugins-config-volume\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:37:41.219223 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:41.219184    2558 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89da9e44-caa5-4acf-9abb-90a220879254-plugins-config-volume\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:37:41.293800 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:41.293764    2558 generic.go:356] "Generic (PLEG): container finished" podID="827a8b2e-8691-4577-a2b5-6f6b0f84aea9" containerID="481af599442c48332f7ab62868b9910a4dd4a318792762186c7a0d3612a072e2" exitCode=0
Sep 16 02:37:41.293981 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:41.293833    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-2e69f8df-667fdbfbf8-bbj87"
Sep 16 02:37:41.293981 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:41.293849    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-2e69f8df-667fdbfbf8-bbj87" event={"ID":"827a8b2e-8691-4577-a2b5-6f6b0f84aea9","Type":"ContainerDied","Data":"481af599442c48332f7ab62868b9910a4dd4a318792762186c7a0d3612a072e2"}
Sep 16 02:37:41.293981 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:41.293922    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-2e69f8df-667fdbfbf8-bbj87" event={"ID":"827a8b2e-8691-4577-a2b5-6f6b0f84aea9","Type":"ContainerDied","Data":"2d580c21a6eceabbeadd1b08b07c1c9228a8822c5136ddd0ad8c0d23977c1c6a"}
Sep 16 02:37:41.293981 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:41.293951    2558 scope.go:117] "RemoveContainer" containerID="481af599442c48332f7ab62868b9910a4dd4a318792762186c7a0d3612a072e2"
Sep 16 02:37:41.296058 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:41.296028    2558 generic.go:356] "Generic (PLEG): container finished" podID="89da9e44-caa5-4acf-9abb-90a220879254" containerID="41270fc6d0b0c503355adcdc03446be4db3860248c217a6798ab32cb8458f326" exitCode=0
Sep 16 02:37:41.296178 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:41.296139    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-2e69f8df-65c4c95d57-sltg4" event={"ID":"89da9e44-caa5-4acf-9abb-90a220879254","Type":"ContainerDied","Data":"41270fc6d0b0c503355adcdc03446be4db3860248c217a6798ab32cb8458f326"}
Sep 16 02:37:41.296251 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:41.296186    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-2e69f8df-65c4c95d57-sltg4" event={"ID":"89da9e44-caa5-4acf-9abb-90a220879254","Type":"ContainerDied","Data":"cde1d1df9a034810b33703b5226142e7b8d62f52941fb71da245ab8c95abcd39"}
Sep 16 02:37:41.296309 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:41.296253    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-2e69f8df-65c4c95d57-sltg4"
Sep 16 02:37:41.315679 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:41.309641    2558 scope.go:117] "RemoveContainer" containerID="481af599442c48332f7ab62868b9910a4dd4a318792762186c7a0d3612a072e2"
Sep 16 02:37:41.322331 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:37:41.322276    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"481af599442c48332f7ab62868b9910a4dd4a318792762186c7a0d3612a072e2\": container with ID starting with 481af599442c48332f7ab62868b9910a4dd4a318792762186c7a0d3612a072e2 not found: ID does not exist" containerID="481af599442c48332f7ab62868b9910a4dd4a318792762186c7a0d3612a072e2"
Sep 16 02:37:41.322529 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:41.322327    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"481af599442c48332f7ab62868b9910a4dd4a318792762186c7a0d3612a072e2"} err="failed to get container status \"481af599442c48332f7ab62868b9910a4dd4a318792762186c7a0d3612a072e2\": rpc error: code = NotFound desc = could not find container \"481af599442c48332f7ab62868b9910a4dd4a318792762186c7a0d3612a072e2\": container with ID starting with 481af599442c48332f7ab62868b9910a4dd4a318792762186c7a0d3612a072e2 not found: ID does not exist"
Sep 16 02:37:41.322529 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:41.322365    2558 scope.go:117] "RemoveContainer" containerID="41270fc6d0b0c503355adcdc03446be4db3860248c217a6798ab32cb8458f326"
Sep 16 02:37:41.344909 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:41.344872    2558 scope.go:117] "RemoveContainer" containerID="41270fc6d0b0c503355adcdc03446be4db3860248c217a6798ab32cb8458f326"
Sep 16 02:37:41.345498 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:37:41.345418    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41270fc6d0b0c503355adcdc03446be4db3860248c217a6798ab32cb8458f326\": container with ID starting with 41270fc6d0b0c503355adcdc03446be4db3860248c217a6798ab32cb8458f326 not found: ID does not exist" containerID="41270fc6d0b0c503355adcdc03446be4db3860248c217a6798ab32cb8458f326"
Sep 16 02:37:41.345498 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:41.345461    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41270fc6d0b0c503355adcdc03446be4db3860248c217a6798ab32cb8458f326"} err="failed to get container status \"41270fc6d0b0c503355adcdc03446be4db3860248c217a6798ab32cb8458f326\": rpc error: code = NotFound desc = could not find container \"41270fc6d0b0c503355adcdc03446be4db3860248c217a6798ab32cb8458f326\": container with ID starting with 41270fc6d0b0c503355adcdc03446be4db3860248c217a6798ab32cb8458f326 not found: ID does not exist"
Sep 16 02:37:41.348520 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:41.348496    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-2e69f8df-667fdbfbf8-bbj87"]
Sep 16 02:37:41.353191 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:41.353166    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-2e69f8df-667fdbfbf8-bbj87"]
Sep 16 02:37:41.364265 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:41.364226    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-2e69f8df-65c4c95d57-sltg4"]
Sep 16 02:37:41.366917 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:41.366897    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-2e69f8df-65c4c95d57-sltg4"]
Sep 16 02:37:41.550349 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:41.550254    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="827a8b2e-8691-4577-a2b5-6f6b0f84aea9" path="/var/lib/kubelet/pods/827a8b2e-8691-4577-a2b5-6f6b0f84aea9/volumes"
Sep 16 02:37:41.550793 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:41.550771    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="89da9e44-caa5-4acf-9abb-90a220879254" path="/var/lib/kubelet/pods/89da9e44-caa5-4acf-9abb-90a220879254/volumes"
Sep 16 02:37:41.551146 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:41.551129    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fc3ab2d4-becc-4aa9-95ed-6f5cf8aec1e7" path="/var/lib/kubelet/pods/fc3ab2d4-becc-4aa9-95ed-6f5cf8aec1e7/volumes"
Sep 16 02:37:43.158870 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:43.158817    2558 patch_prober.go:28] interesting pod/e2e-shared-b-w2-a53a44-openshift-default-75f7864f96-gh2ct container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 16 02:37:43.159352 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:43.158901    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-a53a44-openshift-default-75f7864f96-gh2ct" podUID="1e8ae5f7-3d2a-4725-81c3-a2ae2fdaaaa2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 16 02:37:43.552196 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:43.552167    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-2e69f8df-openshift-default-6b45f469f6-v2kbk"
Sep 16 02:37:43.648042 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:43.648007    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a9173c4-9b43-4203-8287-1b34dfd261b1-istio-token\") pod \"5a9173c4-9b43-4203-8287-1b34dfd261b1\" (UID: \"5a9173c4-9b43-4203-8287-1b34dfd261b1\") "
Sep 16 02:37:43.648208 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:43.648056    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a9173c4-9b43-4203-8287-1b34dfd261b1-istiod-ca-cert\") pod \"5a9173c4-9b43-4203-8287-1b34dfd261b1\" (UID: \"5a9173c4-9b43-4203-8287-1b34dfd261b1\") "
Sep 16 02:37:43.648208 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:43.648103    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a9173c4-9b43-4203-8287-1b34dfd261b1-workload-certs\") pod \"5a9173c4-9b43-4203-8287-1b34dfd261b1\" (UID: \"5a9173c4-9b43-4203-8287-1b34dfd261b1\") "
Sep 16 02:37:43.648208 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:43.648130    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a9173c4-9b43-4203-8287-1b34dfd261b1-istio-podinfo\") pod \"5a9173c4-9b43-4203-8287-1b34dfd261b1\" (UID: \"5a9173c4-9b43-4203-8287-1b34dfd261b1\") "
Sep 16 02:37:43.648208 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:43.648151    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a9173c4-9b43-4203-8287-1b34dfd261b1-credential-socket\") pod \"5a9173c4-9b43-4203-8287-1b34dfd261b1\" (UID: \"5a9173c4-9b43-4203-8287-1b34dfd261b1\") "
Sep 16 02:37:43.648208 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:43.648204    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4jstg\" (UniqueName: \"kubernetes.io/projected/5a9173c4-9b43-4203-8287-1b34dfd261b1-kube-api-access-4jstg\") pod \"5a9173c4-9b43-4203-8287-1b34dfd261b1\" (UID: \"5a9173c4-9b43-4203-8287-1b34dfd261b1\") "
Sep 16 02:37:43.648489 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:43.648256    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a9173c4-9b43-4203-8287-1b34dfd261b1-istio-envoy\") pod \"5a9173c4-9b43-4203-8287-1b34dfd261b1\" (UID: \"5a9173c4-9b43-4203-8287-1b34dfd261b1\") "
Sep 16 02:37:43.648489 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:43.648324    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a9173c4-9b43-4203-8287-1b34dfd261b1-workload-socket\") pod \"5a9173c4-9b43-4203-8287-1b34dfd261b1\" (UID: \"5a9173c4-9b43-4203-8287-1b34dfd261b1\") "
Sep 16 02:37:43.648489 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:43.648362    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a9173c4-9b43-4203-8287-1b34dfd261b1-istio-data\") pod \"5a9173c4-9b43-4203-8287-1b34dfd261b1\" (UID: \"5a9173c4-9b43-4203-8287-1b34dfd261b1\") "
Sep 16 02:37:43.648489 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:43.648455    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a9173c4-9b43-4203-8287-1b34dfd261b1-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "5a9173c4-9b43-4203-8287-1b34dfd261b1" (UID: "5a9173c4-9b43-4203-8287-1b34dfd261b1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:37:43.648690 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:43.648549    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a9173c4-9b43-4203-8287-1b34dfd261b1-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "5a9173c4-9b43-4203-8287-1b34dfd261b1" (UID: "5a9173c4-9b43-4203-8287-1b34dfd261b1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 02:37:43.648690 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:43.648669    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a9173c4-9b43-4203-8287-1b34dfd261b1-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "5a9173c4-9b43-4203-8287-1b34dfd261b1" (UID: "5a9173c4-9b43-4203-8287-1b34dfd261b1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:37:43.648838 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:43.648791    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a9173c4-9b43-4203-8287-1b34dfd261b1-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "5a9173c4-9b43-4203-8287-1b34dfd261b1" (UID: "5a9173c4-9b43-4203-8287-1b34dfd261b1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:37:43.648931 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:43.648905    2558 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a9173c4-9b43-4203-8287-1b34dfd261b1-istiod-ca-cert\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:37:43.648931 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:43.648897    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a9173c4-9b43-4203-8287-1b34dfd261b1-istio-data" (OuterVolumeSpecName: "istio-data") pod "5a9173c4-9b43-4203-8287-1b34dfd261b1" (UID: "5a9173c4-9b43-4203-8287-1b34dfd261b1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:37:43.649071 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:43.648926    2558 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a9173c4-9b43-4203-8287-1b34dfd261b1-workload-certs\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:37:43.649071 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:43.648960    2558 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a9173c4-9b43-4203-8287-1b34dfd261b1-credential-socket\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:37:43.649071 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:43.648976    2558 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a9173c4-9b43-4203-8287-1b34dfd261b1-workload-socket\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:37:43.650516 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:43.650487    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5a9173c4-9b43-4203-8287-1b34dfd261b1-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "5a9173c4-9b43-4203-8287-1b34dfd261b1" (UID: "5a9173c4-9b43-4203-8287-1b34dfd261b1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 02:37:43.650613 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:43.650514    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a9173c4-9b43-4203-8287-1b34dfd261b1-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "5a9173c4-9b43-4203-8287-1b34dfd261b1" (UID: "5a9173c4-9b43-4203-8287-1b34dfd261b1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:37:43.650613 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:43.650514    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a9173c4-9b43-4203-8287-1b34dfd261b1-kube-api-access-4jstg" (OuterVolumeSpecName: "kube-api-access-4jstg") pod "5a9173c4-9b43-4203-8287-1b34dfd261b1" (UID: "5a9173c4-9b43-4203-8287-1b34dfd261b1"). InnerVolumeSpecName "kube-api-access-4jstg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 02:37:43.650613 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:43.650585    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a9173c4-9b43-4203-8287-1b34dfd261b1-istio-token" (OuterVolumeSpecName: "istio-token") pod "5a9173c4-9b43-4203-8287-1b34dfd261b1" (UID: "5a9173c4-9b43-4203-8287-1b34dfd261b1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 02:37:43.750447 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:43.750354    2558 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a9173c4-9b43-4203-8287-1b34dfd261b1-istio-data\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:37:43.750447 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:43.750389    2558 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a9173c4-9b43-4203-8287-1b34dfd261b1-istio-token\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:37:43.750447 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:43.750398    2558 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a9173c4-9b43-4203-8287-1b34dfd261b1-istio-podinfo\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:37:43.750447 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:43.750406    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4jstg\" (UniqueName: \"kubernetes.io/projected/5a9173c4-9b43-4203-8287-1b34dfd261b1-kube-api-access-4jstg\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:37:43.750447 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:43.750415    2558 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a9173c4-9b43-4203-8287-1b34dfd261b1-istio-envoy\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:37:44.323177 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:44.323139    2558 generic.go:356] "Generic (PLEG): container finished" podID="5a9173c4-9b43-4203-8287-1b34dfd261b1" containerID="324046f3600b748f7c3359c523e456400a772984af0c2afcaa162ffb9e3e84e8" exitCode=0
Sep 16 02:37:44.323629 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:44.323185    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-2e69f8df-openshift-default-6b45f469f6-v2kbk"
Sep 16 02:37:44.323629 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:44.323224    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-2e69f8df-openshift-default-6b45f469f6-v2kbk" event={"ID":"5a9173c4-9b43-4203-8287-1b34dfd261b1","Type":"ContainerDied","Data":"324046f3600b748f7c3359c523e456400a772984af0c2afcaa162ffb9e3e84e8"}
Sep 16 02:37:44.323629 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:44.323295    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-2e69f8df-openshift-default-6b45f469f6-v2kbk" event={"ID":"5a9173c4-9b43-4203-8287-1b34dfd261b1","Type":"ContainerDied","Data":"24c825891f0fdec70a30fa3c580fbbf886bb3dfa49547b96c3615cd70c1724b0"}
Sep 16 02:37:44.323629 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:44.323329    2558 scope.go:117] "RemoveContainer" containerID="324046f3600b748f7c3359c523e456400a772984af0c2afcaa162ffb9e3e84e8"
Sep 16 02:37:44.325053 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:44.325028    2558 generic.go:356] "Generic (PLEG): container finished" podID="1e8ae5f7-3d2a-4725-81c3-a2ae2fdaaaa2" containerID="2abbcfbc772a22c012be5618e2976bf43fc84c1176e7305eb471f6b44ca9298a" exitCode=0
Sep 16 02:37:44.325152 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:44.325072    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-a53a44-openshift-default-75f7864f96-gh2ct" event={"ID":"1e8ae5f7-3d2a-4725-81c3-a2ae2fdaaaa2","Type":"ContainerDied","Data":"2abbcfbc772a22c012be5618e2976bf43fc84c1176e7305eb471f6b44ca9298a"}
Sep 16 02:37:44.325443 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:44.325431    2558 scope.go:117] "RemoveContainer" containerID="2abbcfbc772a22c012be5618e2976bf43fc84c1176e7305eb471f6b44ca9298a"
Sep 16 02:37:44.326686 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:44.326647    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 16 02:37:44.326902 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:44.326767    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 16 02:37:44.326902 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:44.326815    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 16 02:37:44.333540 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:44.333520    2558 scope.go:117] "RemoveContainer" containerID="324046f3600b748f7c3359c523e456400a772984af0c2afcaa162ffb9e3e84e8"
Sep 16 02:37:44.333847 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:37:44.333825    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"324046f3600b748f7c3359c523e456400a772984af0c2afcaa162ffb9e3e84e8\": container with ID starting with 324046f3600b748f7c3359c523e456400a772984af0c2afcaa162ffb9e3e84e8 not found: ID does not exist" containerID="324046f3600b748f7c3359c523e456400a772984af0c2afcaa162ffb9e3e84e8"
Sep 16 02:37:44.333920 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:44.333858    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"324046f3600b748f7c3359c523e456400a772984af0c2afcaa162ffb9e3e84e8"} err="failed to get container status \"324046f3600b748f7c3359c523e456400a772984af0c2afcaa162ffb9e3e84e8\": rpc error: code = NotFound desc = could not find container \"324046f3600b748f7c3359c523e456400a772984af0c2afcaa162ffb9e3e84e8\": container with ID starting with 324046f3600b748f7c3359c523e456400a772984af0c2afcaa162ffb9e3e84e8 not found: ID does not exist"
Sep 16 02:37:44.361053 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:44.361025    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-2e69f8df-openshift-default-6b45f469f6-v2kbk"]
Sep 16 02:37:44.364455 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:44.364430    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-2e69f8df-openshift-default-6b45f469f6-v2kbk"]
Sep 16 02:37:45.333000 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:45.332963    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-a53a44-openshift-default-75f7864f96-gh2ct" event={"ID":"1e8ae5f7-3d2a-4725-81c3-a2ae2fdaaaa2","Type":"ContainerStarted","Data":"8ce56a48cc53fc4f138764b7304ac5ef6a829e3271279499037c1777bbfab333"}
Sep 16 02:37:45.554429 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:45.554381    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5a9173c4-9b43-4203-8287-1b34dfd261b1" path="/var/lib/kubelet/pods/5a9173c4-9b43-4203-8287-1b34dfd261b1/volumes"
Sep 16 02:37:45.866842 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:45.866737    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-a53a44-openshift-default-75f7864f96-gh2ct"
Sep 16 02:37:45.867450 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:45.867423    2558 patch_prober.go:28] interesting pod/e2e-shared-b-w2-a53a44-openshift-default-75f7864f96-gh2ct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body=
Sep 16 02:37:45.867625 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:45.867596    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-a53a44-openshift-default-75f7864f96-gh2ct" podUID="1e8ae5f7-3d2a-4725-81c3-a2ae2fdaaaa2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Sep 16 02:37:46.867374 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:46.867330    2558 patch_prober.go:28] interesting pod/e2e-shared-b-w2-a53a44-openshift-default-75f7864f96-gh2ct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body=
Sep 16 02:37:46.867907 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:46.867399    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-a53a44-openshift-default-75f7864f96-gh2ct" podUID="1e8ae5f7-3d2a-4725-81c3-a2ae2fdaaaa2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Sep 16 02:37:47.867635 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:47.867600    2558 patch_prober.go:28] interesting pod/e2e-shared-b-w2-a53a44-openshift-default-75f7864f96-gh2ct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body=
Sep 16 02:37:47.868235 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:47.868188    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-a53a44-openshift-default-75f7864f96-gh2ct" podUID="1e8ae5f7-3d2a-4725-81c3-a2ae2fdaaaa2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Sep 16 02:37:48.866981 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:48.866935    2558 patch_prober.go:28] interesting pod/e2e-shared-b-w2-a53a44-openshift-default-75f7864f96-gh2ct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body=
Sep 16 02:37:48.867215 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:48.867013    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-a53a44-openshift-default-75f7864f96-gh2ct" podUID="1e8ae5f7-3d2a-4725-81c3-a2ae2fdaaaa2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Sep 16 02:37:49.867467 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:49.867421    2558 patch_prober.go:28] interesting pod/e2e-shared-b-w2-a53a44-openshift-default-75f7864f96-gh2ct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body=
Sep 16 02:37:49.868086 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:49.868049    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-a53a44-openshift-default-75f7864f96-gh2ct" podUID="1e8ae5f7-3d2a-4725-81c3-a2ae2fdaaaa2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Sep 16 02:37:50.911759 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:50.911720    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-a53a44-openshift-default-75f7864f96-gh2ct"
Sep 16 02:37:50.912256 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:50.912166    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-a53a44-openshift-default-75f7864f96-gh2ct"
Sep 16 02:37:50.913079 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:37:50.913057    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-a53a44-openshift-default-75f7864f96-gh2ct"
Sep 16 02:38:36.180664 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:38:36.180577    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-619550/embedding-w0-619550-kserve-7f858bb94-cs6ff"]
Sep 16 02:38:36.181231 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:38:36.181210    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="827a8b2e-8691-4577-a2b5-6f6b0f84aea9" containerName="payload-pre-processing"
Sep 16 02:38:36.181276 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:38:36.181236    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="827a8b2e-8691-4577-a2b5-6f6b0f84aea9" containerName="payload-pre-processing"
Sep 16 02:38:36.181276 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:38:36.181264    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89da9e44-caa5-4acf-9abb-90a220879254" containerName="payload-processing"
Sep 16 02:38:36.181276 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:38:36.181273    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="89da9e44-caa5-4acf-9abb-90a220879254" containerName="payload-processing"
Sep 16 02:38:36.181376 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:38:36.181288    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc3ab2d4-becc-4aa9-95ed-6f5cf8aec1e7" containerName="revoke-keys"
Sep 16 02:38:36.181376 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:38:36.181297    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc3ab2d4-becc-4aa9-95ed-6f5cf8aec1e7" containerName="revoke-keys"
Sep 16 02:38:36.181376 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:38:36.181313    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a9173c4-9b43-4203-8287-1b34dfd261b1" containerName="istio-proxy"
Sep 16 02:38:36.181376 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:38:36.181322    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a9173c4-9b43-4203-8287-1b34dfd261b1" containerName="istio-proxy"
Sep 16 02:38:36.181495 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:38:36.181428    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="5a9173c4-9b43-4203-8287-1b34dfd261b1" containerName="istio-proxy"
Sep 16 02:38:36.181495 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:38:36.181443    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="89da9e44-caa5-4acf-9abb-90a220879254" containerName="payload-processing"
Sep 16 02:38:36.181495 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:38:36.181453    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="827a8b2e-8691-4577-a2b5-6f6b0f84aea9" containerName="payload-pre-processing"
Sep 16 02:38:36.181495 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:38:36.181463    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="fc3ab2d4-becc-4aa9-95ed-6f5cf8aec1e7" containerName="revoke-keys"
Sep 16 02:38:36.185101 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:38:36.185078    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-619550/embedding-w0-619550-kserve-7f858bb94-cs6ff"
Sep 16 02:38:36.187386 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:38:36.187350    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-619550\"/\"kube-root-ca.crt\""
Sep 16 02:38:36.189061 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:38:36.189037    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-619550\"/\"default-dockercfg-s94hk\""
Sep 16 02:38:36.189291 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:38:36.189271    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-619550\"/\"embedding-w0-619550-kserve-self-signed-certs\""
Sep 16 02:38:36.189509 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:38:36.189490    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-619550\"/\"openshift-service-ca.crt\""
Sep 16 02:38:36.191864 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:38:36.191838    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-619550/embedding-w0-619550-kserve-7f858bb94-cs6ff"]
Sep 16 02:38:36.277450 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:38:36.277416    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a3c3acf6-e578-441b-acaa-ae1063718cd9-model-cache\") pod \"embedding-w0-619550-kserve-7f858bb94-cs6ff\" (UID: \"a3c3acf6-e578-441b-acaa-ae1063718cd9\") " pod="e2e-models-e2e-worker-w0-619550/embedding-w0-619550-kserve-7f858bb94-cs6ff"
Sep 16 02:38:36.277450 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:38:36.277455    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3c3acf6-e578-441b-acaa-ae1063718cd9-tmp-dir\") pod \"embedding-w0-619550-kserve-7f858bb94-cs6ff\" (UID: \"a3c3acf6-e578-441b-acaa-ae1063718cd9\") " pod="e2e-models-e2e-worker-w0-619550/embedding-w0-619550-kserve-7f858bb94-cs6ff"
Sep 16 02:38:36.277713 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:38:36.277552    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkb4b\" (UniqueName: \"kubernetes.io/projected/a3c3acf6-e578-441b-acaa-ae1063718cd9-kube-api-access-lkb4b\") pod \"embedding-w0-619550-kserve-7f858bb94-cs6ff\" (UID: \"a3c3acf6-e578-441b-acaa-ae1063718cd9\") " pod="e2e-models-e2e-worker-w0-619550/embedding-w0-619550-kserve-7f858bb94-cs6ff"
Sep 16 02:38:36.277713 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:38:36.277602    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a3c3acf6-e578-441b-acaa-ae1063718cd9-dshm\") pod \"embedding-w0-619550-kserve-7f858bb94-cs6ff\" (UID: \"a3c3acf6-e578-441b-acaa-ae1063718cd9\") " pod="e2e-models-e2e-worker-w0-619550/embedding-w0-619550-kserve-7f858bb94-cs6ff"
Sep 16 02:38:36.277713 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:38:36.277628    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3c3acf6-e578-441b-acaa-ae1063718cd9-tls-certs\") pod \"embedding-w0-619550-kserve-7f858bb94-cs6ff\" (UID: \"a3c3acf6-e578-441b-acaa-ae1063718cd9\") " pod="e2e-models-e2e-worker-w0-619550/embedding-w0-619550-kserve-7f858bb94-cs6ff"
Sep 16 02:38:36.277874 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:38:36.277741    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a3c3acf6-e578-441b-acaa-ae1063718cd9-home\") pod \"embedding-w0-619550-kserve-7f858bb94-cs6ff\" (UID: \"a3c3acf6-e578-441b-acaa-ae1063718cd9\") " pod="e2e-models-e2e-worker-w0-619550/embedding-w0-619550-kserve-7f858bb94-cs6ff"
Sep 16 02:38:36.378950 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:38:36.378913    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkb4b\" (UniqueName: \"kubernetes.io/projected/a3c3acf6-e578-441b-acaa-ae1063718cd9-kube-api-access-lkb4b\") pod \"embedding-w0-619550-kserve-7f858bb94-cs6ff\" (UID: \"a3c3acf6-e578-441b-acaa-ae1063718cd9\") " pod="e2e-models-e2e-worker-w0-619550/embedding-w0-619550-kserve-7f858bb94-cs6ff"
Sep 16 02:38:36.379141 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:38:36.378980    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a3c3acf6-e578-441b-acaa-ae1063718cd9-dshm\") pod \"embedding-w0-619550-kserve-7f858bb94-cs6ff\" (UID: \"a3c3acf6-e578-441b-acaa-ae1063718cd9\") " pod="e2e-models-e2e-worker-w0-619550/embedding-w0-619550-kserve-7f858bb94-cs6ff"
Sep 16 02:38:36.379141 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:38:36.379004    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3c3acf6-e578-441b-acaa-ae1063718cd9-tls-certs\") pod \"embedding-w0-619550-kserve-7f858bb94-cs6ff\" (UID: \"a3c3acf6-e578-441b-acaa-ae1063718cd9\") " pod="e2e-models-e2e-worker-w0-619550/embedding-w0-619550-kserve-7f858bb94-cs6ff"
Sep 16 02:38:36.379141 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:38:36.379065    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a3c3acf6-e578-441b-acaa-ae1063718cd9-home\") pod \"embedding-w0-619550-kserve-7f858bb94-cs6ff\" (UID: \"a3c3acf6-e578-441b-acaa-ae1063718cd9\") " pod="e2e-models-e2e-worker-w0-619550/embedding-w0-619550-kserve-7f858bb94-cs6ff"
Sep 16 02:38:36.379141 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:38:36.379096    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a3c3acf6-e578-441b-acaa-ae1063718cd9-model-cache\") pod \"embedding-w0-619550-kserve-7f858bb94-cs6ff\" (UID: \"a3c3acf6-e578-441b-acaa-ae1063718cd9\") " pod="e2e-models-e2e-worker-w0-619550/embedding-w0-619550-kserve-7f858bb94-cs6ff"
Sep 16 02:38:36.379141 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:38:36.379117    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3c3acf6-e578-441b-acaa-ae1063718cd9-tmp-dir\") pod \"embedding-w0-619550-kserve-7f858bb94-cs6ff\" (UID: \"a3c3acf6-e578-441b-acaa-ae1063718cd9\") " pod="e2e-models-e2e-worker-w0-619550/embedding-w0-619550-kserve-7f858bb94-cs6ff"
Sep 16 02:38:36.379564 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:38:36.379517    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a3c3acf6-e578-441b-acaa-ae1063718cd9-home\") pod \"embedding-w0-619550-kserve-7f858bb94-cs6ff\" (UID: \"a3c3acf6-e578-441b-acaa-ae1063718cd9\") " pod="e2e-models-e2e-worker-w0-619550/embedding-w0-619550-kserve-7f858bb94-cs6ff"
Sep 16 02:38:36.379564 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:38:36.379557    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a3c3acf6-e578-441b-acaa-ae1063718cd9-model-cache\") pod \"embedding-w0-619550-kserve-7f858bb94-cs6ff\" (UID: \"a3c3acf6-e578-441b-acaa-ae1063718cd9\") " pod="e2e-models-e2e-worker-w0-619550/embedding-w0-619550-kserve-7f858bb94-cs6ff"
Sep 16 02:38:36.379791 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:38:36.379573    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3c3acf6-e578-441b-acaa-ae1063718cd9-tmp-dir\") pod \"embedding-w0-619550-kserve-7f858bb94-cs6ff\" (UID: \"a3c3acf6-e578-441b-acaa-ae1063718cd9\") " pod="e2e-models-e2e-worker-w0-619550/embedding-w0-619550-kserve-7f858bb94-cs6ff"
Sep 16 02:38:36.381309 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:38:36.381283    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a3c3acf6-e578-441b-acaa-ae1063718cd9-dshm\") pod \"embedding-w0-619550-kserve-7f858bb94-cs6ff\" (UID: \"a3c3acf6-e578-441b-acaa-ae1063718cd9\") " pod="e2e-models-e2e-worker-w0-619550/embedding-w0-619550-kserve-7f858bb94-cs6ff"
Sep 16 02:38:36.381639 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:38:36.381620    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3c3acf6-e578-441b-acaa-ae1063718cd9-tls-certs\") pod \"embedding-w0-619550-kserve-7f858bb94-cs6ff\" (UID: \"a3c3acf6-e578-441b-acaa-ae1063718cd9\") " pod="e2e-models-e2e-worker-w0-619550/embedding-w0-619550-kserve-7f858bb94-cs6ff"
Sep 16 02:38:36.387173 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:38:36.387143    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkb4b\" (UniqueName: \"kubernetes.io/projected/a3c3acf6-e578-441b-acaa-ae1063718cd9-kube-api-access-lkb4b\") pod \"embedding-w0-619550-kserve-7f858bb94-cs6ff\" (UID: \"a3c3acf6-e578-441b-acaa-ae1063718cd9\") " pod="e2e-models-e2e-worker-w0-619550/embedding-w0-619550-kserve-7f858bb94-cs6ff"
Sep 16 02:38:36.498899 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:38:36.498799    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-619550/embedding-w0-619550-kserve-7f858bb94-cs6ff"
Sep 16 02:38:36.787535 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:38:36.787512    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-619550/embedding-w0-619550-kserve-7f858bb94-cs6ff"]
Sep 16 02:38:36.789727 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:38:36.789674    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda3c3acf6_e578_441b_acaa_ae1063718cd9.slice/crio-8b84c184091cb9af93652d75a1065989a43d6ac8bd7e75cd95321b13df1ebf25 WatchSource:0}: Error finding container 8b84c184091cb9af93652d75a1065989a43d6ac8bd7e75cd95321b13df1ebf25: Status 404 returned error can't find the container with id 8b84c184091cb9af93652d75a1065989a43d6ac8bd7e75cd95321b13df1ebf25
Sep 16 02:38:37.612008 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:38:37.611966    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-619550/embedding-w0-619550-kserve-7f858bb94-cs6ff" event={"ID":"a3c3acf6-e578-441b-acaa-ae1063718cd9","Type":"ContainerStarted","Data":"6f8614de46596ff199e36784d0e5f65521e0327d809781059b123cf22eb35994"}
Sep 16 02:38:37.612399 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:38:37.612017    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-619550/embedding-w0-619550-kserve-7f858bb94-cs6ff" event={"ID":"a3c3acf6-e578-441b-acaa-ae1063718cd9","Type":"ContainerStarted","Data":"8b84c184091cb9af93652d75a1065989a43d6ac8bd7e75cd95321b13df1ebf25"}
Sep 16 02:38:37.627828 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:38:37.627767    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-619550/embedding-w0-619550-kserve-7f858bb94-cs6ff" podStartSLOduration=1.6277470630000002 podStartE2EDuration="1.627747063s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 02:38:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:38:37.62557511 +0000 UTC m=+1530.544064353" watchObservedRunningTime="2026-09-16 02:38:37.627747063 +0000 UTC m=+1530.546236313"
Sep 16 02:38:40.544493 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:38:40.544453    2558 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-processing-5d7dbbd56b-pltkr" secret="" err="secret \"payload-processing-dockercfg-zzx6j\" not found"
Sep 16 02:38:46.499236 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:38:46.499193    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-619550/embedding-w0-619550-kserve-7f858bb94-cs6ff"
Sep 16 02:38:46.499749 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:38:46.499263    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-619550/embedding-w0-619550-kserve-7f858bb94-cs6ff"
Sep 16 02:38:46.511817 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:38:46.511790    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-619550/embedding-w0-619550-kserve-7f858bb94-cs6ff"
Sep 16 02:38:46.671204 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:38:46.671173    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-619550/embedding-w0-619550-kserve-7f858bb94-cs6ff"
Sep 16 02:39:16.756590 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:16.756553    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-542ed5-7gzgm"]
Sep 16 02:39:16.760410 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:16.760388    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-542ed5-7gzgm"
Sep 16 02:39:16.762466 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:16.762441    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 16 02:39:16.762588 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:16.762486    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jpd4f\""
Sep 16 02:39:16.767027 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:16.766948    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-542ed5-7gzgm"]
Sep 16 02:39:16.878542 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:16.878506    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2bb210c-599f-4fd3-8567-cd4ce69199ee-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-542ed5-7gzgm\" (UID: \"c2bb210c-599f-4fd3-8567-cd4ce69199ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-542ed5-7gzgm"
Sep 16 02:39:16.980043 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:16.979984    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2bb210c-599f-4fd3-8567-cd4ce69199ee-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-542ed5-7gzgm\" (UID: \"c2bb210c-599f-4fd3-8567-cd4ce69199ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-542ed5-7gzgm"
Sep 16 02:39:16.980617 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:16.980596    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2bb210c-599f-4fd3-8567-cd4ce69199ee-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-542ed5-7gzgm\" (UID: \"c2bb210c-599f-4fd3-8567-cd4ce69199ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-542ed5-7gzgm"
Sep 16 02:39:17.070560 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:17.070482    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-e415ca22-openshift-default-6f5546c844-vlld5"]
Sep 16 02:39:17.073189 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:17.073164    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-542ed5-7gzgm"
Sep 16 02:39:17.074208 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:17.074186    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e415ca22-openshift-default-6f5546c844-vlld5"
Sep 16 02:39:17.076536 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:17.076387    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-e415ca22-openshift-default-dockercfg-tvmjd\""
Sep 16 02:39:17.085450 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:17.085427    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-e415ca22-openshift-default-6f5546c844-vlld5"]
Sep 16 02:39:17.185380 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:17.185335    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73651cfe-d120-4ab1-a3be-d779025b1fae-istio-podinfo\") pod \"e2e-mt-e415ca22-openshift-default-6f5546c844-vlld5\" (UID: \"73651cfe-d120-4ab1-a3be-d779025b1fae\") " pod="openshift-ingress/e2e-mt-e415ca22-openshift-default-6f5546c844-vlld5"
Sep 16 02:39:17.185575 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:17.185396    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7fdv5\" (UniqueName: \"kubernetes.io/projected/73651cfe-d120-4ab1-a3be-d779025b1fae-kube-api-access-7fdv5\") pod \"e2e-mt-e415ca22-openshift-default-6f5546c844-vlld5\" (UID: \"73651cfe-d120-4ab1-a3be-d779025b1fae\") " pod="openshift-ingress/e2e-mt-e415ca22-openshift-default-6f5546c844-vlld5"
Sep 16 02:39:17.185575 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:17.185460    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73651cfe-d120-4ab1-a3be-d779025b1fae-istio-data\") pod \"e2e-mt-e415ca22-openshift-default-6f5546c844-vlld5\" (UID: \"73651cfe-d120-4ab1-a3be-d779025b1fae\") " pod="openshift-ingress/e2e-mt-e415ca22-openshift-default-6f5546c844-vlld5"
Sep 16 02:39:17.185575 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:17.185492    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73651cfe-d120-4ab1-a3be-d779025b1fae-workload-socket\") pod \"e2e-mt-e415ca22-openshift-default-6f5546c844-vlld5\" (UID: \"73651cfe-d120-4ab1-a3be-d779025b1fae\") " pod="openshift-ingress/e2e-mt-e415ca22-openshift-default-6f5546c844-vlld5"
Sep 16 02:39:17.185575 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:17.185532    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73651cfe-d120-4ab1-a3be-d779025b1fae-credential-socket\") pod \"e2e-mt-e415ca22-openshift-default-6f5546c844-vlld5\" (UID: \"73651cfe-d120-4ab1-a3be-d779025b1fae\") " pod="openshift-ingress/e2e-mt-e415ca22-openshift-default-6f5546c844-vlld5"
Sep 16 02:39:17.185575 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:17.185558    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73651cfe-d120-4ab1-a3be-d779025b1fae-workload-certs\") pod \"e2e-mt-e415ca22-openshift-default-6f5546c844-vlld5\" (UID: \"73651cfe-d120-4ab1-a3be-d779025b1fae\") " pod="openshift-ingress/e2e-mt-e415ca22-openshift-default-6f5546c844-vlld5"
Sep 16 02:39:17.185880 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:17.185625    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73651cfe-d120-4ab1-a3be-d779025b1fae-istiod-ca-cert\") pod \"e2e-mt-e415ca22-openshift-default-6f5546c844-vlld5\" (UID: \"73651cfe-d120-4ab1-a3be-d779025b1fae\") " pod="openshift-ingress/e2e-mt-e415ca22-openshift-default-6f5546c844-vlld5"
Sep 16 02:39:17.185880 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:17.185691    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73651cfe-d120-4ab1-a3be-d779025b1fae-istio-token\") pod \"e2e-mt-e415ca22-openshift-default-6f5546c844-vlld5\" (UID: \"73651cfe-d120-4ab1-a3be-d779025b1fae\") " pod="openshift-ingress/e2e-mt-e415ca22-openshift-default-6f5546c844-vlld5"
Sep 16 02:39:17.185880 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:17.185747    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73651cfe-d120-4ab1-a3be-d779025b1fae-istio-envoy\") pod \"e2e-mt-e415ca22-openshift-default-6f5546c844-vlld5\" (UID: \"73651cfe-d120-4ab1-a3be-d779025b1fae\") " pod="openshift-ingress/e2e-mt-e415ca22-openshift-default-6f5546c844-vlld5"
Sep 16 02:39:17.287020 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:17.286980    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73651cfe-d120-4ab1-a3be-d779025b1fae-istio-podinfo\") pod \"e2e-mt-e415ca22-openshift-default-6f5546c844-vlld5\" (UID: \"73651cfe-d120-4ab1-a3be-d779025b1fae\") " pod="openshift-ingress/e2e-mt-e415ca22-openshift-default-6f5546c844-vlld5"
Sep 16 02:39:17.287020 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:17.287024    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7fdv5\" (UniqueName: \"kubernetes.io/projected/73651cfe-d120-4ab1-a3be-d779025b1fae-kube-api-access-7fdv5\") pod \"e2e-mt-e415ca22-openshift-default-6f5546c844-vlld5\" (UID: \"73651cfe-d120-4ab1-a3be-d779025b1fae\") " pod="openshift-ingress/e2e-mt-e415ca22-openshift-default-6f5546c844-vlld5"
Sep 16 02:39:17.287283 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:17.287079    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73651cfe-d120-4ab1-a3be-d779025b1fae-istio-data\") pod \"e2e-mt-e415ca22-openshift-default-6f5546c844-vlld5\" (UID: \"73651cfe-d120-4ab1-a3be-d779025b1fae\") " pod="openshift-ingress/e2e-mt-e415ca22-openshift-default-6f5546c844-vlld5"
Sep 16 02:39:17.287283 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:17.287107    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73651cfe-d120-4ab1-a3be-d779025b1fae-workload-socket\") pod \"e2e-mt-e415ca22-openshift-default-6f5546c844-vlld5\" (UID: \"73651cfe-d120-4ab1-a3be-d779025b1fae\") " pod="openshift-ingress/e2e-mt-e415ca22-openshift-default-6f5546c844-vlld5"
Sep 16 02:39:17.287283 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:17.287141    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73651cfe-d120-4ab1-a3be-d779025b1fae-credential-socket\") pod \"e2e-mt-e415ca22-openshift-default-6f5546c844-vlld5\" (UID: \"73651cfe-d120-4ab1-a3be-d779025b1fae\") " pod="openshift-ingress/e2e-mt-e415ca22-openshift-default-6f5546c844-vlld5"
Sep 16 02:39:17.287283 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:17.287168    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73651cfe-d120-4ab1-a3be-d779025b1fae-workload-certs\") pod \"e2e-mt-e415ca22-openshift-default-6f5546c844-vlld5\" (UID: \"73651cfe-d120-4ab1-a3be-d779025b1fae\") " pod="openshift-ingress/e2e-mt-e415ca22-openshift-default-6f5546c844-vlld5"
Sep 16 02:39:17.287283 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:17.287225    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73651cfe-d120-4ab1-a3be-d779025b1fae-istiod-ca-cert\") pod \"e2e-mt-e415ca22-openshift-default-6f5546c844-vlld5\" (UID: \"73651cfe-d120-4ab1-a3be-d779025b1fae\") " pod="openshift-ingress/e2e-mt-e415ca22-openshift-default-6f5546c844-vlld5"
Sep 16 02:39:17.287542 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:17.287286    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73651cfe-d120-4ab1-a3be-d779025b1fae-istio-token\") pod \"e2e-mt-e415ca22-openshift-default-6f5546c844-vlld5\" (UID: \"73651cfe-d120-4ab1-a3be-d779025b1fae\") " pod="openshift-ingress/e2e-mt-e415ca22-openshift-default-6f5546c844-vlld5"
Sep 16 02:39:17.287542 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:17.287320    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73651cfe-d120-4ab1-a3be-d779025b1fae-istio-envoy\") pod \"e2e-mt-e415ca22-openshift-default-6f5546c844-vlld5\" (UID: \"73651cfe-d120-4ab1-a3be-d779025b1fae\") " pod="openshift-ingress/e2e-mt-e415ca22-openshift-default-6f5546c844-vlld5"
Sep 16 02:39:17.287646 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:17.287560    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73651cfe-d120-4ab1-a3be-d779025b1fae-credential-socket\") pod \"e2e-mt-e415ca22-openshift-default-6f5546c844-vlld5\" (UID: \"73651cfe-d120-4ab1-a3be-d779025b1fae\") " pod="openshift-ingress/e2e-mt-e415ca22-openshift-default-6f5546c844-vlld5"
Sep 16 02:39:17.287646 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:17.287602    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73651cfe-d120-4ab1-a3be-d779025b1fae-workload-socket\") pod \"e2e-mt-e415ca22-openshift-default-6f5546c844-vlld5\" (UID: \"73651cfe-d120-4ab1-a3be-d779025b1fae\") " pod="openshift-ingress/e2e-mt-e415ca22-openshift-default-6f5546c844-vlld5"
Sep 16 02:39:17.287770 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:17.287642    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73651cfe-d120-4ab1-a3be-d779025b1fae-istio-data\") pod \"e2e-mt-e415ca22-openshift-default-6f5546c844-vlld5\" (UID: \"73651cfe-d120-4ab1-a3be-d779025b1fae\") " pod="openshift-ingress/e2e-mt-e415ca22-openshift-default-6f5546c844-vlld5"
Sep 16 02:39:17.287770 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:17.287681    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73651cfe-d120-4ab1-a3be-d779025b1fae-workload-certs\") pod \"e2e-mt-e415ca22-openshift-default-6f5546c844-vlld5\" (UID: \"73651cfe-d120-4ab1-a3be-d779025b1fae\") " pod="openshift-ingress/e2e-mt-e415ca22-openshift-default-6f5546c844-vlld5"
Sep 16 02:39:17.288012 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:17.287991    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73651cfe-d120-4ab1-a3be-d779025b1fae-istiod-ca-cert\") pod \"e2e-mt-e415ca22-openshift-default-6f5546c844-vlld5\" (UID: \"73651cfe-d120-4ab1-a3be-d779025b1fae\") " pod="openshift-ingress/e2e-mt-e415ca22-openshift-default-6f5546c844-vlld5"
Sep 16 02:39:17.289610 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:17.289582    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73651cfe-d120-4ab1-a3be-d779025b1fae-istio-envoy\") pod \"e2e-mt-e415ca22-openshift-default-6f5546c844-vlld5\" (UID: \"73651cfe-d120-4ab1-a3be-d779025b1fae\") " pod="openshift-ingress/e2e-mt-e415ca22-openshift-default-6f5546c844-vlld5"
Sep 16 02:39:17.289904 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:17.289885    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73651cfe-d120-4ab1-a3be-d779025b1fae-istio-podinfo\") pod \"e2e-mt-e415ca22-openshift-default-6f5546c844-vlld5\" (UID: \"73651cfe-d120-4ab1-a3be-d779025b1fae\") " pod="openshift-ingress/e2e-mt-e415ca22-openshift-default-6f5546c844-vlld5"
Sep 16 02:39:17.299784 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:17.299349    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73651cfe-d120-4ab1-a3be-d779025b1fae-istio-token\") pod \"e2e-mt-e415ca22-openshift-default-6f5546c844-vlld5\" (UID: \"73651cfe-d120-4ab1-a3be-d779025b1fae\") " pod="openshift-ingress/e2e-mt-e415ca22-openshift-default-6f5546c844-vlld5"
Sep 16 02:39:17.301954 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:17.300870    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7fdv5\" (UniqueName: \"kubernetes.io/projected/73651cfe-d120-4ab1-a3be-d779025b1fae-kube-api-access-7fdv5\") pod \"e2e-mt-e415ca22-openshift-default-6f5546c844-vlld5\" (UID: \"73651cfe-d120-4ab1-a3be-d779025b1fae\") " pod="openshift-ingress/e2e-mt-e415ca22-openshift-default-6f5546c844-vlld5"
Sep 16 02:39:17.348786 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:17.348762    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-542ed5-7gzgm"]
Sep 16 02:39:17.351118 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:39:17.351091    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2bb210c_599f_4fd3_8567_cd4ce69199ee.slice/crio-7a904c7964eecdfa34f70f73477a73ba484946b3a15ebfeb1ac76937157169d2 WatchSource:0}: Error finding container 7a904c7964eecdfa34f70f73477a73ba484946b3a15ebfeb1ac76937157169d2: Status 404 returned error can't find the container with id 7a904c7964eecdfa34f70f73477a73ba484946b3a15ebfeb1ac76937157169d2
Sep 16 02:39:17.407963 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:17.407933    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e415ca22-openshift-default-6f5546c844-vlld5"
Sep 16 02:39:17.680437 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:17.680409    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-e415ca22-openshift-default-6f5546c844-vlld5"]
Sep 16 02:39:17.683518 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:39:17.683493    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73651cfe_d120_4ab1_a3be_d779025b1fae.slice/crio-e4f0f11ec45646c3e3e9e79dd3b3f4ba309ddd6682f67e851363c0d8b900fe18 WatchSource:0}: Error finding container e4f0f11ec45646c3e3e9e79dd3b3f4ba309ddd6682f67e851363c0d8b900fe18: Status 404 returned error can't find the container with id e4f0f11ec45646c3e3e9e79dd3b3f4ba309ddd6682f67e851363c0d8b900fe18
Sep 16 02:39:17.685836 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:17.685806    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 16 02:39:17.685914 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:17.685868    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 16 02:39:17.685914 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:17.685899    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 16 02:39:17.822498 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:17.822456    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e415ca22-openshift-default-6f5546c844-vlld5" event={"ID":"73651cfe-d120-4ab1-a3be-d779025b1fae","Type":"ContainerStarted","Data":"a17399c6b2d72e6210577ce30d23b66d15e0131e45e9403cc52f560869e05d44"}
Sep 16 02:39:17.822959 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:17.822524    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e415ca22-openshift-default-6f5546c844-vlld5" event={"ID":"73651cfe-d120-4ab1-a3be-d779025b1fae","Type":"ContainerStarted","Data":"e4f0f11ec45646c3e3e9e79dd3b3f4ba309ddd6682f67e851363c0d8b900fe18"}
Sep 16 02:39:17.824243 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:17.824219    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-542ed5-7gzgm" event={"ID":"c2bb210c-599f-4fd3-8567-cd4ce69199ee","Type":"ContainerStarted","Data":"fd638446489c61ca8f37b0ca7264e66c3e0d458f329e82e4c90daafd7d7361ef"}
Sep 16 02:39:17.824357 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:17.824262    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-542ed5-7gzgm" event={"ID":"c2bb210c-599f-4fd3-8567-cd4ce69199ee","Type":"ContainerStarted","Data":"7a904c7964eecdfa34f70f73477a73ba484946b3a15ebfeb1ac76937157169d2"}
Sep 16 02:39:17.844560 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:17.844481    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-e415ca22-openshift-default-6f5546c844-vlld5" podStartSLOduration=0.844460326 podStartE2EDuration="844.460326ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 02:39:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:39:17.841472345 +0000 UTC m=+1570.759961606" watchObservedRunningTime="2026-09-16 02:39:17.844460326 +0000 UTC m=+1570.762949576"
Sep 16 02:39:17.854835 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:17.854779    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-542ed5-7gzgm" podStartSLOduration=1.854761125 podStartE2EDuration="1.854761125s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 02:39:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:39:17.853677625 +0000 UTC m=+1570.772166875" watchObservedRunningTime="2026-09-16 02:39:17.854761125 +0000 UTC m=+1570.773250375"
Sep 16 02:39:18.408879 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:18.408846    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-e415ca22-openshift-default-6f5546c844-vlld5"
Sep 16 02:39:18.414180 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:18.414147    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-e415ca22-openshift-default-6f5546c844-vlld5"
Sep 16 02:39:18.831505 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:18.831412    2558 generic.go:356] "Generic (PLEG): container finished" podID="c2bb210c-599f-4fd3-8567-cd4ce69199ee" containerID="fd638446489c61ca8f37b0ca7264e66c3e0d458f329e82e4c90daafd7d7361ef" exitCode=0
Sep 16 02:39:18.831986 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:18.831508    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-542ed5-7gzgm" event={"ID":"c2bb210c-599f-4fd3-8567-cd4ce69199ee","Type":"ContainerDied","Data":"fd638446489c61ca8f37b0ca7264e66c3e0d458f329e82e4c90daafd7d7361ef"}
Sep 16 02:39:18.831986 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:18.831770    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-e415ca22-openshift-default-6f5546c844-vlld5"
Sep 16 02:39:18.832744 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:18.832725    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-e415ca22-openshift-default-6f5546c844-vlld5"
Sep 16 02:39:19.971877 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:19.971854    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-542ed5-7gzgm"
Sep 16 02:39:20.022283 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:20.022247    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2bb210c-599f-4fd3-8567-cd4ce69199ee-maas-api-service-ca\") pod \"c2bb210c-599f-4fd3-8567-cd4ce69199ee\" (UID: \"c2bb210c-599f-4fd3-8567-cd4ce69199ee\") "
Sep 16 02:39:20.022622 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:20.022599    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2bb210c-599f-4fd3-8567-cd4ce69199ee-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c2bb210c-599f-4fd3-8567-cd4ce69199ee" (UID: "c2bb210c-599f-4fd3-8567-cd4ce69199ee"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 02:39:20.123511 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:20.123433    2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2bb210c-599f-4fd3-8567-cd4ce69199ee-maas-api-service-ca\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:39:20.843298 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:20.843266    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-542ed5-7gzgm"
Sep 16 02:39:20.843479 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:20.843297    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-542ed5-7gzgm" event={"ID":"c2bb210c-599f-4fd3-8567-cd4ce69199ee","Type":"ContainerDied","Data":"7a904c7964eecdfa34f70f73477a73ba484946b3a15ebfeb1ac76937157169d2"}
Sep 16 02:39:20.843479 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:20.843345    2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7a904c7964eecdfa34f70f73477a73ba484946b3a15ebfeb1ac76937157169d2"
Sep 16 02:39:21.586550 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:21.586519    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-542ed5/distinct-2-w3-542ed5-kserve-57dc75646d-x4qrb"]
Sep 16 02:39:23.195906 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:23.195869    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e415ca22-67b7f75bd6-kn49p"]
Sep 16 02:39:23.196276 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:23.196261    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c2bb210c-599f-4fd3-8567-cd4ce69199ee" containerName="revoke-keys"
Sep 16 02:39:23.196276 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:23.196272    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="c2bb210c-599f-4fd3-8567-cd4ce69199ee" containerName="revoke-keys"
Sep 16 02:39:23.196367 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:23.196357    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="c2bb210c-599f-4fd3-8567-cd4ce69199ee" containerName="revoke-keys"
Sep 16 02:39:23.199561 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:23.199544    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-e415ca22-67b7f75bd6-kn49p"
Sep 16 02:39:23.201689 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:23.201669    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-e415ca22\""
Sep 16 02:39:23.201829 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:23.201810    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-e415ca22-dockercfg-fltzm\""
Sep 16 02:39:23.210102 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:23.210078    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e415ca22-67b7f75bd6-kn49p"]
Sep 16 02:39:23.229755 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:23.229727    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-e415ca22-59d4f8f697-bgn5c"]
Sep 16 02:39:23.234798 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:23.234776    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-e415ca22-59d4f8f697-bgn5c"
Sep 16 02:39:23.243091 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:23.242999    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-e415ca22-59d4f8f697-bgn5c"]
Sep 16 02:39:23.253164 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:23.253135    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q82hj\" (UniqueName: \"kubernetes.io/projected/08f5eb62-09b3-4f64-aa9c-9b2cb5ebe430-kube-api-access-q82hj\") pod \"payload-pre-processing-e2e-mt-e415ca22-67b7f75bd6-kn49p\" (UID: \"08f5eb62-09b3-4f64-aa9c-9b2cb5ebe430\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e415ca22-67b7f75bd6-kn49p"
Sep 16 02:39:23.253292 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:23.253274    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08f5eb62-09b3-4f64-aa9c-9b2cb5ebe430-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-e415ca22-67b7f75bd6-kn49p\" (UID: \"08f5eb62-09b3-4f64-aa9c-9b2cb5ebe430\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e415ca22-67b7f75bd6-kn49p"
Sep 16 02:39:23.354741 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:23.354679    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08f5eb62-09b3-4f64-aa9c-9b2cb5ebe430-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-e415ca22-67b7f75bd6-kn49p\" (UID: \"08f5eb62-09b3-4f64-aa9c-9b2cb5ebe430\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e415ca22-67b7f75bd6-kn49p"
Sep 16 02:39:23.354931 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:23.354776    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c03643a4-41ac-46bc-8685-c85ad0bc49a5-plugins-config-volume\") pod \"payload-processing-e2e-mt-e415ca22-59d4f8f697-bgn5c\" (UID: \"c03643a4-41ac-46bc-8685-c85ad0bc49a5\") " pod="openshift-ingress/payload-processing-e2e-mt-e415ca22-59d4f8f697-bgn5c"
Sep 16 02:39:23.354931 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:23.354834    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q82hj\" (UniqueName: \"kubernetes.io/projected/08f5eb62-09b3-4f64-aa9c-9b2cb5ebe430-kube-api-access-q82hj\") pod \"payload-pre-processing-e2e-mt-e415ca22-67b7f75bd6-kn49p\" (UID: \"08f5eb62-09b3-4f64-aa9c-9b2cb5ebe430\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e415ca22-67b7f75bd6-kn49p"
Sep 16 02:39:23.354931 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:23.354852    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ln24h\" (UniqueName: \"kubernetes.io/projected/c03643a4-41ac-46bc-8685-c85ad0bc49a5-kube-api-access-ln24h\") pod \"payload-processing-e2e-mt-e415ca22-59d4f8f697-bgn5c\" (UID: \"c03643a4-41ac-46bc-8685-c85ad0bc49a5\") " pod="openshift-ingress/payload-processing-e2e-mt-e415ca22-59d4f8f697-bgn5c"
Sep 16 02:39:23.355462 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:23.355435    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08f5eb62-09b3-4f64-aa9c-9b2cb5ebe430-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-e415ca22-67b7f75bd6-kn49p\" (UID: \"08f5eb62-09b3-4f64-aa9c-9b2cb5ebe430\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e415ca22-67b7f75bd6-kn49p"
Sep 16 02:39:23.362270 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:23.362246    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q82hj\" (UniqueName: \"kubernetes.io/projected/08f5eb62-09b3-4f64-aa9c-9b2cb5ebe430-kube-api-access-q82hj\") pod \"payload-pre-processing-e2e-mt-e415ca22-67b7f75bd6-kn49p\" (UID: \"08f5eb62-09b3-4f64-aa9c-9b2cb5ebe430\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e415ca22-67b7f75bd6-kn49p"
Sep 16 02:39:23.455610 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:23.455529    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c03643a4-41ac-46bc-8685-c85ad0bc49a5-plugins-config-volume\") pod \"payload-processing-e2e-mt-e415ca22-59d4f8f697-bgn5c\" (UID: \"c03643a4-41ac-46bc-8685-c85ad0bc49a5\") " pod="openshift-ingress/payload-processing-e2e-mt-e415ca22-59d4f8f697-bgn5c"
Sep 16 02:39:23.455610 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:23.455596    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ln24h\" (UniqueName: \"kubernetes.io/projected/c03643a4-41ac-46bc-8685-c85ad0bc49a5-kube-api-access-ln24h\") pod \"payload-processing-e2e-mt-e415ca22-59d4f8f697-bgn5c\" (UID: \"c03643a4-41ac-46bc-8685-c85ad0bc49a5\") " pod="openshift-ingress/payload-processing-e2e-mt-e415ca22-59d4f8f697-bgn5c"
Sep 16 02:39:23.456233 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:23.456213    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c03643a4-41ac-46bc-8685-c85ad0bc49a5-plugins-config-volume\") pod \"payload-processing-e2e-mt-e415ca22-59d4f8f697-bgn5c\" (UID: \"c03643a4-41ac-46bc-8685-c85ad0bc49a5\") " pod="openshift-ingress/payload-processing-e2e-mt-e415ca22-59d4f8f697-bgn5c"
Sep 16 02:39:23.463069 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:23.463041    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ln24h\" (UniqueName: \"kubernetes.io/projected/c03643a4-41ac-46bc-8685-c85ad0bc49a5-kube-api-access-ln24h\") pod \"payload-processing-e2e-mt-e415ca22-59d4f8f697-bgn5c\" (UID: \"c03643a4-41ac-46bc-8685-c85ad0bc49a5\") " pod="openshift-ingress/payload-processing-e2e-mt-e415ca22-59d4f8f697-bgn5c"
Sep 16 02:39:23.510951 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:23.510913    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-e415ca22-67b7f75bd6-kn49p"
Sep 16 02:39:23.547945 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:23.547908    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-e415ca22-59d4f8f697-bgn5c"
Sep 16 02:39:23.780272 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:23.780173    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e415ca22-67b7f75bd6-kn49p"]
Sep 16 02:39:23.782627 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:39:23.782598    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08f5eb62_09b3_4f64_aa9c_9b2cb5ebe430.slice/crio-69a99cef5de5af4aca50ff63a0fceee16a314370017237dc6b763b71916b7e30 WatchSource:0}: Error finding container 69a99cef5de5af4aca50ff63a0fceee16a314370017237dc6b763b71916b7e30: Status 404 returned error can't find the container with id 69a99cef5de5af4aca50ff63a0fceee16a314370017237dc6b763b71916b7e30
Sep 16 02:39:23.829084 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:23.829054    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-e415ca22-59d4f8f697-bgn5c"]
Sep 16 02:39:23.832833 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:39:23.832806    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc03643a4_41ac_46bc_8685_c85ad0bc49a5.slice/crio-10db080ebf5a39510b157a6b564f4233cf4e7302826fbc02c93c5b9f62d10aee WatchSource:0}: Error finding container 10db080ebf5a39510b157a6b564f4233cf4e7302826fbc02c93c5b9f62d10aee: Status 404 returned error can't find the container with id 10db080ebf5a39510b157a6b564f4233cf4e7302826fbc02c93c5b9f62d10aee
Sep 16 02:39:23.860502 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:23.860476    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-e415ca22-59d4f8f697-bgn5c" event={"ID":"c03643a4-41ac-46bc-8685-c85ad0bc49a5","Type":"ContainerStarted","Data":"10db080ebf5a39510b157a6b564f4233cf4e7302826fbc02c93c5b9f62d10aee"}
Sep 16 02:39:23.862128 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:23.862105    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-e415ca22-67b7f75bd6-kn49p" event={"ID":"08f5eb62-09b3-4f64-aa9c-9b2cb5ebe430","Type":"ContainerStarted","Data":"c0781b810a269139118704766de2456a4389cecb9b53888de05825745b85064f"}
Sep 16 02:39:23.862253 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:23.862145    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-e415ca22-67b7f75bd6-kn49p" event={"ID":"08f5eb62-09b3-4f64-aa9c-9b2cb5ebe430","Type":"ContainerStarted","Data":"69a99cef5de5af4aca50ff63a0fceee16a314370017237dc6b763b71916b7e30"}
Sep 16 02:39:23.862253 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:23.862212    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-e415ca22-67b7f75bd6-kn49p"
Sep 16 02:39:23.879308 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:23.879267    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-e415ca22-67b7f75bd6-kn49p" podStartSLOduration=0.879253168 podStartE2EDuration="879.253168ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 02:39:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:39:23.878043562 +0000 UTC m=+1576.796532815" watchObservedRunningTime="2026-09-16 02:39:23.879253168 +0000 UTC m=+1576.797742416"
Sep 16 02:39:24.869204 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:24.869168    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-e415ca22-59d4f8f697-bgn5c" event={"ID":"c03643a4-41ac-46bc-8685-c85ad0bc49a5","Type":"ContainerStarted","Data":"09c9bc1c186c8ac498a49b82a568a3f21c040858e8797babaf5d89334bf08b94"}
Sep 16 02:39:24.869719 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:24.869239    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-e415ca22-59d4f8f697-bgn5c"
Sep 16 02:39:24.886857 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:24.886790    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-e415ca22-59d4f8f697-bgn5c" podStartSLOduration=1.886767578 podStartE2EDuration="1.886767578s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 02:39:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:39:24.884522282 +0000 UTC m=+1577.803011533" watchObservedRunningTime="2026-09-16 02:39:24.886767578 +0000 UTC m=+1577.805256827"
Sep 16 02:39:32.635256 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:32.635224    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-542ed5-6bc6cbdb75-dlvtl"]
Sep 16 02:39:32.635753 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:32.635428    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w3-542ed5-6bc6cbdb75-dlvtl" podUID="6833535b-d82b-40b4-8f53-ebf02c5214be" containerName="payload-processing" containerID="cri-o://0d9c8ad9ad972d871297062de3f7e828b0d238bfa83c4e0d6c3cfaff9f290b15" gracePeriod=30
Sep 16 02:39:32.785888 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:32.785851    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-542ed5-7gzgm"]
Sep 16 02:39:32.788111 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:32.788083    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-542ed5-7gzgm"]
Sep 16 02:39:32.916089 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:32.916054    2558 generic.go:356] "Generic (PLEG): container finished" podID="6833535b-d82b-40b4-8f53-ebf02c5214be" containerID="0d9c8ad9ad972d871297062de3f7e828b0d238bfa83c4e0d6c3cfaff9f290b15" exitCode=0
Sep 16 02:39:32.916224 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:32.916129    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-542ed5-6bc6cbdb75-dlvtl" event={"ID":"6833535b-d82b-40b4-8f53-ebf02c5214be","Type":"ContainerDied","Data":"0d9c8ad9ad972d871297062de3f7e828b0d238bfa83c4e0d6c3cfaff9f290b15"}
Sep 16 02:39:32.916282 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:32.916231    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-542ed5-6bc6cbdb75-dlvtl" event={"ID":"6833535b-d82b-40b4-8f53-ebf02c5214be","Type":"ContainerDied","Data":"eff1e74ed7f9bdc653a5ffe90536c107b30caba7a660f3079c1b41974358ec5b"}
Sep 16 02:39:32.916282 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:32.916256    2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eff1e74ed7f9bdc653a5ffe90536c107b30caba7a660f3079c1b41974358ec5b"
Sep 16 02:39:32.929540 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:32.929516    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-542ed5-6bc6cbdb75-dlvtl"
Sep 16 02:39:33.066833 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:33.066797    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6833535b-d82b-40b4-8f53-ebf02c5214be-plugins-config-volume\") pod \"6833535b-d82b-40b4-8f53-ebf02c5214be\" (UID: \"6833535b-d82b-40b4-8f53-ebf02c5214be\") "
Sep 16 02:39:33.067027 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:33.066874    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c56k7\" (UniqueName: \"kubernetes.io/projected/6833535b-d82b-40b4-8f53-ebf02c5214be-kube-api-access-c56k7\") pod \"6833535b-d82b-40b4-8f53-ebf02c5214be\" (UID: \"6833535b-d82b-40b4-8f53-ebf02c5214be\") "
Sep 16 02:39:33.067189 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:33.067164    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6833535b-d82b-40b4-8f53-ebf02c5214be-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6833535b-d82b-40b4-8f53-ebf02c5214be" (UID: "6833535b-d82b-40b4-8f53-ebf02c5214be"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 02:39:33.068923 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:33.068899    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6833535b-d82b-40b4-8f53-ebf02c5214be-kube-api-access-c56k7" (OuterVolumeSpecName: "kube-api-access-c56k7") pod "6833535b-d82b-40b4-8f53-ebf02c5214be" (UID: "6833535b-d82b-40b4-8f53-ebf02c5214be"). InnerVolumeSpecName "kube-api-access-c56k7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 02:39:33.168220 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:33.168142    2558 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6833535b-d82b-40b4-8f53-ebf02c5214be-plugins-config-volume\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:39:33.168220 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:33.168178    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c56k7\" (UniqueName: \"kubernetes.io/projected/6833535b-d82b-40b4-8f53-ebf02c5214be-kube-api-access-c56k7\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:39:33.551199 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:33.551114    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c2bb210c-599f-4fd3-8567-cd4ce69199ee" path="/var/lib/kubelet/pods/c2bb210c-599f-4fd3-8567-cd4ce69199ee/volumes"
Sep 16 02:39:33.921459 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:33.921428    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-542ed5-6bc6cbdb75-dlvtl"
Sep 16 02:39:33.944277 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:33.944209    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-542ed5-6bc6cbdb75-dlvtl"]
Sep 16 02:39:33.946539 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:33.946510    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-542ed5-6bc6cbdb75-dlvtl"]
Sep 16 02:39:34.871882 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:34.871855    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-e415ca22-67b7f75bd6-kn49p"
Sep 16 02:39:35.550150 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:35.550117    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6833535b-d82b-40b4-8f53-ebf02c5214be" path="/var/lib/kubelet/pods/6833535b-d82b-40b4-8f53-ebf02c5214be/volumes"
Sep 16 02:39:35.878026 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:35.877948    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-e415ca22-59d4f8f697-bgn5c"
Sep 16 02:39:36.384642 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:36.384594    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-619550-lg77r"]
Sep 16 02:39:36.385037 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:36.385022    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6833535b-d82b-40b4-8f53-ebf02c5214be" containerName="payload-processing"
Sep 16 02:39:36.385037 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:36.385038    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="6833535b-d82b-40b4-8f53-ebf02c5214be" containerName="payload-processing"
Sep 16 02:39:36.385122 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:36.385106    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="6833535b-d82b-40b4-8f53-ebf02c5214be" containerName="payload-processing"
Sep 16 02:39:36.389431 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:36.389414    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-619550-lg77r"
Sep 16 02:39:36.391584 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:36.391556    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 16 02:39:36.391782 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:36.391611    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jpd4f\""
Sep 16 02:39:36.395149 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:36.395126    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-619550-lg77r"]
Sep 16 02:39:36.499630 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:36.499593    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2fe11042-4069-4f91-9f0d-1cbc33028f77-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-619550-lg77r\" (UID: \"2fe11042-4069-4f91-9f0d-1cbc33028f77\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-619550-lg77r"
Sep 16 02:39:36.601215 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:36.601176    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2fe11042-4069-4f91-9f0d-1cbc33028f77-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-619550-lg77r\" (UID: \"2fe11042-4069-4f91-9f0d-1cbc33028f77\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-619550-lg77r"
Sep 16 02:39:36.601843 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:36.601823    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2fe11042-4069-4f91-9f0d-1cbc33028f77-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-619550-lg77r\" (UID: \"2fe11042-4069-4f91-9f0d-1cbc33028f77\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-619550-lg77r"
Sep 16 02:39:36.606305 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:36.606275    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-542ed5/distinct-2-w3-542ed5-kserve-57dc75646d-x4qrb" podUID="ff6a30ad-287e-40df-aab7-455812130778" containerName="main" containerID="cri-o://e38a085d80dd4fb1b8cae4d9398b5031570c400cce6c254591c9c5106c3e958d" gracePeriod=45
Sep 16 02:39:36.702928 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:36.702900    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-619550-lg77r"
Sep 16 02:39:36.873470 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:36.873436    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-542ed5/distinct-2-w3-542ed5-kserve-57dc75646d-x4qrb"
Sep 16 02:39:36.940273 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:36.940236    2558 generic.go:356] "Generic (PLEG): container finished" podID="ff6a30ad-287e-40df-aab7-455812130778" containerID="e38a085d80dd4fb1b8cae4d9398b5031570c400cce6c254591c9c5106c3e958d" exitCode=0
Sep 16 02:39:36.940273 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:36.940272    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-542ed5/distinct-2-w3-542ed5-kserve-57dc75646d-x4qrb"
Sep 16 02:39:36.940470 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:36.940311    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-542ed5/distinct-2-w3-542ed5-kserve-57dc75646d-x4qrb" event={"ID":"ff6a30ad-287e-40df-aab7-455812130778","Type":"ContainerDied","Data":"e38a085d80dd4fb1b8cae4d9398b5031570c400cce6c254591c9c5106c3e958d"}
Sep 16 02:39:36.940470 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:36.940356    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-542ed5/distinct-2-w3-542ed5-kserve-57dc75646d-x4qrb" event={"ID":"ff6a30ad-287e-40df-aab7-455812130778","Type":"ContainerDied","Data":"2be640db84bd7689f1f8c19feb1e9db2aeb7cd2214d866bed98176e04b6a19c1"}
Sep 16 02:39:36.940470 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:36.940373    2558 scope.go:117] "RemoveContainer" containerID="e38a085d80dd4fb1b8cae4d9398b5031570c400cce6c254591c9c5106c3e958d"
Sep 16 02:39:36.950815 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:36.950792    2558 scope.go:117] "RemoveContainer" containerID="e38a085d80dd4fb1b8cae4d9398b5031570c400cce6c254591c9c5106c3e958d"
Sep 16 02:39:36.951178 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:39:36.951147    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e38a085d80dd4fb1b8cae4d9398b5031570c400cce6c254591c9c5106c3e958d\": container with ID starting with e38a085d80dd4fb1b8cae4d9398b5031570c400cce6c254591c9c5106c3e958d not found: ID does not exist" containerID="e38a085d80dd4fb1b8cae4d9398b5031570c400cce6c254591c9c5106c3e958d"
Sep 16 02:39:36.951313 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:36.951192    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e38a085d80dd4fb1b8cae4d9398b5031570c400cce6c254591c9c5106c3e958d"} err="failed to get container status \"e38a085d80dd4fb1b8cae4d9398b5031570c400cce6c254591c9c5106c3e958d\": rpc error: code = NotFound desc = could not find container \"e38a085d80dd4fb1b8cae4d9398b5031570c400cce6c254591c9c5106c3e958d\": container with ID starting with e38a085d80dd4fb1b8cae4d9398b5031570c400cce6c254591c9c5106c3e958d not found: ID does not exist"
Sep 16 02:39:36.990239 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:39:36.990138    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fe11042_4069_4f91_9f0d_1cbc33028f77.slice/crio-9e55752215ad388f93650bbb986e6de70d98bc72db14354ac65f363025ab252b WatchSource:0}: Error finding container 9e55752215ad388f93650bbb986e6de70d98bc72db14354ac65f363025ab252b: Status 404 returned error can't find the container with id 9e55752215ad388f93650bbb986e6de70d98bc72db14354ac65f363025ab252b
Sep 16 02:39:36.990650 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:36.990620    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-619550-lg77r"]
Sep 16 02:39:37.005578 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:37.005558    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8ngl5\" (UniqueName: \"kubernetes.io/projected/ff6a30ad-287e-40df-aab7-455812130778-kube-api-access-8ngl5\") pod \"ff6a30ad-287e-40df-aab7-455812130778\" (UID: \"ff6a30ad-287e-40df-aab7-455812130778\") "
Sep 16 02:39:37.005678 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:37.005630    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ff6a30ad-287e-40df-aab7-455812130778-home\") pod \"ff6a30ad-287e-40df-aab7-455812130778\" (UID: \"ff6a30ad-287e-40df-aab7-455812130778\") "
Sep 16 02:39:37.005747 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:37.005685    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff6a30ad-287e-40df-aab7-455812130778-tls-certs\") pod \"ff6a30ad-287e-40df-aab7-455812130778\" (UID: \"ff6a30ad-287e-40df-aab7-455812130778\") "
Sep 16 02:39:37.005747 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:37.005734    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff6a30ad-287e-40df-aab7-455812130778-model-cache\") pod \"ff6a30ad-287e-40df-aab7-455812130778\" (UID: \"ff6a30ad-287e-40df-aab7-455812130778\") "
Sep 16 02:39:37.005866 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:37.005787    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff6a30ad-287e-40df-aab7-455812130778-tmp-dir\") pod \"ff6a30ad-287e-40df-aab7-455812130778\" (UID: \"ff6a30ad-287e-40df-aab7-455812130778\") "
Sep 16 02:39:37.005976 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:37.005910    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff6a30ad-287e-40df-aab7-455812130778-dshm\") pod \"ff6a30ad-287e-40df-aab7-455812130778\" (UID: \"ff6a30ad-287e-40df-aab7-455812130778\") "
Sep 16 02:39:37.005976 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:37.005919    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff6a30ad-287e-40df-aab7-455812130778-home" (OuterVolumeSpecName: "home") pod "ff6a30ad-287e-40df-aab7-455812130778" (UID: "ff6a30ad-287e-40df-aab7-455812130778"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:39:37.006168 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:37.006135    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff6a30ad-287e-40df-aab7-455812130778-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ff6a30ad-287e-40df-aab7-455812130778" (UID: "ff6a30ad-287e-40df-aab7-455812130778"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:39:37.006231 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:37.006158    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff6a30ad-287e-40df-aab7-455812130778-model-cache" (OuterVolumeSpecName: "model-cache") pod "ff6a30ad-287e-40df-aab7-455812130778" (UID: "ff6a30ad-287e-40df-aab7-455812130778"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:39:37.006599 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:37.006575    2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ff6a30ad-287e-40df-aab7-455812130778-home\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:39:37.006753 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:37.006603    2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff6a30ad-287e-40df-aab7-455812130778-model-cache\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:39:37.006753 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:37.006618    2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff6a30ad-287e-40df-aab7-455812130778-tmp-dir\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:39:37.008205 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:37.008131    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff6a30ad-287e-40df-aab7-455812130778-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ff6a30ad-287e-40df-aab7-455812130778" (UID: "ff6a30ad-287e-40df-aab7-455812130778"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 02:39:37.008337 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:37.008321    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff6a30ad-287e-40df-aab7-455812130778-kube-api-access-8ngl5" (OuterVolumeSpecName: "kube-api-access-8ngl5") pod "ff6a30ad-287e-40df-aab7-455812130778" (UID: "ff6a30ad-287e-40df-aab7-455812130778"). InnerVolumeSpecName "kube-api-access-8ngl5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 02:39:37.008574 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:37.008544    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff6a30ad-287e-40df-aab7-455812130778-dshm" (OuterVolumeSpecName: "dshm") pod "ff6a30ad-287e-40df-aab7-455812130778" (UID: "ff6a30ad-287e-40df-aab7-455812130778"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:39:37.107812 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:37.107778    2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff6a30ad-287e-40df-aab7-455812130778-dshm\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:39:37.107812 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:37.107806    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8ngl5\" (UniqueName: \"kubernetes.io/projected/ff6a30ad-287e-40df-aab7-455812130778-kube-api-access-8ngl5\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:39:37.107812 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:37.107816    2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff6a30ad-287e-40df-aab7-455812130778-tls-certs\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:39:37.260285 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:37.260261    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-542ed5/distinct-2-w3-542ed5-kserve-57dc75646d-x4qrb"]
Sep 16 02:39:37.265475 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:37.265452    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-542ed5/distinct-2-w3-542ed5-kserve-57dc75646d-x4qrb"]
Sep 16 02:39:37.551229 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:37.551149    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ff6a30ad-287e-40df-aab7-455812130778" path="/var/lib/kubelet/pods/ff6a30ad-287e-40df-aab7-455812130778/volumes"
Sep 16 02:39:37.946969 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:37.946938    2558 generic.go:356] "Generic (PLEG): container finished" podID="2fe11042-4069-4f91-9f0d-1cbc33028f77" containerID="2861abdb269844cd5ece908d80426a3211c4e8ff9c369ed668f017b696d940e7" exitCode=0
Sep 16 02:39:37.947354 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:37.947022    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-619550-lg77r" event={"ID":"2fe11042-4069-4f91-9f0d-1cbc33028f77","Type":"ContainerDied","Data":"2861abdb269844cd5ece908d80426a3211c4e8ff9c369ed668f017b696d940e7"}
Sep 16 02:39:37.947354 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:37.947073    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-619550-lg77r" event={"ID":"2fe11042-4069-4f91-9f0d-1cbc33028f77","Type":"ContainerStarted","Data":"9e55752215ad388f93650bbb986e6de70d98bc72db14354ac65f363025ab252b"}
Sep 16 02:39:39.089623 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:39.089594    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-619550-lg77r"
Sep 16 02:39:39.126172 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:39.126128    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2fe11042-4069-4f91-9f0d-1cbc33028f77-maas-api-service-ca\") pod \"2fe11042-4069-4f91-9f0d-1cbc33028f77\" (UID: \"2fe11042-4069-4f91-9f0d-1cbc33028f77\") "
Sep 16 02:39:39.126479 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:39.126456    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2fe11042-4069-4f91-9f0d-1cbc33028f77-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2fe11042-4069-4f91-9f0d-1cbc33028f77" (UID: "2fe11042-4069-4f91-9f0d-1cbc33028f77"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 02:39:39.227211 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:39.227129    2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2fe11042-4069-4f91-9f0d-1cbc33028f77-maas-api-service-ca\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:39:39.286285 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:39.286245    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-542ed5-openshift-default-7dc67cb65b-gcjgl"]
Sep 16 02:39:39.286511 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:39.286483    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-542ed5-openshift-default-7dc67cb65b-gcjgl" podUID="556c7c3b-f739-4660-95e6-3f4348276d47" containerName="istio-proxy" containerID="cri-o://c6a5d86755d56a2090e7af9115b4541a5d4f6aa58533695334e8f077ad59d9a7" gracePeriod=30
Sep 16 02:39:39.960764 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:39.960736    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-619550-lg77r"
Sep 16 02:39:39.960934 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:39.960789    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-619550-lg77r" event={"ID":"2fe11042-4069-4f91-9f0d-1cbc33028f77","Type":"ContainerDied","Data":"9e55752215ad388f93650bbb986e6de70d98bc72db14354ac65f363025ab252b"}
Sep 16 02:39:39.960934 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:39.960839    2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9e55752215ad388f93650bbb986e6de70d98bc72db14354ac65f363025ab252b"
Sep 16 02:39:41.346621 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:41.346579    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-619550/embedding-w0-619550-kserve-7f858bb94-cs6ff"]
Sep 16 02:39:44.538045 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:44.538022    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-542ed5-openshift-default-7dc67cb65b-gcjgl"
Sep 16 02:39:44.574141 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:44.574077    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/556c7c3b-f739-4660-95e6-3f4348276d47-workload-socket\") pod \"556c7c3b-f739-4660-95e6-3f4348276d47\" (UID: \"556c7c3b-f739-4660-95e6-3f4348276d47\") "
Sep 16 02:39:44.574141 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:44.574117    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/556c7c3b-f739-4660-95e6-3f4348276d47-istio-podinfo\") pod \"556c7c3b-f739-4660-95e6-3f4348276d47\" (UID: \"556c7c3b-f739-4660-95e6-3f4348276d47\") "
Sep 16 02:39:44.574141 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:44.574134    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/556c7c3b-f739-4660-95e6-3f4348276d47-istio-token\") pod \"556c7c3b-f739-4660-95e6-3f4348276d47\" (UID: \"556c7c3b-f739-4660-95e6-3f4348276d47\") "
Sep 16 02:39:44.574351 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:44.574170    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pz4h7\" (UniqueName: \"kubernetes.io/projected/556c7c3b-f739-4660-95e6-3f4348276d47-kube-api-access-pz4h7\") pod \"556c7c3b-f739-4660-95e6-3f4348276d47\" (UID: \"556c7c3b-f739-4660-95e6-3f4348276d47\") "
Sep 16 02:39:44.574351 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:44.574258    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/556c7c3b-f739-4660-95e6-3f4348276d47-istiod-ca-cert\") pod \"556c7c3b-f739-4660-95e6-3f4348276d47\" (UID: \"556c7c3b-f739-4660-95e6-3f4348276d47\") "
Sep 16 02:39:44.574351 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:44.574298    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/556c7c3b-f739-4660-95e6-3f4348276d47-workload-certs\") pod \"556c7c3b-f739-4660-95e6-3f4348276d47\" (UID: \"556c7c3b-f739-4660-95e6-3f4348276d47\") "
Sep 16 02:39:44.574351 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:44.574322    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/556c7c3b-f739-4660-95e6-3f4348276d47-istio-envoy\") pod \"556c7c3b-f739-4660-95e6-3f4348276d47\" (UID: \"556c7c3b-f739-4660-95e6-3f4348276d47\") "
Sep 16 02:39:44.574543 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:44.574365    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/556c7c3b-f739-4660-95e6-3f4348276d47-credential-socket\") pod \"556c7c3b-f739-4660-95e6-3f4348276d47\" (UID: \"556c7c3b-f739-4660-95e6-3f4348276d47\") "
Sep 16 02:39:44.574543 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:44.574382    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/556c7c3b-f739-4660-95e6-3f4348276d47-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "556c7c3b-f739-4660-95e6-3f4348276d47" (UID: "556c7c3b-f739-4660-95e6-3f4348276d47"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:39:44.574543 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:44.574393    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/556c7c3b-f739-4660-95e6-3f4348276d47-istio-data\") pod \"556c7c3b-f739-4660-95e6-3f4348276d47\" (UID: \"556c7c3b-f739-4660-95e6-3f4348276d47\") "
Sep 16 02:39:44.574719 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:44.574595    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/556c7c3b-f739-4660-95e6-3f4348276d47-istio-data" (OuterVolumeSpecName: "istio-data") pod "556c7c3b-f739-4660-95e6-3f4348276d47" (UID: "556c7c3b-f739-4660-95e6-3f4348276d47"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:39:44.574856 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:44.574781    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/556c7c3b-f739-4660-95e6-3f4348276d47-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "556c7c3b-f739-4660-95e6-3f4348276d47" (UID: "556c7c3b-f739-4660-95e6-3f4348276d47"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:39:44.574931 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:44.574863    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/556c7c3b-f739-4660-95e6-3f4348276d47-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "556c7c3b-f739-4660-95e6-3f4348276d47" (UID: "556c7c3b-f739-4660-95e6-3f4348276d47"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:39:44.574995 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:44.574927    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/556c7c3b-f739-4660-95e6-3f4348276d47-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "556c7c3b-f739-4660-95e6-3f4348276d47" (UID: "556c7c3b-f739-4660-95e6-3f4348276d47"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 02:39:44.575221 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:44.575145    2558 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/556c7c3b-f739-4660-95e6-3f4348276d47-istio-data\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:39:44.575221 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:44.575170    2558 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/556c7c3b-f739-4660-95e6-3f4348276d47-workload-socket\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:39:44.575221 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:44.575183    2558 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/556c7c3b-f739-4660-95e6-3f4348276d47-istiod-ca-cert\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:39:44.575221 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:44.575197    2558 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/556c7c3b-f739-4660-95e6-3f4348276d47-workload-certs\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:39:44.575221 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:44.575209    2558 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/556c7c3b-f739-4660-95e6-3f4348276d47-credential-socket\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:39:44.576790 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:44.576760    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/556c7c3b-f739-4660-95e6-3f4348276d47-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "556c7c3b-f739-4660-95e6-3f4348276d47" (UID: "556c7c3b-f739-4660-95e6-3f4348276d47"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 02:39:44.576904 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:44.576809    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/556c7c3b-f739-4660-95e6-3f4348276d47-istio-token" (OuterVolumeSpecName: "istio-token") pod "556c7c3b-f739-4660-95e6-3f4348276d47" (UID: "556c7c3b-f739-4660-95e6-3f4348276d47"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 02:39:44.576904 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:44.576835    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/556c7c3b-f739-4660-95e6-3f4348276d47-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "556c7c3b-f739-4660-95e6-3f4348276d47" (UID: "556c7c3b-f739-4660-95e6-3f4348276d47"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:39:44.576983 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:44.576901    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/556c7c3b-f739-4660-95e6-3f4348276d47-kube-api-access-pz4h7" (OuterVolumeSpecName: "kube-api-access-pz4h7") pod "556c7c3b-f739-4660-95e6-3f4348276d47" (UID: "556c7c3b-f739-4660-95e6-3f4348276d47"). InnerVolumeSpecName "kube-api-access-pz4h7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 02:39:44.675675 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:44.675644    2558 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/556c7c3b-f739-4660-95e6-3f4348276d47-istio-envoy\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:39:44.675675 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:44.675669    2558 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/556c7c3b-f739-4660-95e6-3f4348276d47-istio-podinfo\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:39:44.675675 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:44.675678    2558 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/556c7c3b-f739-4660-95e6-3f4348276d47-istio-token\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:39:44.675895 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:44.675686    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pz4h7\" (UniqueName: \"kubernetes.io/projected/556c7c3b-f739-4660-95e6-3f4348276d47-kube-api-access-pz4h7\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:39:44.990623 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:44.990589    2558 generic.go:356] "Generic (PLEG): container finished" podID="556c7c3b-f739-4660-95e6-3f4348276d47" containerID="c6a5d86755d56a2090e7af9115b4541a5d4f6aa58533695334e8f077ad59d9a7" exitCode=0
Sep 16 02:39:44.990800 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:44.990640    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-542ed5-openshift-default-7dc67cb65b-gcjgl" event={"ID":"556c7c3b-f739-4660-95e6-3f4348276d47","Type":"ContainerDied","Data":"c6a5d86755d56a2090e7af9115b4541a5d4f6aa58533695334e8f077ad59d9a7"}
Sep 16 02:39:44.990800 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:44.990643    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-542ed5-openshift-default-7dc67cb65b-gcjgl"
Sep 16 02:39:44.990800 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:44.990682    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-542ed5-openshift-default-7dc67cb65b-gcjgl" event={"ID":"556c7c3b-f739-4660-95e6-3f4348276d47","Type":"ContainerDied","Data":"5add03532bfcb78d3ff5cc3859df08070e476794ec176a9b0186fafdfbc54cb9"}
Sep 16 02:39:44.990800 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:44.990724    2558 scope.go:117] "RemoveContainer" containerID="c6a5d86755d56a2090e7af9115b4541a5d4f6aa58533695334e8f077ad59d9a7"
Sep 16 02:39:45.004746 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:45.004623    2558 scope.go:117] "RemoveContainer" containerID="39aea0fac2e04c00fa9cf4c3bc63d72d737ffa22269698207df720fb0a2de7c2"
Sep 16 02:39:45.014577 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:45.014546    2558 scope.go:117] "RemoveContainer" containerID="c6a5d86755d56a2090e7af9115b4541a5d4f6aa58533695334e8f077ad59d9a7"
Sep 16 02:39:45.014876 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:39:45.014854    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6a5d86755d56a2090e7af9115b4541a5d4f6aa58533695334e8f077ad59d9a7\": container with ID starting with c6a5d86755d56a2090e7af9115b4541a5d4f6aa58533695334e8f077ad59d9a7 not found: ID does not exist" containerID="c6a5d86755d56a2090e7af9115b4541a5d4f6aa58533695334e8f077ad59d9a7"
Sep 16 02:39:45.014926 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:45.014884    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6a5d86755d56a2090e7af9115b4541a5d4f6aa58533695334e8f077ad59d9a7"} err="failed to get container status \"c6a5d86755d56a2090e7af9115b4541a5d4f6aa58533695334e8f077ad59d9a7\": rpc error: code = NotFound desc = could not find container \"c6a5d86755d56a2090e7af9115b4541a5d4f6aa58533695334e8f077ad59d9a7\": container with ID starting with c6a5d86755d56a2090e7af9115b4541a5d4f6aa58533695334e8f077ad59d9a7 not found: ID does not exist"
Sep 16 02:39:45.014926 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:45.014902    2558 scope.go:117] "RemoveContainer" containerID="39aea0fac2e04c00fa9cf4c3bc63d72d737ffa22269698207df720fb0a2de7c2"
Sep 16 02:39:45.015170 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:39:45.015146    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39aea0fac2e04c00fa9cf4c3bc63d72d737ffa22269698207df720fb0a2de7c2\": container with ID starting with 39aea0fac2e04c00fa9cf4c3bc63d72d737ffa22269698207df720fb0a2de7c2 not found: ID does not exist" containerID="39aea0fac2e04c00fa9cf4c3bc63d72d737ffa22269698207df720fb0a2de7c2"
Sep 16 02:39:45.015217 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:45.015179    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39aea0fac2e04c00fa9cf4c3bc63d72d737ffa22269698207df720fb0a2de7c2"} err="failed to get container status \"39aea0fac2e04c00fa9cf4c3bc63d72d737ffa22269698207df720fb0a2de7c2\": rpc error: code = NotFound desc = could not find container \"39aea0fac2e04c00fa9cf4c3bc63d72d737ffa22269698207df720fb0a2de7c2\": container with ID starting with 39aea0fac2e04c00fa9cf4c3bc63d72d737ffa22269698207df720fb0a2de7c2 not found: ID does not exist"
Sep 16 02:39:45.016449 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:45.016432    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-542ed5-openshift-default-7dc67cb65b-gcjgl"]
Sep 16 02:39:45.022043 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:45.022022    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-542ed5-openshift-default-7dc67cb65b-gcjgl"]
Sep 16 02:39:45.549648 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:45.549613    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="556c7c3b-f739-4660-95e6-3f4348276d47" path="/var/lib/kubelet/pods/556c7c3b-f739-4660-95e6-3f4348276d47/volumes"
Sep 16 02:39:48.190859 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:48.190817    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e415ca22-p9qpf"]
Sep 16 02:39:48.191271 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:48.191233    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="556c7c3b-f739-4660-95e6-3f4348276d47" containerName="istio-proxy"
Sep 16 02:39:48.191271 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:48.191245    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="556c7c3b-f739-4660-95e6-3f4348276d47" containerName="istio-proxy"
Sep 16 02:39:48.191271 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:48.191255    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff6a30ad-287e-40df-aab7-455812130778" containerName="main"
Sep 16 02:39:48.191271 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:48.191261    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff6a30ad-287e-40df-aab7-455812130778" containerName="main"
Sep 16 02:39:48.191396 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:48.191280    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2fe11042-4069-4f91-9f0d-1cbc33028f77" containerName="revoke-keys"
Sep 16 02:39:48.191396 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:48.191288    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="2fe11042-4069-4f91-9f0d-1cbc33028f77" containerName="revoke-keys"
Sep 16 02:39:48.191396 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:48.191359    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="2fe11042-4069-4f91-9f0d-1cbc33028f77" containerName="revoke-keys"
Sep 16 02:39:48.191396 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:48.191370    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="556c7c3b-f739-4660-95e6-3f4348276d47" containerName="istio-proxy"
Sep 16 02:39:48.191396 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:48.191379    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="556c7c3b-f739-4660-95e6-3f4348276d47" containerName="istio-proxy"
Sep 16 02:39:48.191396 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:48.191386    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="ff6a30ad-287e-40df-aab7-455812130778" containerName="main"
Sep 16 02:39:48.196124 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:48.196103    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e415ca22-p9qpf"
Sep 16 02:39:48.198440 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:48.198411    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 16 02:39:48.198575 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:48.198484    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jpd4f\""
Sep 16 02:39:48.200138 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:48.200118    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e415ca22-p9qpf"]
Sep 16 02:39:48.310688 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:48.310655    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a3a6508-8f9a-4b0f-b4d6-cf0a4fc08045-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-e415ca22-p9qpf\" (UID: \"9a3a6508-8f9a-4b0f-b4d6-cf0a4fc08045\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e415ca22-p9qpf"
Sep 16 02:39:48.412988 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:48.412837    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a3a6508-8f9a-4b0f-b4d6-cf0a4fc08045-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-e415ca22-p9qpf\" (UID: \"9a3a6508-8f9a-4b0f-b4d6-cf0a4fc08045\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e415ca22-p9qpf"
Sep 16 02:39:48.414577 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:48.414380    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a3a6508-8f9a-4b0f-b4d6-cf0a4fc08045-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-e415ca22-p9qpf\" (UID: \"9a3a6508-8f9a-4b0f-b4d6-cf0a4fc08045\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e415ca22-p9qpf"
Sep 16 02:39:48.507349 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:48.507269    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e415ca22-p9qpf"
Sep 16 02:39:48.544847 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:48.544199    2558 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-processing-5d7dbbd56b-pltkr" secret="" err="secret \"payload-processing-dockercfg-zzx6j\" not found"
Sep 16 02:39:48.786260 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:48.786230    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e415ca22-p9qpf"]
Sep 16 02:39:48.788027 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:39:48.787991    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a3a6508_8f9a_4b0f_b4d6_cf0a4fc08045.slice/crio-739d4e5dfe01f6d0b8043d5760b5562be2ea745715433ba52213be8b65080ec5 WatchSource:0}: Error finding container 739d4e5dfe01f6d0b8043d5760b5562be2ea745715433ba52213be8b65080ec5: Status 404 returned error can't find the container with id 739d4e5dfe01f6d0b8043d5760b5562be2ea745715433ba52213be8b65080ec5
Sep 16 02:39:49.016876 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:49.016833    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e415ca22-p9qpf" event={"ID":"9a3a6508-8f9a-4b0f-b4d6-cf0a4fc08045","Type":"ContainerStarted","Data":"739d4e5dfe01f6d0b8043d5760b5562be2ea745715433ba52213be8b65080ec5"}
Sep 16 02:39:50.023555 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:50.023463    2558 generic.go:356] "Generic (PLEG): container finished" podID="9a3a6508-8f9a-4b0f-b4d6-cf0a4fc08045" containerID="4a847856a382090a1271bd0879e60ebc17b3b4e8e48bc50824b56c852e7ac82d" exitCode=0
Sep 16 02:39:50.023951 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:50.023555    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e415ca22-p9qpf" event={"ID":"9a3a6508-8f9a-4b0f-b4d6-cf0a4fc08045","Type":"ContainerDied","Data":"4a847856a382090a1271bd0879e60ebc17b3b4e8e48bc50824b56c852e7ac82d"}
Sep 16 02:39:51.179064 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:51.179033    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-619550-lg77r"]
Sep 16 02:39:51.196291 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:51.191583    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-619550-lg77r"]
Sep 16 02:39:51.206198 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:51.206173    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e415ca22-p9qpf"
Sep 16 02:39:51.241208 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:51.241171    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a3a6508-8f9a-4b0f-b4d6-cf0a4fc08045-maas-api-service-ca\") pod \"9a3a6508-8f9a-4b0f-b4d6-cf0a4fc08045\" (UID: \"9a3a6508-8f9a-4b0f-b4d6-cf0a4fc08045\") "
Sep 16 02:39:51.241652 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:51.241632    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a3a6508-8f9a-4b0f-b4d6-cf0a4fc08045-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "9a3a6508-8f9a-4b0f-b4d6-cf0a4fc08045" (UID: "9a3a6508-8f9a-4b0f-b4d6-cf0a4fc08045"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 02:39:51.342994 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:51.342898    2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a3a6508-8f9a-4b0f-b4d6-cf0a4fc08045-maas-api-service-ca\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:39:51.558982 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:51.558946    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2fe11042-4069-4f91-9f0d-1cbc33028f77" path="/var/lib/kubelet/pods/2fe11042-4069-4f91-9f0d-1cbc33028f77/volumes"
Sep 16 02:39:52.035920 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:52.035886    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e415ca22-p9qpf" event={"ID":"9a3a6508-8f9a-4b0f-b4d6-cf0a4fc08045","Type":"ContainerDied","Data":"739d4e5dfe01f6d0b8043d5760b5562be2ea745715433ba52213be8b65080ec5"}
Sep 16 02:39:52.036090 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:52.035934    2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="739d4e5dfe01f6d0b8043d5760b5562be2ea745715433ba52213be8b65080ec5"
Sep 16 02:39:52.036090 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:52.035902    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e415ca22-p9qpf"
Sep 16 02:39:56.365444 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:56.365404    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-619550/embedding-w0-619550-kserve-7f858bb94-cs6ff" podUID="a3c3acf6-e578-441b-acaa-ae1063718cd9" containerName="main" containerID="cri-o://6f8614de46596ff199e36784d0e5f65521e0327d809781059b123cf22eb35994" gracePeriod=45
Sep 16 02:39:56.624664 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:56.624604    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-619550/embedding-w0-619550-kserve-7f858bb94-cs6ff"
Sep 16 02:39:56.703504 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:56.703472    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a3c3acf6-e578-441b-acaa-ae1063718cd9-home\") pod \"a3c3acf6-e578-441b-acaa-ae1063718cd9\" (UID: \"a3c3acf6-e578-441b-acaa-ae1063718cd9\") "
Sep 16 02:39:56.703670 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:56.703532    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3c3acf6-e578-441b-acaa-ae1063718cd9-tls-certs\") pod \"a3c3acf6-e578-441b-acaa-ae1063718cd9\" (UID: \"a3c3acf6-e578-441b-acaa-ae1063718cd9\") "
Sep 16 02:39:56.703670 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:56.703576    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3c3acf6-e578-441b-acaa-ae1063718cd9-tmp-dir\") pod \"a3c3acf6-e578-441b-acaa-ae1063718cd9\" (UID: \"a3c3acf6-e578-441b-acaa-ae1063718cd9\") "
Sep 16 02:39:56.703819 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:56.703686    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a3c3acf6-e578-441b-acaa-ae1063718cd9-model-cache\") pod \"a3c3acf6-e578-441b-acaa-ae1063718cd9\" (UID: \"a3c3acf6-e578-441b-acaa-ae1063718cd9\") "
Sep 16 02:39:56.703872 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:56.703848    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lkb4b\" (UniqueName: \"kubernetes.io/projected/a3c3acf6-e578-441b-acaa-ae1063718cd9-kube-api-access-lkb4b\") pod \"a3c3acf6-e578-441b-acaa-ae1063718cd9\" (UID: \"a3c3acf6-e578-441b-acaa-ae1063718cd9\") "
Sep 16 02:39:56.703931 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:56.703856    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3c3acf6-e578-441b-acaa-ae1063718cd9-home" (OuterVolumeSpecName: "home") pod "a3c3acf6-e578-441b-acaa-ae1063718cd9" (UID: "a3c3acf6-e578-441b-acaa-ae1063718cd9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:39:56.703986 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:56.703941    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a3c3acf6-e578-441b-acaa-ae1063718cd9-dshm\") pod \"a3c3acf6-e578-441b-acaa-ae1063718cd9\" (UID: \"a3c3acf6-e578-441b-acaa-ae1063718cd9\") "
Sep 16 02:39:56.704139 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:56.704100    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3c3acf6-e578-441b-acaa-ae1063718cd9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a3c3acf6-e578-441b-acaa-ae1063718cd9" (UID: "a3c3acf6-e578-441b-acaa-ae1063718cd9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:39:56.704303 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:56.704266    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3c3acf6-e578-441b-acaa-ae1063718cd9-model-cache" (OuterVolumeSpecName: "model-cache") pod "a3c3acf6-e578-441b-acaa-ae1063718cd9" (UID: "a3c3acf6-e578-441b-acaa-ae1063718cd9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:39:56.705078 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:56.705052    2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a3c3acf6-e578-441b-acaa-ae1063718cd9-home\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:39:56.705194 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:56.705081    2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3c3acf6-e578-441b-acaa-ae1063718cd9-tmp-dir\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:39:56.705194 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:56.705095    2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a3c3acf6-e578-441b-acaa-ae1063718cd9-model-cache\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:39:56.706614 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:56.706549    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3c3acf6-e578-441b-acaa-ae1063718cd9-dshm" (OuterVolumeSpecName: "dshm") pod "a3c3acf6-e578-441b-acaa-ae1063718cd9" (UID: "a3c3acf6-e578-441b-acaa-ae1063718cd9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:39:56.706614 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:56.706597    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3c3acf6-e578-441b-acaa-ae1063718cd9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a3c3acf6-e578-441b-acaa-ae1063718cd9" (UID: "a3c3acf6-e578-441b-acaa-ae1063718cd9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 02:39:56.706804 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:56.706736    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3c3acf6-e578-441b-acaa-ae1063718cd9-kube-api-access-lkb4b" (OuterVolumeSpecName: "kube-api-access-lkb4b") pod "a3c3acf6-e578-441b-acaa-ae1063718cd9" (UID: "a3c3acf6-e578-441b-acaa-ae1063718cd9"). InnerVolumeSpecName "kube-api-access-lkb4b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 02:39:56.805649 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:56.805610    2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3c3acf6-e578-441b-acaa-ae1063718cd9-tls-certs\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:39:56.805649 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:56.805643    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lkb4b\" (UniqueName: \"kubernetes.io/projected/a3c3acf6-e578-441b-acaa-ae1063718cd9-kube-api-access-lkb4b\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:39:56.805649 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:56.805656    2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a3c3acf6-e578-441b-acaa-ae1063718cd9-dshm\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:39:57.066410 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:57.066372    2558 generic.go:356] "Generic (PLEG): container finished" podID="a3c3acf6-e578-441b-acaa-ae1063718cd9" containerID="6f8614de46596ff199e36784d0e5f65521e0327d809781059b123cf22eb35994" exitCode=0
Sep 16 02:39:57.066410 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:57.066412    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-619550/embedding-w0-619550-kserve-7f858bb94-cs6ff"
Sep 16 02:39:57.066607 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:57.066467    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-619550/embedding-w0-619550-kserve-7f858bb94-cs6ff" event={"ID":"a3c3acf6-e578-441b-acaa-ae1063718cd9","Type":"ContainerDied","Data":"6f8614de46596ff199e36784d0e5f65521e0327d809781059b123cf22eb35994"}
Sep 16 02:39:57.066607 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:57.066504    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-619550/embedding-w0-619550-kserve-7f858bb94-cs6ff" event={"ID":"a3c3acf6-e578-441b-acaa-ae1063718cd9","Type":"ContainerDied","Data":"8b84c184091cb9af93652d75a1065989a43d6ac8bd7e75cd95321b13df1ebf25"}
Sep 16 02:39:57.066607 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:57.066526    2558 scope.go:117] "RemoveContainer" containerID="6f8614de46596ff199e36784d0e5f65521e0327d809781059b123cf22eb35994"
Sep 16 02:39:57.083489 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:57.083468    2558 scope.go:117] "RemoveContainer" containerID="6f8614de46596ff199e36784d0e5f65521e0327d809781059b123cf22eb35994"
Sep 16 02:39:57.083828 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:39:57.083809    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f8614de46596ff199e36784d0e5f65521e0327d809781059b123cf22eb35994\": container with ID starting with 6f8614de46596ff199e36784d0e5f65521e0327d809781059b123cf22eb35994 not found: ID does not exist" containerID="6f8614de46596ff199e36784d0e5f65521e0327d809781059b123cf22eb35994"
Sep 16 02:39:57.083892 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:57.083838    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f8614de46596ff199e36784d0e5f65521e0327d809781059b123cf22eb35994"} err="failed to get container status \"6f8614de46596ff199e36784d0e5f65521e0327d809781059b123cf22eb35994\": rpc error: code = NotFound desc = could not find container \"6f8614de46596ff199e36784d0e5f65521e0327d809781059b123cf22eb35994\": container with ID starting with 6f8614de46596ff199e36784d0e5f65521e0327d809781059b123cf22eb35994 not found: ID does not exist"
Sep 16 02:39:57.091216 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:57.091192    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-619550/embedding-w0-619550-kserve-7f858bb94-cs6ff"]
Sep 16 02:39:57.094564 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:57.094540    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-619550/embedding-w0-619550-kserve-7f858bb94-cs6ff"]
Sep 16 02:39:57.549873 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:57.549803    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a3c3acf6-e578-441b-acaa-ae1063718cd9" path="/var/lib/kubelet/pods/a3c3acf6-e578-441b-acaa-ae1063718cd9/volumes"
Sep 16 02:39:57.743387 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:57.743350    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-619550-openshift-default-7787b6c55b-5jrjc"]
Sep 16 02:39:57.743637 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:57.743607    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-619550-openshift-default-7787b6c55b-5jrjc" podUID="8b5f2556-ebb9-4089-b0f5-9cc37357b31f" containerName="istio-proxy" containerID="cri-o://0a54bd28fb3fdf28b0c5cee0ccceb41d0cbbe48235a2a4e790ea0d674b23170b" gracePeriod=30
Sep 16 02:39:58.012254 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:58.012170    2558 patch_prober.go:28] interesting pod/e2e-worker-w0-619550-openshift-default-7787b6c55b-5jrjc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 16 02:39:58.012254 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:58.012240    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-619550-openshift-default-7787b6c55b-5jrjc" podUID="8b5f2556-ebb9-4089-b0f5-9cc37357b31f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 02:39:58.075877 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:58.075836    2558 generic.go:356] "Generic (PLEG): container finished" podID="73651cfe-d120-4ab1-a3be-d779025b1fae" containerID="a17399c6b2d72e6210577ce30d23b66d15e0131e45e9403cc52f560869e05d44" exitCode=0
Sep 16 02:39:58.076166 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:58.075923    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e415ca22-openshift-default-6f5546c844-vlld5" event={"ID":"73651cfe-d120-4ab1-a3be-d779025b1fae","Type":"ContainerDied","Data":"a17399c6b2d72e6210577ce30d23b66d15e0131e45e9403cc52f560869e05d44"}
Sep 16 02:39:58.076812 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:58.076793    2558 scope.go:117] "RemoveContainer" containerID="a17399c6b2d72e6210577ce30d23b66d15e0131e45e9403cc52f560869e05d44"
Sep 16 02:39:58.078177 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:58.078132    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 16 02:39:58.078371 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:58.078348    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 16 02:39:58.078529 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:58.078496    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 16 02:39:59.085571 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:59.085538    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e415ca22-openshift-default-6f5546c844-vlld5" event={"ID":"73651cfe-d120-4ab1-a3be-d779025b1fae","Type":"ContainerStarted","Data":"5c83cfdb018cba1e0ddcc5273b3b40420da2029fc6ef82919ca8e03ec8ad0c47"}
Sep 16 02:39:59.409120 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:59.409090    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-e415ca22-openshift-default-6f5546c844-vlld5"
Sep 16 02:39:59.410232 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:59.410207    2558 patch_prober.go:28] interesting pod/e2e-mt-e415ca22-openshift-default-6f5546c844-vlld5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Sep 16 02:39:59.410339 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:39:59.410254    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e415ca22-openshift-default-6f5546c844-vlld5" podUID="73651cfe-d120-4ab1-a3be-d779025b1fae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Sep 16 02:40:00.408881 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:00.408848    2558 patch_prober.go:28] interesting pod/e2e-mt-e415ca22-openshift-default-6f5546c844-vlld5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Sep 16 02:40:00.409240 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:00.408904    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e415ca22-openshift-default-6f5546c844-vlld5" podUID="73651cfe-d120-4ab1-a3be-d779025b1fae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Sep 16 02:40:01.408559 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:01.408522    2558 patch_prober.go:28] interesting pod/e2e-mt-e415ca22-openshift-default-6f5546c844-vlld5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Sep 16 02:40:01.408811 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:01.408582    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e415ca22-openshift-default-6f5546c844-vlld5" podUID="73651cfe-d120-4ab1-a3be-d779025b1fae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Sep 16 02:40:01.739663 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:01.739565    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-e415ca22-59d4f8f697-bgn5c"]
Sep 16 02:40:01.740155 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:01.739809    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-e415ca22-59d4f8f697-bgn5c" podUID="c03643a4-41ac-46bc-8685-c85ad0bc49a5" containerName="payload-processing" containerID="cri-o://09c9bc1c186c8ac498a49b82a568a3f21c040858e8797babaf5d89334bf08b94" gracePeriod=30
Sep 16 02:40:01.755782 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:01.743753    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e415ca22-67b7f75bd6-kn49p"]
Sep 16 02:40:01.755782 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:01.743979    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-e415ca22-67b7f75bd6-kn49p" podUID="08f5eb62-09b3-4f64-aa9c-9b2cb5ebe430" containerName="payload-pre-processing" containerID="cri-o://c0781b810a269139118704766de2456a4389cecb9b53888de05825745b85064f" gracePeriod=30
Sep 16 02:40:01.887124 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:01.887080    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e415ca22-p9qpf"]
Sep 16 02:40:01.889563 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:01.889535    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e415ca22-p9qpf"]
Sep 16 02:40:02.074508 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:02.074204    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-e415ca22-59d4f8f697-bgn5c"
Sep 16 02:40:02.107990 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:02.107950    2558 generic.go:356] "Generic (PLEG): container finished" podID="08f5eb62-09b3-4f64-aa9c-9b2cb5ebe430" containerID="c0781b810a269139118704766de2456a4389cecb9b53888de05825745b85064f" exitCode=0
Sep 16 02:40:02.108306 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:02.108209    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-e415ca22-67b7f75bd6-kn49p" event={"ID":"08f5eb62-09b3-4f64-aa9c-9b2cb5ebe430","Type":"ContainerDied","Data":"c0781b810a269139118704766de2456a4389cecb9b53888de05825745b85064f"}
Sep 16 02:40:02.108386 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:02.108360    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-e415ca22-67b7f75bd6-kn49p" event={"ID":"08f5eb62-09b3-4f64-aa9c-9b2cb5ebe430","Type":"ContainerDied","Data":"69a99cef5de5af4aca50ff63a0fceee16a314370017237dc6b763b71916b7e30"}
Sep 16 02:40:02.110347 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:02.110271    2558 generic.go:356] "Generic (PLEG): container finished" podID="c03643a4-41ac-46bc-8685-c85ad0bc49a5" containerID="09c9bc1c186c8ac498a49b82a568a3f21c040858e8797babaf5d89334bf08b94" exitCode=0
Sep 16 02:40:02.110448 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:02.110387    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-e415ca22-59d4f8f697-bgn5c"
Sep 16 02:40:02.110867 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:02.108390    2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="69a99cef5de5af4aca50ff63a0fceee16a314370017237dc6b763b71916b7e30"
Sep 16 02:40:02.111050 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:02.111028    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-e415ca22-59d4f8f697-bgn5c" event={"ID":"c03643a4-41ac-46bc-8685-c85ad0bc49a5","Type":"ContainerDied","Data":"09c9bc1c186c8ac498a49b82a568a3f21c040858e8797babaf5d89334bf08b94"}
Sep 16 02:40:02.111224 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:02.111197    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-e415ca22-59d4f8f697-bgn5c" event={"ID":"c03643a4-41ac-46bc-8685-c85ad0bc49a5","Type":"ContainerDied","Data":"10db080ebf5a39510b157a6b564f4233cf4e7302826fbc02c93c5b9f62d10aee"}
Sep 16 02:40:02.111416 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:02.111346    2558 scope.go:117] "RemoveContainer" containerID="09c9bc1c186c8ac498a49b82a568a3f21c040858e8797babaf5d89334bf08b94"
Sep 16 02:40:02.122669 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:02.122591    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-e415ca22-67b7f75bd6-kn49p"
Sep 16 02:40:02.122884 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:02.122777    2558 scope.go:117] "RemoveContainer" containerID="09c9bc1c186c8ac498a49b82a568a3f21c040858e8797babaf5d89334bf08b94"
Sep 16 02:40:02.123064 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:40:02.123045    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09c9bc1c186c8ac498a49b82a568a3f21c040858e8797babaf5d89334bf08b94\": container with ID starting with 09c9bc1c186c8ac498a49b82a568a3f21c040858e8797babaf5d89334bf08b94 not found: ID does not exist" containerID="09c9bc1c186c8ac498a49b82a568a3f21c040858e8797babaf5d89334bf08b94"
Sep 16 02:40:02.123117 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:02.123074    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09c9bc1c186c8ac498a49b82a568a3f21c040858e8797babaf5d89334bf08b94"} err="failed to get container status \"09c9bc1c186c8ac498a49b82a568a3f21c040858e8797babaf5d89334bf08b94\": rpc error: code = NotFound desc = could not find container \"09c9bc1c186c8ac498a49b82a568a3f21c040858e8797babaf5d89334bf08b94\": container with ID starting with 09c9bc1c186c8ac498a49b82a568a3f21c040858e8797babaf5d89334bf08b94 not found: ID does not exist"
Sep 16 02:40:02.171662 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:02.170486    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c03643a4-41ac-46bc-8685-c85ad0bc49a5-plugins-config-volume\") pod \"c03643a4-41ac-46bc-8685-c85ad0bc49a5\" (UID: \"c03643a4-41ac-46bc-8685-c85ad0bc49a5\") "
Sep 16 02:40:02.171662 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:02.170621    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ln24h\" (UniqueName: \"kubernetes.io/projected/c03643a4-41ac-46bc-8685-c85ad0bc49a5-kube-api-access-ln24h\") pod \"c03643a4-41ac-46bc-8685-c85ad0bc49a5\" (UID: \"c03643a4-41ac-46bc-8685-c85ad0bc49a5\") "
Sep 16 02:40:02.171662 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:02.171608    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c03643a4-41ac-46bc-8685-c85ad0bc49a5-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c03643a4-41ac-46bc-8685-c85ad0bc49a5" (UID: "c03643a4-41ac-46bc-8685-c85ad0bc49a5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 02:40:02.173677 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:02.173650    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c03643a4-41ac-46bc-8685-c85ad0bc49a5-kube-api-access-ln24h" (OuterVolumeSpecName: "kube-api-access-ln24h") pod "c03643a4-41ac-46bc-8685-c85ad0bc49a5" (UID: "c03643a4-41ac-46bc-8685-c85ad0bc49a5"). InnerVolumeSpecName "kube-api-access-ln24h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 02:40:02.271914 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:02.271808    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q82hj\" (UniqueName: \"kubernetes.io/projected/08f5eb62-09b3-4f64-aa9c-9b2cb5ebe430-kube-api-access-q82hj\") pod \"08f5eb62-09b3-4f64-aa9c-9b2cb5ebe430\" (UID: \"08f5eb62-09b3-4f64-aa9c-9b2cb5ebe430\") "
Sep 16 02:40:02.271914 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:02.271865    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08f5eb62-09b3-4f64-aa9c-9b2cb5ebe430-plugins-config-volume\") pod \"08f5eb62-09b3-4f64-aa9c-9b2cb5ebe430\" (UID: \"08f5eb62-09b3-4f64-aa9c-9b2cb5ebe430\") "
Sep 16 02:40:02.272245 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:02.272061    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ln24h\" (UniqueName: \"kubernetes.io/projected/c03643a4-41ac-46bc-8685-c85ad0bc49a5-kube-api-access-ln24h\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:40:02.272245 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:02.272071    2558 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c03643a4-41ac-46bc-8685-c85ad0bc49a5-plugins-config-volume\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:40:02.272245 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:02.272212    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08f5eb62-09b3-4f64-aa9c-9b2cb5ebe430-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "08f5eb62-09b3-4f64-aa9c-9b2cb5ebe430" (UID: "08f5eb62-09b3-4f64-aa9c-9b2cb5ebe430"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 02:40:02.274007 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:02.273978    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08f5eb62-09b3-4f64-aa9c-9b2cb5ebe430-kube-api-access-q82hj" (OuterVolumeSpecName: "kube-api-access-q82hj") pod "08f5eb62-09b3-4f64-aa9c-9b2cb5ebe430" (UID: "08f5eb62-09b3-4f64-aa9c-9b2cb5ebe430"). InnerVolumeSpecName "kube-api-access-q82hj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 02:40:02.373395 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:02.373352    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q82hj\" (UniqueName: \"kubernetes.io/projected/08f5eb62-09b3-4f64-aa9c-9b2cb5ebe430-kube-api-access-q82hj\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:40:02.373395 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:02.373387    2558 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08f5eb62-09b3-4f64-aa9c-9b2cb5ebe430-plugins-config-volume\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:40:02.408661 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:02.408620    2558 patch_prober.go:28] interesting pod/e2e-mt-e415ca22-openshift-default-6f5546c844-vlld5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Sep 16 02:40:02.408826 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:02.408681    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e415ca22-openshift-default-6f5546c844-vlld5" podUID="73651cfe-d120-4ab1-a3be-d779025b1fae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Sep 16 02:40:02.430804 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:02.430760    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-e415ca22-59d4f8f697-bgn5c"]
Sep 16 02:40:02.434034 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:02.434005    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-e415ca22-59d4f8f697-bgn5c"]
Sep 16 02:40:03.029275 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:03.029167    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-619550-openshift-default-7787b6c55b-5jrjc"
Sep 16 02:40:03.118964 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:03.118931    2558 generic.go:356] "Generic (PLEG): container finished" podID="8b5f2556-ebb9-4089-b0f5-9cc37357b31f" containerID="0a54bd28fb3fdf28b0c5cee0ccceb41d0cbbe48235a2a4e790ea0d674b23170b" exitCode=0
Sep 16 02:40:03.119140 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:03.118985    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-619550-openshift-default-7787b6c55b-5jrjc"
Sep 16 02:40:03.119140 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:03.118977    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-619550-openshift-default-7787b6c55b-5jrjc" event={"ID":"8b5f2556-ebb9-4089-b0f5-9cc37357b31f","Type":"ContainerDied","Data":"0a54bd28fb3fdf28b0c5cee0ccceb41d0cbbe48235a2a4e790ea0d674b23170b"}
Sep 16 02:40:03.119140 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:03.119109    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-619550-openshift-default-7787b6c55b-5jrjc" event={"ID":"8b5f2556-ebb9-4089-b0f5-9cc37357b31f","Type":"ContainerDied","Data":"0ea2ddd137bad541944ee77f109130007bed7f1947f39f52a5d431e4d66d8173"}
Sep 16 02:40:03.119140 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:03.119132    2558 scope.go:117] "RemoveContainer" containerID="0a54bd28fb3fdf28b0c5cee0ccceb41d0cbbe48235a2a4e790ea0d674b23170b"
Sep 16 02:40:03.119376 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:03.119159    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-e415ca22-67b7f75bd6-kn49p"
Sep 16 02:40:03.129290 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:03.129267    2558 scope.go:117] "RemoveContainer" containerID="be7f09b2d6a914f953c5af311e8fcae270069eb0e4847cc070c2409eae8a274c"
Sep 16 02:40:03.143203 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:03.143062    2558 scope.go:117] "RemoveContainer" containerID="0a54bd28fb3fdf28b0c5cee0ccceb41d0cbbe48235a2a4e790ea0d674b23170b"
Sep 16 02:40:03.144181 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:40:03.144025    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a54bd28fb3fdf28b0c5cee0ccceb41d0cbbe48235a2a4e790ea0d674b23170b\": container with ID starting with 0a54bd28fb3fdf28b0c5cee0ccceb41d0cbbe48235a2a4e790ea0d674b23170b not found: ID does not exist" containerID="0a54bd28fb3fdf28b0c5cee0ccceb41d0cbbe48235a2a4e790ea0d674b23170b"
Sep 16 02:40:03.144181 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:03.144071    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a54bd28fb3fdf28b0c5cee0ccceb41d0cbbe48235a2a4e790ea0d674b23170b"} err="failed to get container status \"0a54bd28fb3fdf28b0c5cee0ccceb41d0cbbe48235a2a4e790ea0d674b23170b\": rpc error: code = NotFound desc = could not find container \"0a54bd28fb3fdf28b0c5cee0ccceb41d0cbbe48235a2a4e790ea0d674b23170b\": container with ID starting with 0a54bd28fb3fdf28b0c5cee0ccceb41d0cbbe48235a2a4e790ea0d674b23170b not found: ID does not exist"
Sep 16 02:40:03.144181 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:03.144096    2558 scope.go:117] "RemoveContainer" containerID="be7f09b2d6a914f953c5af311e8fcae270069eb0e4847cc070c2409eae8a274c"
Sep 16 02:40:03.148742 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:40:03.144980    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be7f09b2d6a914f953c5af311e8fcae270069eb0e4847cc070c2409eae8a274c\": container with ID starting with be7f09b2d6a914f953c5af311e8fcae270069eb0e4847cc070c2409eae8a274c not found: ID does not exist" containerID="be7f09b2d6a914f953c5af311e8fcae270069eb0e4847cc070c2409eae8a274c"
Sep 16 02:40:03.148742 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:03.145021    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be7f09b2d6a914f953c5af311e8fcae270069eb0e4847cc070c2409eae8a274c"} err="failed to get container status \"be7f09b2d6a914f953c5af311e8fcae270069eb0e4847cc070c2409eae8a274c\": rpc error: code = NotFound desc = could not find container \"be7f09b2d6a914f953c5af311e8fcae270069eb0e4847cc070c2409eae8a274c\": container with ID starting with be7f09b2d6a914f953c5af311e8fcae270069eb0e4847cc070c2409eae8a274c not found: ID does not exist"
Sep 16 02:40:03.167728 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:03.166075    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e415ca22-67b7f75bd6-kn49p"]
Sep 16 02:40:03.170090 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:03.170052    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e415ca22-67b7f75bd6-kn49p"]
Sep 16 02:40:03.180414 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:03.180380    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b5f2556-ebb9-4089-b0f5-9cc37357b31f-istio-podinfo\") pod \"8b5f2556-ebb9-4089-b0f5-9cc37357b31f\" (UID: \"8b5f2556-ebb9-4089-b0f5-9cc37357b31f\") "
Sep 16 02:40:03.180565 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:03.180418    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b5f2556-ebb9-4089-b0f5-9cc37357b31f-istiod-ca-cert\") pod \"8b5f2556-ebb9-4089-b0f5-9cc37357b31f\" (UID: \"8b5f2556-ebb9-4089-b0f5-9cc37357b31f\") "
Sep 16 02:40:03.180565 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:03.180449    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b5f2556-ebb9-4089-b0f5-9cc37357b31f-istio-envoy\") pod \"8b5f2556-ebb9-4089-b0f5-9cc37357b31f\" (UID: \"8b5f2556-ebb9-4089-b0f5-9cc37357b31f\") "
Sep 16 02:40:03.180565 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:03.180470    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b5f2556-ebb9-4089-b0f5-9cc37357b31f-workload-certs\") pod \"8b5f2556-ebb9-4089-b0f5-9cc37357b31f\" (UID: \"8b5f2556-ebb9-4089-b0f5-9cc37357b31f\") "
Sep 16 02:40:03.180565 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:03.180508    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b5f2556-ebb9-4089-b0f5-9cc37357b31f-workload-socket\") pod \"8b5f2556-ebb9-4089-b0f5-9cc37357b31f\" (UID: \"8b5f2556-ebb9-4089-b0f5-9cc37357b31f\") "
Sep 16 02:40:03.180565 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:03.180543    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b5f2556-ebb9-4089-b0f5-9cc37357b31f-istio-data\") pod \"8b5f2556-ebb9-4089-b0f5-9cc37357b31f\" (UID: \"8b5f2556-ebb9-4089-b0f5-9cc37357b31f\") "
Sep 16 02:40:03.180847 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:03.180579    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qhmfj\" (UniqueName: \"kubernetes.io/projected/8b5f2556-ebb9-4089-b0f5-9cc37357b31f-kube-api-access-qhmfj\") pod \"8b5f2556-ebb9-4089-b0f5-9cc37357b31f\" (UID: \"8b5f2556-ebb9-4089-b0f5-9cc37357b31f\") "
Sep 16 02:40:03.180847 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:03.180645    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b5f2556-ebb9-4089-b0f5-9cc37357b31f-credential-socket\") pod \"8b5f2556-ebb9-4089-b0f5-9cc37357b31f\" (UID: \"8b5f2556-ebb9-4089-b0f5-9cc37357b31f\") "
Sep 16 02:40:03.180847 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:03.180683    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b5f2556-ebb9-4089-b0f5-9cc37357b31f-istio-token\") pod \"8b5f2556-ebb9-4089-b0f5-9cc37357b31f\" (UID: \"8b5f2556-ebb9-4089-b0f5-9cc37357b31f\") "
Sep 16 02:40:03.180847 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:03.180778    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b5f2556-ebb9-4089-b0f5-9cc37357b31f-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "8b5f2556-ebb9-4089-b0f5-9cc37357b31f" (UID: "8b5f2556-ebb9-4089-b0f5-9cc37357b31f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:40:03.180847 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:03.180792    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b5f2556-ebb9-4089-b0f5-9cc37357b31f-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "8b5f2556-ebb9-4089-b0f5-9cc37357b31f" (UID: "8b5f2556-ebb9-4089-b0f5-9cc37357b31f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:40:03.180847 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:03.180829    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b5f2556-ebb9-4089-b0f5-9cc37357b31f-istio-data" (OuterVolumeSpecName: "istio-data") pod "8b5f2556-ebb9-4089-b0f5-9cc37357b31f" (UID: "8b5f2556-ebb9-4089-b0f5-9cc37357b31f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:40:03.181130 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:03.180888    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b5f2556-ebb9-4089-b0f5-9cc37357b31f-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "8b5f2556-ebb9-4089-b0f5-9cc37357b31f" (UID: "8b5f2556-ebb9-4089-b0f5-9cc37357b31f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 02:40:03.181130 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:03.181028    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b5f2556-ebb9-4089-b0f5-9cc37357b31f-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "8b5f2556-ebb9-4089-b0f5-9cc37357b31f" (UID: "8b5f2556-ebb9-4089-b0f5-9cc37357b31f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:40:03.181130 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:03.181105    2558 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b5f2556-ebb9-4089-b0f5-9cc37357b31f-istiod-ca-cert\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:40:03.181130 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:03.181128    2558 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b5f2556-ebb9-4089-b0f5-9cc37357b31f-workload-certs\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:40:03.181328 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:03.181142    2558 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b5f2556-ebb9-4089-b0f5-9cc37357b31f-workload-socket\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:40:03.181328 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:03.181155    2558 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b5f2556-ebb9-4089-b0f5-9cc37357b31f-istio-data\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:40:03.182987 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:03.182957    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b5f2556-ebb9-4089-b0f5-9cc37357b31f-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "8b5f2556-ebb9-4089-b0f5-9cc37357b31f" (UID: "8b5f2556-ebb9-4089-b0f5-9cc37357b31f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:40:03.183088 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:03.182985    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8b5f2556-ebb9-4089-b0f5-9cc37357b31f-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "8b5f2556-ebb9-4089-b0f5-9cc37357b31f" (UID: "8b5f2556-ebb9-4089-b0f5-9cc37357b31f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 02:40:03.183088 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:03.182963    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b5f2556-ebb9-4089-b0f5-9cc37357b31f-istio-token" (OuterVolumeSpecName: "istio-token") pod "8b5f2556-ebb9-4089-b0f5-9cc37357b31f" (UID: "8b5f2556-ebb9-4089-b0f5-9cc37357b31f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 02:40:03.183088 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:03.182963    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b5f2556-ebb9-4089-b0f5-9cc37357b31f-kube-api-access-qhmfj" (OuterVolumeSpecName: "kube-api-access-qhmfj") pod "8b5f2556-ebb9-4089-b0f5-9cc37357b31f" (UID: "8b5f2556-ebb9-4089-b0f5-9cc37357b31f"). InnerVolumeSpecName "kube-api-access-qhmfj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 02:40:03.282737 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:03.282599    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qhmfj\" (UniqueName: \"kubernetes.io/projected/8b5f2556-ebb9-4089-b0f5-9cc37357b31f-kube-api-access-qhmfj\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:40:03.282737 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:03.282638    2558 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b5f2556-ebb9-4089-b0f5-9cc37357b31f-credential-socket\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:40:03.282737 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:03.282649    2558 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b5f2556-ebb9-4089-b0f5-9cc37357b31f-istio-token\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:40:03.282737 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:03.282658    2558 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b5f2556-ebb9-4089-b0f5-9cc37357b31f-istio-podinfo\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:40:03.282737 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:03.282666    2558 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b5f2556-ebb9-4089-b0f5-9cc37357b31f-istio-envoy\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:40:03.408864 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:03.408823    2558 patch_prober.go:28] interesting pod/e2e-mt-e415ca22-openshift-default-6f5546c844-vlld5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Sep 16 02:40:03.409041 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:03.408887    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e415ca22-openshift-default-6f5546c844-vlld5" podUID="73651cfe-d120-4ab1-a3be-d779025b1fae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Sep 16 02:40:03.453416 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:03.449558    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-619550-openshift-default-7787b6c55b-5jrjc"]
Sep 16 02:40:03.453416 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:03.453187    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-619550-openshift-default-7787b6c55b-5jrjc"]
Sep 16 02:40:03.554586 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:03.554477    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="08f5eb62-09b3-4f64-aa9c-9b2cb5ebe430" path="/var/lib/kubelet/pods/08f5eb62-09b3-4f64-aa9c-9b2cb5ebe430/volumes"
Sep 16 02:40:03.555197 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:03.555169    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8b5f2556-ebb9-4089-b0f5-9cc37357b31f" path="/var/lib/kubelet/pods/8b5f2556-ebb9-4089-b0f5-9cc37357b31f/volumes"
Sep 16 02:40:03.555667 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:03.555643    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9a3a6508-8f9a-4b0f-b4d6-cf0a4fc08045" path="/var/lib/kubelet/pods/9a3a6508-8f9a-4b0f-b4d6-cf0a4fc08045/volumes"
Sep 16 02:40:03.556598 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:03.556567    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c03643a4-41ac-46bc-8685-c85ad0bc49a5" path="/var/lib/kubelet/pods/c03643a4-41ac-46bc-8685-c85ad0bc49a5/volumes"
Sep 16 02:40:03.832186 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:03.832087    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-e415ca22-openshift-default-6f5546c844-vlld5"
Sep 16 02:40:04.408432 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:04.408391    2558 patch_prober.go:28] interesting pod/e2e-mt-e415ca22-openshift-default-6f5546c844-vlld5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Sep 16 02:40:04.408867 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:04.408459    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e415ca22-openshift-default-6f5546c844-vlld5" podUID="73651cfe-d120-4ab1-a3be-d779025b1fae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Sep 16 02:40:05.413216 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:05.413182    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-e415ca22-openshift-default-6f5546c844-vlld5"
Sep 16 02:40:05.414358 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:05.414336    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-e415ca22-openshift-default-6f5546c844-vlld5"
Sep 16 02:40:07.726734 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:07.726682    2558 scope.go:117] "RemoveContainer" containerID="0d9c8ad9ad972d871297062de3f7e828b0d238bfa83c4e0d6c3cfaff9f290b15"
Sep 16 02:40:08.246955 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:08.246922    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-e415ca22-openshift-default-6f5546c844-vlld5"]
Sep 16 02:40:08.247195 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:08.247161    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-e415ca22-openshift-default-6f5546c844-vlld5" podUID="73651cfe-d120-4ab1-a3be-d779025b1fae" containerName="istio-proxy" containerID="cri-o://5c83cfdb018cba1e0ddcc5273b3b40420da2029fc6ef82919ca8e03ec8ad0c47" gracePeriod=30
Sep 16 02:40:13.492114 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:13.492084    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e415ca22-openshift-default-6f5546c844-vlld5"
Sep 16 02:40:13.685655 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:13.685624    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73651cfe-d120-4ab1-a3be-d779025b1fae-credential-socket\") pod \"73651cfe-d120-4ab1-a3be-d779025b1fae\" (UID: \"73651cfe-d120-4ab1-a3be-d779025b1fae\") "
Sep 16 02:40:13.685846 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:13.685661    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73651cfe-d120-4ab1-a3be-d779025b1fae-istiod-ca-cert\") pod \"73651cfe-d120-4ab1-a3be-d779025b1fae\" (UID: \"73651cfe-d120-4ab1-a3be-d779025b1fae\") "
Sep 16 02:40:13.685846 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:13.685720    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73651cfe-d120-4ab1-a3be-d779025b1fae-istio-podinfo\") pod \"73651cfe-d120-4ab1-a3be-d779025b1fae\" (UID: \"73651cfe-d120-4ab1-a3be-d779025b1fae\") "
Sep 16 02:40:13.685846 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:13.685740    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7fdv5\" (UniqueName: \"kubernetes.io/projected/73651cfe-d120-4ab1-a3be-d779025b1fae-kube-api-access-7fdv5\") pod \"73651cfe-d120-4ab1-a3be-d779025b1fae\" (UID: \"73651cfe-d120-4ab1-a3be-d779025b1fae\") "
Sep 16 02:40:13.685846 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:13.685757    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73651cfe-d120-4ab1-a3be-d779025b1fae-istio-data\") pod \"73651cfe-d120-4ab1-a3be-d779025b1fae\" (UID: \"73651cfe-d120-4ab1-a3be-d779025b1fae\") "
Sep 16 02:40:13.685846 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:13.685801    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73651cfe-d120-4ab1-a3be-d779025b1fae-istio-token\") pod \"73651cfe-d120-4ab1-a3be-d779025b1fae\" (UID: \"73651cfe-d120-4ab1-a3be-d779025b1fae\") "
Sep 16 02:40:13.685846 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:13.685837    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73651cfe-d120-4ab1-a3be-d779025b1fae-istio-envoy\") pod \"73651cfe-d120-4ab1-a3be-d779025b1fae\" (UID: \"73651cfe-d120-4ab1-a3be-d779025b1fae\") "
Sep 16 02:40:13.686183 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:13.685901    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73651cfe-d120-4ab1-a3be-d779025b1fae-workload-socket\") pod \"73651cfe-d120-4ab1-a3be-d779025b1fae\" (UID: \"73651cfe-d120-4ab1-a3be-d779025b1fae\") "
Sep 16 02:40:13.686183 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:13.685933    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73651cfe-d120-4ab1-a3be-d779025b1fae-workload-certs\") pod \"73651cfe-d120-4ab1-a3be-d779025b1fae\" (UID: \"73651cfe-d120-4ab1-a3be-d779025b1fae\") "
Sep 16 02:40:13.686183 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:13.686004    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73651cfe-d120-4ab1-a3be-d779025b1fae-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "73651cfe-d120-4ab1-a3be-d779025b1fae" (UID: "73651cfe-d120-4ab1-a3be-d779025b1fae"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:40:13.686335 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:13.686220    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73651cfe-d120-4ab1-a3be-d779025b1fae-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "73651cfe-d120-4ab1-a3be-d779025b1fae" (UID: "73651cfe-d120-4ab1-a3be-d779025b1fae"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 02:40:13.686335 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:13.686235    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73651cfe-d120-4ab1-a3be-d779025b1fae-istio-data" (OuterVolumeSpecName: "istio-data") pod "73651cfe-d120-4ab1-a3be-d779025b1fae" (UID: "73651cfe-d120-4ab1-a3be-d779025b1fae"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:40:13.686335 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:13.686248    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73651cfe-d120-4ab1-a3be-d779025b1fae-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "73651cfe-d120-4ab1-a3be-d779025b1fae" (UID: "73651cfe-d120-4ab1-a3be-d779025b1fae"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:40:13.686540 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:13.686393    2558 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73651cfe-d120-4ab1-a3be-d779025b1fae-workload-socket\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:40:13.686540 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:13.686412    2558 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73651cfe-d120-4ab1-a3be-d779025b1fae-credential-socket\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:40:13.686540 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:13.686414    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73651cfe-d120-4ab1-a3be-d779025b1fae-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "73651cfe-d120-4ab1-a3be-d779025b1fae" (UID: "73651cfe-d120-4ab1-a3be-d779025b1fae"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:40:13.686540 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:13.686428    2558 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73651cfe-d120-4ab1-a3be-d779025b1fae-istiod-ca-cert\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:40:13.686540 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:13.686442    2558 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73651cfe-d120-4ab1-a3be-d779025b1fae-istio-data\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:40:13.688108 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:13.688073    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73651cfe-d120-4ab1-a3be-d779025b1fae-istio-token" (OuterVolumeSpecName: "istio-token") pod "73651cfe-d120-4ab1-a3be-d779025b1fae" (UID: "73651cfe-d120-4ab1-a3be-d779025b1fae"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 02:40:13.688204 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:13.688144    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/73651cfe-d120-4ab1-a3be-d779025b1fae-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "73651cfe-d120-4ab1-a3be-d779025b1fae" (UID: "73651cfe-d120-4ab1-a3be-d779025b1fae"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 02:40:13.688583 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:13.688559    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73651cfe-d120-4ab1-a3be-d779025b1fae-kube-api-access-7fdv5" (OuterVolumeSpecName: "kube-api-access-7fdv5") pod "73651cfe-d120-4ab1-a3be-d779025b1fae" (UID: "73651cfe-d120-4ab1-a3be-d779025b1fae"). InnerVolumeSpecName "kube-api-access-7fdv5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 02:40:13.688583 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:13.688557    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73651cfe-d120-4ab1-a3be-d779025b1fae-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "73651cfe-d120-4ab1-a3be-d779025b1fae" (UID: "73651cfe-d120-4ab1-a3be-d779025b1fae"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:40:13.787272 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:13.787233    2558 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73651cfe-d120-4ab1-a3be-d779025b1fae-istio-podinfo\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:40:13.787272 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:13.787265    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7fdv5\" (UniqueName: \"kubernetes.io/projected/73651cfe-d120-4ab1-a3be-d779025b1fae-kube-api-access-7fdv5\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:40:13.787272 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:13.787276    2558 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73651cfe-d120-4ab1-a3be-d779025b1fae-istio-token\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:40:13.787272 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:13.787285    2558 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73651cfe-d120-4ab1-a3be-d779025b1fae-istio-envoy\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:40:13.787518 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:13.787292    2558 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73651cfe-d120-4ab1-a3be-d779025b1fae-workload-certs\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:40:14.180413 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:14.180383    2558 generic.go:356] "Generic (PLEG): container finished" podID="73651cfe-d120-4ab1-a3be-d779025b1fae" containerID="5c83cfdb018cba1e0ddcc5273b3b40420da2029fc6ef82919ca8e03ec8ad0c47" exitCode=0
Sep 16 02:40:14.180596 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:14.180433    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e415ca22-openshift-default-6f5546c844-vlld5"
Sep 16 02:40:14.180596 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:14.180473    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e415ca22-openshift-default-6f5546c844-vlld5" event={"ID":"73651cfe-d120-4ab1-a3be-d779025b1fae","Type":"ContainerDied","Data":"5c83cfdb018cba1e0ddcc5273b3b40420da2029fc6ef82919ca8e03ec8ad0c47"}
Sep 16 02:40:14.180596 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:14.180526    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e415ca22-openshift-default-6f5546c844-vlld5" event={"ID":"73651cfe-d120-4ab1-a3be-d779025b1fae","Type":"ContainerDied","Data":"e4f0f11ec45646c3e3e9e79dd3b3f4ba309ddd6682f67e851363c0d8b900fe18"}
Sep 16 02:40:14.180596 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:14.180549    2558 scope.go:117] "RemoveContainer" containerID="5c83cfdb018cba1e0ddcc5273b3b40420da2029fc6ef82919ca8e03ec8ad0c47"
Sep 16 02:40:14.190149 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:14.190129    2558 scope.go:117] "RemoveContainer" containerID="a17399c6b2d72e6210577ce30d23b66d15e0131e45e9403cc52f560869e05d44"
Sep 16 02:40:14.202187 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:14.202155    2558 scope.go:117] "RemoveContainer" containerID="5c83cfdb018cba1e0ddcc5273b3b40420da2029fc6ef82919ca8e03ec8ad0c47"
Sep 16 02:40:14.202594 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:40:14.202569    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c83cfdb018cba1e0ddcc5273b3b40420da2029fc6ef82919ca8e03ec8ad0c47\": container with ID starting with 5c83cfdb018cba1e0ddcc5273b3b40420da2029fc6ef82919ca8e03ec8ad0c47 not found: ID does not exist" containerID="5c83cfdb018cba1e0ddcc5273b3b40420da2029fc6ef82919ca8e03ec8ad0c47"
Sep 16 02:40:14.202680 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:14.202606    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c83cfdb018cba1e0ddcc5273b3b40420da2029fc6ef82919ca8e03ec8ad0c47"} err="failed to get container status \"5c83cfdb018cba1e0ddcc5273b3b40420da2029fc6ef82919ca8e03ec8ad0c47\": rpc error: code = NotFound desc = could not find container \"5c83cfdb018cba1e0ddcc5273b3b40420da2029fc6ef82919ca8e03ec8ad0c47\": container with ID starting with 5c83cfdb018cba1e0ddcc5273b3b40420da2029fc6ef82919ca8e03ec8ad0c47 not found: ID does not exist"
Sep 16 02:40:14.202680 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:14.202633    2558 scope.go:117] "RemoveContainer" containerID="a17399c6b2d72e6210577ce30d23b66d15e0131e45e9403cc52f560869e05d44"
Sep 16 02:40:14.202981 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:40:14.202962    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a17399c6b2d72e6210577ce30d23b66d15e0131e45e9403cc52f560869e05d44\": container with ID starting with a17399c6b2d72e6210577ce30d23b66d15e0131e45e9403cc52f560869e05d44 not found: ID does not exist" containerID="a17399c6b2d72e6210577ce30d23b66d15e0131e45e9403cc52f560869e05d44"
Sep 16 02:40:14.203029 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:14.202990    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a17399c6b2d72e6210577ce30d23b66d15e0131e45e9403cc52f560869e05d44"} err="failed to get container status \"a17399c6b2d72e6210577ce30d23b66d15e0131e45e9403cc52f560869e05d44\": rpc error: code = NotFound desc = could not find container \"a17399c6b2d72e6210577ce30d23b66d15e0131e45e9403cc52f560869e05d44\": container with ID starting with a17399c6b2d72e6210577ce30d23b66d15e0131e45e9403cc52f560869e05d44 not found: ID does not exist"
Sep 16 02:40:14.203086 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:14.203068    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-e415ca22-openshift-default-6f5546c844-vlld5"]
Sep 16 02:40:14.206785 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:14.206759    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-e415ca22-openshift-default-6f5546c844-vlld5"]
Sep 16 02:40:15.549488 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:15.549440    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="73651cfe-d120-4ab1-a3be-d779025b1fae" path="/var/lib/kubelet/pods/73651cfe-d120-4ab1-a3be-d779025b1fae/volumes"
Sep 16 02:40:22.372327 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:22.372288    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-bd0fb8-openshift-default-84f4588456-hdfcv"]
Sep 16 02:40:22.373215 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:22.372745    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a3c3acf6-e578-441b-acaa-ae1063718cd9" containerName="main"
Sep 16 02:40:22.373215 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:22.372763    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="a3c3acf6-e578-441b-acaa-ae1063718cd9" containerName="main"
Sep 16 02:40:22.373215 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:22.372788    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a3a6508-8f9a-4b0f-b4d6-cf0a4fc08045" containerName="revoke-keys"
Sep 16 02:40:22.373215 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:22.372797    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a3a6508-8f9a-4b0f-b4d6-cf0a4fc08045" containerName="revoke-keys"
Sep 16 02:40:22.373215 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:22.372808    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b5f2556-ebb9-4089-b0f5-9cc37357b31f" containerName="istio-proxy"
Sep 16 02:40:22.373215 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:22.372816    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b5f2556-ebb9-4089-b0f5-9cc37357b31f" containerName="istio-proxy"
Sep 16 02:40:22.373215 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:22.372826    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="73651cfe-d120-4ab1-a3be-d779025b1fae" containerName="istio-proxy"
Sep 16 02:40:22.373215 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:22.372834    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="73651cfe-d120-4ab1-a3be-d779025b1fae" containerName="istio-proxy"
Sep 16 02:40:22.373215 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:22.372845    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="556c7c3b-f739-4660-95e6-3f4348276d47" containerName="istio-proxy"
Sep 16 02:40:22.373215 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:22.372853    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="556c7c3b-f739-4660-95e6-3f4348276d47" containerName="istio-proxy"
Sep 16 02:40:22.373215 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:22.372865    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c03643a4-41ac-46bc-8685-c85ad0bc49a5" containerName="payload-processing"
Sep 16 02:40:22.373215 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:22.372874    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="c03643a4-41ac-46bc-8685-c85ad0bc49a5" containerName="payload-processing"
Sep 16 02:40:22.373215 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:22.372884    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="08f5eb62-09b3-4f64-aa9c-9b2cb5ebe430" containerName="payload-pre-processing"
Sep 16 02:40:22.373215 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:22.372890    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="08f5eb62-09b3-4f64-aa9c-9b2cb5ebe430" containerName="payload-pre-processing"
Sep 16 02:40:22.373215 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:22.372900    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="73651cfe-d120-4ab1-a3be-d779025b1fae" containerName="istio-proxy"
Sep 16 02:40:22.373215 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:22.372904    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="73651cfe-d120-4ab1-a3be-d779025b1fae" containerName="istio-proxy"
Sep 16 02:40:22.373215 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:22.372999    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="73651cfe-d120-4ab1-a3be-d779025b1fae" containerName="istio-proxy"
Sep 16 02:40:22.373215 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:22.373010    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="a3c3acf6-e578-441b-acaa-ae1063718cd9" containerName="main"
Sep 16 02:40:22.373215 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:22.373022    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="8b5f2556-ebb9-4089-b0f5-9cc37357b31f" containerName="istio-proxy"
Sep 16 02:40:22.373215 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:22.373027    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="9a3a6508-8f9a-4b0f-b4d6-cf0a4fc08045" containerName="revoke-keys"
Sep 16 02:40:22.373215 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:22.373033    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="c03643a4-41ac-46bc-8685-c85ad0bc49a5" containerName="payload-processing"
Sep 16 02:40:22.373215 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:22.373041    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="08f5eb62-09b3-4f64-aa9c-9b2cb5ebe430" containerName="payload-pre-processing"
Sep 16 02:40:22.377982 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:22.377959    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-bd0fb8-openshift-default-84f4588456-hdfcv"
Sep 16 02:40:22.380390 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:22.380366    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w1-bd0fb8-openshift-default-dockercfg-9slqf\""
Sep 16 02:40:22.385126 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:22.385103    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-bd0fb8-openshift-default-84f4588456-hdfcv"]
Sep 16 02:40:22.455875 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:22.455825    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4a618df-1dcc-42ed-ac72-e519ee2c5515-istio-podinfo\") pod \"e2e-shared-a-w1-bd0fb8-openshift-default-84f4588456-hdfcv\" (UID: \"f4a618df-1dcc-42ed-ac72-e519ee2c5515\") " pod="openshift-ingress/e2e-shared-a-w1-bd0fb8-openshift-default-84f4588456-hdfcv"
Sep 16 02:40:22.455875 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:22.455868    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4a618df-1dcc-42ed-ac72-e519ee2c5515-istio-envoy\") pod \"e2e-shared-a-w1-bd0fb8-openshift-default-84f4588456-hdfcv\" (UID: \"f4a618df-1dcc-42ed-ac72-e519ee2c5515\") " pod="openshift-ingress/e2e-shared-a-w1-bd0fb8-openshift-default-84f4588456-hdfcv"
Sep 16 02:40:22.456152 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:22.455895    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4a618df-1dcc-42ed-ac72-e519ee2c5515-workload-socket\") pod \"e2e-shared-a-w1-bd0fb8-openshift-default-84f4588456-hdfcv\" (UID: \"f4a618df-1dcc-42ed-ac72-e519ee2c5515\") " pod="openshift-ingress/e2e-shared-a-w1-bd0fb8-openshift-default-84f4588456-hdfcv"
Sep 16 02:40:22.456152 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:22.455923    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4a618df-1dcc-42ed-ac72-e519ee2c5515-credential-socket\") pod \"e2e-shared-a-w1-bd0fb8-openshift-default-84f4588456-hdfcv\" (UID: \"f4a618df-1dcc-42ed-ac72-e519ee2c5515\") " pod="openshift-ingress/e2e-shared-a-w1-bd0fb8-openshift-default-84f4588456-hdfcv"
Sep 16 02:40:22.456152 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:22.455957    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqblf\" (UniqueName: \"kubernetes.io/projected/f4a618df-1dcc-42ed-ac72-e519ee2c5515-kube-api-access-nqblf\") pod \"e2e-shared-a-w1-bd0fb8-openshift-default-84f4588456-hdfcv\" (UID: \"f4a618df-1dcc-42ed-ac72-e519ee2c5515\") " pod="openshift-ingress/e2e-shared-a-w1-bd0fb8-openshift-default-84f4588456-hdfcv"
Sep 16 02:40:22.456152 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:22.455994    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4a618df-1dcc-42ed-ac72-e519ee2c5515-workload-certs\") pod \"e2e-shared-a-w1-bd0fb8-openshift-default-84f4588456-hdfcv\" (UID: \"f4a618df-1dcc-42ed-ac72-e519ee2c5515\") " pod="openshift-ingress/e2e-shared-a-w1-bd0fb8-openshift-default-84f4588456-hdfcv"
Sep 16 02:40:22.456152 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:22.456030    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4a618df-1dcc-42ed-ac72-e519ee2c5515-istiod-ca-cert\") pod \"e2e-shared-a-w1-bd0fb8-openshift-default-84f4588456-hdfcv\" (UID: \"f4a618df-1dcc-42ed-ac72-e519ee2c5515\") " pod="openshift-ingress/e2e-shared-a-w1-bd0fb8-openshift-default-84f4588456-hdfcv"
Sep 16 02:40:22.456152 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:22.456077    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4a618df-1dcc-42ed-ac72-e519ee2c5515-istio-token\") pod \"e2e-shared-a-w1-bd0fb8-openshift-default-84f4588456-hdfcv\" (UID: \"f4a618df-1dcc-42ed-ac72-e519ee2c5515\") " pod="openshift-ingress/e2e-shared-a-w1-bd0fb8-openshift-default-84f4588456-hdfcv"
Sep 16 02:40:22.456152 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:22.456102    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4a618df-1dcc-42ed-ac72-e519ee2c5515-istio-data\") pod \"e2e-shared-a-w1-bd0fb8-openshift-default-84f4588456-hdfcv\" (UID: \"f4a618df-1dcc-42ed-ac72-e519ee2c5515\") " pod="openshift-ingress/e2e-shared-a-w1-bd0fb8-openshift-default-84f4588456-hdfcv"
Sep 16 02:40:22.557399 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:22.557364    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4a618df-1dcc-42ed-ac72-e519ee2c5515-workload-certs\") pod \"e2e-shared-a-w1-bd0fb8-openshift-default-84f4588456-hdfcv\" (UID: \"f4a618df-1dcc-42ed-ac72-e519ee2c5515\") " pod="openshift-ingress/e2e-shared-a-w1-bd0fb8-openshift-default-84f4588456-hdfcv"
Sep 16 02:40:22.557399 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:22.557406    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4a618df-1dcc-42ed-ac72-e519ee2c5515-istiod-ca-cert\") pod \"e2e-shared-a-w1-bd0fb8-openshift-default-84f4588456-hdfcv\" (UID: \"f4a618df-1dcc-42ed-ac72-e519ee2c5515\") " pod="openshift-ingress/e2e-shared-a-w1-bd0fb8-openshift-default-84f4588456-hdfcv"
Sep 16 02:40:22.557632 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:22.557445    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4a618df-1dcc-42ed-ac72-e519ee2c5515-istio-token\") pod \"e2e-shared-a-w1-bd0fb8-openshift-default-84f4588456-hdfcv\" (UID: \"f4a618df-1dcc-42ed-ac72-e519ee2c5515\") " pod="openshift-ingress/e2e-shared-a-w1-bd0fb8-openshift-default-84f4588456-hdfcv"
Sep 16 02:40:22.557632 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:22.557469    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4a618df-1dcc-42ed-ac72-e519ee2c5515-istio-data\") pod \"e2e-shared-a-w1-bd0fb8-openshift-default-84f4588456-hdfcv\" (UID: \"f4a618df-1dcc-42ed-ac72-e519ee2c5515\") " pod="openshift-ingress/e2e-shared-a-w1-bd0fb8-openshift-default-84f4588456-hdfcv"
Sep 16 02:40:22.557632 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:22.557501    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4a618df-1dcc-42ed-ac72-e519ee2c5515-istio-podinfo\") pod \"e2e-shared-a-w1-bd0fb8-openshift-default-84f4588456-hdfcv\" (UID: \"f4a618df-1dcc-42ed-ac72-e519ee2c5515\") " pod="openshift-ingress/e2e-shared-a-w1-bd0fb8-openshift-default-84f4588456-hdfcv"
Sep 16 02:40:22.557632 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:22.557553    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4a618df-1dcc-42ed-ac72-e519ee2c5515-istio-envoy\") pod \"e2e-shared-a-w1-bd0fb8-openshift-default-84f4588456-hdfcv\" (UID: \"f4a618df-1dcc-42ed-ac72-e519ee2c5515\") " pod="openshift-ingress/e2e-shared-a-w1-bd0fb8-openshift-default-84f4588456-hdfcv"
Sep 16 02:40:22.557632 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:22.557622    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4a618df-1dcc-42ed-ac72-e519ee2c5515-workload-socket\") pod \"e2e-shared-a-w1-bd0fb8-openshift-default-84f4588456-hdfcv\" (UID: \"f4a618df-1dcc-42ed-ac72-e519ee2c5515\") " pod="openshift-ingress/e2e-shared-a-w1-bd0fb8-openshift-default-84f4588456-hdfcv"
Sep 16 02:40:22.557905 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:22.557733    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4a618df-1dcc-42ed-ac72-e519ee2c5515-credential-socket\") pod \"e2e-shared-a-w1-bd0fb8-openshift-default-84f4588456-hdfcv\" (UID: \"f4a618df-1dcc-42ed-ac72-e519ee2c5515\") " pod="openshift-ingress/e2e-shared-a-w1-bd0fb8-openshift-default-84f4588456-hdfcv"
Sep 16 02:40:22.557905 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:22.557783    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqblf\" (UniqueName: \"kubernetes.io/projected/f4a618df-1dcc-42ed-ac72-e519ee2c5515-kube-api-access-nqblf\") pod \"e2e-shared-a-w1-bd0fb8-openshift-default-84f4588456-hdfcv\" (UID: \"f4a618df-1dcc-42ed-ac72-e519ee2c5515\") " pod="openshift-ingress/e2e-shared-a-w1-bd0fb8-openshift-default-84f4588456-hdfcv"
Sep 16 02:40:22.557905 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:22.557841    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4a618df-1dcc-42ed-ac72-e519ee2c5515-workload-certs\") pod \"e2e-shared-a-w1-bd0fb8-openshift-default-84f4588456-hdfcv\" (UID: \"f4a618df-1dcc-42ed-ac72-e519ee2c5515\") " pod="openshift-ingress/e2e-shared-a-w1-bd0fb8-openshift-default-84f4588456-hdfcv"
Sep 16 02:40:22.557905 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:22.557857    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4a618df-1dcc-42ed-ac72-e519ee2c5515-istio-data\") pod \"e2e-shared-a-w1-bd0fb8-openshift-default-84f4588456-hdfcv\" (UID: \"f4a618df-1dcc-42ed-ac72-e519ee2c5515\") " pod="openshift-ingress/e2e-shared-a-w1-bd0fb8-openshift-default-84f4588456-hdfcv"
Sep 16 02:40:22.558122 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:22.557945    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4a618df-1dcc-42ed-ac72-e519ee2c5515-workload-socket\") pod \"e2e-shared-a-w1-bd0fb8-openshift-default-84f4588456-hdfcv\" (UID: \"f4a618df-1dcc-42ed-ac72-e519ee2c5515\") " pod="openshift-ingress/e2e-shared-a-w1-bd0fb8-openshift-default-84f4588456-hdfcv"
Sep 16 02:40:22.558197 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:22.558149    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4a618df-1dcc-42ed-ac72-e519ee2c5515-credential-socket\") pod \"e2e-shared-a-w1-bd0fb8-openshift-default-84f4588456-hdfcv\" (UID: \"f4a618df-1dcc-42ed-ac72-e519ee2c5515\") " pod="openshift-ingress/e2e-shared-a-w1-bd0fb8-openshift-default-84f4588456-hdfcv"
Sep 16 02:40:22.558197 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:22.558190    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4a618df-1dcc-42ed-ac72-e519ee2c5515-istiod-ca-cert\") pod \"e2e-shared-a-w1-bd0fb8-openshift-default-84f4588456-hdfcv\" (UID: \"f4a618df-1dcc-42ed-ac72-e519ee2c5515\") " pod="openshift-ingress/e2e-shared-a-w1-bd0fb8-openshift-default-84f4588456-hdfcv"
Sep 16 02:40:22.559996 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:22.559974    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4a618df-1dcc-42ed-ac72-e519ee2c5515-istio-podinfo\") pod \"e2e-shared-a-w1-bd0fb8-openshift-default-84f4588456-hdfcv\" (UID: \"f4a618df-1dcc-42ed-ac72-e519ee2c5515\") " pod="openshift-ingress/e2e-shared-a-w1-bd0fb8-openshift-default-84f4588456-hdfcv"
Sep 16 02:40:22.560146 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:22.560130    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4a618df-1dcc-42ed-ac72-e519ee2c5515-istio-envoy\") pod \"e2e-shared-a-w1-bd0fb8-openshift-default-84f4588456-hdfcv\" (UID: \"f4a618df-1dcc-42ed-ac72-e519ee2c5515\") " pod="openshift-ingress/e2e-shared-a-w1-bd0fb8-openshift-default-84f4588456-hdfcv"
Sep 16 02:40:22.568789 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:22.568760    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4a618df-1dcc-42ed-ac72-e519ee2c5515-istio-token\") pod \"e2e-shared-a-w1-bd0fb8-openshift-default-84f4588456-hdfcv\" (UID: \"f4a618df-1dcc-42ed-ac72-e519ee2c5515\") " pod="openshift-ingress/e2e-shared-a-w1-bd0fb8-openshift-default-84f4588456-hdfcv"
Sep 16 02:40:22.568902 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:22.568892    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqblf\" (UniqueName: \"kubernetes.io/projected/f4a618df-1dcc-42ed-ac72-e519ee2c5515-kube-api-access-nqblf\") pod \"e2e-shared-a-w1-bd0fb8-openshift-default-84f4588456-hdfcv\" (UID: \"f4a618df-1dcc-42ed-ac72-e519ee2c5515\") " pod="openshift-ingress/e2e-shared-a-w1-bd0fb8-openshift-default-84f4588456-hdfcv"
Sep 16 02:40:22.689744 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:22.689667    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-bd0fb8-openshift-default-84f4588456-hdfcv"
Sep 16 02:40:22.965282 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:22.965255    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-bd0fb8-openshift-default-84f4588456-hdfcv"]
Sep 16 02:40:22.967064 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:40:22.967037    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4a618df_1dcc_42ed_ac72_e519ee2c5515.slice/crio-53410d63c6a6fc782e4f7eb479fe09ed2a4875e76879b0c939d1e8f1aa37e7b7 WatchSource:0}: Error finding container 53410d63c6a6fc782e4f7eb479fe09ed2a4875e76879b0c939d1e8f1aa37e7b7: Status 404 returned error can't find the container with id 53410d63c6a6fc782e4f7eb479fe09ed2a4875e76879b0c939d1e8f1aa37e7b7
Sep 16 02:40:22.969196 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:22.969158    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 16 02:40:22.969281 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:22.969227    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 16 02:40:22.969281 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:22.969254    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 16 02:40:23.233787 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:23.233693    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-bd0fb8-openshift-default-84f4588456-hdfcv" event={"ID":"f4a618df-1dcc-42ed-ac72-e519ee2c5515","Type":"ContainerStarted","Data":"f0fb1f59a7d7d0d73262d9156bcf6336d94a22a2b413eddef7118743709163cd"}
Sep 16 02:40:23.233787 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:23.233759    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-bd0fb8-openshift-default-84f4588456-hdfcv" event={"ID":"f4a618df-1dcc-42ed-ac72-e519ee2c5515","Type":"ContainerStarted","Data":"53410d63c6a6fc782e4f7eb479fe09ed2a4875e76879b0c939d1e8f1aa37e7b7"}
Sep 16 02:40:23.253508 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:23.253444    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-bd0fb8-openshift-default-84f4588456-hdfcv" podStartSLOduration=1.25342393 podStartE2EDuration="1.25342393s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 02:40:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:40:23.250179667 +0000 UTC m=+1636.168668917" watchObservedRunningTime="2026-09-16 02:40:23.25342393 +0000 UTC m=+1636.171913180"
Sep 16 02:40:23.690088 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:23.690049    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-bd0fb8-openshift-default-84f4588456-hdfcv"
Sep 16 02:40:24.694805 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:24.694771    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-bd0fb8-openshift-default-84f4588456-hdfcv"
Sep 16 02:40:25.243669 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:25.243634    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-bd0fb8-openshift-default-84f4588456-hdfcv"
Sep 16 02:40:25.244483 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:25.244464    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-bd0fb8-openshift-default-84f4588456-hdfcv"
Sep 16 02:40:26.974192 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:26.974147    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-bd0fb8-55f956944b-tl98x"]
Sep 16 02:40:26.974575 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:26.974532    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b5f2556-ebb9-4089-b0f5-9cc37357b31f" containerName="istio-proxy"
Sep 16 02:40:26.974575 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:26.974544    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b5f2556-ebb9-4089-b0f5-9cc37357b31f" containerName="istio-proxy"
Sep 16 02:40:26.974647 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:26.974596    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="8b5f2556-ebb9-4089-b0f5-9cc37357b31f" containerName="istio-proxy"
Sep 16 02:40:26.974647 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:26.974609    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="73651cfe-d120-4ab1-a3be-d779025b1fae" containerName="istio-proxy"
Sep 16 02:40:26.977831 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:26.977813    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-bd0fb8-55f956944b-tl98x"
Sep 16 02:40:26.980083 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:26.980063    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w1-bd0fb8\""
Sep 16 02:40:26.980192 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:26.980097    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w1-bd0fb8-dockercfg-5zkg5\""
Sep 16 02:40:26.987820 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:26.987794    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-bd0fb8-55f956944b-tl98x"]
Sep 16 02:40:26.989670 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:26.989641    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c298d7e3-7484-4e0b-814e-6b4151d40a16-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-bd0fb8-55f956944b-tl98x\" (UID: \"c298d7e3-7484-4e0b-814e-6b4151d40a16\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-bd0fb8-55f956944b-tl98x"
Sep 16 02:40:26.989809 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:26.989727    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6d2b\" (UniqueName: \"kubernetes.io/projected/c298d7e3-7484-4e0b-814e-6b4151d40a16-kube-api-access-m6d2b\") pod \"payload-pre-processing-e2e-shared-a-w1-bd0fb8-55f956944b-tl98x\" (UID: \"c298d7e3-7484-4e0b-814e-6b4151d40a16\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-bd0fb8-55f956944b-tl98x"
Sep 16 02:40:27.009275 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:27.009246    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-bd0fb8-6c4f6f65f-sd7gz"]
Sep 16 02:40:27.012618 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:27.012600    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-bd0fb8-6c4f6f65f-sd7gz"
Sep 16 02:40:27.022487 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:27.022466    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-bd0fb8-6c4f6f65f-sd7gz"]
Sep 16 02:40:27.091109 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:27.091065    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c298d7e3-7484-4e0b-814e-6b4151d40a16-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-bd0fb8-55f956944b-tl98x\" (UID: \"c298d7e3-7484-4e0b-814e-6b4151d40a16\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-bd0fb8-55f956944b-tl98x"
Sep 16 02:40:27.091315 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:27.091133    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d8288a9-39c6-41bf-aa3e-392304de982d-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-bd0fb8-6c4f6f65f-sd7gz\" (UID: \"4d8288a9-39c6-41bf-aa3e-392304de982d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-bd0fb8-6c4f6f65f-sd7gz"
Sep 16 02:40:27.091315 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:27.091172    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6d2b\" (UniqueName: \"kubernetes.io/projected/c298d7e3-7484-4e0b-814e-6b4151d40a16-kube-api-access-m6d2b\") pod \"payload-pre-processing-e2e-shared-a-w1-bd0fb8-55f956944b-tl98x\" (UID: \"c298d7e3-7484-4e0b-814e-6b4151d40a16\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-bd0fb8-55f956944b-tl98x"
Sep 16 02:40:27.091315 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:27.091212    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grz5l\" (UniqueName: \"kubernetes.io/projected/4d8288a9-39c6-41bf-aa3e-392304de982d-kube-api-access-grz5l\") pod \"payload-processing-e2e-shared-a-w1-bd0fb8-6c4f6f65f-sd7gz\" (UID: \"4d8288a9-39c6-41bf-aa3e-392304de982d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-bd0fb8-6c4f6f65f-sd7gz"
Sep 16 02:40:27.091962 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:27.091940    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c298d7e3-7484-4e0b-814e-6b4151d40a16-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-bd0fb8-55f956944b-tl98x\" (UID: \"c298d7e3-7484-4e0b-814e-6b4151d40a16\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-bd0fb8-55f956944b-tl98x"
Sep 16 02:40:27.100713 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:27.100662    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6d2b\" (UniqueName: \"kubernetes.io/projected/c298d7e3-7484-4e0b-814e-6b4151d40a16-kube-api-access-m6d2b\") pod \"payload-pre-processing-e2e-shared-a-w1-bd0fb8-55f956944b-tl98x\" (UID: \"c298d7e3-7484-4e0b-814e-6b4151d40a16\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-bd0fb8-55f956944b-tl98x"
Sep 16 02:40:27.192051 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:27.192005    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d8288a9-39c6-41bf-aa3e-392304de982d-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-bd0fb8-6c4f6f65f-sd7gz\" (UID: \"4d8288a9-39c6-41bf-aa3e-392304de982d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-bd0fb8-6c4f6f65f-sd7gz"
Sep 16 02:40:27.192242 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:27.192155    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grz5l\" (UniqueName: \"kubernetes.io/projected/4d8288a9-39c6-41bf-aa3e-392304de982d-kube-api-access-grz5l\") pod \"payload-processing-e2e-shared-a-w1-bd0fb8-6c4f6f65f-sd7gz\" (UID: \"4d8288a9-39c6-41bf-aa3e-392304de982d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-bd0fb8-6c4f6f65f-sd7gz"
Sep 16 02:40:27.192936 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:27.192908    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d8288a9-39c6-41bf-aa3e-392304de982d-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-bd0fb8-6c4f6f65f-sd7gz\" (UID: \"4d8288a9-39c6-41bf-aa3e-392304de982d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-bd0fb8-6c4f6f65f-sd7gz"
Sep 16 02:40:27.200048 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:27.200021    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-grz5l\" (UniqueName: \"kubernetes.io/projected/4d8288a9-39c6-41bf-aa3e-392304de982d-kube-api-access-grz5l\") pod \"payload-processing-e2e-shared-a-w1-bd0fb8-6c4f6f65f-sd7gz\" (UID: \"4d8288a9-39c6-41bf-aa3e-392304de982d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-bd0fb8-6c4f6f65f-sd7gz"
Sep 16 02:40:27.289763 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:27.289670    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-bd0fb8-55f956944b-tl98x"
Sep 16 02:40:27.322871 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:27.322831    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-bd0fb8-6c4f6f65f-sd7gz"
Sep 16 02:40:27.558282 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:27.558207    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-bd0fb8-55f956944b-tl98x"]
Sep 16 02:40:27.561030 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:40:27.561005    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc298d7e3_7484_4e0b_814e_6b4151d40a16.slice/crio-293dc7fe6063488d4276ca6d03aae112a8e8720537c0a42f2020c03017d4ac24 WatchSource:0}: Error finding container 293dc7fe6063488d4276ca6d03aae112a8e8720537c0a42f2020c03017d4ac24: Status 404 returned error can't find the container with id 293dc7fe6063488d4276ca6d03aae112a8e8720537c0a42f2020c03017d4ac24
Sep 16 02:40:27.602777 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:27.602755    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-bd0fb8-6c4f6f65f-sd7gz"]
Sep 16 02:40:27.604505 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:40:27.604481    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d8288a9_39c6_41bf_aa3e_392304de982d.slice/crio-01fc67e322f9f4aed209b4dee2f8cbd7742a8b742b9cbf9daee5a6c091c6a187 WatchSource:0}: Error finding container 01fc67e322f9f4aed209b4dee2f8cbd7742a8b742b9cbf9daee5a6c091c6a187: Status 404 returned error can't find the container with id 01fc67e322f9f4aed209b4dee2f8cbd7742a8b742b9cbf9daee5a6c091c6a187
Sep 16 02:40:28.265742 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:28.265682    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-bd0fb8-6c4f6f65f-sd7gz" event={"ID":"4d8288a9-39c6-41bf-aa3e-392304de982d","Type":"ContainerStarted","Data":"62a258f67803eb84215061f1abfcc7bcdf981d22ac59bc0fd4fba0aa2de69d0c"}
Sep 16 02:40:28.266218 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:28.265815    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-bd0fb8-6c4f6f65f-sd7gz" event={"ID":"4d8288a9-39c6-41bf-aa3e-392304de982d","Type":"ContainerStarted","Data":"01fc67e322f9f4aed209b4dee2f8cbd7742a8b742b9cbf9daee5a6c091c6a187"}
Sep 16 02:40:28.266218 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:28.265869    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-bd0fb8-6c4f6f65f-sd7gz"
Sep 16 02:40:28.268328 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:28.268295    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-bd0fb8-55f956944b-tl98x" event={"ID":"c298d7e3-7484-4e0b-814e-6b4151d40a16","Type":"ContainerStarted","Data":"fa2eb16e42820247631269ab1a30bea94feba395f440ad0745fc6f2e61a1e222"}
Sep 16 02:40:28.268444 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:28.268354    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-bd0fb8-55f956944b-tl98x" event={"ID":"c298d7e3-7484-4e0b-814e-6b4151d40a16","Type":"ContainerStarted","Data":"293dc7fe6063488d4276ca6d03aae112a8e8720537c0a42f2020c03017d4ac24"}
Sep 16 02:40:28.268590 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:28.268566    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-bd0fb8-55f956944b-tl98x"
Sep 16 02:40:28.283557 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:28.283492    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-bd0fb8-6c4f6f65f-sd7gz" podStartSLOduration=2.283459816 podStartE2EDuration="2.283459816s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 02:40:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:40:28.280553748 +0000 UTC m=+1641.199043000" watchObservedRunningTime="2026-09-16 02:40:28.283459816 +0000 UTC m=+1641.201949066"
Sep 16 02:40:28.297231 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:28.297182    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-bd0fb8-55f956944b-tl98x" podStartSLOduration=2.29716692 podStartE2EDuration="2.29716692s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 02:40:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:40:28.295120933 +0000 UTC m=+1641.213610182" watchObservedRunningTime="2026-09-16 02:40:28.29716692 +0000 UTC m=+1641.215656169"
Sep 16 02:40:39.276053 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:39.276016    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-bd0fb8-6c4f6f65f-sd7gz"
Sep 16 02:40:39.276527 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:39.276089    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-bd0fb8-55f956944b-tl98x"
Sep 16 02:40:46.376740 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:46.376678    2558 generic.go:356] "Generic (PLEG): container finished" podID="f4a618df-1dcc-42ed-ac72-e519ee2c5515" containerID="f0fb1f59a7d7d0d73262d9156bcf6336d94a22a2b413eddef7118743709163cd" exitCode=0
Sep 16 02:40:46.377172 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:46.376750    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-bd0fb8-openshift-default-84f4588456-hdfcv" event={"ID":"f4a618df-1dcc-42ed-ac72-e519ee2c5515","Type":"ContainerDied","Data":"f0fb1f59a7d7d0d73262d9156bcf6336d94a22a2b413eddef7118743709163cd"}
Sep 16 02:40:46.377172 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:46.377147    2558 scope.go:117] "RemoveContainer" containerID="f0fb1f59a7d7d0d73262d9156bcf6336d94a22a2b413eddef7118743709163cd"
Sep 16 02:40:46.378231 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:46.378201    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 16 02:40:46.378293 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:46.378274    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 16 02:40:46.378337 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:46.378312    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 16 02:40:47.384352 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:47.384316    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-bd0fb8-openshift-default-84f4588456-hdfcv" event={"ID":"f4a618df-1dcc-42ed-ac72-e519ee2c5515","Type":"ContainerStarted","Data":"d547325cdc13aeac9bfe39a340d29152b7f4a03b1e7289fea7e446d97aebbb71"}
Sep 16 02:40:47.690764 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:47.690729    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-bd0fb8-openshift-default-84f4588456-hdfcv"
Sep 16 02:40:47.690981 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:47.690960    2558 patch_prober.go:28] interesting pod/e2e-shared-a-w1-bd0fb8-openshift-default-84f4588456-hdfcv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Sep 16 02:40:47.691031 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:47.691017    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-bd0fb8-openshift-default-84f4588456-hdfcv" podUID="f4a618df-1dcc-42ed-ac72-e519ee2c5515" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Sep 16 02:40:48.690095 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:48.690053    2558 patch_prober.go:28] interesting pod/e2e-shared-a-w1-bd0fb8-openshift-default-84f4588456-hdfcv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Sep 16 02:40:48.690577 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:48.690121    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-bd0fb8-openshift-default-84f4588456-hdfcv" podUID="f4a618df-1dcc-42ed-ac72-e519ee2c5515" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Sep 16 02:40:49.690986 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:49.690949    2558 patch_prober.go:28] interesting pod/e2e-shared-a-w1-bd0fb8-openshift-default-84f4588456-hdfcv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Sep 16 02:40:49.691379 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:49.691022    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-bd0fb8-openshift-default-84f4588456-hdfcv" podUID="f4a618df-1dcc-42ed-ac72-e519ee2c5515" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Sep 16 02:40:50.690476 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:50.690438    2558 patch_prober.go:28] interesting pod/e2e-shared-a-w1-bd0fb8-openshift-default-84f4588456-hdfcv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Sep 16 02:40:50.690681 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:50.690494    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-bd0fb8-openshift-default-84f4588456-hdfcv" podUID="f4a618df-1dcc-42ed-ac72-e519ee2c5515" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Sep 16 02:40:51.161166 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:51.161129    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-8e10f1-openshift-default-7596f6d7c6-88psf"]
Sep 16 02:40:51.167128 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:51.167101    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-8e10f1-openshift-default-7596f6d7c6-88psf"
Sep 16 02:40:51.171761 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:51.171733    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w1-8e10f1-openshift-default-dockercfg-6jdkq\""
Sep 16 02:40:51.179722 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:51.177671    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-8e10f1-openshift-default-7596f6d7c6-88psf"]
Sep 16 02:40:51.220230 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:51.220196    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de5a022a-4dfa-4881-b370-ef83e1129056-istio-data\") pod \"e2e-shared-b-w1-8e10f1-openshift-default-7596f6d7c6-88psf\" (UID: \"de5a022a-4dfa-4881-b370-ef83e1129056\") " pod="openshift-ingress/e2e-shared-b-w1-8e10f1-openshift-default-7596f6d7c6-88psf"
Sep 16 02:40:51.220380 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:51.220249    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jz25x\" (UniqueName: \"kubernetes.io/projected/de5a022a-4dfa-4881-b370-ef83e1129056-kube-api-access-jz25x\") pod \"e2e-shared-b-w1-8e10f1-openshift-default-7596f6d7c6-88psf\" (UID: \"de5a022a-4dfa-4881-b370-ef83e1129056\") " pod="openshift-ingress/e2e-shared-b-w1-8e10f1-openshift-default-7596f6d7c6-88psf"
Sep 16 02:40:51.220380 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:51.220281    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de5a022a-4dfa-4881-b370-ef83e1129056-istiod-ca-cert\") pod \"e2e-shared-b-w1-8e10f1-openshift-default-7596f6d7c6-88psf\" (UID: \"de5a022a-4dfa-4881-b370-ef83e1129056\") " pod="openshift-ingress/e2e-shared-b-w1-8e10f1-openshift-default-7596f6d7c6-88psf"
Sep 16 02:40:51.220380 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:51.220303    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de5a022a-4dfa-4881-b370-ef83e1129056-workload-certs\") pod \"e2e-shared-b-w1-8e10f1-openshift-default-7596f6d7c6-88psf\" (UID: \"de5a022a-4dfa-4881-b370-ef83e1129056\") " pod="openshift-ingress/e2e-shared-b-w1-8e10f1-openshift-default-7596f6d7c6-88psf"
Sep 16 02:40:51.220380 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:51.220321    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de5a022a-4dfa-4881-b370-ef83e1129056-credential-socket\") pod \"e2e-shared-b-w1-8e10f1-openshift-default-7596f6d7c6-88psf\" (UID: \"de5a022a-4dfa-4881-b370-ef83e1129056\") " pod="openshift-ingress/e2e-shared-b-w1-8e10f1-openshift-default-7596f6d7c6-88psf"
Sep 16 02:40:51.220535 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:51.220410    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de5a022a-4dfa-4881-b370-ef83e1129056-istio-podinfo\") pod \"e2e-shared-b-w1-8e10f1-openshift-default-7596f6d7c6-88psf\" (UID: \"de5a022a-4dfa-4881-b370-ef83e1129056\") " pod="openshift-ingress/e2e-shared-b-w1-8e10f1-openshift-default-7596f6d7c6-88psf"
Sep 16 02:40:51.220535 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:51.220425    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de5a022a-4dfa-4881-b370-ef83e1129056-istio-envoy\") pod \"e2e-shared-b-w1-8e10f1-openshift-default-7596f6d7c6-88psf\" (UID: \"de5a022a-4dfa-4881-b370-ef83e1129056\") " pod="openshift-ingress/e2e-shared-b-w1-8e10f1-openshift-default-7596f6d7c6-88psf"
Sep 16 02:40:51.220535 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:51.220443    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de5a022a-4dfa-4881-b370-ef83e1129056-istio-token\") pod \"e2e-shared-b-w1-8e10f1-openshift-default-7596f6d7c6-88psf\" (UID: \"de5a022a-4dfa-4881-b370-ef83e1129056\") " pod="openshift-ingress/e2e-shared-b-w1-8e10f1-openshift-default-7596f6d7c6-88psf"
Sep 16 02:40:51.220535 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:51.220462    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de5a022a-4dfa-4881-b370-ef83e1129056-workload-socket\") pod \"e2e-shared-b-w1-8e10f1-openshift-default-7596f6d7c6-88psf\" (UID: \"de5a022a-4dfa-4881-b370-ef83e1129056\") " pod="openshift-ingress/e2e-shared-b-w1-8e10f1-openshift-default-7596f6d7c6-88psf"
Sep 16 02:40:51.321068 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:51.321035    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jz25x\" (UniqueName: \"kubernetes.io/projected/de5a022a-4dfa-4881-b370-ef83e1129056-kube-api-access-jz25x\") pod \"e2e-shared-b-w1-8e10f1-openshift-default-7596f6d7c6-88psf\" (UID: \"de5a022a-4dfa-4881-b370-ef83e1129056\") " pod="openshift-ingress/e2e-shared-b-w1-8e10f1-openshift-default-7596f6d7c6-88psf"
Sep 16 02:40:51.321259 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:51.321075    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de5a022a-4dfa-4881-b370-ef83e1129056-istiod-ca-cert\") pod \"e2e-shared-b-w1-8e10f1-openshift-default-7596f6d7c6-88psf\" (UID: \"de5a022a-4dfa-4881-b370-ef83e1129056\") " pod="openshift-ingress/e2e-shared-b-w1-8e10f1-openshift-default-7596f6d7c6-88psf"
Sep 16 02:40:51.321259 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:51.321100    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de5a022a-4dfa-4881-b370-ef83e1129056-workload-certs\") pod \"e2e-shared-b-w1-8e10f1-openshift-default-7596f6d7c6-88psf\" (UID: \"de5a022a-4dfa-4881-b370-ef83e1129056\") " pod="openshift-ingress/e2e-shared-b-w1-8e10f1-openshift-default-7596f6d7c6-88psf"
Sep 16 02:40:51.321259 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:51.321126    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de5a022a-4dfa-4881-b370-ef83e1129056-credential-socket\") pod \"e2e-shared-b-w1-8e10f1-openshift-default-7596f6d7c6-88psf\" (UID: \"de5a022a-4dfa-4881-b370-ef83e1129056\") " pod="openshift-ingress/e2e-shared-b-w1-8e10f1-openshift-default-7596f6d7c6-88psf"
Sep 16 02:40:51.321259 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:51.321193    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de5a022a-4dfa-4881-b370-ef83e1129056-istio-podinfo\") pod \"e2e-shared-b-w1-8e10f1-openshift-default-7596f6d7c6-88psf\" (UID: \"de5a022a-4dfa-4881-b370-ef83e1129056\") " pod="openshift-ingress/e2e-shared-b-w1-8e10f1-openshift-default-7596f6d7c6-88psf"
Sep 16 02:40:51.321259 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:51.321219    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de5a022a-4dfa-4881-b370-ef83e1129056-istio-envoy\") pod \"e2e-shared-b-w1-8e10f1-openshift-default-7596f6d7c6-88psf\" (UID: \"de5a022a-4dfa-4881-b370-ef83e1129056\") " pod="openshift-ingress/e2e-shared-b-w1-8e10f1-openshift-default-7596f6d7c6-88psf"
Sep 16 02:40:51.321464 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:51.321351    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de5a022a-4dfa-4881-b370-ef83e1129056-istio-token\") pod \"e2e-shared-b-w1-8e10f1-openshift-default-7596f6d7c6-88psf\" (UID: \"de5a022a-4dfa-4881-b370-ef83e1129056\") " pod="openshift-ingress/e2e-shared-b-w1-8e10f1-openshift-default-7596f6d7c6-88psf"
Sep 16 02:40:51.321464 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:51.321397    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de5a022a-4dfa-4881-b370-ef83e1129056-workload-socket\") pod \"e2e-shared-b-w1-8e10f1-openshift-default-7596f6d7c6-88psf\" (UID: \"de5a022a-4dfa-4881-b370-ef83e1129056\") " pod="openshift-ingress/e2e-shared-b-w1-8e10f1-openshift-default-7596f6d7c6-88psf"
Sep 16 02:40:51.321556 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:51.321489    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de5a022a-4dfa-4881-b370-ef83e1129056-istio-data\") pod \"e2e-shared-b-w1-8e10f1-openshift-default-7596f6d7c6-88psf\" (UID: \"de5a022a-4dfa-4881-b370-ef83e1129056\") " pod="openshift-ingress/e2e-shared-b-w1-8e10f1-openshift-default-7596f6d7c6-88psf"
Sep 16 02:40:51.321686 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:51.321665    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de5a022a-4dfa-4881-b370-ef83e1129056-credential-socket\") pod \"e2e-shared-b-w1-8e10f1-openshift-default-7596f6d7c6-88psf\" (UID: \"de5a022a-4dfa-4881-b370-ef83e1129056\") " pod="openshift-ingress/e2e-shared-b-w1-8e10f1-openshift-default-7596f6d7c6-88psf"
Sep 16 02:40:51.321777 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:51.321690    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de5a022a-4dfa-4881-b370-ef83e1129056-workload-certs\") pod \"e2e-shared-b-w1-8e10f1-openshift-default-7596f6d7c6-88psf\" (UID: \"de5a022a-4dfa-4881-b370-ef83e1129056\") " pod="openshift-ingress/e2e-shared-b-w1-8e10f1-openshift-default-7596f6d7c6-88psf"
Sep 16 02:40:51.321868 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:51.321843    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de5a022a-4dfa-4881-b370-ef83e1129056-workload-socket\") pod \"e2e-shared-b-w1-8e10f1-openshift-default-7596f6d7c6-88psf\" (UID: \"de5a022a-4dfa-4881-b370-ef83e1129056\") " pod="openshift-ingress/e2e-shared-b-w1-8e10f1-openshift-default-7596f6d7c6-88psf"
Sep 16 02:40:51.321912 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:51.321864    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de5a022a-4dfa-4881-b370-ef83e1129056-istio-data\") pod \"e2e-shared-b-w1-8e10f1-openshift-default-7596f6d7c6-88psf\" (UID: \"de5a022a-4dfa-4881-b370-ef83e1129056\") " pod="openshift-ingress/e2e-shared-b-w1-8e10f1-openshift-default-7596f6d7c6-88psf"
Sep 16 02:40:51.321951 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:51.321938    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de5a022a-4dfa-4881-b370-ef83e1129056-istiod-ca-cert\") pod \"e2e-shared-b-w1-8e10f1-openshift-default-7596f6d7c6-88psf\" (UID: \"de5a022a-4dfa-4881-b370-ef83e1129056\") " pod="openshift-ingress/e2e-shared-b-w1-8e10f1-openshift-default-7596f6d7c6-88psf"
Sep 16 02:40:51.323667 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:51.323642    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de5a022a-4dfa-4881-b370-ef83e1129056-istio-envoy\") pod \"e2e-shared-b-w1-8e10f1-openshift-default-7596f6d7c6-88psf\" (UID: \"de5a022a-4dfa-4881-b370-ef83e1129056\") " pod="openshift-ingress/e2e-shared-b-w1-8e10f1-openshift-default-7596f6d7c6-88psf"
Sep 16 02:40:51.323941 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:51.323922    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de5a022a-4dfa-4881-b370-ef83e1129056-istio-podinfo\") pod \"e2e-shared-b-w1-8e10f1-openshift-default-7596f6d7c6-88psf\" (UID: \"de5a022a-4dfa-4881-b370-ef83e1129056\") " pod="openshift-ingress/e2e-shared-b-w1-8e10f1-openshift-default-7596f6d7c6-88psf"
Sep 16 02:40:51.328029 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:51.328009    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de5a022a-4dfa-4881-b370-ef83e1129056-istio-token\") pod \"e2e-shared-b-w1-8e10f1-openshift-default-7596f6d7c6-88psf\" (UID: \"de5a022a-4dfa-4881-b370-ef83e1129056\") " pod="openshift-ingress/e2e-shared-b-w1-8e10f1-openshift-default-7596f6d7c6-88psf"
Sep 16 02:40:51.328591 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:51.328566    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jz25x\" (UniqueName: \"kubernetes.io/projected/de5a022a-4dfa-4881-b370-ef83e1129056-kube-api-access-jz25x\") pod \"e2e-shared-b-w1-8e10f1-openshift-default-7596f6d7c6-88psf\" (UID: \"de5a022a-4dfa-4881-b370-ef83e1129056\") " pod="openshift-ingress/e2e-shared-b-w1-8e10f1-openshift-default-7596f6d7c6-88psf"
Sep 16 02:40:51.489573 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:51.489476    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-8e10f1-openshift-default-7596f6d7c6-88psf"
Sep 16 02:40:51.690209 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:51.690173    2558 patch_prober.go:28] interesting pod/e2e-shared-a-w1-bd0fb8-openshift-default-84f4588456-hdfcv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Sep 16 02:40:51.690377 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:51.690233    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-bd0fb8-openshift-default-84f4588456-hdfcv" podUID="f4a618df-1dcc-42ed-ac72-e519ee2c5515" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Sep 16 02:40:51.789373 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:51.789348    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-8e10f1-openshift-default-7596f6d7c6-88psf"]
Sep 16 02:40:51.791823 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:40:51.791801    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde5a022a_4dfa_4881_b370_ef83e1129056.slice/crio-889ba7bddf2705902f9abe77589305821e8c5245a77e177ed51ef26e5f614c44 WatchSource:0}: Error finding container 889ba7bddf2705902f9abe77589305821e8c5245a77e177ed51ef26e5f614c44: Status 404 returned error can't find the container with id 889ba7bddf2705902f9abe77589305821e8c5245a77e177ed51ef26e5f614c44
Sep 16 02:40:51.794111 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:51.794078    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 16 02:40:51.794195 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:51.794140    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 16 02:40:51.794195 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:51.794178    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 16 02:40:52.414141 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:52.414106    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-8e10f1-openshift-default-7596f6d7c6-88psf" event={"ID":"de5a022a-4dfa-4881-b370-ef83e1129056","Type":"ContainerStarted","Data":"4265b8c81d8c3a9475be8a47d2efd65c89590dad5678176f49fe345e3435b643"}
Sep 16 02:40:52.414770 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:52.414180    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-8e10f1-openshift-default-7596f6d7c6-88psf" event={"ID":"de5a022a-4dfa-4881-b370-ef83e1129056","Type":"ContainerStarted","Data":"889ba7bddf2705902f9abe77589305821e8c5245a77e177ed51ef26e5f614c44"}
Sep 16 02:40:52.434068 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:52.433791    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-8e10f1-openshift-default-7596f6d7c6-88psf" podStartSLOduration=1.4337710430000001 podStartE2EDuration="1.433771043s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 02:40:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:40:52.432411843 +0000 UTC m=+1665.350901092" watchObservedRunningTime="2026-09-16 02:40:52.433771043 +0000 UTC m=+1665.352260292"
Sep 16 02:40:52.490265 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:52.490221    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-8e10f1-openshift-default-7596f6d7c6-88psf"
Sep 16 02:40:52.496927 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:52.496891    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-8e10f1-openshift-default-7596f6d7c6-88psf"
Sep 16 02:40:52.694281 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:52.694201    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-bd0fb8-openshift-default-84f4588456-hdfcv"
Sep 16 02:40:52.694576 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:52.694555    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-bd0fb8-openshift-default-84f4588456-hdfcv"
Sep 16 02:40:52.695282 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:52.695259    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-bd0fb8-openshift-default-84f4588456-hdfcv"
Sep 16 02:40:53.419277 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:53.419247    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-8e10f1-openshift-default-7596f6d7c6-88psf"
Sep 16 02:40:53.420268 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:40:53.420249    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-8e10f1-openshift-default-7596f6d7c6-88psf"
Sep 16 02:41:10.544152 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:41:10.544072    2558 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-processing-5d7dbbd56b-pltkr" secret="" err="secret \"payload-processing-dockercfg-zzx6j\" not found"
Sep 16 02:41:13.525597 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:41:13.525487    2558 generic.go:356] "Generic (PLEG): container finished" podID="de5a022a-4dfa-4881-b370-ef83e1129056" containerID="4265b8c81d8c3a9475be8a47d2efd65c89590dad5678176f49fe345e3435b643" exitCode=0
Sep 16 02:41:13.525597 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:41:13.525567    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-8e10f1-openshift-default-7596f6d7c6-88psf" event={"ID":"de5a022a-4dfa-4881-b370-ef83e1129056","Type":"ContainerDied","Data":"4265b8c81d8c3a9475be8a47d2efd65c89590dad5678176f49fe345e3435b643"}
Sep 16 02:41:13.526202 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:41:13.526122    2558 scope.go:117] "RemoveContainer" containerID="4265b8c81d8c3a9475be8a47d2efd65c89590dad5678176f49fe345e3435b643"
Sep 16 02:41:13.527325 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:41:13.527296    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 16 02:41:13.527389 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:41:13.527368    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 16 02:41:13.527427 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:41:13.527398    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 16 02:41:14.532934 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:41:14.532901    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-8e10f1-openshift-default-7596f6d7c6-88psf" event={"ID":"de5a022a-4dfa-4881-b370-ef83e1129056","Type":"ContainerStarted","Data":"bf63b47f4de688b06683f339141b276decb756f82e02b1504102a7f13fd5985c"}
Sep 16 02:41:15.490149 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:41:15.490104    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-8e10f1-openshift-default-7596f6d7c6-88psf"
Sep 16 02:41:15.490396 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:41:15.490374    2558 patch_prober.go:28] interesting pod/e2e-shared-b-w1-8e10f1-openshift-default-7596f6d7c6-88psf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Sep 16 02:41:15.490451 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:41:15.490424    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-8e10f1-openshift-default-7596f6d7c6-88psf" podUID="de5a022a-4dfa-4881-b370-ef83e1129056" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Sep 16 02:41:16.490873 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:41:16.490832    2558 patch_prober.go:28] interesting pod/e2e-shared-b-w1-8e10f1-openshift-default-7596f6d7c6-88psf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Sep 16 02:41:16.491295 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:41:16.490890    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-8e10f1-openshift-default-7596f6d7c6-88psf" podUID="de5a022a-4dfa-4881-b370-ef83e1129056" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Sep 16 02:41:17.490121 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:41:17.490080    2558 patch_prober.go:28] interesting pod/e2e-shared-b-w1-8e10f1-openshift-default-7596f6d7c6-88psf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Sep 16 02:41:17.490323 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:41:17.490155    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-8e10f1-openshift-default-7596f6d7c6-88psf" podUID="de5a022a-4dfa-4881-b370-ef83e1129056" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Sep 16 02:41:18.490040 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:41:18.490004    2558 patch_prober.go:28] interesting pod/e2e-shared-b-w1-8e10f1-openshift-default-7596f6d7c6-88psf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Sep 16 02:41:18.490447 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:41:18.490061    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-8e10f1-openshift-default-7596f6d7c6-88psf" podUID="de5a022a-4dfa-4881-b370-ef83e1129056" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Sep 16 02:41:19.490851 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:41:19.490813    2558 patch_prober.go:28] interesting pod/e2e-shared-b-w1-8e10f1-openshift-default-7596f6d7c6-88psf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Sep 16 02:41:19.491241 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:41:19.490869    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-8e10f1-openshift-default-7596f6d7c6-88psf" podUID="de5a022a-4dfa-4881-b370-ef83e1129056" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Sep 16 02:41:20.494126 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:41:20.494093    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-8e10f1-openshift-default-7596f6d7c6-88psf"
Sep 16 02:41:20.494550 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:41:20.494366    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-8e10f1-openshift-default-7596f6d7c6-88psf"
Sep 16 02:41:20.495040 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:41:20.495024    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-8e10f1-openshift-default-7596f6d7c6-88psf"
Sep 16 02:41:29.774166 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:41:29.774047    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-77f43ab3-openshift-default-ccbc9c496-ngb48"]
Sep 16 02:41:29.778223 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:41:29.778198    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-77f43ab3-openshift-default-ccbc9c496-ngb48"
Sep 16 02:41:29.780559 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:41:29.780537    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-77f43ab3-openshift-default-dockercfg-vwfzm\""
Sep 16 02:41:29.789098 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:41:29.789037    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-77f43ab3-openshift-default-ccbc9c496-ngb48"]
Sep 16 02:41:29.907047 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:41:29.907007    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8114a0e-c5be-4502-9c31-4b8421406542-istiod-ca-cert\") pod \"e2e-mt-77f43ab3-openshift-default-ccbc9c496-ngb48\" (UID: \"a8114a0e-c5be-4502-9c31-4b8421406542\") " pod="openshift-ingress/e2e-mt-77f43ab3-openshift-default-ccbc9c496-ngb48"
Sep 16 02:41:29.907249 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:41:29.907059    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8114a0e-c5be-4502-9c31-4b8421406542-workload-socket\") pod \"e2e-mt-77f43ab3-openshift-default-ccbc9c496-ngb48\" (UID: \"a8114a0e-c5be-4502-9c31-4b8421406542\") " pod="openshift-ingress/e2e-mt-77f43ab3-openshift-default-ccbc9c496-ngb48"
Sep 16 02:41:29.907249 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:41:29.907084    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8114a0e-c5be-4502-9c31-4b8421406542-credential-socket\") pod \"e2e-mt-77f43ab3-openshift-default-ccbc9c496-ngb48\" (UID: \"a8114a0e-c5be-4502-9c31-4b8421406542\") " pod="openshift-ingress/e2e-mt-77f43ab3-openshift-default-ccbc9c496-ngb48"
Sep 16 02:41:29.907249 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:41:29.907104    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8114a0e-c5be-4502-9c31-4b8421406542-istio-data\") pod \"e2e-mt-77f43ab3-openshift-default-ccbc9c496-ngb48\" (UID: \"a8114a0e-c5be-4502-9c31-4b8421406542\") " pod="openshift-ingress/e2e-mt-77f43ab3-openshift-default-ccbc9c496-ngb48"
Sep 16 02:41:29.907249 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:41:29.907146    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8114a0e-c5be-4502-9c31-4b8421406542-istio-token\") pod \"e2e-mt-77f43ab3-openshift-default-ccbc9c496-ngb48\" (UID: \"a8114a0e-c5be-4502-9c31-4b8421406542\") " pod="openshift-ingress/e2e-mt-77f43ab3-openshift-default-ccbc9c496-ngb48"
Sep 16 02:41:29.907412 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:41:29.907267    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8114a0e-c5be-4502-9c31-4b8421406542-istio-podinfo\") pod \"e2e-mt-77f43ab3-openshift-default-ccbc9c496-ngb48\" (UID: \"a8114a0e-c5be-4502-9c31-4b8421406542\") " pod="openshift-ingress/e2e-mt-77f43ab3-openshift-default-ccbc9c496-ngb48"
Sep 16 02:41:29.907412 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:41:29.907332    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8114a0e-c5be-4502-9c31-4b8421406542-workload-certs\") pod \"e2e-mt-77f43ab3-openshift-default-ccbc9c496-ngb48\" (UID: \"a8114a0e-c5be-4502-9c31-4b8421406542\") " pod="openshift-ingress/e2e-mt-77f43ab3-openshift-default-ccbc9c496-ngb48"
Sep 16 02:41:29.907412 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:41:29.907359    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnq2c\" (UniqueName: \"kubernetes.io/projected/a8114a0e-c5be-4502-9c31-4b8421406542-kube-api-access-gnq2c\") pod \"e2e-mt-77f43ab3-openshift-default-ccbc9c496-ngb48\" (UID: \"a8114a0e-c5be-4502-9c31-4b8421406542\") " pod="openshift-ingress/e2e-mt-77f43ab3-openshift-default-ccbc9c496-ngb48"
Sep 16 02:41:29.907412 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:41:29.907396    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8114a0e-c5be-4502-9c31-4b8421406542-istio-envoy\") pod \"e2e-mt-77f43ab3-openshift-default-ccbc9c496-ngb48\" (UID: \"a8114a0e-c5be-4502-9c31-4b8421406542\") " pod="openshift-ingress/e2e-mt-77f43ab3-openshift-default-ccbc9c496-ngb48"
Sep 16 02:41:30.008857 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:41:30.008813    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnq2c\" (UniqueName: \"kubernetes.io/projected/a8114a0e-c5be-4502-9c31-4b8421406542-kube-api-access-gnq2c\") pod \"e2e-mt-77f43ab3-openshift-default-ccbc9c496-ngb48\" (UID: \"a8114a0e-c5be-4502-9c31-4b8421406542\") " pod="openshift-ingress/e2e-mt-77f43ab3-openshift-default-ccbc9c496-ngb48"
Sep 16 02:41:30.009047 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:41:30.008868    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8114a0e-c5be-4502-9c31-4b8421406542-istio-envoy\") pod \"e2e-mt-77f43ab3-openshift-default-ccbc9c496-ngb48\" (UID: \"a8114a0e-c5be-4502-9c31-4b8421406542\") " pod="openshift-ingress/e2e-mt-77f43ab3-openshift-default-ccbc9c496-ngb48"
Sep 16 02:41:30.009047 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:41:30.008915    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8114a0e-c5be-4502-9c31-4b8421406542-istiod-ca-cert\") pod \"e2e-mt-77f43ab3-openshift-default-ccbc9c496-ngb48\" (UID: \"a8114a0e-c5be-4502-9c31-4b8421406542\") " pod="openshift-ingress/e2e-mt-77f43ab3-openshift-default-ccbc9c496-ngb48"
Sep 16 02:41:30.009047 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:41:30.008945    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8114a0e-c5be-4502-9c31-4b8421406542-workload-socket\") pod \"e2e-mt-77f43ab3-openshift-default-ccbc9c496-ngb48\" (UID: \"a8114a0e-c5be-4502-9c31-4b8421406542\") " pod="openshift-ingress/e2e-mt-77f43ab3-openshift-default-ccbc9c496-ngb48"
Sep 16 02:41:30.009047 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:41:30.008967    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8114a0e-c5be-4502-9c31-4b8421406542-credential-socket\") pod \"e2e-mt-77f43ab3-openshift-default-ccbc9c496-ngb48\" (UID: \"a8114a0e-c5be-4502-9c31-4b8421406542\") " pod="openshift-ingress/e2e-mt-77f43ab3-openshift-default-ccbc9c496-ngb48"
Sep 16 02:41:30.009271 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:41:30.009064    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8114a0e-c5be-4502-9c31-4b8421406542-istio-data\") pod \"e2e-mt-77f43ab3-openshift-default-ccbc9c496-ngb48\" (UID: \"a8114a0e-c5be-4502-9c31-4b8421406542\") " pod="openshift-ingress/e2e-mt-77f43ab3-openshift-default-ccbc9c496-ngb48"
Sep 16 02:41:30.009271 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:41:30.009115    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8114a0e-c5be-4502-9c31-4b8421406542-istio-token\") pod \"e2e-mt-77f43ab3-openshift-default-ccbc9c496-ngb48\" (UID: \"a8114a0e-c5be-4502-9c31-4b8421406542\") " pod="openshift-ingress/e2e-mt-77f43ab3-openshift-default-ccbc9c496-ngb48"
Sep 16 02:41:30.009271 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:41:30.009243    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8114a0e-c5be-4502-9c31-4b8421406542-istio-podinfo\") pod \"e2e-mt-77f43ab3-openshift-default-ccbc9c496-ngb48\" (UID: \"a8114a0e-c5be-4502-9c31-4b8421406542\") " pod="openshift-ingress/e2e-mt-77f43ab3-openshift-default-ccbc9c496-ngb48"
Sep 16 02:41:30.009428 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:41:30.009341    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8114a0e-c5be-4502-9c31-4b8421406542-workload-certs\") pod \"e2e-mt-77f43ab3-openshift-default-ccbc9c496-ngb48\" (UID: \"a8114a0e-c5be-4502-9c31-4b8421406542\") " pod="openshift-ingress/e2e-mt-77f43ab3-openshift-default-ccbc9c496-ngb48"
Sep 16 02:41:30.009476 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:41:30.009426    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8114a0e-c5be-4502-9c31-4b8421406542-credential-socket\") pod \"e2e-mt-77f43ab3-openshift-default-ccbc9c496-ngb48\" (UID: \"a8114a0e-c5be-4502-9c31-4b8421406542\") " pod="openshift-ingress/e2e-mt-77f43ab3-openshift-default-ccbc9c496-ngb48"
Sep 16 02:41:30.009476 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:41:30.009461    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8114a0e-c5be-4502-9c31-4b8421406542-istio-data\") pod \"e2e-mt-77f43ab3-openshift-default-ccbc9c496-ngb48\" (UID: \"a8114a0e-c5be-4502-9c31-4b8421406542\") " pod="openshift-ingress/e2e-mt-77f43ab3-openshift-default-ccbc9c496-ngb48"
Sep 16 02:41:30.009637 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:41:30.009617    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8114a0e-c5be-4502-9c31-4b8421406542-workload-socket\") pod \"e2e-mt-77f43ab3-openshift-default-ccbc9c496-ngb48\" (UID: \"a8114a0e-c5be-4502-9c31-4b8421406542\") " pod="openshift-ingress/e2e-mt-77f43ab3-openshift-default-ccbc9c496-ngb48"
Sep 16 02:41:30.009684 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:41:30.009653    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8114a0e-c5be-4502-9c31-4b8421406542-workload-certs\") pod \"e2e-mt-77f43ab3-openshift-default-ccbc9c496-ngb48\" (UID: \"a8114a0e-c5be-4502-9c31-4b8421406542\") " pod="openshift-ingress/e2e-mt-77f43ab3-openshift-default-ccbc9c496-ngb48"
Sep 16 02:41:30.009785 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:41:30.009760    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8114a0e-c5be-4502-9c31-4b8421406542-istiod-ca-cert\") pod \"e2e-mt-77f43ab3-openshift-default-ccbc9c496-ngb48\" (UID: \"a8114a0e-c5be-4502-9c31-4b8421406542\") " pod="openshift-ingress/e2e-mt-77f43ab3-openshift-default-ccbc9c496-ngb48"
Sep 16 02:41:30.011270 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:41:30.011241    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8114a0e-c5be-4502-9c31-4b8421406542-istio-envoy\") pod \"e2e-mt-77f43ab3-openshift-default-ccbc9c496-ngb48\" (UID: \"a8114a0e-c5be-4502-9c31-4b8421406542\") " pod="openshift-ingress/e2e-mt-77f43ab3-openshift-default-ccbc9c496-ngb48"
Sep 16 02:41:30.011578 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:41:30.011555    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8114a0e-c5be-4502-9c31-4b8421406542-istio-podinfo\") pod \"e2e-mt-77f43ab3-openshift-default-ccbc9c496-ngb48\" (UID: \"a8114a0e-c5be-4502-9c31-4b8421406542\") " pod="openshift-ingress/e2e-mt-77f43ab3-openshift-default-ccbc9c496-ngb48"
Sep 16 02:41:30.018063 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:41:30.018044    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8114a0e-c5be-4502-9c31-4b8421406542-istio-token\") pod \"e2e-mt-77f43ab3-openshift-default-ccbc9c496-ngb48\" (UID: \"a8114a0e-c5be-4502-9c31-4b8421406542\") " pod="openshift-ingress/e2e-mt-77f43ab3-openshift-default-ccbc9c496-ngb48"
Sep 16 02:41:30.018151 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:41:30.018103    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnq2c\" (UniqueName: \"kubernetes.io/projected/a8114a0e-c5be-4502-9c31-4b8421406542-kube-api-access-gnq2c\") pod \"e2e-mt-77f43ab3-openshift-default-ccbc9c496-ngb48\" (UID: \"a8114a0e-c5be-4502-9c31-4b8421406542\") " pod="openshift-ingress/e2e-mt-77f43ab3-openshift-default-ccbc9c496-ngb48"
Sep 16 02:41:30.090560 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:41:30.090476    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-77f43ab3-openshift-default-ccbc9c496-ngb48"
Sep 16 02:41:30.364051 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:41:30.364024    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-77f43ab3-openshift-default-ccbc9c496-ngb48"]
Sep 16 02:41:30.365600 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:41:30.365575    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8114a0e_c5be_4502_9c31_4b8421406542.slice/crio-e9ab55ce371665be4c45d677ba2a23b5c964f496262b74b9ba35bc59cbfc784d WatchSource:0}: Error finding container e9ab55ce371665be4c45d677ba2a23b5c964f496262b74b9ba35bc59cbfc784d: Status 404 returned error can't find the container with id e9ab55ce371665be4c45d677ba2a23b5c964f496262b74b9ba35bc59cbfc784d
Sep 16 02:41:30.368016 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:41:30.367980    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 16 02:41:30.368100 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:41:30.368048    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 16 02:41:30.368100 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:41:30.368080    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 16 02:41:30.656227 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:41:30.656189    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-77f43ab3-openshift-default-ccbc9c496-ngb48" event={"ID":"a8114a0e-c5be-4502-9c31-4b8421406542","Type":"ContainerStarted","Data":"07d34ae69ecd7473ed7c18863646ff2eca3ad9ca9d811fb8a9e30c9cf7c35711"}
Sep 16 02:41:30.656425 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:41:30.656249    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-77f43ab3-openshift-default-ccbc9c496-ngb48" event={"ID":"a8114a0e-c5be-4502-9c31-4b8421406542","Type":"ContainerStarted","Data":"e9ab55ce371665be4c45d677ba2a23b5c964f496262b74b9ba35bc59cbfc784d"}
Sep 16 02:41:30.680863 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:41:30.680805    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-77f43ab3-openshift-default-ccbc9c496-ngb48" podStartSLOduration=1.680786769 podStartE2EDuration="1.680786769s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 02:41:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:41:30.677233517 +0000 UTC m=+1703.595722767" watchObservedRunningTime="2026-09-16 02:41:30.680786769 +0000 UTC m=+1703.599276020"
Sep 16 02:41:31.091391 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:41:31.091299    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-77f43ab3-openshift-default-ccbc9c496-ngb48"
Sep 16 02:41:31.096937 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:41:31.096912    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-77f43ab3-openshift-default-ccbc9c496-ngb48"
Sep 16 02:41:31.662171 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:41:31.662130    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-77f43ab3-openshift-default-ccbc9c496-ngb48"
Sep 16 02:41:31.663003 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:41:31.662984    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-77f43ab3-openshift-default-ccbc9c496-ngb48"
Sep 16 02:41:51.789285 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:41:51.789248    2558 generic.go:356] "Generic (PLEG): container finished" podID="a8114a0e-c5be-4502-9c31-4b8421406542" containerID="07d34ae69ecd7473ed7c18863646ff2eca3ad9ca9d811fb8a9e30c9cf7c35711" exitCode=0
Sep 16 02:41:51.789766 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:41:51.789325    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-77f43ab3-openshift-default-ccbc9c496-ngb48" event={"ID":"a8114a0e-c5be-4502-9c31-4b8421406542","Type":"ContainerDied","Data":"07d34ae69ecd7473ed7c18863646ff2eca3ad9ca9d811fb8a9e30c9cf7c35711"}
Sep 16 02:41:51.789848 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:41:51.789801    2558 scope.go:117] "RemoveContainer" containerID="07d34ae69ecd7473ed7c18863646ff2eca3ad9ca9d811fb8a9e30c9cf7c35711"
Sep 16 02:41:51.790912 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:41:51.790882    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 16 02:41:51.790991 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:41:51.790949    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 16 02:41:51.790991 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:41:51.790980    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 16 02:41:52.796851 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:41:52.796816    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-77f43ab3-openshift-default-ccbc9c496-ngb48" event={"ID":"a8114a0e-c5be-4502-9c31-4b8421406542","Type":"ContainerStarted","Data":"1c3f1f9790aaf20bde78e311dd8d962c6e90f40164e78db31320e63a3c347cc2"}
Sep 16 02:41:53.090827 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:41:53.090739    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-77f43ab3-openshift-default-ccbc9c496-ngb48"
Sep 16 02:41:53.091092 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:41:53.091068    2558 patch_prober.go:28] interesting pod/e2e-mt-77f43ab3-openshift-default-ccbc9c496-ngb48 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Sep 16 02:41:53.091151 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:41:53.091117    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-77f43ab3-openshift-default-ccbc9c496-ngb48" podUID="a8114a0e-c5be-4502-9c31-4b8421406542" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Sep 16 02:41:54.091404 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:41:54.091364    2558 patch_prober.go:28] interesting pod/e2e-mt-77f43ab3-openshift-default-ccbc9c496-ngb48 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Sep 16 02:41:54.091846 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:41:54.091424    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-77f43ab3-openshift-default-ccbc9c496-ngb48" podUID="a8114a0e-c5be-4502-9c31-4b8421406542" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Sep 16 02:41:55.090848 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:41:55.090812    2558 patch_prober.go:28] interesting pod/e2e-mt-77f43ab3-openshift-default-ccbc9c496-ngb48 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Sep 16 02:41:55.091035 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:41:55.090863    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-77f43ab3-openshift-default-ccbc9c496-ngb48" podUID="a8114a0e-c5be-4502-9c31-4b8421406542" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Sep 16 02:41:56.090924 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:41:56.090887    2558 patch_prober.go:28] interesting pod/e2e-mt-77f43ab3-openshift-default-ccbc9c496-ngb48 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Sep 16 02:41:56.091319 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:41:56.090941    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-77f43ab3-openshift-default-ccbc9c496-ngb48" podUID="a8114a0e-c5be-4502-9c31-4b8421406542" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Sep 16 02:41:57.091408 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:41:57.091367    2558 patch_prober.go:28] interesting pod/e2e-mt-77f43ab3-openshift-default-ccbc9c496-ngb48 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Sep 16 02:41:57.091889 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:41:57.091445    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-77f43ab3-openshift-default-ccbc9c496-ngb48" podUID="a8114a0e-c5be-4502-9c31-4b8421406542" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Sep 16 02:41:58.091497 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:41:58.091457    2558 patch_prober.go:28] interesting pod/e2e-mt-77f43ab3-openshift-default-ccbc9c496-ngb48 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Sep 16 02:41:58.092032 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:41:58.091512    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-77f43ab3-openshift-default-ccbc9c496-ngb48" podUID="a8114a0e-c5be-4502-9c31-4b8421406542" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Sep 16 02:41:59.095240 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:41:59.095205    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-77f43ab3-openshift-default-ccbc9c496-ngb48"
Sep 16 02:41:59.095687 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:41:59.095496    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-77f43ab3-openshift-default-ccbc9c496-ngb48"
Sep 16 02:41:59.096169 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:41:59.096153    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-77f43ab3-openshift-default-ccbc9c496-ngb48"
Sep 16 02:42:07.782094 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:42:07.782059    2558 scope.go:117] "RemoveContainer" containerID="6ba257f9fa9a6db8acfcb229fac839b9c2c095ad70a4e84d5544ac3325429347"
Sep 16 02:42:23.544496 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:42:23.544455    2558 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-processing-5d7dbbd56b-pltkr" secret="" err="secret \"payload-processing-dockercfg-zzx6j\" not found"
Sep 16 02:42:58.221282 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:42:58.221250    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-77f43ab3-rwb2l"]
Sep 16 02:42:58.224865 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:42:58.224848    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-77f43ab3-rwb2l"
Sep 16 02:42:58.226999 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:42:58.226974    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 16 02:42:58.227126 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:42:58.227070    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jpd4f\""
Sep 16 02:42:58.231634 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:42:58.231605    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-77f43ab3-rwb2l"]
Sep 16 02:42:58.392028 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:42:58.391998    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/354cd26d-662f-445b-8bce-10158d55d60d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-77f43ab3-rwb2l\" (UID: \"354cd26d-662f-445b-8bce-10158d55d60d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-77f43ab3-rwb2l"
Sep 16 02:42:58.492991 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:42:58.492918    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/354cd26d-662f-445b-8bce-10158d55d60d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-77f43ab3-rwb2l\" (UID: \"354cd26d-662f-445b-8bce-10158d55d60d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-77f43ab3-rwb2l"
Sep 16 02:42:58.493528 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:42:58.493504    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/354cd26d-662f-445b-8bce-10158d55d60d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-77f43ab3-rwb2l\" (UID: \"354cd26d-662f-445b-8bce-10158d55d60d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-77f43ab3-rwb2l"
Sep 16 02:42:58.538438 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:42:58.538409    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-77f43ab3-rwb2l"
Sep 16 02:42:58.812112 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:42:58.812090    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-77f43ab3-rwb2l"]
Sep 16 02:42:58.813929 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:42:58.813901    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod354cd26d_662f_445b_8bce_10158d55d60d.slice/crio-39ae5be5028eec29c241210305deaef66fc95dc34a06dd2d6b7e462cf7eb59b6 WatchSource:0}: Error finding container 39ae5be5028eec29c241210305deaef66fc95dc34a06dd2d6b7e462cf7eb59b6: Status 404 returned error can't find the container with id 39ae5be5028eec29c241210305deaef66fc95dc34a06dd2d6b7e462cf7eb59b6
Sep 16 02:42:59.164063 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:42:59.164026    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-77f43ab3-rwb2l" event={"ID":"354cd26d-662f-445b-8bce-10158d55d60d","Type":"ContainerStarted","Data":"d2bb20ae412d51cc9513cca0fa74941c4a056ebffca4f95b5ee82ef93b6dec84"}
Sep 16 02:42:59.164230 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:42:59.164077    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-77f43ab3-rwb2l" event={"ID":"354cd26d-662f-445b-8bce-10158d55d60d","Type":"ContainerStarted","Data":"39ae5be5028eec29c241210305deaef66fc95dc34a06dd2d6b7e462cf7eb59b6"}
Sep 16 02:42:59.191654 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:42:59.191602    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-77f43ab3-rwb2l" podStartSLOduration=1.191586214 podStartE2EDuration="1.191586214s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 02:42:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:42:59.190578753 +0000 UTC m=+1792.109068016" watchObservedRunningTime="2026-09-16 02:42:59.191586214 +0000 UTC m=+1792.110075482"
Sep 16 02:43:00.171031 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:43:00.170999    2558 generic.go:356] "Generic (PLEG): container finished" podID="354cd26d-662f-445b-8bce-10158d55d60d" containerID="d2bb20ae412d51cc9513cca0fa74941c4a056ebffca4f95b5ee82ef93b6dec84" exitCode=0
Sep 16 02:43:00.171506 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:43:00.171084    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-77f43ab3-rwb2l" event={"ID":"354cd26d-662f-445b-8bce-10158d55d60d","Type":"ContainerDied","Data":"d2bb20ae412d51cc9513cca0fa74941c4a056ebffca4f95b5ee82ef93b6dec84"}
Sep 16 02:43:01.318203 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:43:01.318168    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-77f43ab3-rwb2l"
Sep 16 02:43:01.422788 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:43:01.422751    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/354cd26d-662f-445b-8bce-10158d55d60d-maas-api-service-ca\") pod \"354cd26d-662f-445b-8bce-10158d55d60d\" (UID: \"354cd26d-662f-445b-8bce-10158d55d60d\") "
Sep 16 02:43:01.423113 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:43:01.423086    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/354cd26d-662f-445b-8bce-10158d55d60d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "354cd26d-662f-445b-8bce-10158d55d60d" (UID: "354cd26d-662f-445b-8bce-10158d55d60d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 02:43:01.524477 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:43:01.524384    2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/354cd26d-662f-445b-8bce-10158d55d60d-maas-api-service-ca\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:43:02.182916 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:43:02.182879    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-77f43ab3-rwb2l"
Sep 16 02:43:02.183092 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:43:02.182916    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-77f43ab3-rwb2l" event={"ID":"354cd26d-662f-445b-8bce-10158d55d60d","Type":"ContainerDied","Data":"39ae5be5028eec29c241210305deaef66fc95dc34a06dd2d6b7e462cf7eb59b6"}
Sep 16 02:43:02.183092 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:43:02.182964    2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="39ae5be5028eec29c241210305deaef66fc95dc34a06dd2d6b7e462cf7eb59b6"
Sep 16 02:43:07.803256 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:43:07.803222    2558 scope.go:117] "RemoveContainer" containerID="aefe1b4a34baee9387c9397e9d0bf71bea316c74b02625fbb6502fb30451f122"
Sep 16 02:43:07.811660 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:43:07.811637    2558 scope.go:117] "RemoveContainer" containerID="2d2afcaf99d50ae1fff28690713ba0e4e1a351feeb0ba3d1a6d62e8b31343f3c"
Sep 16 02:43:14.047031 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:43:14.046992    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-77f43ab3-rwb2l"]
Sep 16 02:43:14.048966 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:43:14.048944    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-77f43ab3-rwb2l"]
Sep 16 02:43:15.549947 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:43:15.549913    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="354cd26d-662f-445b-8bce-10158d55d60d" path="/var/lib/kubelet/pods/354cd26d-662f-445b-8bce-10158d55d60d/volumes"
Sep 16 02:43:30.544194 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:43:30.544159    2558 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-processing-5d7dbbd56b-pltkr" secret="" err="secret \"payload-processing-dockercfg-zzx6j\" not found"
Sep 16 02:44:07.836390 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:44:07.836355    2558 scope.go:117] "RemoveContainer" containerID="f94d0d0a7b37b8afba7d2431f9143811533648bc0c7d28ab63e380adcecf1f22"
Sep 16 02:44:11.017260 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:44:11.017221    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-77f43ab3-openshift-default-ccbc9c496-ngb48"]
Sep 16 02:44:11.017718 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:44:11.017506    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-77f43ab3-openshift-default-ccbc9c496-ngb48" podUID="a8114a0e-c5be-4502-9c31-4b8421406542" containerName="istio-proxy" containerID="cri-o://1c3f1f9790aaf20bde78e311dd8d962c6e90f40164e78db31320e63a3c347cc2" gracePeriod=30
Sep 16 02:44:13.117078 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:44:13.117041    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-fd614e-openshift-default-649576fb58-sg4bp"]
Sep 16 02:44:13.117645 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:44:13.117621    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="354cd26d-662f-445b-8bce-10158d55d60d" containerName="revoke-keys"
Sep 16 02:44:13.117762 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:44:13.117647    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="354cd26d-662f-445b-8bce-10158d55d60d" containerName="revoke-keys"
Sep 16 02:44:13.117835 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:44:13.117815    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="354cd26d-662f-445b-8bce-10158d55d60d" containerName="revoke-keys"
Sep 16 02:44:13.120448 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:44:13.120425    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-fd614e-openshift-default-649576fb58-sg4bp"
Sep 16 02:44:13.122854 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:44:13.122826    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-fd614e-openshift-default-dockercfg-tl96r\""
Sep 16 02:44:13.133032 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:44:13.133002    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-fd614e-openshift-default-649576fb58-sg4bp"]
Sep 16 02:44:13.316964 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:44:13.316925    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvczb\" (UniqueName: \"kubernetes.io/projected/aa9e2461-89e4-4edc-8637-8f8934e91072-kube-api-access-jvczb\") pod \"e2e-ipp-a-fd614e-openshift-default-649576fb58-sg4bp\" (UID: \"aa9e2461-89e4-4edc-8637-8f8934e91072\") " pod="openshift-ingress/e2e-ipp-a-fd614e-openshift-default-649576fb58-sg4bp"
Sep 16 02:44:13.317169 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:44:13.316982    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa9e2461-89e4-4edc-8637-8f8934e91072-istio-token\") pod \"e2e-ipp-a-fd614e-openshift-default-649576fb58-sg4bp\" (UID: \"aa9e2461-89e4-4edc-8637-8f8934e91072\") " pod="openshift-ingress/e2e-ipp-a-fd614e-openshift-default-649576fb58-sg4bp"
Sep 16 02:44:13.317169 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:44:13.317076    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa9e2461-89e4-4edc-8637-8f8934e91072-istiod-ca-cert\") pod \"e2e-ipp-a-fd614e-openshift-default-649576fb58-sg4bp\" (UID: \"aa9e2461-89e4-4edc-8637-8f8934e91072\") " pod="openshift-ingress/e2e-ipp-a-fd614e-openshift-default-649576fb58-sg4bp"
Sep 16 02:44:13.317169 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:44:13.317158    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa9e2461-89e4-4edc-8637-8f8934e91072-istio-podinfo\") pod \"e2e-ipp-a-fd614e-openshift-default-649576fb58-sg4bp\" (UID: \"aa9e2461-89e4-4edc-8637-8f8934e91072\") " pod="openshift-ingress/e2e-ipp-a-fd614e-openshift-default-649576fb58-sg4bp"
Sep 16 02:44:13.317353 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:44:13.317184    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa9e2461-89e4-4edc-8637-8f8934e91072-workload-socket\") pod \"e2e-ipp-a-fd614e-openshift-default-649576fb58-sg4bp\" (UID: \"aa9e2461-89e4-4edc-8637-8f8934e91072\") " pod="openshift-ingress/e2e-ipp-a-fd614e-openshift-default-649576fb58-sg4bp"
Sep 16 02:44:13.317353 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:44:13.317236    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa9e2461-89e4-4edc-8637-8f8934e91072-credential-socket\") pod \"e2e-ipp-a-fd614e-openshift-default-649576fb58-sg4bp\" (UID: \"aa9e2461-89e4-4edc-8637-8f8934e91072\") " pod="openshift-ingress/e2e-ipp-a-fd614e-openshift-default-649576fb58-sg4bp"
Sep 16 02:44:13.317353 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:44:13.317284    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa9e2461-89e4-4edc-8637-8f8934e91072-workload-certs\") pod \"e2e-ipp-a-fd614e-openshift-default-649576fb58-sg4bp\" (UID: \"aa9e2461-89e4-4edc-8637-8f8934e91072\") " pod="openshift-ingress/e2e-ipp-a-fd614e-openshift-default-649576fb58-sg4bp"
Sep 16 02:44:13.317353 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:44:13.317319    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa9e2461-89e4-4edc-8637-8f8934e91072-istio-envoy\") pod \"e2e-ipp-a-fd614e-openshift-default-649576fb58-sg4bp\" (UID: \"aa9e2461-89e4-4edc-8637-8f8934e91072\") " pod="openshift-ingress/e2e-ipp-a-fd614e-openshift-default-649576fb58-sg4bp"
Sep 16 02:44:13.317543 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:44:13.317365    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa9e2461-89e4-4edc-8637-8f8934e91072-istio-data\") pod \"e2e-ipp-a-fd614e-openshift-default-649576fb58-sg4bp\" (UID: \"aa9e2461-89e4-4edc-8637-8f8934e91072\") " pod="openshift-ingress/e2e-ipp-a-fd614e-openshift-default-649576fb58-sg4bp"
Sep 16 02:44:13.417900 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:44:13.417865    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa9e2461-89e4-4edc-8637-8f8934e91072-credential-socket\") pod \"e2e-ipp-a-fd614e-openshift-default-649576fb58-sg4bp\" (UID: \"aa9e2461-89e4-4edc-8637-8f8934e91072\") " pod="openshift-ingress/e2e-ipp-a-fd614e-openshift-default-649576fb58-sg4bp"
Sep 16 02:44:13.418100 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:44:13.417911    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa9e2461-89e4-4edc-8637-8f8934e91072-workload-certs\") pod \"e2e-ipp-a-fd614e-openshift-default-649576fb58-sg4bp\" (UID: \"aa9e2461-89e4-4edc-8637-8f8934e91072\") " pod="openshift-ingress/e2e-ipp-a-fd614e-openshift-default-649576fb58-sg4bp"
Sep 16 02:44:13.418100 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:44:13.417930    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa9e2461-89e4-4edc-8637-8f8934e91072-istio-envoy\") pod \"e2e-ipp-a-fd614e-openshift-default-649576fb58-sg4bp\" (UID: \"aa9e2461-89e4-4edc-8637-8f8934e91072\") " pod="openshift-ingress/e2e-ipp-a-fd614e-openshift-default-649576fb58-sg4bp"
Sep 16 02:44:13.418100 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:44:13.417967    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa9e2461-89e4-4edc-8637-8f8934e91072-istio-data\") pod \"e2e-ipp-a-fd614e-openshift-default-649576fb58-sg4bp\" (UID: \"aa9e2461-89e4-4edc-8637-8f8934e91072\") " pod="openshift-ingress/e2e-ipp-a-fd614e-openshift-default-649576fb58-sg4bp"
Sep 16 02:44:13.418100 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:44:13.418005    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvczb\" (UniqueName: \"kubernetes.io/projected/aa9e2461-89e4-4edc-8637-8f8934e91072-kube-api-access-jvczb\") pod \"e2e-ipp-a-fd614e-openshift-default-649576fb58-sg4bp\" (UID: \"aa9e2461-89e4-4edc-8637-8f8934e91072\") " pod="openshift-ingress/e2e-ipp-a-fd614e-openshift-default-649576fb58-sg4bp"
Sep 16 02:44:13.418448 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:44:13.418173    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa9e2461-89e4-4edc-8637-8f8934e91072-istio-token\") pod \"e2e-ipp-a-fd614e-openshift-default-649576fb58-sg4bp\" (UID: \"aa9e2461-89e4-4edc-8637-8f8934e91072\") " pod="openshift-ingress/e2e-ipp-a-fd614e-openshift-default-649576fb58-sg4bp"
Sep 16 02:44:13.418448 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:44:13.418233    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa9e2461-89e4-4edc-8637-8f8934e91072-istiod-ca-cert\") pod \"e2e-ipp-a-fd614e-openshift-default-649576fb58-sg4bp\" (UID: \"aa9e2461-89e4-4edc-8637-8f8934e91072\") " pod="openshift-ingress/e2e-ipp-a-fd614e-openshift-default-649576fb58-sg4bp"
Sep 16 02:44:13.418448 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:44:13.418312    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa9e2461-89e4-4edc-8637-8f8934e91072-istio-podinfo\") pod \"e2e-ipp-a-fd614e-openshift-default-649576fb58-sg4bp\" (UID: \"aa9e2461-89e4-4edc-8637-8f8934e91072\") " pod="openshift-ingress/e2e-ipp-a-fd614e-openshift-default-649576fb58-sg4bp"
Sep 16 02:44:13.418448 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:44:13.418340    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa9e2461-89e4-4edc-8637-8f8934e91072-workload-socket\") pod \"e2e-ipp-a-fd614e-openshift-default-649576fb58-sg4bp\" (UID: \"aa9e2461-89e4-4edc-8637-8f8934e91072\") " pod="openshift-ingress/e2e-ipp-a-fd614e-openshift-default-649576fb58-sg4bp"
Sep 16 02:44:13.418448 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:44:13.418358    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa9e2461-89e4-4edc-8637-8f8934e91072-workload-certs\") pod \"e2e-ipp-a-fd614e-openshift-default-649576fb58-sg4bp\" (UID: \"aa9e2461-89e4-4edc-8637-8f8934e91072\") " pod="openshift-ingress/e2e-ipp-a-fd614e-openshift-default-649576fb58-sg4bp"
Sep 16 02:44:13.418448 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:44:13.418426    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa9e2461-89e4-4edc-8637-8f8934e91072-istio-data\") pod \"e2e-ipp-a-fd614e-openshift-default-649576fb58-sg4bp\" (UID: \"aa9e2461-89e4-4edc-8637-8f8934e91072\") " pod="openshift-ingress/e2e-ipp-a-fd614e-openshift-default-649576fb58-sg4bp"
Sep 16 02:44:13.418762 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:44:13.418512    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa9e2461-89e4-4edc-8637-8f8934e91072-credential-socket\") pod \"e2e-ipp-a-fd614e-openshift-default-649576fb58-sg4bp\" (UID: \"aa9e2461-89e4-4edc-8637-8f8934e91072\") " pod="openshift-ingress/e2e-ipp-a-fd614e-openshift-default-649576fb58-sg4bp"
Sep 16 02:44:13.418762 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:44:13.418675    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa9e2461-89e4-4edc-8637-8f8934e91072-workload-socket\") pod \"e2e-ipp-a-fd614e-openshift-default-649576fb58-sg4bp\" (UID: \"aa9e2461-89e4-4edc-8637-8f8934e91072\") " pod="openshift-ingress/e2e-ipp-a-fd614e-openshift-default-649576fb58-sg4bp"
Sep 16 02:44:13.419053 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:44:13.419030    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa9e2461-89e4-4edc-8637-8f8934e91072-istiod-ca-cert\") pod \"e2e-ipp-a-fd614e-openshift-default-649576fb58-sg4bp\" (UID: \"aa9e2461-89e4-4edc-8637-8f8934e91072\") " pod="openshift-ingress/e2e-ipp-a-fd614e-openshift-default-649576fb58-sg4bp"
Sep 16 02:44:13.420557 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:44:13.420533    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa9e2461-89e4-4edc-8637-8f8934e91072-istio-envoy\") pod \"e2e-ipp-a-fd614e-openshift-default-649576fb58-sg4bp\" (UID: \"aa9e2461-89e4-4edc-8637-8f8934e91072\") " pod="openshift-ingress/e2e-ipp-a-fd614e-openshift-default-649576fb58-sg4bp"
Sep 16 02:44:13.421116 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:44:13.421096    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa9e2461-89e4-4edc-8637-8f8934e91072-istio-podinfo\") pod \"e2e-ipp-a-fd614e-openshift-default-649576fb58-sg4bp\" (UID: \"aa9e2461-89e4-4edc-8637-8f8934e91072\") " pod="openshift-ingress/e2e-ipp-a-fd614e-openshift-default-649576fb58-sg4bp"
Sep 16 02:44:13.428569 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:44:13.428531    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa9e2461-89e4-4edc-8637-8f8934e91072-istio-token\") pod \"e2e-ipp-a-fd614e-openshift-default-649576fb58-sg4bp\" (UID: \"aa9e2461-89e4-4edc-8637-8f8934e91072\") " pod="openshift-ingress/e2e-ipp-a-fd614e-openshift-default-649576fb58-sg4bp"
Sep 16 02:44:13.428711 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:44:13.428564    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvczb\" (UniqueName: \"kubernetes.io/projected/aa9e2461-89e4-4edc-8637-8f8934e91072-kube-api-access-jvczb\") pod \"e2e-ipp-a-fd614e-openshift-default-649576fb58-sg4bp\" (UID: \"aa9e2461-89e4-4edc-8637-8f8934e91072\") " pod="openshift-ingress/e2e-ipp-a-fd614e-openshift-default-649576fb58-sg4bp"
Sep 16 02:44:13.431547 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:44:13.431527    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-fd614e-openshift-default-649576fb58-sg4bp"
Sep 16 02:44:13.715969 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:44:13.711923    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-fd614e-openshift-default-649576fb58-sg4bp"]
Sep 16 02:44:13.719189 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:44:13.719153    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa9e2461_89e4_4edc_8637_8f8934e91072.slice/crio-39a1c9d7defcce857ab07a45c976db27ceeea84ee8ecf434890aaab8a42e3fc9 WatchSource:0}: Error finding container 39a1c9d7defcce857ab07a45c976db27ceeea84ee8ecf434890aaab8a42e3fc9: Status 404 returned error can't find the container with id 39a1c9d7defcce857ab07a45c976db27ceeea84ee8ecf434890aaab8a42e3fc9
Sep 16 02:44:13.721686 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:44:13.721640    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 16 02:44:13.721804 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:44:13.721770    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 16 02:44:13.721847 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:44:13.721820    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 16 02:44:14.097883 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:44:14.097797    2558 patch_prober.go:28] interesting pod/e2e-mt-77f43ab3-openshift-default-ccbc9c496-ngb48 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 16 02:44:14.097883 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:44:14.097872    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-77f43ab3-openshift-default-ccbc9c496-ngb48" podUID="a8114a0e-c5be-4502-9c31-4b8421406542" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 02:44:14.581753 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:44:14.581712    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-fd614e-openshift-default-649576fb58-sg4bp" event={"ID":"aa9e2461-89e4-4edc-8637-8f8934e91072","Type":"ContainerStarted","Data":"9b437313db209b820f8eb8427639104959de2e41a51e41dfdf1d5f7c00daeb71"}
Sep 16 02:44:14.582148 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:44:14.581770    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-fd614e-openshift-default-649576fb58-sg4bp" event={"ID":"aa9e2461-89e4-4edc-8637-8f8934e91072","Type":"ContainerStarted","Data":"39a1c9d7defcce857ab07a45c976db27ceeea84ee8ecf434890aaab8a42e3fc9"}
Sep 16 02:44:14.602598 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:44:14.602542    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-fd614e-openshift-default-649576fb58-sg4bp" podStartSLOduration=1.602525137 podStartE2EDuration="1.602525137s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 02:44:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:44:14.600787304 +0000 UTC m=+1867.519276556" watchObservedRunningTime="2026-09-16 02:44:14.602525137 +0000 UTC m=+1867.521014386"
Sep 16 02:44:15.432370 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:44:15.432328    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-fd614e-openshift-default-649576fb58-sg4bp"
Sep 16 02:44:15.437582 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:44:15.437556    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-fd614e-openshift-default-649576fb58-sg4bp"
Sep 16 02:44:15.588234 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:44:15.588205    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-fd614e-openshift-default-649576fb58-sg4bp"
Sep 16 02:44:15.589362 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:44:15.589340    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-fd614e-openshift-default-649576fb58-sg4bp"
Sep 16 02:44:16.277980 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:44:16.277942    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-77f43ab3-openshift-default-ccbc9c496-ngb48"
Sep 16 02:44:16.347440 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:44:16.347349    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8114a0e-c5be-4502-9c31-4b8421406542-istio-data\") pod \"a8114a0e-c5be-4502-9c31-4b8421406542\" (UID: \"a8114a0e-c5be-4502-9c31-4b8421406542\") "
Sep 16 02:44:16.347440 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:44:16.347405    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8114a0e-c5be-4502-9c31-4b8421406542-istio-envoy\") pod \"a8114a0e-c5be-4502-9c31-4b8421406542\" (UID: \"a8114a0e-c5be-4502-9c31-4b8421406542\") "
Sep 16 02:44:16.347440 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:44:16.347433    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8114a0e-c5be-4502-9c31-4b8421406542-workload-certs\") pod \"a8114a0e-c5be-4502-9c31-4b8421406542\" (UID: \"a8114a0e-c5be-4502-9c31-4b8421406542\") "
Sep 16 02:44:16.347681 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:44:16.347460    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8114a0e-c5be-4502-9c31-4b8421406542-credential-socket\") pod \"a8114a0e-c5be-4502-9c31-4b8421406542\" (UID: \"a8114a0e-c5be-4502-9c31-4b8421406542\") "
Sep 16 02:44:16.347681 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:44:16.347546    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gnq2c\" (UniqueName: \"kubernetes.io/projected/a8114a0e-c5be-4502-9c31-4b8421406542-kube-api-access-gnq2c\") pod \"a8114a0e-c5be-4502-9c31-4b8421406542\" (UID: \"a8114a0e-c5be-4502-9c31-4b8421406542\") "
Sep 16 02:44:16.347681 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:44:16.347607    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8114a0e-c5be-4502-9c31-4b8421406542-istio-token\") pod \"a8114a0e-c5be-4502-9c31-4b8421406542\" (UID: \"a8114a0e-c5be-4502-9c31-4b8421406542\") "
Sep 16 02:44:16.347681 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:44:16.347645    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8114a0e-c5be-4502-9c31-4b8421406542-istio-podinfo\") pod \"a8114a0e-c5be-4502-9c31-4b8421406542\" (UID: \"a8114a0e-c5be-4502-9c31-4b8421406542\") "
Sep 16 02:44:16.347681 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:44:16.347675    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8114a0e-c5be-4502-9c31-4b8421406542-istiod-ca-cert\") pod \"a8114a0e-c5be-4502-9c31-4b8421406542\" (UID: \"a8114a0e-c5be-4502-9c31-4b8421406542\") "
Sep 16 02:44:16.347987 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:44:16.347731    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8114a0e-c5be-4502-9c31-4b8421406542-istio-data" (OuterVolumeSpecName: "istio-data") pod "a8114a0e-c5be-4502-9c31-4b8421406542" (UID: "a8114a0e-c5be-4502-9c31-4b8421406542"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:44:16.347987 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:44:16.347742    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8114a0e-c5be-4502-9c31-4b8421406542-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "a8114a0e-c5be-4502-9c31-4b8421406542" (UID: "a8114a0e-c5be-4502-9c31-4b8421406542"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:44:16.347987 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:44:16.347812    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8114a0e-c5be-4502-9c31-4b8421406542-workload-socket\") pod \"a8114a0e-c5be-4502-9c31-4b8421406542\" (UID: \"a8114a0e-c5be-4502-9c31-4b8421406542\") "
Sep 16 02:44:16.347987 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:44:16.347850    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8114a0e-c5be-4502-9c31-4b8421406542-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "a8114a0e-c5be-4502-9c31-4b8421406542" (UID: "a8114a0e-c5be-4502-9c31-4b8421406542"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:44:16.348199 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:44:16.348067    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8114a0e-c5be-4502-9c31-4b8421406542-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "a8114a0e-c5be-4502-9c31-4b8421406542" (UID: "a8114a0e-c5be-4502-9c31-4b8421406542"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:44:16.348199 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:44:16.348163    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8114a0e-c5be-4502-9c31-4b8421406542-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "a8114a0e-c5be-4502-9c31-4b8421406542" (UID: "a8114a0e-c5be-4502-9c31-4b8421406542"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 02:44:16.348305 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:44:16.348227    2558 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8114a0e-c5be-4502-9c31-4b8421406542-workload-socket\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:44:16.348305 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:44:16.348246    2558 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8114a0e-c5be-4502-9c31-4b8421406542-istio-data\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:44:16.348305 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:44:16.348259    2558 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8114a0e-c5be-4502-9c31-4b8421406542-workload-certs\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:44:16.348305 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:44:16.348271    2558 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8114a0e-c5be-4502-9c31-4b8421406542-credential-socket\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:44:16.349920 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:44:16.349892    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8114a0e-c5be-4502-9c31-4b8421406542-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "a8114a0e-c5be-4502-9c31-4b8421406542" (UID: "a8114a0e-c5be-4502-9c31-4b8421406542"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:44:16.350111 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:44:16.350087    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a8114a0e-c5be-4502-9c31-4b8421406542-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "a8114a0e-c5be-4502-9c31-4b8421406542" (UID: "a8114a0e-c5be-4502-9c31-4b8421406542"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 02:44:16.350187 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:44:16.350124    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8114a0e-c5be-4502-9c31-4b8421406542-istio-token" (OuterVolumeSpecName: "istio-token") pod "a8114a0e-c5be-4502-9c31-4b8421406542" (UID: "a8114a0e-c5be-4502-9c31-4b8421406542"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 02:44:16.350229 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:44:16.350182    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8114a0e-c5be-4502-9c31-4b8421406542-kube-api-access-gnq2c" (OuterVolumeSpecName: "kube-api-access-gnq2c") pod "a8114a0e-c5be-4502-9c31-4b8421406542" (UID: "a8114a0e-c5be-4502-9c31-4b8421406542"). InnerVolumeSpecName "kube-api-access-gnq2c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 02:44:16.448929 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:44:16.448891    2558 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8114a0e-c5be-4502-9c31-4b8421406542-istio-token\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:44:16.448929 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:44:16.448925    2558 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8114a0e-c5be-4502-9c31-4b8421406542-istio-podinfo\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:44:16.449191 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:44:16.449085    2558 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8114a0e-c5be-4502-9c31-4b8421406542-istiod-ca-cert\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:44:16.449191 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:44:16.449094    2558 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8114a0e-c5be-4502-9c31-4b8421406542-istio-envoy\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:44:16.449191 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:44:16.449102    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gnq2c\" (UniqueName: \"kubernetes.io/projected/a8114a0e-c5be-4502-9c31-4b8421406542-kube-api-access-gnq2c\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:44:16.595862 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:44:16.595827    2558 generic.go:356] "Generic (PLEG): container finished" podID="a8114a0e-c5be-4502-9c31-4b8421406542" containerID="1c3f1f9790aaf20bde78e311dd8d962c6e90f40164e78db31320e63a3c347cc2" exitCode=0
Sep 16 02:44:16.596290 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:44:16.595882    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-77f43ab3-openshift-default-ccbc9c496-ngb48"
Sep 16 02:44:16.596290 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:44:16.595909    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-77f43ab3-openshift-default-ccbc9c496-ngb48" event={"ID":"a8114a0e-c5be-4502-9c31-4b8421406542","Type":"ContainerDied","Data":"1c3f1f9790aaf20bde78e311dd8d962c6e90f40164e78db31320e63a3c347cc2"}
Sep 16 02:44:16.596290 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:44:16.595968    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-77f43ab3-openshift-default-ccbc9c496-ngb48" event={"ID":"a8114a0e-c5be-4502-9c31-4b8421406542","Type":"ContainerDied","Data":"e9ab55ce371665be4c45d677ba2a23b5c964f496262b74b9ba35bc59cbfc784d"}
Sep 16 02:44:16.596290 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:44:16.595991    2558 scope.go:117] "RemoveContainer" containerID="1c3f1f9790aaf20bde78e311dd8d962c6e90f40164e78db31320e63a3c347cc2"
Sep 16 02:44:16.606731 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:44:16.606690    2558 scope.go:117] "RemoveContainer" containerID="07d34ae69ecd7473ed7c18863646ff2eca3ad9ca9d811fb8a9e30c9cf7c35711"
Sep 16 02:44:16.615098 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:44:16.615073    2558 scope.go:117] "RemoveContainer" containerID="1c3f1f9790aaf20bde78e311dd8d962c6e90f40164e78db31320e63a3c347cc2"
Sep 16 02:44:16.615403 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:44:16.615379    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c3f1f9790aaf20bde78e311dd8d962c6e90f40164e78db31320e63a3c347cc2\": container with ID starting with 1c3f1f9790aaf20bde78e311dd8d962c6e90f40164e78db31320e63a3c347cc2 not found: ID does not exist" containerID="1c3f1f9790aaf20bde78e311dd8d962c6e90f40164e78db31320e63a3c347cc2"
Sep 16 02:44:16.615466 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:44:16.615410    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c3f1f9790aaf20bde78e311dd8d962c6e90f40164e78db31320e63a3c347cc2"} err="failed to get container status \"1c3f1f9790aaf20bde78e311dd8d962c6e90f40164e78db31320e63a3c347cc2\": rpc error: code = NotFound desc = could not find container \"1c3f1f9790aaf20bde78e311dd8d962c6e90f40164e78db31320e63a3c347cc2\": container with ID starting with 1c3f1f9790aaf20bde78e311dd8d962c6e90f40164e78db31320e63a3c347cc2 not found: ID does not exist"
Sep 16 02:44:16.615466 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:44:16.615428    2558 scope.go:117] "RemoveContainer" containerID="07d34ae69ecd7473ed7c18863646ff2eca3ad9ca9d811fb8a9e30c9cf7c35711"
Sep 16 02:44:16.615669 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:44:16.615652    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07d34ae69ecd7473ed7c18863646ff2eca3ad9ca9d811fb8a9e30c9cf7c35711\": container with ID starting with 07d34ae69ecd7473ed7c18863646ff2eca3ad9ca9d811fb8a9e30c9cf7c35711 not found: ID does not exist" containerID="07d34ae69ecd7473ed7c18863646ff2eca3ad9ca9d811fb8a9e30c9cf7c35711"
Sep 16 02:44:16.615741 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:44:16.615674    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07d34ae69ecd7473ed7c18863646ff2eca3ad9ca9d811fb8a9e30c9cf7c35711"} err="failed to get container status \"07d34ae69ecd7473ed7c18863646ff2eca3ad9ca9d811fb8a9e30c9cf7c35711\": rpc error: code = NotFound desc = could not find container \"07d34ae69ecd7473ed7c18863646ff2eca3ad9ca9d811fb8a9e30c9cf7c35711\": container with ID starting with 07d34ae69ecd7473ed7c18863646ff2eca3ad9ca9d811fb8a9e30c9cf7c35711 not found: ID does not exist"
Sep 16 02:44:16.626184 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:44:16.626134    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-77f43ab3-openshift-default-ccbc9c496-ngb48"]
Sep 16 02:44:16.628142 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:44:16.628119    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-77f43ab3-openshift-default-ccbc9c496-ngb48"]
Sep 16 02:44:17.553814 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:44:17.553781    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a8114a0e-c5be-4502-9c31-4b8421406542" path="/var/lib/kubelet/pods/a8114a0e-c5be-4502-9c31-4b8421406542/volumes"
Sep 16 02:44:35.712288 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:44:35.712258    2558 generic.go:356] "Generic (PLEG): container finished" podID="aa9e2461-89e4-4edc-8637-8f8934e91072" containerID="9b437313db209b820f8eb8427639104959de2e41a51e41dfdf1d5f7c00daeb71" exitCode=0
Sep 16 02:44:35.712711 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:44:35.712328    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-fd614e-openshift-default-649576fb58-sg4bp" event={"ID":"aa9e2461-89e4-4edc-8637-8f8934e91072","Type":"ContainerDied","Data":"9b437313db209b820f8eb8427639104959de2e41a51e41dfdf1d5f7c00daeb71"}
Sep 16 02:44:35.712711 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:44:35.712681    2558 scope.go:117] "RemoveContainer" containerID="9b437313db209b820f8eb8427639104959de2e41a51e41dfdf1d5f7c00daeb71"
Sep 16 02:44:35.713767 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:44:35.713740    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 16 02:44:35.713832 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:44:35.713803    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 16 02:44:35.713869 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:44:35.713834    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 16 02:44:36.719750 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:44:36.719714    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-fd614e-openshift-default-649576fb58-sg4bp" event={"ID":"aa9e2461-89e4-4edc-8637-8f8934e91072","Type":"ContainerStarted","Data":"98890aeae1f27d25a7fc5e7fb62f762e6ec102a5df04bc7fd5df7337a0125b4c"}
Sep 16 02:44:37.431769 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:44:37.431735    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-fd614e-openshift-default-649576fb58-sg4bp"
Sep 16 02:44:37.431956 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:44:37.431931    2558 patch_prober.go:28] interesting pod/e2e-ipp-a-fd614e-openshift-default-649576fb58-sg4bp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Sep 16 02:44:37.432023 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:44:37.432000    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-fd614e-openshift-default-649576fb58-sg4bp" podUID="aa9e2461-89e4-4edc-8637-8f8934e91072" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Sep 16 02:44:38.432907 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:44:38.432865    2558 patch_prober.go:28] interesting pod/e2e-ipp-a-fd614e-openshift-default-649576fb58-sg4bp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Sep 16 02:44:38.433404 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:44:38.432939    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-fd614e-openshift-default-649576fb58-sg4bp" podUID="aa9e2461-89e4-4edc-8637-8f8934e91072" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Sep 16 02:44:39.432972 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:44:39.432930    2558 patch_prober.go:28] interesting pod/e2e-ipp-a-fd614e-openshift-default-649576fb58-sg4bp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Sep 16 02:44:39.433501 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:44:39.432993    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-fd614e-openshift-default-649576fb58-sg4bp" podUID="aa9e2461-89e4-4edc-8637-8f8934e91072" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Sep 16 02:44:40.432719 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:44:40.432663    2558 patch_prober.go:28] interesting pod/e2e-ipp-a-fd614e-openshift-default-649576fb58-sg4bp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Sep 16 02:44:40.432897 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:44:40.432755    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-fd614e-openshift-default-649576fb58-sg4bp" podUID="aa9e2461-89e4-4edc-8637-8f8934e91072" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Sep 16 02:44:41.432790 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:44:41.432757    2558 patch_prober.go:28] interesting pod/e2e-ipp-a-fd614e-openshift-default-649576fb58-sg4bp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Sep 16 02:44:41.433169 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:44:41.432823    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-fd614e-openshift-default-649576fb58-sg4bp" podUID="aa9e2461-89e4-4edc-8637-8f8934e91072" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Sep 16 02:44:42.436029 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:44:42.435998    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-fd614e-openshift-default-649576fb58-sg4bp"
Sep 16 02:44:42.436472 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:44:42.436264    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-fd614e-openshift-default-649576fb58-sg4bp"
Sep 16 02:44:42.436969 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:44:42.436949    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-fd614e-openshift-default-649576fb58-sg4bp"
Sep 16 02:44:42.470924 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:44:42.470014    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-d3d202-openshift-default-6745f5bd6-gvffh"]
Sep 16 02:44:42.471152 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:44:42.471057    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a8114a0e-c5be-4502-9c31-4b8421406542" containerName="istio-proxy"
Sep 16 02:44:42.471152 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:44:42.471081    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="a8114a0e-c5be-4502-9c31-4b8421406542" containerName="istio-proxy"
Sep 16 02:44:42.471152 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:44:42.471118    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a8114a0e-c5be-4502-9c31-4b8421406542" containerName="istio-proxy"
Sep 16 02:44:42.471152 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:44:42.471127    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="a8114a0e-c5be-4502-9c31-4b8421406542" containerName="istio-proxy"
Sep 16 02:44:42.471409 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:44:42.471338    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="a8114a0e-c5be-4502-9c31-4b8421406542" containerName="istio-proxy"
Sep 16 02:44:42.477866 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:44:42.477841    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-d3d202-openshift-default-6745f5bd6-gvffh"
Sep 16 02:44:42.480470 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:44:42.480424    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-d3d202-openshift-default-dockercfg-rf686\""
Sep 16 02:44:42.489369 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:44:42.487615    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-d3d202-openshift-default-6745f5bd6-gvffh"]
Sep 16 02:44:42.625656 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:44:42.625610    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cdd07dc7-673e-46cc-b248-4838609577c1-istio-podinfo\") pod \"e2e-ipp-b-d3d202-openshift-default-6745f5bd6-gvffh\" (UID: \"cdd07dc7-673e-46cc-b248-4838609577c1\") " pod="openshift-ingress/e2e-ipp-b-d3d202-openshift-default-6745f5bd6-gvffh"
Sep 16 02:44:42.625971 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:44:42.625948    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdd07dc7-673e-46cc-b248-4838609577c1-workload-socket\") pod \"e2e-ipp-b-d3d202-openshift-default-6745f5bd6-gvffh\" (UID: \"cdd07dc7-673e-46cc-b248-4838609577c1\") " pod="openshift-ingress/e2e-ipp-b-d3d202-openshift-default-6745f5bd6-gvffh"
Sep 16 02:44:42.626110 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:44:42.626093    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wznjd\" (UniqueName: \"kubernetes.io/projected/cdd07dc7-673e-46cc-b248-4838609577c1-kube-api-access-wznjd\") pod \"e2e-ipp-b-d3d202-openshift-default-6745f5bd6-gvffh\" (UID: \"cdd07dc7-673e-46cc-b248-4838609577c1\") " pod="openshift-ingress/e2e-ipp-b-d3d202-openshift-default-6745f5bd6-gvffh"
Sep 16 02:44:42.626323 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:44:42.626306    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cdd07dc7-673e-46cc-b248-4838609577c1-istio-token\") pod \"e2e-ipp-b-d3d202-openshift-default-6745f5bd6-gvffh\" (UID: \"cdd07dc7-673e-46cc-b248-4838609577c1\") " pod="openshift-ingress/e2e-ipp-b-d3d202-openshift-default-6745f5bd6-gvffh"
Sep 16 02:44:42.626436 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:44:42.626422    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cdd07dc7-673e-46cc-b248-4838609577c1-istio-data\") pod \"e2e-ipp-b-d3d202-openshift-default-6745f5bd6-gvffh\" (UID: \"cdd07dc7-673e-46cc-b248-4838609577c1\") " pod="openshift-ingress/e2e-ipp-b-d3d202-openshift-default-6745f5bd6-gvffh"
Sep 16 02:44:42.626556 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:44:42.626543    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cdd07dc7-673e-46cc-b248-4838609577c1-istio-envoy\") pod \"e2e-ipp-b-d3d202-openshift-default-6745f5bd6-gvffh\" (UID: \"cdd07dc7-673e-46cc-b248-4838609577c1\") " pod="openshift-ingress/e2e-ipp-b-d3d202-openshift-default-6745f5bd6-gvffh"
Sep 16 02:44:42.626678 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:44:42.626665    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdd07dc7-673e-46cc-b248-4838609577c1-credential-socket\") pod \"e2e-ipp-b-d3d202-openshift-default-6745f5bd6-gvffh\" (UID: \"cdd07dc7-673e-46cc-b248-4838609577c1\") " pod="openshift-ingress/e2e-ipp-b-d3d202-openshift-default-6745f5bd6-gvffh"
Sep 16 02:44:42.626806 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:44:42.626789    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cdd07dc7-673e-46cc-b248-4838609577c1-istiod-ca-cert\") pod \"e2e-ipp-b-d3d202-openshift-default-6745f5bd6-gvffh\" (UID: \"cdd07dc7-673e-46cc-b248-4838609577c1\") " pod="openshift-ingress/e2e-ipp-b-d3d202-openshift-default-6745f5bd6-gvffh"
Sep 16 02:44:42.626910 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:44:42.626896    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cdd07dc7-673e-46cc-b248-4838609577c1-workload-certs\") pod \"e2e-ipp-b-d3d202-openshift-default-6745f5bd6-gvffh\" (UID: \"cdd07dc7-673e-46cc-b248-4838609577c1\") " pod="openshift-ingress/e2e-ipp-b-d3d202-openshift-default-6745f5bd6-gvffh"
Sep 16 02:44:42.728360 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:44:42.728268    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cdd07dc7-673e-46cc-b248-4838609577c1-istio-token\") pod \"e2e-ipp-b-d3d202-openshift-default-6745f5bd6-gvffh\" (UID: \"cdd07dc7-673e-46cc-b248-4838609577c1\") " pod="openshift-ingress/e2e-ipp-b-d3d202-openshift-default-6745f5bd6-gvffh"
Sep 16 02:44:42.728360 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:44:42.728310    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cdd07dc7-673e-46cc-b248-4838609577c1-istio-data\") pod \"e2e-ipp-b-d3d202-openshift-default-6745f5bd6-gvffh\" (UID: \"cdd07dc7-673e-46cc-b248-4838609577c1\") " pod="openshift-ingress/e2e-ipp-b-d3d202-openshift-default-6745f5bd6-gvffh"
Sep 16 02:44:42.728360 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:44:42.728336    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cdd07dc7-673e-46cc-b248-4838609577c1-istio-envoy\") pod \"e2e-ipp-b-d3d202-openshift-default-6745f5bd6-gvffh\" (UID: \"cdd07dc7-673e-46cc-b248-4838609577c1\") " pod="openshift-ingress/e2e-ipp-b-d3d202-openshift-default-6745f5bd6-gvffh"
Sep 16 02:44:42.728612 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:44:42.728447    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdd07dc7-673e-46cc-b248-4838609577c1-credential-socket\") pod \"e2e-ipp-b-d3d202-openshift-default-6745f5bd6-gvffh\" (UID: \"cdd07dc7-673e-46cc-b248-4838609577c1\") " pod="openshift-ingress/e2e-ipp-b-d3d202-openshift-default-6745f5bd6-gvffh"
Sep 16 02:44:42.728612 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:44:42.728481    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cdd07dc7-673e-46cc-b248-4838609577c1-istiod-ca-cert\") pod \"e2e-ipp-b-d3d202-openshift-default-6745f5bd6-gvffh\" (UID: \"cdd07dc7-673e-46cc-b248-4838609577c1\") " pod="openshift-ingress/e2e-ipp-b-d3d202-openshift-default-6745f5bd6-gvffh"
Sep 16 02:44:42.728612 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:44:42.728509    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cdd07dc7-673e-46cc-b248-4838609577c1-workload-certs\") pod \"e2e-ipp-b-d3d202-openshift-default-6745f5bd6-gvffh\" (UID: \"cdd07dc7-673e-46cc-b248-4838609577c1\") " pod="openshift-ingress/e2e-ipp-b-d3d202-openshift-default-6745f5bd6-gvffh"
Sep 16 02:44:42.728612 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:44:42.728563    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cdd07dc7-673e-46cc-b248-4838609577c1-istio-podinfo\") pod \"e2e-ipp-b-d3d202-openshift-default-6745f5bd6-gvffh\" (UID: \"cdd07dc7-673e-46cc-b248-4838609577c1\") " pod="openshift-ingress/e2e-ipp-b-d3d202-openshift-default-6745f5bd6-gvffh"
Sep 16 02:44:42.728873 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:44:42.728621    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdd07dc7-673e-46cc-b248-4838609577c1-workload-socket\") pod \"e2e-ipp-b-d3d202-openshift-default-6745f5bd6-gvffh\" (UID: \"cdd07dc7-673e-46cc-b248-4838609577c1\") " pod="openshift-ingress/e2e-ipp-b-d3d202-openshift-default-6745f5bd6-gvffh"
Sep 16 02:44:42.728873 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:44:42.728649    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wznjd\" (UniqueName: \"kubernetes.io/projected/cdd07dc7-673e-46cc-b248-4838609577c1-kube-api-access-wznjd\") pod \"e2e-ipp-b-d3d202-openshift-default-6745f5bd6-gvffh\" (UID: \"cdd07dc7-673e-46cc-b248-4838609577c1\") " pod="openshift-ingress/e2e-ipp-b-d3d202-openshift-default-6745f5bd6-gvffh"
Sep 16 02:44:42.728873 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:44:42.728812    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cdd07dc7-673e-46cc-b248-4838609577c1-istio-data\") pod \"e2e-ipp-b-d3d202-openshift-default-6745f5bd6-gvffh\" (UID: \"cdd07dc7-673e-46cc-b248-4838609577c1\") " pod="openshift-ingress/e2e-ipp-b-d3d202-openshift-default-6745f5bd6-gvffh"
Sep 16 02:44:42.728873 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:44:42.728868    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdd07dc7-673e-46cc-b248-4838609577c1-credential-socket\") pod \"e2e-ipp-b-d3d202-openshift-default-6745f5bd6-gvffh\" (UID: \"cdd07dc7-673e-46cc-b248-4838609577c1\") " pod="openshift-ingress/e2e-ipp-b-d3d202-openshift-default-6745f5bd6-gvffh"
Sep 16 02:44:42.729082 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:44:42.729017    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cdd07dc7-673e-46cc-b248-4838609577c1-workload-certs\") pod \"e2e-ipp-b-d3d202-openshift-default-6745f5bd6-gvffh\" (UID: \"cdd07dc7-673e-46cc-b248-4838609577c1\") " pod="openshift-ingress/e2e-ipp-b-d3d202-openshift-default-6745f5bd6-gvffh"
Sep 16 02:44:42.729237 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:44:42.729216    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdd07dc7-673e-46cc-b248-4838609577c1-workload-socket\") pod \"e2e-ipp-b-d3d202-openshift-default-6745f5bd6-gvffh\" (UID: \"cdd07dc7-673e-46cc-b248-4838609577c1\") " pod="openshift-ingress/e2e-ipp-b-d3d202-openshift-default-6745f5bd6-gvffh"
Sep 16 02:44:42.729391 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:44:42.729375    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cdd07dc7-673e-46cc-b248-4838609577c1-istiod-ca-cert\") pod \"e2e-ipp-b-d3d202-openshift-default-6745f5bd6-gvffh\" (UID: \"cdd07dc7-673e-46cc-b248-4838609577c1\") " pod="openshift-ingress/e2e-ipp-b-d3d202-openshift-default-6745f5bd6-gvffh"
Sep 16 02:44:42.730667 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:44:42.730645    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cdd07dc7-673e-46cc-b248-4838609577c1-istio-envoy\") pod \"e2e-ipp-b-d3d202-openshift-default-6745f5bd6-gvffh\" (UID: \"cdd07dc7-673e-46cc-b248-4838609577c1\") " pod="openshift-ingress/e2e-ipp-b-d3d202-openshift-default-6745f5bd6-gvffh"
Sep 16 02:44:42.731144 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:44:42.731123    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cdd07dc7-673e-46cc-b248-4838609577c1-istio-podinfo\") pod \"e2e-ipp-b-d3d202-openshift-default-6745f5bd6-gvffh\" (UID: \"cdd07dc7-673e-46cc-b248-4838609577c1\") " pod="openshift-ingress/e2e-ipp-b-d3d202-openshift-default-6745f5bd6-gvffh"
Sep 16 02:44:42.735614 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:44:42.735594    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cdd07dc7-673e-46cc-b248-4838609577c1-istio-token\") pod \"e2e-ipp-b-d3d202-openshift-default-6745f5bd6-gvffh\" (UID: \"cdd07dc7-673e-46cc-b248-4838609577c1\") " pod="openshift-ingress/e2e-ipp-b-d3d202-openshift-default-6745f5bd6-gvffh"
Sep 16 02:44:42.735993 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:44:42.735973    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wznjd\" (UniqueName: \"kubernetes.io/projected/cdd07dc7-673e-46cc-b248-4838609577c1-kube-api-access-wznjd\") pod \"e2e-ipp-b-d3d202-openshift-default-6745f5bd6-gvffh\" (UID: \"cdd07dc7-673e-46cc-b248-4838609577c1\") " pod="openshift-ingress/e2e-ipp-b-d3d202-openshift-default-6745f5bd6-gvffh"
Sep 16 02:44:42.791933 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:44:42.791903    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-d3d202-openshift-default-6745f5bd6-gvffh"
Sep 16 02:44:43.065978 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:44:43.065953    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-d3d202-openshift-default-6745f5bd6-gvffh"]
Sep 16 02:44:43.068103 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:44:43.068076    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdd07dc7_673e_46cc_b248_4838609577c1.slice/crio-3a929b051d3759aa0dc65b16eaf94920f01581d6dedb7712b9c1dda6df3eba33 WatchSource:0}: Error finding container 3a929b051d3759aa0dc65b16eaf94920f01581d6dedb7712b9c1dda6df3eba33: Status 404 returned error can't find the container with id 3a929b051d3759aa0dc65b16eaf94920f01581d6dedb7712b9c1dda6df3eba33
Sep 16 02:44:43.070334 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:44:43.070304    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 16 02:44:43.070404 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:44:43.070364    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 16 02:44:43.070404 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:44:43.070392    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 16 02:44:43.757340 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:44:43.757300    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-d3d202-openshift-default-6745f5bd6-gvffh" event={"ID":"cdd07dc7-673e-46cc-b248-4838609577c1","Type":"ContainerStarted","Data":"f52358d4482f3fe79785a205eceb5a13bd47b795c73f8bc2e3eac96c043a0cbf"}
Sep 16 02:44:43.757848 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:44:43.757362    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-d3d202-openshift-default-6745f5bd6-gvffh" event={"ID":"cdd07dc7-673e-46cc-b248-4838609577c1","Type":"ContainerStarted","Data":"3a929b051d3759aa0dc65b16eaf94920f01581d6dedb7712b9c1dda6df3eba33"}
Sep 16 02:44:43.780255 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:44:43.780202    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-d3d202-openshift-default-6745f5bd6-gvffh" podStartSLOduration=1.7801865239999999 podStartE2EDuration="1.780186524s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 02:44:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:44:43.778014447 +0000 UTC m=+1896.696503698" watchObservedRunningTime="2026-09-16 02:44:43.780186524 +0000 UTC m=+1896.698675773"
Sep 16 02:44:43.792795 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:44:43.792766    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-d3d202-openshift-default-6745f5bd6-gvffh"
Sep 16 02:44:44.798021 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:44:44.797994    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-d3d202-openshift-default-6745f5bd6-gvffh"
Sep 16 02:44:45.767772 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:44:45.767737    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-d3d202-openshift-default-6745f5bd6-gvffh"
Sep 16 02:44:45.768731 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:44:45.768685    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-d3d202-openshift-default-6745f5bd6-gvffh"
Sep 16 02:44:51.544076 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:44:51.544033    2558 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-processing-5d7dbbd56b-pltkr" secret="" err="secret \"payload-processing-dockercfg-zzx6j\" not found"
Sep 16 02:45:00.137985 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:45:00.137949    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29825445-pzbjg"]
Sep 16 02:45:00.138710 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:45:00.138681    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="a8114a0e-c5be-4502-9c31-4b8421406542" containerName="istio-proxy"
Sep 16 02:45:00.142405 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:45:00.142379    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-a53a44-29825445-j986r"]
Sep 16 02:45:00.142557 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:45:00.142503    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29825445-pzbjg"
Sep 16 02:45:00.145284 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:45:00.145256    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jpd4f\""
Sep 16 02:45:00.148855 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:45:00.148826    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-24aabe-29825445-trdjt"]
Sep 16 02:45:00.149300 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:45:00.149280    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-a53a44-29825445-j986r"
Sep 16 02:45:00.152882 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:45:00.152766    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29825445-pzbjg"]
Sep 16 02:45:00.153065 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:45:00.152856    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-24aabe-29825445-trdjt"
Sep 16 02:45:00.153534 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:45:00.153519    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-a53a44-29825445-j986r"]
Sep 16 02:45:00.165324 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:45:00.165298    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29825445-pzbjg"
Sep 16 02:45:00.165771 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:45:00.165752    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-24aabe-29825445-trdjt"]
Sep 16 02:45:00.215293 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:45:00.214862    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-a53a44-29825445-j986r"
Sep 16 02:45:00.221492 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:45:00.221468    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-24aabe-29825445-trdjt"
Sep 16 02:45:00.484003 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:45:00.483959    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29825445-pzbjg"]
Sep 16 02:45:00.486240 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:45:00.486215    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f2fe101_6b05_4f4d_80d2_086ba82212c9.slice/crio-a5f1db22e09acc26a8c2c9ab4921119aae55b15ccbb0f0e2787fbd37eb2d5ae4 WatchSource:0}: Error finding container a5f1db22e09acc26a8c2c9ab4921119aae55b15ccbb0f0e2787fbd37eb2d5ae4: Status 404 returned error can't find the container with id a5f1db22e09acc26a8c2c9ab4921119aae55b15ccbb0f0e2787fbd37eb2d5ae4
Sep 16 02:45:00.530632 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:45:00.530602    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-a53a44-29825445-j986r"]
Sep 16 02:45:00.532339 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:45:00.532312    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34d8f74b_ffb7_40b1_9d88_cc958b9db01a.slice/crio-92b711cd5801eedd040e3ce46c22769b165c3f1ac986849de79e274da65f42cb WatchSource:0}: Error finding container 92b711cd5801eedd040e3ce46c22769b165c3f1ac986849de79e274da65f42cb: Status 404 returned error can't find the container with id 92b711cd5801eedd040e3ce46c22769b165c3f1ac986849de79e274da65f42cb
Sep 16 02:45:00.558771 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:45:00.558665    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-24aabe-29825445-trdjt"]
Sep 16 02:45:00.575173 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:45:00.575144    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod111ee69d_f9d4_421b_a6e3_ad5f828b2eaf.slice/crio-b426f285b1b8158f6a11f387f0dbf2d4a8b9d0a068e8206e1f9113f6a459c4c3 WatchSource:0}: Error finding container b426f285b1b8158f6a11f387f0dbf2d4a8b9d0a068e8206e1f9113f6a459c4c3: Status 404 returned error can't find the container with id b426f285b1b8158f6a11f387f0dbf2d4a8b9d0a068e8206e1f9113f6a459c4c3
Sep 16 02:45:00.859794 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:45:00.859756    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-a53a44-29825445-j986r" event={"ID":"34d8f74b-ffb7-40b1-9d88-cc958b9db01a","Type":"ContainerStarted","Data":"92b711cd5801eedd040e3ce46c22769b165c3f1ac986849de79e274da65f42cb"}
Sep 16 02:45:00.862046 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:45:00.862005    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29825445-pzbjg" event={"ID":"3f2fe101-6b05-4f4d-80d2-086ba82212c9","Type":"ContainerStarted","Data":"4dca00b4a9ad8d043a1a6d935fbc0b1485cc136d5534d9d39509827104163ff4"}
Sep 16 02:45:00.862204 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:45:00.862074    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29825445-pzbjg" event={"ID":"3f2fe101-6b05-4f4d-80d2-086ba82212c9","Type":"ContainerStarted","Data":"a5f1db22e09acc26a8c2c9ab4921119aae55b15ccbb0f0e2787fbd37eb2d5ae4"}
Sep 16 02:45:00.864685 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:45:00.864652    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-24aabe-29825445-trdjt" event={"ID":"111ee69d-f9d4-421b-a6e3-ad5f828b2eaf","Type":"ContainerStarted","Data":"b426f285b1b8158f6a11f387f0dbf2d4a8b9d0a068e8206e1f9113f6a459c4c3"}
Sep 16 02:45:00.877632 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:45:00.877571    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29825445-pzbjg" podStartSLOduration=0.877552267 podStartE2EDuration="877.552267ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 02:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:45:00.875379369 +0000 UTC m=+1913.793868618" watchObservedRunningTime="2026-09-16 02:45:00.877552267 +0000 UTC m=+1913.796041520"
Sep 16 02:45:01.875548 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:45:01.875502    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-a53a44-29825445-j986r" event={"ID":"34d8f74b-ffb7-40b1-9d88-cc958b9db01a","Type":"ContainerStarted","Data":"ad67f4280e9de0539fe86156c28e4c397291a773e39d05ddfe1bac5809f0698f"}
Sep 16 02:45:01.877744 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:45:01.877688    2558 generic.go:356] "Generic (PLEG): container finished" podID="3f2fe101-6b05-4f4d-80d2-086ba82212c9" containerID="4dca00b4a9ad8d043a1a6d935fbc0b1485cc136d5534d9d39509827104163ff4" exitCode=0
Sep 16 02:45:01.877891 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:45:01.877802    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29825445-pzbjg" event={"ID":"3f2fe101-6b05-4f4d-80d2-086ba82212c9","Type":"ContainerDied","Data":"4dca00b4a9ad8d043a1a6d935fbc0b1485cc136d5534d9d39509827104163ff4"}
Sep 16 02:45:01.879477 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:45:01.879452    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-24aabe-29825445-trdjt" event={"ID":"111ee69d-f9d4-421b-a6e3-ad5f828b2eaf","Type":"ContainerStarted","Data":"bfb87fe09448a28336c3a98c966f9e0336268200b3f68f0d595b5da1c9ff28f6"}
Sep 16 02:45:01.892262 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:45:01.892207    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-a53a44-29825445-j986r" podStartSLOduration=1.892192019 podStartE2EDuration="1.892192019s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 02:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:45:01.89195393 +0000 UTC m=+1914.810443179" watchObservedRunningTime="2026-09-16 02:45:01.892192019 +0000 UTC m=+1914.810681321"
Sep 16 02:45:01.922117 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:45:01.922060    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-24aabe-29825445-trdjt" podStartSLOduration=1.9220419309999999 podStartE2EDuration="1.922041931s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 02:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:45:01.92005165 +0000 UTC m=+1914.838540900" watchObservedRunningTime="2026-09-16 02:45:01.922041931 +0000 UTC m=+1914.840531214"
Sep 16 02:45:02.888474 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:45:02.888432    2558 generic.go:356] "Generic (PLEG): container finished" podID="111ee69d-f9d4-421b-a6e3-ad5f828b2eaf" containerID="bfb87fe09448a28336c3a98c966f9e0336268200b3f68f0d595b5da1c9ff28f6" exitCode=0
Sep 16 02:45:02.888991 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:45:02.888528    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-24aabe-29825445-trdjt" event={"ID":"111ee69d-f9d4-421b-a6e3-ad5f828b2eaf","Type":"ContainerDied","Data":"bfb87fe09448a28336c3a98c966f9e0336268200b3f68f0d595b5da1c9ff28f6"}
Sep 16 02:45:02.890339 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:45:02.890309    2558 generic.go:356] "Generic (PLEG): container finished" podID="34d8f74b-ffb7-40b1-9d88-cc958b9db01a" containerID="ad67f4280e9de0539fe86156c28e4c397291a773e39d05ddfe1bac5809f0698f" exitCode=0
Sep 16 02:45:02.890470 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:45:02.890362    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-a53a44-29825445-j986r" event={"ID":"34d8f74b-ffb7-40b1-9d88-cc958b9db01a","Type":"ContainerDied","Data":"ad67f4280e9de0539fe86156c28e4c397291a773e39d05ddfe1bac5809f0698f"}
Sep 16 02:45:03.037254 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:45:03.037222    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29825445-pzbjg"
Sep 16 02:45:03.899045 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:45:03.899010    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29825445-pzbjg"
Sep 16 02:45:03.899508 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:45:03.899038    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29825445-pzbjg" event={"ID":"3f2fe101-6b05-4f4d-80d2-086ba82212c9","Type":"ContainerDied","Data":"a5f1db22e09acc26a8c2c9ab4921119aae55b15ccbb0f0e2787fbd37eb2d5ae4"}
Sep 16 02:45:03.899508 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:45:03.899097    2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a5f1db22e09acc26a8c2c9ab4921119aae55b15ccbb0f0e2787fbd37eb2d5ae4"
Sep 16 02:45:04.046361 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:45:04.046328    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-24aabe-29825445-trdjt"
Sep 16 02:45:04.087195 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:45:04.087159    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-a53a44-29825445-j986r"
Sep 16 02:45:04.906830 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:45:04.906795    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-a53a44-29825445-j986r"
Sep 16 02:45:04.906830 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:45:04.906816    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-a53a44-29825445-j986r" event={"ID":"34d8f74b-ffb7-40b1-9d88-cc958b9db01a","Type":"ContainerDied","Data":"92b711cd5801eedd040e3ce46c22769b165c3f1ac986849de79e274da65f42cb"}
Sep 16 02:45:04.907313 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:45:04.906870    2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="92b711cd5801eedd040e3ce46c22769b165c3f1ac986849de79e274da65f42cb"
Sep 16 02:45:04.908334 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:45:04.908311    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-24aabe-29825445-trdjt"
Sep 16 02:45:04.908467 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:45:04.908324    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-24aabe-29825445-trdjt" event={"ID":"111ee69d-f9d4-421b-a6e3-ad5f828b2eaf","Type":"ContainerDied","Data":"b426f285b1b8158f6a11f387f0dbf2d4a8b9d0a068e8206e1f9113f6a459c4c3"}
Sep 16 02:45:04.908467 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:45:04.908372    2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b426f285b1b8158f6a11f387f0dbf2d4a8b9d0a068e8206e1f9113f6a459c4c3"
Sep 16 02:45:05.221119 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:45:05.221026    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-7448c4-openshift-default-7884d9db5b-g56df"]
Sep 16 02:45:05.221737 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:45:05.221712    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="111ee69d-f9d4-421b-a6e3-ad5f828b2eaf" containerName="cleanup"
Sep 16 02:45:05.221856 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:45:05.221739    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="111ee69d-f9d4-421b-a6e3-ad5f828b2eaf" containerName="cleanup"
Sep 16 02:45:05.221856 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:45:05.221781    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f2fe101-6b05-4f4d-80d2-086ba82212c9" containerName="cleanup"
Sep 16 02:45:05.221856 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:45:05.221790    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f2fe101-6b05-4f4d-80d2-086ba82212c9" containerName="cleanup"
Sep 16 02:45:05.221856 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:45:05.221802    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34d8f74b-ffb7-40b1-9d88-cc958b9db01a" containerName="cleanup"
Sep 16 02:45:05.221856 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:45:05.221811    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="34d8f74b-ffb7-40b1-9d88-cc958b9db01a" containerName="cleanup"
Sep 16 02:45:05.222118 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:45:05.221918    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f2fe101-6b05-4f4d-80d2-086ba82212c9" containerName="cleanup"
Sep 16 02:45:05.222118 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:45:05.221930    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="34d8f74b-ffb7-40b1-9d88-cc958b9db01a" containerName="cleanup"
Sep 16 02:45:05.222118 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:45:05.221942    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="111ee69d-f9d4-421b-a6e3-ad5f828b2eaf" containerName="cleanup"
Sep 16 02:45:05.225589 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:45:05.225557    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-7448c4-openshift-default-7884d9db5b-g56df"
Sep 16 02:45:05.228128 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:45:05.228094    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-7448c4-openshift-default-dockercfg-8v4t2\""
Sep 16 02:45:05.236512 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:45:05.236481    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-7448c4-openshift-default-7884d9db5b-g56df"]
Sep 16 02:45:05.263152 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:45:05.263103    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b7f764c-be16-4f85-996c-9b3383b90c23-istio-envoy\") pod \"e2e-api-a-7448c4-openshift-default-7884d9db5b-g56df\" (UID: \"0b7f764c-be16-4f85-996c-9b3383b90c23\") " pod="openshift-ingress/e2e-api-a-7448c4-openshift-default-7884d9db5b-g56df"
Sep 16 02:45:05.263334 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:45:05.263176    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b7f764c-be16-4f85-996c-9b3383b90c23-workload-certs\") pod \"e2e-api-a-7448c4-openshift-default-7884d9db5b-g56df\" (UID: \"0b7f764c-be16-4f85-996c-9b3383b90c23\") " pod="openshift-ingress/e2e-api-a-7448c4-openshift-default-7884d9db5b-g56df"
Sep 16 02:45:05.263334 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:45:05.263212    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b7f764c-be16-4f85-996c-9b3383b90c23-istiod-ca-cert\") pod \"e2e-api-a-7448c4-openshift-default-7884d9db5b-g56df\" (UID: \"0b7f764c-be16-4f85-996c-9b3383b90c23\") " pod="openshift-ingress/e2e-api-a-7448c4-openshift-default-7884d9db5b-g56df"
Sep 16 02:45:05.263334 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:45:05.263256    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b7f764c-be16-4f85-996c-9b3383b90c23-istio-token\") pod \"e2e-api-a-7448c4-openshift-default-7884d9db5b-g56df\" (UID: \"0b7f764c-be16-4f85-996c-9b3383b90c23\") " pod="openshift-ingress/e2e-api-a-7448c4-openshift-default-7884d9db5b-g56df"
Sep 16 02:45:05.263500 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:45:05.263349    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b7f764c-be16-4f85-996c-9b3383b90c23-workload-socket\") pod \"e2e-api-a-7448c4-openshift-default-7884d9db5b-g56df\" (UID: \"0b7f764c-be16-4f85-996c-9b3383b90c23\") " pod="openshift-ingress/e2e-api-a-7448c4-openshift-default-7884d9db5b-g56df"
Sep 16 02:45:05.263500 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:45:05.263378    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ffbc\" (UniqueName: \"kubernetes.io/projected/0b7f764c-be16-4f85-996c-9b3383b90c23-kube-api-access-9ffbc\") pod \"e2e-api-a-7448c4-openshift-default-7884d9db5b-g56df\" (UID: \"0b7f764c-be16-4f85-996c-9b3383b90c23\") " pod="openshift-ingress/e2e-api-a-7448c4-openshift-default-7884d9db5b-g56df"
Sep 16 02:45:05.263591 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:45:05.263486    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b7f764c-be16-4f85-996c-9b3383b90c23-credential-socket\") pod \"e2e-api-a-7448c4-openshift-default-7884d9db5b-g56df\" (UID: \"0b7f764c-be16-4f85-996c-9b3383b90c23\") " pod="openshift-ingress/e2e-api-a-7448c4-openshift-default-7884d9db5b-g56df"
Sep 16 02:45:05.263591 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:45:05.263531    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b7f764c-be16-4f85-996c-9b3383b90c23-istio-data\") pod \"e2e-api-a-7448c4-openshift-default-7884d9db5b-g56df\" (UID: \"0b7f764c-be16-4f85-996c-9b3383b90c23\") " pod="openshift-ingress/e2e-api-a-7448c4-openshift-default-7884d9db5b-g56df"
Sep 16 02:45:05.263659 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:45:05.263583    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b7f764c-be16-4f85-996c-9b3383b90c23-istio-podinfo\") pod \"e2e-api-a-7448c4-openshift-default-7884d9db5b-g56df\" (UID: \"0b7f764c-be16-4f85-996c-9b3383b90c23\") " pod="openshift-ingress/e2e-api-a-7448c4-openshift-default-7884d9db5b-g56df"
Sep 16 02:45:05.364868 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:45:05.364816    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b7f764c-be16-4f85-996c-9b3383b90c23-workload-certs\") pod \"e2e-api-a-7448c4-openshift-default-7884d9db5b-g56df\" (UID: \"0b7f764c-be16-4f85-996c-9b3383b90c23\") " pod="openshift-ingress/e2e-api-a-7448c4-openshift-default-7884d9db5b-g56df"
Sep 16 02:45:05.364868 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:45:05.364870    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b7f764c-be16-4f85-996c-9b3383b90c23-istiod-ca-cert\") pod \"e2e-api-a-7448c4-openshift-default-7884d9db5b-g56df\" (UID: \"0b7f764c-be16-4f85-996c-9b3383b90c23\") " pod="openshift-ingress/e2e-api-a-7448c4-openshift-default-7884d9db5b-g56df"
Sep 16 02:45:05.365131 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:45:05.364915    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b7f764c-be16-4f85-996c-9b3383b90c23-istio-token\") pod \"e2e-api-a-7448c4-openshift-default-7884d9db5b-g56df\" (UID: \"0b7f764c-be16-4f85-996c-9b3383b90c23\") " pod="openshift-ingress/e2e-api-a-7448c4-openshift-default-7884d9db5b-g56df"
Sep 16 02:45:05.365131 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:45:05.364991    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b7f764c-be16-4f85-996c-9b3383b90c23-workload-socket\") pod \"e2e-api-a-7448c4-openshift-default-7884d9db5b-g56df\" (UID: \"0b7f764c-be16-4f85-996c-9b3383b90c23\") " pod="openshift-ingress/e2e-api-a-7448c4-openshift-default-7884d9db5b-g56df"
Sep 16 02:45:05.365131 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:45:05.365017    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ffbc\" (UniqueName: \"kubernetes.io/projected/0b7f764c-be16-4f85-996c-9b3383b90c23-kube-api-access-9ffbc\") pod \"e2e-api-a-7448c4-openshift-default-7884d9db5b-g56df\" (UID: \"0b7f764c-be16-4f85-996c-9b3383b90c23\") " pod="openshift-ingress/e2e-api-a-7448c4-openshift-default-7884d9db5b-g56df"
Sep 16 02:45:05.365131 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:45:05.365076    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b7f764c-be16-4f85-996c-9b3383b90c23-credential-socket\") pod \"e2e-api-a-7448c4-openshift-default-7884d9db5b-g56df\" (UID: \"0b7f764c-be16-4f85-996c-9b3383b90c23\") " pod="openshift-ingress/e2e-api-a-7448c4-openshift-default-7884d9db5b-g56df"
Sep 16 02:45:05.365131 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:45:05.365104    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b7f764c-be16-4f85-996c-9b3383b90c23-istio-data\") pod \"e2e-api-a-7448c4-openshift-default-7884d9db5b-g56df\" (UID: \"0b7f764c-be16-4f85-996c-9b3383b90c23\") " pod="openshift-ingress/e2e-api-a-7448c4-openshift-default-7884d9db5b-g56df"
Sep 16 02:45:05.365131 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:45:05.365131    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b7f764c-be16-4f85-996c-9b3383b90c23-istio-podinfo\") pod \"e2e-api-a-7448c4-openshift-default-7884d9db5b-g56df\" (UID: \"0b7f764c-be16-4f85-996c-9b3383b90c23\") " pod="openshift-ingress/e2e-api-a-7448c4-openshift-default-7884d9db5b-g56df"
Sep 16 02:45:05.365443 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:45:05.365206    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b7f764c-be16-4f85-996c-9b3383b90c23-istio-envoy\") pod \"e2e-api-a-7448c4-openshift-default-7884d9db5b-g56df\" (UID: \"0b7f764c-be16-4f85-996c-9b3383b90c23\") " pod="openshift-ingress/e2e-api-a-7448c4-openshift-default-7884d9db5b-g56df"
Sep 16 02:45:05.365443 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:45:05.365281    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b7f764c-be16-4f85-996c-9b3383b90c23-workload-certs\") pod \"e2e-api-a-7448c4-openshift-default-7884d9db5b-g56df\" (UID: \"0b7f764c-be16-4f85-996c-9b3383b90c23\") " pod="openshift-ingress/e2e-api-a-7448c4-openshift-default-7884d9db5b-g56df"
Sep 16 02:45:05.365623 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:45:05.365598    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b7f764c-be16-4f85-996c-9b3383b90c23-credential-socket\") pod \"e2e-api-a-7448c4-openshift-default-7884d9db5b-g56df\" (UID: \"0b7f764c-be16-4f85-996c-9b3383b90c23\") " pod="openshift-ingress/e2e-api-a-7448c4-openshift-default-7884d9db5b-g56df"
Sep 16 02:45:05.365725 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:45:05.365665    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b7f764c-be16-4f85-996c-9b3383b90c23-workload-socket\") pod \"e2e-api-a-7448c4-openshift-default-7884d9db5b-g56df\" (UID: \"0b7f764c-be16-4f85-996c-9b3383b90c23\") " pod="openshift-ingress/e2e-api-a-7448c4-openshift-default-7884d9db5b-g56df"
Sep 16 02:45:05.365725 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:45:05.365685    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b7f764c-be16-4f85-996c-9b3383b90c23-istio-data\") pod \"e2e-api-a-7448c4-openshift-default-7884d9db5b-g56df\" (UID: \"0b7f764c-be16-4f85-996c-9b3383b90c23\") " pod="openshift-ingress/e2e-api-a-7448c4-openshift-default-7884d9db5b-g56df"
Sep 16 02:45:05.366553 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:45:05.366337    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b7f764c-be16-4f85-996c-9b3383b90c23-istiod-ca-cert\") pod \"e2e-api-a-7448c4-openshift-default-7884d9db5b-g56df\" (UID: \"0b7f764c-be16-4f85-996c-9b3383b90c23\") " pod="openshift-ingress/e2e-api-a-7448c4-openshift-default-7884d9db5b-g56df"
Sep 16 02:45:05.370089 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:45:05.370055    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b7f764c-be16-4f85-996c-9b3383b90c23-istio-podinfo\") pod \"e2e-api-a-7448c4-openshift-default-7884d9db5b-g56df\" (UID: \"0b7f764c-be16-4f85-996c-9b3383b90c23\") " pod="openshift-ingress/e2e-api-a-7448c4-openshift-default-7884d9db5b-g56df"
Sep 16 02:45:05.370369 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:45:05.370348    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b7f764c-be16-4f85-996c-9b3383b90c23-istio-envoy\") pod \"e2e-api-a-7448c4-openshift-default-7884d9db5b-g56df\" (UID: \"0b7f764c-be16-4f85-996c-9b3383b90c23\") " pod="openshift-ingress/e2e-api-a-7448c4-openshift-default-7884d9db5b-g56df"
Sep 16 02:45:05.373418 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:45:05.373397    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b7f764c-be16-4f85-996c-9b3383b90c23-istio-token\") pod \"e2e-api-a-7448c4-openshift-default-7884d9db5b-g56df\" (UID: \"0b7f764c-be16-4f85-996c-9b3383b90c23\") " pod="openshift-ingress/e2e-api-a-7448c4-openshift-default-7884d9db5b-g56df"
Sep 16 02:45:05.373516 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:45:05.373399    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ffbc\" (UniqueName: \"kubernetes.io/projected/0b7f764c-be16-4f85-996c-9b3383b90c23-kube-api-access-9ffbc\") pod \"e2e-api-a-7448c4-openshift-default-7884d9db5b-g56df\" (UID: \"0b7f764c-be16-4f85-996c-9b3383b90c23\") " pod="openshift-ingress/e2e-api-a-7448c4-openshift-default-7884d9db5b-g56df"
Sep 16 02:45:05.540405 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:45:05.540310    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-7448c4-openshift-default-7884d9db5b-g56df"
Sep 16 02:45:05.825564 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:45:05.825535    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-7448c4-openshift-default-7884d9db5b-g56df"]
Sep 16 02:45:05.854759 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:45:05.854731    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b7f764c_be16_4f85_996c_9b3383b90c23.slice/crio-0ab3e477b8902481c8d5dfdae6450311afb7c1a2a34861bdd1cb4acbde482eff WatchSource:0}: Error finding container 0ab3e477b8902481c8d5dfdae6450311afb7c1a2a34861bdd1cb4acbde482eff: Status 404 returned error can't find the container with id 0ab3e477b8902481c8d5dfdae6450311afb7c1a2a34861bdd1cb4acbde482eff
Sep 16 02:45:05.856988 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:45:05.856956    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 16 02:45:05.857054 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:45:05.857024    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 16 02:45:05.857091 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:45:05.857053    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 16 02:45:05.915147 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:45:05.915075    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-7448c4-openshift-default-7884d9db5b-g56df" event={"ID":"0b7f764c-be16-4f85-996c-9b3383b90c23","Type":"ContainerStarted","Data":"0ab3e477b8902481c8d5dfdae6450311afb7c1a2a34861bdd1cb4acbde482eff"}
Sep 16 02:45:05.916949 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:45:05.916921    2558 generic.go:356] "Generic (PLEG): container finished" podID="cdd07dc7-673e-46cc-b248-4838609577c1" containerID="f52358d4482f3fe79785a205eceb5a13bd47b795c73f8bc2e3eac96c043a0cbf" exitCode=0
Sep 16 02:45:05.917045 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:45:05.916988    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-d3d202-openshift-default-6745f5bd6-gvffh" event={"ID":"cdd07dc7-673e-46cc-b248-4838609577c1","Type":"ContainerDied","Data":"f52358d4482f3fe79785a205eceb5a13bd47b795c73f8bc2e3eac96c043a0cbf"}
Sep 16 02:45:05.917392 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:45:05.917377    2558 scope.go:117] "RemoveContainer" containerID="f52358d4482f3fe79785a205eceb5a13bd47b795c73f8bc2e3eac96c043a0cbf"
Sep 16 02:45:05.918722 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:45:05.918674    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 16 02:45:05.918810 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:45:05.918797    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 16 02:45:05.918908 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:45:05.918842    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 16 02:45:06.923792 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:45:06.923752    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-7448c4-openshift-default-7884d9db5b-g56df" event={"ID":"0b7f764c-be16-4f85-996c-9b3383b90c23","Type":"ContainerStarted","Data":"f2981c57f7b9b44fbae9667c29b5ed0644608227accf505649d35d500b459bae"}
Sep 16 02:45:06.925620 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:45:06.925597    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-d3d202-openshift-default-6745f5bd6-gvffh" event={"ID":"cdd07dc7-673e-46cc-b248-4838609577c1","Type":"ContainerStarted","Data":"dcbb49801615e5ebc2a3831e48668a17d3acb46f51081fa2dc3b6b4c3da5f4f2"}
Sep 16 02:45:06.946472 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:45:06.946411    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-7448c4-openshift-default-7884d9db5b-g56df" podStartSLOduration=1.946389291 podStartE2EDuration="1.946389291s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 02:45:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:45:06.942494015 +0000 UTC m=+1919.860983264" watchObservedRunningTime="2026-09-16 02:45:06.946389291 +0000 UTC m=+1919.864878542"
Sep 16 02:45:07.540898 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:45:07.540870    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-7448c4-openshift-default-7884d9db5b-g56df"
Sep 16 02:45:07.573615 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:45:07.573387    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-7448c4-openshift-default-7884d9db5b-g56df"
Sep 16 02:45:07.792311 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:45:07.792204    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-d3d202-openshift-default-6745f5bd6-gvffh"
Sep 16 02:45:07.792495 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:45:07.792448    2558 patch_prober.go:28] interesting pod/e2e-ipp-b-d3d202-openshift-default-6745f5bd6-gvffh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Sep 16 02:45:07.792578 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:45:07.792493    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-d3d202-openshift-default-6745f5bd6-gvffh" podUID="cdd07dc7-673e-46cc-b248-4838609577c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Sep 16 02:45:07.931318 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:45:07.931288    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-7448c4-openshift-default-7884d9db5b-g56df"
Sep 16 02:45:07.932165 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:45:07.932145    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-7448c4-openshift-default-7884d9db5b-g56df"
Sep 16 02:45:08.792449 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:45:08.792416    2558 patch_prober.go:28] interesting pod/e2e-ipp-b-d3d202-openshift-default-6745f5bd6-gvffh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Sep 16 02:45:08.792678 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:45:08.792470    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-d3d202-openshift-default-6745f5bd6-gvffh" podUID="cdd07dc7-673e-46cc-b248-4838609577c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Sep 16 02:45:09.793060 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:45:09.793020    2558 patch_prober.go:28] interesting pod/e2e-ipp-b-d3d202-openshift-default-6745f5bd6-gvffh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Sep 16 02:45:09.793517 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:45:09.793079    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-d3d202-openshift-default-6745f5bd6-gvffh" podUID="cdd07dc7-673e-46cc-b248-4838609577c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Sep 16 02:45:10.792280 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:45:10.792240    2558 patch_prober.go:28] interesting pod/e2e-ipp-b-d3d202-openshift-default-6745f5bd6-gvffh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Sep 16 02:45:10.792501 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:45:10.792304    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-d3d202-openshift-default-6745f5bd6-gvffh" podUID="cdd07dc7-673e-46cc-b248-4838609577c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Sep 16 02:45:11.793094 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:45:11.793054    2558 patch_prober.go:28] interesting pod/e2e-ipp-b-d3d202-openshift-default-6745f5bd6-gvffh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Sep 16 02:45:11.793481 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:45:11.793113    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-d3d202-openshift-default-6745f5bd6-gvffh" podUID="cdd07dc7-673e-46cc-b248-4838609577c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Sep 16 02:45:12.802065 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:45:12.802021    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-d3d202-openshift-default-6745f5bd6-gvffh"
Sep 16 02:45:12.802611 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:45:12.802573    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-d3d202-openshift-default-6745f5bd6-gvffh"
Sep 16 02:45:12.804354 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:45:12.804332    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-d3d202-openshift-default-6745f5bd6-gvffh"
Sep 16 02:45:20.997058 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:45:20.997021    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5d7dbbd56b-pltkr_4189cfcd-c3ba-4bad-813f-5a6851cb5317/payload-processing/0.log"
Sep 16 02:45:21.124965 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:45:21.124939    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5d7dbbd56b-pltkr_4189cfcd-c3ba-4bad-813f-5a6851cb5317/payload-processing/0.log"
Sep 16 02:45:30.067915 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:45:30.067827    2558 generic.go:356] "Generic (PLEG): container finished" podID="0b7f764c-be16-4f85-996c-9b3383b90c23" containerID="f2981c57f7b9b44fbae9667c29b5ed0644608227accf505649d35d500b459bae" exitCode=0
Sep 16 02:45:30.067915 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:45:30.067899    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-7448c4-openshift-default-7884d9db5b-g56df" event={"ID":"0b7f764c-be16-4f85-996c-9b3383b90c23","Type":"ContainerDied","Data":"f2981c57f7b9b44fbae9667c29b5ed0644608227accf505649d35d500b459bae"}
Sep 16 02:45:30.068341 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:45:30.068295    2558 scope.go:117] "RemoveContainer" containerID="f2981c57f7b9b44fbae9667c29b5ed0644608227accf505649d35d500b459bae"
Sep 16 02:45:30.069447 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:45:30.069420    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 16 02:45:30.069510 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:45:30.069483    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 16 02:45:30.069547 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:45:30.069512    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 16 02:45:31.075895 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:45:31.075856    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-7448c4-openshift-default-7884d9db5b-g56df" event={"ID":"0b7f764c-be16-4f85-996c-9b3383b90c23","Type":"ContainerStarted","Data":"bd6318476479f00951f9bf4c06664b0ecf05535608ac2a2a954e319aea5ff83f"}
Sep 16 02:45:31.541225 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:45:31.541185    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-7448c4-openshift-default-7884d9db5b-g56df"
Sep 16 02:45:31.541428 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:45:31.541397    2558 patch_prober.go:28] interesting pod/e2e-api-a-7448c4-openshift-default-7884d9db5b-g56df container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Sep 16 02:45:31.541503 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:45:31.541449    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-7448c4-openshift-default-7884d9db5b-g56df" podUID="0b7f764c-be16-4f85-996c-9b3383b90c23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Sep 16 02:45:32.540820 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:45:32.540783    2558 patch_prober.go:28] interesting pod/e2e-api-a-7448c4-openshift-default-7884d9db5b-g56df container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Sep 16 02:45:32.541201 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:45:32.540842    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-7448c4-openshift-default-7884d9db5b-g56df" podUID="0b7f764c-be16-4f85-996c-9b3383b90c23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Sep 16 02:45:33.541747 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:45:33.541674    2558 patch_prober.go:28] interesting pod/e2e-api-a-7448c4-openshift-default-7884d9db5b-g56df container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Sep 16 02:45:33.542141 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:45:33.541772    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-7448c4-openshift-default-7884d9db5b-g56df" podUID="0b7f764c-be16-4f85-996c-9b3383b90c23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Sep 16 02:45:34.541577 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:45:34.541535    2558 patch_prober.go:28] interesting pod/e2e-api-a-7448c4-openshift-default-7884d9db5b-g56df container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Sep 16 02:45:34.541781 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:45:34.541600    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-7448c4-openshift-default-7884d9db5b-g56df" podUID="0b7f764c-be16-4f85-996c-9b3383b90c23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Sep 16 02:45:35.541355 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:45:35.541322    2558 patch_prober.go:28] interesting pod/e2e-api-a-7448c4-openshift-default-7884d9db5b-g56df container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Sep 16 02:45:35.541538 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:45:35.541378    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-7448c4-openshift-default-7884d9db5b-g56df" podUID="0b7f764c-be16-4f85-996c-9b3383b90c23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Sep 16 02:45:36.549615 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:45:36.549580    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-7448c4-openshift-default-7884d9db5b-g56df"
Sep 16 02:45:36.550091 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:45:36.549942    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-7448c4-openshift-default-7884d9db5b-g56df"
Sep 16 02:45:36.550957 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:45:36.550934    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-7448c4-openshift-default-7884d9db5b-g56df"
Sep 16 02:45:39.620724 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:45:39.620674    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-e21dae-openshift-default-78c8949998-cn6ks"]
Sep 16 02:45:39.624688 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:45:39.624659    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-e21dae-openshift-default-78c8949998-cn6ks"
Sep 16 02:45:39.626865 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:45:39.626842    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-e21dae-openshift-default-dockercfg-qvxjm\""
Sep 16 02:45:39.635052 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:45:39.635025    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-e21dae-openshift-default-78c8949998-cn6ks"]
Sep 16 02:45:39.743947 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:45:39.743914    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb94df1b-32d7-4c01-b581-8e09b4766544-credential-socket\") pod \"e2e-api-b-e21dae-openshift-default-78c8949998-cn6ks\" (UID: \"eb94df1b-32d7-4c01-b581-8e09b4766544\") " pod="openshift-ingress/e2e-api-b-e21dae-openshift-default-78c8949998-cn6ks"
Sep 16 02:45:39.744081 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:45:39.743960    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb94df1b-32d7-4c01-b581-8e09b4766544-istio-data\") pod \"e2e-api-b-e21dae-openshift-default-78c8949998-cn6ks\" (UID: \"eb94df1b-32d7-4c01-b581-8e09b4766544\") " pod="openshift-ingress/e2e-api-b-e21dae-openshift-default-78c8949998-cn6ks"
Sep 16 02:45:39.744081 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:45:39.744061    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb94df1b-32d7-4c01-b581-8e09b4766544-istio-token\") pod \"e2e-api-b-e21dae-openshift-default-78c8949998-cn6ks\" (UID: \"eb94df1b-32d7-4c01-b581-8e09b4766544\") " pod="openshift-ingress/e2e-api-b-e21dae-openshift-default-78c8949998-cn6ks"
Sep 16 02:45:39.744170 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:45:39.744103    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb94df1b-32d7-4c01-b581-8e09b4766544-istiod-ca-cert\") pod \"e2e-api-b-e21dae-openshift-default-78c8949998-cn6ks\" (UID: \"eb94df1b-32d7-4c01-b581-8e09b4766544\") " pod="openshift-ingress/e2e-api-b-e21dae-openshift-default-78c8949998-cn6ks"
Sep 16 02:45:39.744170 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:45:39.744128    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xc2xg\" (UniqueName: \"kubernetes.io/projected/eb94df1b-32d7-4c01-b581-8e09b4766544-kube-api-access-xc2xg\") pod \"e2e-api-b-e21dae-openshift-default-78c8949998-cn6ks\" (UID: \"eb94df1b-32d7-4c01-b581-8e09b4766544\") " pod="openshift-ingress/e2e-api-b-e21dae-openshift-default-78c8949998-cn6ks"
Sep 16 02:45:39.744170 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:45:39.744155    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb94df1b-32d7-4c01-b581-8e09b4766544-istio-podinfo\") pod \"e2e-api-b-e21dae-openshift-default-78c8949998-cn6ks\" (UID: \"eb94df1b-32d7-4c01-b581-8e09b4766544\") " pod="openshift-ingress/e2e-api-b-e21dae-openshift-default-78c8949998-cn6ks"
Sep 16 02:45:39.744270 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:45:39.744187    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb94df1b-32d7-4c01-b581-8e09b4766544-workload-socket\") pod \"e2e-api-b-e21dae-openshift-default-78c8949998-cn6ks\" (UID: \"eb94df1b-32d7-4c01-b581-8e09b4766544\") " pod="openshift-ingress/e2e-api-b-e21dae-openshift-default-78c8949998-cn6ks"
Sep 16 02:45:39.744270 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:45:39.744221    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb94df1b-32d7-4c01-b581-8e09b4766544-istio-envoy\") pod \"e2e-api-b-e21dae-openshift-default-78c8949998-cn6ks\" (UID: \"eb94df1b-32d7-4c01-b581-8e09b4766544\") " pod="openshift-ingress/e2e-api-b-e21dae-openshift-default-78c8949998-cn6ks"
Sep 16 02:45:39.744337 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:45:39.744267    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb94df1b-32d7-4c01-b581-8e09b4766544-workload-certs\") pod \"e2e-api-b-e21dae-openshift-default-78c8949998-cn6ks\" (UID: \"eb94df1b-32d7-4c01-b581-8e09b4766544\") " pod="openshift-ingress/e2e-api-b-e21dae-openshift-default-78c8949998-cn6ks"
Sep 16 02:45:39.846062 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:45:39.846015    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb94df1b-32d7-4c01-b581-8e09b4766544-workload-socket\") pod \"e2e-api-b-e21dae-openshift-default-78c8949998-cn6ks\" (UID: \"eb94df1b-32d7-4c01-b581-8e09b4766544\") " pod="openshift-ingress/e2e-api-b-e21dae-openshift-default-78c8949998-cn6ks"
Sep 16 02:45:39.846242 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:45:39.846086    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb94df1b-32d7-4c01-b581-8e09b4766544-istio-envoy\") pod \"e2e-api-b-e21dae-openshift-default-78c8949998-cn6ks\" (UID: \"eb94df1b-32d7-4c01-b581-8e09b4766544\") " pod="openshift-ingress/e2e-api-b-e21dae-openshift-default-78c8949998-cn6ks"
Sep 16 02:45:39.846242 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:45:39.846146    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb94df1b-32d7-4c01-b581-8e09b4766544-workload-certs\") pod \"e2e-api-b-e21dae-openshift-default-78c8949998-cn6ks\" (UID: \"eb94df1b-32d7-4c01-b581-8e09b4766544\") " pod="openshift-ingress/e2e-api-b-e21dae-openshift-default-78c8949998-cn6ks"
Sep 16 02:45:39.846242 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:45:39.846230    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb94df1b-32d7-4c01-b581-8e09b4766544-credential-socket\") pod \"e2e-api-b-e21dae-openshift-default-78c8949998-cn6ks\" (UID: \"eb94df1b-32d7-4c01-b581-8e09b4766544\") " pod="openshift-ingress/e2e-api-b-e21dae-openshift-default-78c8949998-cn6ks"
Sep 16 02:45:39.846431 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:45:39.846263    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb94df1b-32d7-4c01-b581-8e09b4766544-istio-data\") pod \"e2e-api-b-e21dae-openshift-default-78c8949998-cn6ks\" (UID: \"eb94df1b-32d7-4c01-b581-8e09b4766544\") " pod="openshift-ingress/e2e-api-b-e21dae-openshift-default-78c8949998-cn6ks"
Sep 16 02:45:39.846431 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:45:39.846320    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb94df1b-32d7-4c01-b581-8e09b4766544-istio-token\") pod \"e2e-api-b-e21dae-openshift-default-78c8949998-cn6ks\" (UID: \"eb94df1b-32d7-4c01-b581-8e09b4766544\") " pod="openshift-ingress/e2e-api-b-e21dae-openshift-default-78c8949998-cn6ks"
Sep 16 02:45:39.846431 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:45:39.846357    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb94df1b-32d7-4c01-b581-8e09b4766544-istiod-ca-cert\") pod \"e2e-api-b-e21dae-openshift-default-78c8949998-cn6ks\" (UID: \"eb94df1b-32d7-4c01-b581-8e09b4766544\") " pod="openshift-ingress/e2e-api-b-e21dae-openshift-default-78c8949998-cn6ks"
Sep 16 02:45:39.846431 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:45:39.846392    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xc2xg\" (UniqueName: \"kubernetes.io/projected/eb94df1b-32d7-4c01-b581-8e09b4766544-kube-api-access-xc2xg\") pod \"e2e-api-b-e21dae-openshift-default-78c8949998-cn6ks\" (UID: \"eb94df1b-32d7-4c01-b581-8e09b4766544\") " pod="openshift-ingress/e2e-api-b-e21dae-openshift-default-78c8949998-cn6ks"
Sep 16 02:45:39.846636 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:45:39.846432    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb94df1b-32d7-4c01-b581-8e09b4766544-istio-podinfo\") pod \"e2e-api-b-e21dae-openshift-default-78c8949998-cn6ks\" (UID: \"eb94df1b-32d7-4c01-b581-8e09b4766544\") " pod="openshift-ingress/e2e-api-b-e21dae-openshift-default-78c8949998-cn6ks"
Sep 16 02:45:39.847249 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:45:39.847218    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb94df1b-32d7-4c01-b581-8e09b4766544-credential-socket\") pod \"e2e-api-b-e21dae-openshift-default-78c8949998-cn6ks\" (UID: \"eb94df1b-32d7-4c01-b581-8e09b4766544\") " pod="openshift-ingress/e2e-api-b-e21dae-openshift-default-78c8949998-cn6ks"
Sep 16 02:45:39.847598 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:45:39.847569    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb94df1b-32d7-4c01-b581-8e09b4766544-workload-socket\") pod \"e2e-api-b-e21dae-openshift-default-78c8949998-cn6ks\" (UID: \"eb94df1b-32d7-4c01-b581-8e09b4766544\") " pod="openshift-ingress/e2e-api-b-e21dae-openshift-default-78c8949998-cn6ks"
Sep 16 02:45:39.848631 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:45:39.848566    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb94df1b-32d7-4c01-b581-8e09b4766544-istio-data\") pod \"e2e-api-b-e21dae-openshift-default-78c8949998-cn6ks\" (UID: \"eb94df1b-32d7-4c01-b581-8e09b4766544\") " pod="openshift-ingress/e2e-api-b-e21dae-openshift-default-78c8949998-cn6ks"
Sep 16 02:45:39.849190 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:45:39.849162    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb94df1b-32d7-4c01-b581-8e09b4766544-istiod-ca-cert\") pod \"e2e-api-b-e21dae-openshift-default-78c8949998-cn6ks\" (UID: \"eb94df1b-32d7-4c01-b581-8e09b4766544\") " pod="openshift-ingress/e2e-api-b-e21dae-openshift-default-78c8949998-cn6ks"
Sep 16 02:45:39.849547 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:45:39.849521    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb94df1b-32d7-4c01-b581-8e09b4766544-workload-certs\") pod \"e2e-api-b-e21dae-openshift-default-78c8949998-cn6ks\" (UID: \"eb94df1b-32d7-4c01-b581-8e09b4766544\") " pod="openshift-ingress/e2e-api-b-e21dae-openshift-default-78c8949998-cn6ks"
Sep 16 02:45:39.853910 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:45:39.853848    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb94df1b-32d7-4c01-b581-8e09b4766544-istio-envoy\") pod \"e2e-api-b-e21dae-openshift-default-78c8949998-cn6ks\" (UID: \"eb94df1b-32d7-4c01-b581-8e09b4766544\") " pod="openshift-ingress/e2e-api-b-e21dae-openshift-default-78c8949998-cn6ks"
Sep 16 02:45:39.856880 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:45:39.856855    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb94df1b-32d7-4c01-b581-8e09b4766544-istio-podinfo\") pod \"e2e-api-b-e21dae-openshift-default-78c8949998-cn6ks\" (UID: \"eb94df1b-32d7-4c01-b581-8e09b4766544\") " pod="openshift-ingress/e2e-api-b-e21dae-openshift-default-78c8949998-cn6ks"
Sep 16 02:45:39.857883 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:45:39.857830    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xc2xg\" (UniqueName: \"kubernetes.io/projected/eb94df1b-32d7-4c01-b581-8e09b4766544-kube-api-access-xc2xg\") pod \"e2e-api-b-e21dae-openshift-default-78c8949998-cn6ks\" (UID: \"eb94df1b-32d7-4c01-b581-8e09b4766544\") " pod="openshift-ingress/e2e-api-b-e21dae-openshift-default-78c8949998-cn6ks"
Sep 16 02:45:39.858143 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:45:39.858114    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb94df1b-32d7-4c01-b581-8e09b4766544-istio-token\") pod \"e2e-api-b-e21dae-openshift-default-78c8949998-cn6ks\" (UID: \"eb94df1b-32d7-4c01-b581-8e09b4766544\") " pod="openshift-ingress/e2e-api-b-e21dae-openshift-default-78c8949998-cn6ks"
Sep 16 02:45:39.937088 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:45:39.937054    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-e21dae-openshift-default-78c8949998-cn6ks"
Sep 16 02:45:40.212295 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:45:40.212271    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-e21dae-openshift-default-78c8949998-cn6ks"]
Sep 16 02:45:40.214000 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:45:40.213969    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb94df1b_32d7_4c01_b581_8e09b4766544.slice/crio-f0e8e774e756e46bca4642cad2ae385393b078719d4818a26d91e2d2750f62bc WatchSource:0}: Error finding container f0e8e774e756e46bca4642cad2ae385393b078719d4818a26d91e2d2750f62bc: Status 404 returned error can't find the container with id f0e8e774e756e46bca4642cad2ae385393b078719d4818a26d91e2d2750f62bc
Sep 16 02:45:40.216411 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:45:40.216371    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 16 02:45:40.216478 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:45:40.216437    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 16 02:45:40.216478 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:45:40.216466    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 16 02:45:41.135173 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:45:41.135136    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e21dae-openshift-default-78c8949998-cn6ks" event={"ID":"eb94df1b-32d7-4c01-b581-8e09b4766544","Type":"ContainerStarted","Data":"129862a294c189e5bae9216779b427b84a51c3c767493c7be7a1df0624b82ff0"}
Sep 16 02:45:41.135623 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:45:41.135231    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e21dae-openshift-default-78c8949998-cn6ks" event={"ID":"eb94df1b-32d7-4c01-b581-8e09b4766544","Type":"ContainerStarted","Data":"f0e8e774e756e46bca4642cad2ae385393b078719d4818a26d91e2d2750f62bc"}
Sep 16 02:45:41.155273 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:45:41.155214    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-e21dae-openshift-default-78c8949998-cn6ks" podStartSLOduration=2.155198025 podStartE2EDuration="2.155198025s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 02:45:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:45:41.154037383 +0000 UTC m=+1954.072526657" watchObservedRunningTime="2026-09-16 02:45:41.155198025 +0000 UTC m=+1954.073687274"
Sep 16 02:45:41.937230 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:45:41.937192    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-e21dae-openshift-default-78c8949998-cn6ks"
Sep 16 02:45:41.942123 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:45:41.942096    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-e21dae-openshift-default-78c8949998-cn6ks"
Sep 16 02:45:42.141134 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:45:42.141107    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-e21dae-openshift-default-78c8949998-cn6ks"
Sep 16 02:45:42.142044 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:45:42.142025    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-e21dae-openshift-default-78c8949998-cn6ks"
Sep 16 02:45:47.919174 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:45:47.919142    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-e21dae-696f5cd9b6-4hl4m"]
Sep 16 02:45:47.922895 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:45:47.922876    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e21dae-696f5cd9b6-4hl4m"
Sep 16 02:45:47.925018 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:45:47.924991    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-e21dae-dockercfg-7xcpw\""
Sep 16 02:45:47.925140 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:45:47.925048    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-e21dae\""
Sep 16 02:45:47.933491 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:45:47.933467    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-e21dae-696f5cd9b6-4hl4m"]
Sep 16 02:45:48.031721 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:45:48.030326    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/916cd446-8e09-4f5c-ad3f-52ce91b98f6c-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-e21dae-696f5cd9b6-4hl4m\" (UID: \"916cd446-8e09-4f5c-ad3f-52ce91b98f6c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-e21dae-696f5cd9b6-4hl4m"
Sep 16 02:45:48.031721 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:45:48.030500    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zplrb\" (UniqueName: \"kubernetes.io/projected/916cd446-8e09-4f5c-ad3f-52ce91b98f6c-kube-api-access-zplrb\") pod \"payload-pre-processing-e2e-api-b-e21dae-696f5cd9b6-4hl4m\" (UID: \"916cd446-8e09-4f5c-ad3f-52ce91b98f6c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-e21dae-696f5cd9b6-4hl4m"
Sep 16 02:45:48.131492 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:45:48.131455    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/916cd446-8e09-4f5c-ad3f-52ce91b98f6c-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-e21dae-696f5cd9b6-4hl4m\" (UID: \"916cd446-8e09-4f5c-ad3f-52ce91b98f6c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-e21dae-696f5cd9b6-4hl4m"
Sep 16 02:45:48.131689 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:45:48.131558    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zplrb\" (UniqueName: \"kubernetes.io/projected/916cd446-8e09-4f5c-ad3f-52ce91b98f6c-kube-api-access-zplrb\") pod \"payload-pre-processing-e2e-api-b-e21dae-696f5cd9b6-4hl4m\" (UID: \"916cd446-8e09-4f5c-ad3f-52ce91b98f6c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-e21dae-696f5cd9b6-4hl4m"
Sep 16 02:45:48.132254 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:45:48.132227    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/916cd446-8e09-4f5c-ad3f-52ce91b98f6c-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-e21dae-696f5cd9b6-4hl4m\" (UID: \"916cd446-8e09-4f5c-ad3f-52ce91b98f6c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-e21dae-696f5cd9b6-4hl4m"
Sep 16 02:45:48.140255 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:45:48.140234    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zplrb\" (UniqueName: \"kubernetes.io/projected/916cd446-8e09-4f5c-ad3f-52ce91b98f6c-kube-api-access-zplrb\") pod \"payload-pre-processing-e2e-api-b-e21dae-696f5cd9b6-4hl4m\" (UID: \"916cd446-8e09-4f5c-ad3f-52ce91b98f6c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-e21dae-696f5cd9b6-4hl4m"
Sep 16 02:45:48.235825 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:45:48.235733    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e21dae-696f5cd9b6-4hl4m"
Sep 16 02:45:48.506240 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:45:48.506212    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-e21dae-696f5cd9b6-4hl4m"]
Sep 16 02:45:48.507518 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:45:48.507492    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod916cd446_8e09_4f5c_ad3f_52ce91b98f6c.slice/crio-a574ed3aaa69da48ed247876aa0ffcc612891596f9add8de34606b8b3f84377b WatchSource:0}: Error finding container a574ed3aaa69da48ed247876aa0ffcc612891596f9add8de34606b8b3f84377b: Status 404 returned error can't find the container with id a574ed3aaa69da48ed247876aa0ffcc612891596f9add8de34606b8b3f84377b
Sep 16 02:45:49.180531 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:45:49.180497    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e21dae-696f5cd9b6-4hl4m" event={"ID":"916cd446-8e09-4f5c-ad3f-52ce91b98f6c","Type":"ContainerStarted","Data":"069a44f0cfea7e55e4b7097de0f48c66107990ecd422009112e12ef091543efd"}
Sep 16 02:45:49.180943 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:45:49.180548    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e21dae-696f5cd9b6-4hl4m" event={"ID":"916cd446-8e09-4f5c-ad3f-52ce91b98f6c","Type":"ContainerStarted","Data":"a574ed3aaa69da48ed247876aa0ffcc612891596f9add8de34606b8b3f84377b"}
Sep 16 02:45:49.180943 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:45:49.180634    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e21dae-696f5cd9b6-4hl4m"
Sep 16 02:45:49.197775 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:45:49.197687    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e21dae-696f5cd9b6-4hl4m" podStartSLOduration=2.197663575 podStartE2EDuration="2.197663575s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 02:45:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:45:49.194849493 +0000 UTC m=+1962.113338743" watchObservedRunningTime="2026-09-16 02:45:49.197663575 +0000 UTC m=+1962.116152826"
Sep 16 02:45:55.543986 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:45:55.543951    2558 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-processing-5d7dbbd56b-pltkr" secret="" err="secret \"payload-processing-dockercfg-zzx6j\" not found"
Sep 16 02:46:00.188065 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:46:00.187985    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e21dae-696f5cd9b6-4hl4m"
Sep 16 02:46:06.286104 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:46:06.286063    2558 generic.go:356] "Generic (PLEG): container finished" podID="eb94df1b-32d7-4c01-b581-8e09b4766544" containerID="129862a294c189e5bae9216779b427b84a51c3c767493c7be7a1df0624b82ff0" exitCode=0
Sep 16 02:46:06.286542 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:46:06.286127    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e21dae-openshift-default-78c8949998-cn6ks" event={"ID":"eb94df1b-32d7-4c01-b581-8e09b4766544","Type":"ContainerDied","Data":"129862a294c189e5bae9216779b427b84a51c3c767493c7be7a1df0624b82ff0"}
Sep 16 02:46:06.286644 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:46:06.286628    2558 scope.go:117] "RemoveContainer" containerID="129862a294c189e5bae9216779b427b84a51c3c767493c7be7a1df0624b82ff0"
Sep 16 02:46:06.287874 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:46:06.287844    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 16 02:46:06.287949 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:46:06.287915    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 16 02:46:06.287986 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:46:06.287948    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 16 02:46:07.294311 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:46:07.294275    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e21dae-openshift-default-78c8949998-cn6ks" event={"ID":"eb94df1b-32d7-4c01-b581-8e09b4766544","Type":"ContainerStarted","Data":"c7d1f443539af7a4bb910d77ae256cfc5d6cc7d380cb310c0d2c5a54316947b0"}
Sep 16 02:46:07.877038 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:46:07.877004    2558 scope.go:117] "RemoveContainer" containerID="4a847856a382090a1271bd0879e60ebc17b3b4e8e48bc50824b56c852e7ac82d"
Sep 16 02:46:07.886261 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:46:07.886237    2558 scope.go:117] "RemoveContainer" containerID="c0781b810a269139118704766de2456a4389cecb9b53888de05825745b85064f"
Sep 16 02:46:07.894401 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:46:07.894376    2558 scope.go:117] "RemoveContainer" containerID="fd638446489c61ca8f37b0ca7264e66c3e0d458f329e82e4c90daafd7d7361ef"
Sep 16 02:46:07.902751 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:46:07.902724    2558 scope.go:117] "RemoveContainer" containerID="2861abdb269844cd5ece908d80426a3211c4e8ff9c369ed668f017b696d940e7"
Sep 16 02:46:07.937301 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:46:07.937276    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-e21dae-openshift-default-78c8949998-cn6ks"
Sep 16 02:46:07.938599 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:46:07.938577    2558 patch_prober.go:28] interesting pod/e2e-api-b-e21dae-openshift-default-78c8949998-cn6ks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Sep 16 02:46:07.938683 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:46:07.938633    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e21dae-openshift-default-78c8949998-cn6ks" podUID="eb94df1b-32d7-4c01-b581-8e09b4766544" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Sep 16 02:46:08.937711 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:46:08.937672    2558 patch_prober.go:28] interesting pod/e2e-api-b-e21dae-openshift-default-78c8949998-cn6ks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Sep 16 02:46:08.938111 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:46:08.937766    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e21dae-openshift-default-78c8949998-cn6ks" podUID="eb94df1b-32d7-4c01-b581-8e09b4766544" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Sep 16 02:46:09.938087 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:46:09.938049    2558 patch_prober.go:28] interesting pod/e2e-api-b-e21dae-openshift-default-78c8949998-cn6ks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Sep 16 02:46:09.938494 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:46:09.938110    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e21dae-openshift-default-78c8949998-cn6ks" podUID="eb94df1b-32d7-4c01-b581-8e09b4766544" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Sep 16 02:46:10.937819 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:46:10.937780    2558 patch_prober.go:28] interesting pod/e2e-api-b-e21dae-openshift-default-78c8949998-cn6ks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Sep 16 02:46:10.938000 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:46:10.937844    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e21dae-openshift-default-78c8949998-cn6ks" podUID="eb94df1b-32d7-4c01-b581-8e09b4766544" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Sep 16 02:46:11.938448 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:46:11.938405    2558 patch_prober.go:28] interesting pod/e2e-api-b-e21dae-openshift-default-78c8949998-cn6ks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Sep 16 02:46:11.938954 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:46:11.938471    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e21dae-openshift-default-78c8949998-cn6ks" podUID="eb94df1b-32d7-4c01-b581-8e09b4766544" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Sep 16 02:46:12.142068 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:46:12.142024    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-e21dae-openshift-default-78c8949998-cn6ks"
Sep 16 02:46:12.942224 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:46:12.942185    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-e21dae-openshift-default-78c8949998-cn6ks"
Sep 16 02:46:12.943350 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:46:12.943330    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-e21dae-openshift-default-78c8949998-cn6ks"
Sep 16 02:46:16.053809 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:46:16.053769    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7448c4-fjntb"]
Sep 16 02:46:16.057590 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:46:16.057565    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7448c4-fjntb"
Sep 16 02:46:16.059717 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:46:16.059677    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jpd4f\""
Sep 16 02:46:16.059969 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:46:16.059790    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 16 02:46:16.063943 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:46:16.063916    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7448c4-fjntb"]
Sep 16 02:46:16.147374 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:46:16.147339    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/23e07a3b-50a0-4f3e-8646-5ebab61a97c8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-7448c4-fjntb\" (UID: \"23e07a3b-50a0-4f3e-8646-5ebab61a97c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7448c4-fjntb"
Sep 16 02:46:16.248632 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:46:16.248593    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/23e07a3b-50a0-4f3e-8646-5ebab61a97c8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-7448c4-fjntb\" (UID: \"23e07a3b-50a0-4f3e-8646-5ebab61a97c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7448c4-fjntb"
Sep 16 02:46:16.249296 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:46:16.249260    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/23e07a3b-50a0-4f3e-8646-5ebab61a97c8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-7448c4-fjntb\" (UID: \"23e07a3b-50a0-4f3e-8646-5ebab61a97c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7448c4-fjntb"
Sep 16 02:46:16.375294 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:46:16.375202    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7448c4-fjntb"
Sep 16 02:46:16.659100 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:46:16.659068    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7448c4-fjntb"]
Sep 16 02:46:16.660935 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:46:16.660906    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23e07a3b_50a0_4f3e_8646_5ebab61a97c8.slice/crio-8dcc0ddf4c9c888dd59aded0f8dd3656370fa13b80db0eecd64cd1626698eb4e WatchSource:0}: Error finding container 8dcc0ddf4c9c888dd59aded0f8dd3656370fa13b80db0eecd64cd1626698eb4e: Status 404 returned error can't find the container with id 8dcc0ddf4c9c888dd59aded0f8dd3656370fa13b80db0eecd64cd1626698eb4e
Sep 16 02:46:17.359622 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:46:17.359593    2558 generic.go:356] "Generic (PLEG): container finished" podID="23e07a3b-50a0-4f3e-8646-5ebab61a97c8" containerID="319df266619bf085f84e853987889086c22266e8d72194a67741c5353180b828" exitCode=0
Sep 16 02:46:17.360022 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:46:17.359674    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7448c4-fjntb" event={"ID":"23e07a3b-50a0-4f3e-8646-5ebab61a97c8","Type":"ContainerDied","Data":"319df266619bf085f84e853987889086c22266e8d72194a67741c5353180b828"}
Sep 16 02:46:17.360022 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:46:17.359780    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7448c4-fjntb" event={"ID":"23e07a3b-50a0-4f3e-8646-5ebab61a97c8","Type":"ContainerStarted","Data":"8dcc0ddf4c9c888dd59aded0f8dd3656370fa13b80db0eecd64cd1626698eb4e"}
Sep 16 02:46:18.502769 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:46:18.502738    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7448c4-fjntb"
Sep 16 02:46:18.573356 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:46:18.573320    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/23e07a3b-50a0-4f3e-8646-5ebab61a97c8-maas-api-service-ca\") pod \"23e07a3b-50a0-4f3e-8646-5ebab61a97c8\" (UID: \"23e07a3b-50a0-4f3e-8646-5ebab61a97c8\") "
Sep 16 02:46:18.573823 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:46:18.573802    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23e07a3b-50a0-4f3e-8646-5ebab61a97c8-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "23e07a3b-50a0-4f3e-8646-5ebab61a97c8" (UID: "23e07a3b-50a0-4f3e-8646-5ebab61a97c8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 02:46:18.675384 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:46:18.675349    2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/23e07a3b-50a0-4f3e-8646-5ebab61a97c8-maas-api-service-ca\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:46:19.373098 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:46:19.373069    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7448c4-fjntb"
Sep 16 02:46:19.373274 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:46:19.373106    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7448c4-fjntb" event={"ID":"23e07a3b-50a0-4f3e-8646-5ebab61a97c8","Type":"ContainerDied","Data":"8dcc0ddf4c9c888dd59aded0f8dd3656370fa13b80db0eecd64cd1626698eb4e"}
Sep 16 02:46:19.373274 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:46:19.373160    2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8dcc0ddf4c9c888dd59aded0f8dd3656370fa13b80db0eecd64cd1626698eb4e"
Sep 16 02:46:30.321836 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:46:30.321806    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5d7dbbd56b-pltkr_4189cfcd-c3ba-4bad-813f-5a6851cb5317/payload-processing/0.log"
Sep 16 02:46:30.439556 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:46:30.439526    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5d7dbbd56b-pltkr_4189cfcd-c3ba-4bad-813f-5a6851cb5317/payload-processing/0.log"
Sep 16 02:46:31.194394 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:46:31.194358    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-5b136e-openshift-default-696f88d9c6-zwnvx"]
Sep 16 02:46:31.194922 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:46:31.194903    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23e07a3b-50a0-4f3e-8646-5ebab61a97c8" containerName="revoke-keys"
Sep 16 02:46:31.194922 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:46:31.194924    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="23e07a3b-50a0-4f3e-8646-5ebab61a97c8" containerName="revoke-keys"
Sep 16 02:46:31.195093 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:46:31.195013    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="23e07a3b-50a0-4f3e-8646-5ebab61a97c8" containerName="revoke-keys"
Sep 16 02:46:31.198567 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:46:31.198542    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-5b136e-openshift-default-696f88d9c6-zwnvx"
Sep 16 02:46:31.201273 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:46:31.201247    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-cleanup-5b136e-openshift-default-dockercfg-k5wqs\""
Sep 16 02:46:31.209082 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:46:31.208996    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-5b136e-openshift-default-696f88d9c6-zwnvx"]
Sep 16 02:46:31.316725 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:46:31.315221    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tlzgz\" (UniqueName: \"kubernetes.io/projected/34139e40-51de-48b1-b460-a6b6a20e82b7-kube-api-access-tlzgz\") pod \"e2e-ipp-cleanup-5b136e-openshift-default-696f88d9c6-zwnvx\" (UID: \"34139e40-51de-48b1-b460-a6b6a20e82b7\") " pod="openshift-ingress/e2e-ipp-cleanup-5b136e-openshift-default-696f88d9c6-zwnvx"
Sep 16 02:46:31.316725 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:46:31.315305    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34139e40-51de-48b1-b460-a6b6a20e82b7-credential-socket\") pod \"e2e-ipp-cleanup-5b136e-openshift-default-696f88d9c6-zwnvx\" (UID: \"34139e40-51de-48b1-b460-a6b6a20e82b7\") " pod="openshift-ingress/e2e-ipp-cleanup-5b136e-openshift-default-696f88d9c6-zwnvx"
Sep 16 02:46:31.316725 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:46:31.315350    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34139e40-51de-48b1-b460-a6b6a20e82b7-workload-socket\") pod \"e2e-ipp-cleanup-5b136e-openshift-default-696f88d9c6-zwnvx\" (UID: \"34139e40-51de-48b1-b460-a6b6a20e82b7\") " pod="openshift-ingress/e2e-ipp-cleanup-5b136e-openshift-default-696f88d9c6-zwnvx"
Sep 16 02:46:31.316725 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:46:31.315384    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34139e40-51de-48b1-b460-a6b6a20e82b7-istio-podinfo\") pod \"e2e-ipp-cleanup-5b136e-openshift-default-696f88d9c6-zwnvx\" (UID: \"34139e40-51de-48b1-b460-a6b6a20e82b7\") " pod="openshift-ingress/e2e-ipp-cleanup-5b136e-openshift-default-696f88d9c6-zwnvx"
Sep 16 02:46:31.316725 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:46:31.315411    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34139e40-51de-48b1-b460-a6b6a20e82b7-istio-data\") pod \"e2e-ipp-cleanup-5b136e-openshift-default-696f88d9c6-zwnvx\" (UID: \"34139e40-51de-48b1-b460-a6b6a20e82b7\") " pod="openshift-ingress/e2e-ipp-cleanup-5b136e-openshift-default-696f88d9c6-zwnvx"
Sep 16 02:46:31.316725 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:46:31.315447    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34139e40-51de-48b1-b460-a6b6a20e82b7-istio-envoy\") pod \"e2e-ipp-cleanup-5b136e-openshift-default-696f88d9c6-zwnvx\" (UID: \"34139e40-51de-48b1-b460-a6b6a20e82b7\") " pod="openshift-ingress/e2e-ipp-cleanup-5b136e-openshift-default-696f88d9c6-zwnvx"
Sep 16 02:46:31.316725 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:46:31.315523    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34139e40-51de-48b1-b460-a6b6a20e82b7-istiod-ca-cert\") pod \"e2e-ipp-cleanup-5b136e-openshift-default-696f88d9c6-zwnvx\" (UID: \"34139e40-51de-48b1-b460-a6b6a20e82b7\") " pod="openshift-ingress/e2e-ipp-cleanup-5b136e-openshift-default-696f88d9c6-zwnvx"
Sep 16 02:46:31.316725 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:46:31.315548    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34139e40-51de-48b1-b460-a6b6a20e82b7-workload-certs\") pod \"e2e-ipp-cleanup-5b136e-openshift-default-696f88d9c6-zwnvx\" (UID: \"34139e40-51de-48b1-b460-a6b6a20e82b7\") " pod="openshift-ingress/e2e-ipp-cleanup-5b136e-openshift-default-696f88d9c6-zwnvx"
Sep 16 02:46:31.316725 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:46:31.315575    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34139e40-51de-48b1-b460-a6b6a20e82b7-istio-token\") pod \"e2e-ipp-cleanup-5b136e-openshift-default-696f88d9c6-zwnvx\" (UID: \"34139e40-51de-48b1-b460-a6b6a20e82b7\") " pod="openshift-ingress/e2e-ipp-cleanup-5b136e-openshift-default-696f88d9c6-zwnvx"
Sep 16 02:46:31.416674 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:46:31.416635    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34139e40-51de-48b1-b460-a6b6a20e82b7-istio-envoy\") pod \"e2e-ipp-cleanup-5b136e-openshift-default-696f88d9c6-zwnvx\" (UID: \"34139e40-51de-48b1-b460-a6b6a20e82b7\") " pod="openshift-ingress/e2e-ipp-cleanup-5b136e-openshift-default-696f88d9c6-zwnvx"
Sep 16 02:46:31.417201 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:46:31.416758    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34139e40-51de-48b1-b460-a6b6a20e82b7-istiod-ca-cert\") pod \"e2e-ipp-cleanup-5b136e-openshift-default-696f88d9c6-zwnvx\" (UID: \"34139e40-51de-48b1-b460-a6b6a20e82b7\") " pod="openshift-ingress/e2e-ipp-cleanup-5b136e-openshift-default-696f88d9c6-zwnvx"
Sep 16 02:46:31.417201 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:46:31.416788    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34139e40-51de-48b1-b460-a6b6a20e82b7-workload-certs\") pod \"e2e-ipp-cleanup-5b136e-openshift-default-696f88d9c6-zwnvx\" (UID: \"34139e40-51de-48b1-b460-a6b6a20e82b7\") " pod="openshift-ingress/e2e-ipp-cleanup-5b136e-openshift-default-696f88d9c6-zwnvx"
Sep 16 02:46:31.417201 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:46:31.416821    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34139e40-51de-48b1-b460-a6b6a20e82b7-istio-token\") pod \"e2e-ipp-cleanup-5b136e-openshift-default-696f88d9c6-zwnvx\" (UID: \"34139e40-51de-48b1-b460-a6b6a20e82b7\") " pod="openshift-ingress/e2e-ipp-cleanup-5b136e-openshift-default-696f88d9c6-zwnvx"
Sep 16 02:46:31.417201 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:46:31.416910    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tlzgz\" (UniqueName: \"kubernetes.io/projected/34139e40-51de-48b1-b460-a6b6a20e82b7-kube-api-access-tlzgz\") pod \"e2e-ipp-cleanup-5b136e-openshift-default-696f88d9c6-zwnvx\" (UID: \"34139e40-51de-48b1-b460-a6b6a20e82b7\") " pod="openshift-ingress/e2e-ipp-cleanup-5b136e-openshift-default-696f88d9c6-zwnvx"
Sep 16 02:46:31.417201 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:46:31.416971    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34139e40-51de-48b1-b460-a6b6a20e82b7-credential-socket\") pod \"e2e-ipp-cleanup-5b136e-openshift-default-696f88d9c6-zwnvx\" (UID: \"34139e40-51de-48b1-b460-a6b6a20e82b7\") " pod="openshift-ingress/e2e-ipp-cleanup-5b136e-openshift-default-696f88d9c6-zwnvx"
Sep 16 02:46:31.417201 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:46:31.417020    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34139e40-51de-48b1-b460-a6b6a20e82b7-workload-socket\") pod \"e2e-ipp-cleanup-5b136e-openshift-default-696f88d9c6-zwnvx\" (UID: \"34139e40-51de-48b1-b460-a6b6a20e82b7\") " pod="openshift-ingress/e2e-ipp-cleanup-5b136e-openshift-default-696f88d9c6-zwnvx"
Sep 16 02:46:31.417201 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:46:31.417051    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34139e40-51de-48b1-b460-a6b6a20e82b7-istio-podinfo\") pod \"e2e-ipp-cleanup-5b136e-openshift-default-696f88d9c6-zwnvx\" (UID: \"34139e40-51de-48b1-b460-a6b6a20e82b7\") " pod="openshift-ingress/e2e-ipp-cleanup-5b136e-openshift-default-696f88d9c6-zwnvx"
Sep 16 02:46:31.417201 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:46:31.417077    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34139e40-51de-48b1-b460-a6b6a20e82b7-istio-data\") pod \"e2e-ipp-cleanup-5b136e-openshift-default-696f88d9c6-zwnvx\" (UID: \"34139e40-51de-48b1-b460-a6b6a20e82b7\") " pod="openshift-ingress/e2e-ipp-cleanup-5b136e-openshift-default-696f88d9c6-zwnvx"
Sep 16 02:46:31.417201 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:46:31.417184    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34139e40-51de-48b1-b460-a6b6a20e82b7-workload-certs\") pod \"e2e-ipp-cleanup-5b136e-openshift-default-696f88d9c6-zwnvx\" (UID: \"34139e40-51de-48b1-b460-a6b6a20e82b7\") " pod="openshift-ingress/e2e-ipp-cleanup-5b136e-openshift-default-696f88d9c6-zwnvx"
Sep 16 02:46:31.417607 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:46:31.417555    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34139e40-51de-48b1-b460-a6b6a20e82b7-istiod-ca-cert\") pod \"e2e-ipp-cleanup-5b136e-openshift-default-696f88d9c6-zwnvx\" (UID: \"34139e40-51de-48b1-b460-a6b6a20e82b7\") " pod="openshift-ingress/e2e-ipp-cleanup-5b136e-openshift-default-696f88d9c6-zwnvx"
Sep 16 02:46:31.417661 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:46:31.417608    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34139e40-51de-48b1-b460-a6b6a20e82b7-istio-data\") pod \"e2e-ipp-cleanup-5b136e-openshift-default-696f88d9c6-zwnvx\" (UID: \"34139e40-51de-48b1-b460-a6b6a20e82b7\") " pod="openshift-ingress/e2e-ipp-cleanup-5b136e-openshift-default-696f88d9c6-zwnvx"
Sep 16 02:46:31.417661 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:46:31.417638    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34139e40-51de-48b1-b460-a6b6a20e82b7-credential-socket\") pod \"e2e-ipp-cleanup-5b136e-openshift-default-696f88d9c6-zwnvx\" (UID: \"34139e40-51de-48b1-b460-a6b6a20e82b7\") " pod="openshift-ingress/e2e-ipp-cleanup-5b136e-openshift-default-696f88d9c6-zwnvx"
Sep 16 02:46:31.417773 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:46:31.417720    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34139e40-51de-48b1-b460-a6b6a20e82b7-workload-socket\") pod \"e2e-ipp-cleanup-5b136e-openshift-default-696f88d9c6-zwnvx\" (UID: \"34139e40-51de-48b1-b460-a6b6a20e82b7\") " pod="openshift-ingress/e2e-ipp-cleanup-5b136e-openshift-default-696f88d9c6-zwnvx"
Sep 16 02:46:31.419149 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:46:31.419131    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34139e40-51de-48b1-b460-a6b6a20e82b7-istio-envoy\") pod \"e2e-ipp-cleanup-5b136e-openshift-default-696f88d9c6-zwnvx\" (UID: \"34139e40-51de-48b1-b460-a6b6a20e82b7\") " pod="openshift-ingress/e2e-ipp-cleanup-5b136e-openshift-default-696f88d9c6-zwnvx"
Sep 16 02:46:31.419412 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:46:31.419394    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34139e40-51de-48b1-b460-a6b6a20e82b7-istio-podinfo\") pod \"e2e-ipp-cleanup-5b136e-openshift-default-696f88d9c6-zwnvx\" (UID: \"34139e40-51de-48b1-b460-a6b6a20e82b7\") " pod="openshift-ingress/e2e-ipp-cleanup-5b136e-openshift-default-696f88d9c6-zwnvx"
Sep 16 02:46:31.424530 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:46:31.424498    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34139e40-51de-48b1-b460-a6b6a20e82b7-istio-token\") pod \"e2e-ipp-cleanup-5b136e-openshift-default-696f88d9c6-zwnvx\" (UID: \"34139e40-51de-48b1-b460-a6b6a20e82b7\") " pod="openshift-ingress/e2e-ipp-cleanup-5b136e-openshift-default-696f88d9c6-zwnvx"
Sep 16 02:46:31.424731 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:46:31.424686    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tlzgz\" (UniqueName: \"kubernetes.io/projected/34139e40-51de-48b1-b460-a6b6a20e82b7-kube-api-access-tlzgz\") pod \"e2e-ipp-cleanup-5b136e-openshift-default-696f88d9c6-zwnvx\" (UID: \"34139e40-51de-48b1-b460-a6b6a20e82b7\") " pod="openshift-ingress/e2e-ipp-cleanup-5b136e-openshift-default-696f88d9c6-zwnvx"
Sep 16 02:46:31.512520 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:46:31.512430    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-5b136e-openshift-default-696f88d9c6-zwnvx"
Sep 16 02:46:31.822767 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:46:31.822674    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-5b136e-openshift-default-696f88d9c6-zwnvx"]
Sep 16 02:46:31.831981 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:46:31.831955    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34139e40_51de_48b1_b460_a6b6a20e82b7.slice/crio-007534df8e3a77d2e6592e6934dc61915f13ec41c66c55fb14db003830b486c2 WatchSource:0}: Error finding container 007534df8e3a77d2e6592e6934dc61915f13ec41c66c55fb14db003830b486c2: Status 404 returned error can't find the container with id 007534df8e3a77d2e6592e6934dc61915f13ec41c66c55fb14db003830b486c2
Sep 16 02:46:31.834430 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:46:31.834395    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 16 02:46:31.834539 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:46:31.834459    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 16 02:46:31.834539 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:46:31.834490    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 16 02:46:31.957211 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:46:31.957162    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7448c4-fjntb"]
Sep 16 02:46:31.959331 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:46:31.959307    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7448c4-fjntb"]
Sep 16 02:46:32.457946 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:46:32.457908    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-5b136e-openshift-default-696f88d9c6-zwnvx" event={"ID":"34139e40-51de-48b1-b460-a6b6a20e82b7","Type":"ContainerStarted","Data":"72902ba8343f4bf161d4fba2aaf3124ac47958df8eefd4096df84dc3f7b224bd"}
Sep 16 02:46:32.458384 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:46:32.458006    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-5b136e-openshift-default-696f88d9c6-zwnvx" event={"ID":"34139e40-51de-48b1-b460-a6b6a20e82b7","Type":"ContainerStarted","Data":"007534df8e3a77d2e6592e6934dc61915f13ec41c66c55fb14db003830b486c2"}
Sep 16 02:46:32.512893 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:46:32.512850    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-5b136e-openshift-default-696f88d9c6-zwnvx"
Sep 16 02:46:32.514614 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:46:32.514585    2558 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-5b136e-openshift-default-696f88d9c6-zwnvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Sep 16 02:46:32.514760 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:46:32.514673    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-5b136e-openshift-default-696f88d9c6-zwnvx" podUID="34139e40-51de-48b1-b460-a6b6a20e82b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Sep 16 02:46:33.517657 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:46:33.517622    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-5b136e-openshift-default-696f88d9c6-zwnvx"
Sep 16 02:46:33.537831 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:46:33.537773    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-cleanup-5b136e-openshift-default-696f88d9c6-zwnvx" podStartSLOduration=2.537755297 podStartE2EDuration="2.537755297s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 02:46:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:46:32.480610469 +0000 UTC m=+2005.399099719" watchObservedRunningTime="2026-09-16 02:46:33.537755297 +0000 UTC m=+2006.456244545"
Sep 16 02:46:33.552041 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:46:33.552005    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="23e07a3b-50a0-4f3e-8646-5ebab61a97c8" path="/var/lib/kubelet/pods/23e07a3b-50a0-4f3e-8646-5ebab61a97c8/volumes"
Sep 16 02:46:34.470026 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:46:34.469987    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-5b136e-openshift-default-696f88d9c6-zwnvx"
Sep 16 02:46:34.470953 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:46:34.470931    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-5b136e-openshift-default-696f88d9c6-zwnvx"
Sep 16 02:46:39.440830 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:46:39.440792    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-7448c4-openshift-default-7884d9db5b-g56df"]
Sep 16 02:46:39.441301 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:46:39.441105    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-7448c4-openshift-default-7884d9db5b-g56df" podUID="0b7f764c-be16-4f85-996c-9b3383b90c23" containerName="istio-proxy" containerID="cri-o://bd6318476479f00951f9bf4c06664b0ecf05535608ac2a2a954e319aea5ff83f" gracePeriod=30
Sep 16 02:46:44.692556 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:46:44.692531    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-7448c4-openshift-default-7884d9db5b-g56df"
Sep 16 02:46:44.766164 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:46:44.766067    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b7f764c-be16-4f85-996c-9b3383b90c23-istiod-ca-cert\") pod \"0b7f764c-be16-4f85-996c-9b3383b90c23\" (UID: \"0b7f764c-be16-4f85-996c-9b3383b90c23\") "
Sep 16 02:46:44.766164 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:46:44.766112    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b7f764c-be16-4f85-996c-9b3383b90c23-istio-token\") pod \"0b7f764c-be16-4f85-996c-9b3383b90c23\" (UID: \"0b7f764c-be16-4f85-996c-9b3383b90c23\") "
Sep 16 02:46:44.766413 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:46:44.766172    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9ffbc\" (UniqueName: \"kubernetes.io/projected/0b7f764c-be16-4f85-996c-9b3383b90c23-kube-api-access-9ffbc\") pod \"0b7f764c-be16-4f85-996c-9b3383b90c23\" (UID: \"0b7f764c-be16-4f85-996c-9b3383b90c23\") "
Sep 16 02:46:44.766413 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:46:44.766208    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b7f764c-be16-4f85-996c-9b3383b90c23-credential-socket\") pod \"0b7f764c-be16-4f85-996c-9b3383b90c23\" (UID: \"0b7f764c-be16-4f85-996c-9b3383b90c23\") "
Sep 16 02:46:44.766413 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:46:44.766264    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b7f764c-be16-4f85-996c-9b3383b90c23-istio-data\") pod \"0b7f764c-be16-4f85-996c-9b3383b90c23\" (UID: \"0b7f764c-be16-4f85-996c-9b3383b90c23\") "
Sep 16 02:46:44.766413 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:46:44.766300    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b7f764c-be16-4f85-996c-9b3383b90c23-istio-podinfo\") pod \"0b7f764c-be16-4f85-996c-9b3383b90c23\" (UID: \"0b7f764c-be16-4f85-996c-9b3383b90c23\") "
Sep 16 02:46:44.766413 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:46:44.766332    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b7f764c-be16-4f85-996c-9b3383b90c23-istio-envoy\") pod \"0b7f764c-be16-4f85-996c-9b3383b90c23\" (UID: \"0b7f764c-be16-4f85-996c-9b3383b90c23\") "
Sep 16 02:46:44.766413 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:46:44.766364    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b7f764c-be16-4f85-996c-9b3383b90c23-workload-certs\") pod \"0b7f764c-be16-4f85-996c-9b3383b90c23\" (UID: \"0b7f764c-be16-4f85-996c-9b3383b90c23\") "
Sep 16 02:46:44.766413 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:46:44.766407    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b7f764c-be16-4f85-996c-9b3383b90c23-workload-socket\") pod \"0b7f764c-be16-4f85-996c-9b3383b90c23\" (UID: \"0b7f764c-be16-4f85-996c-9b3383b90c23\") "
Sep 16 02:46:44.766801 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:46:44.766506    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b7f764c-be16-4f85-996c-9b3383b90c23-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "0b7f764c-be16-4f85-996c-9b3383b90c23" (UID: "0b7f764c-be16-4f85-996c-9b3383b90c23"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 02:46:44.766801 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:46:44.766754    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b7f764c-be16-4f85-996c-9b3383b90c23-istio-data" (OuterVolumeSpecName: "istio-data") pod "0b7f764c-be16-4f85-996c-9b3383b90c23" (UID: "0b7f764c-be16-4f85-996c-9b3383b90c23"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:46:44.766917 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:46:44.766860    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b7f764c-be16-4f85-996c-9b3383b90c23-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "0b7f764c-be16-4f85-996c-9b3383b90c23" (UID: "0b7f764c-be16-4f85-996c-9b3383b90c23"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:46:44.766917 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:46:44.766897    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b7f764c-be16-4f85-996c-9b3383b90c23-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "0b7f764c-be16-4f85-996c-9b3383b90c23" (UID: "0b7f764c-be16-4f85-996c-9b3383b90c23"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:46:44.767034 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:46:44.766937    2558 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b7f764c-be16-4f85-996c-9b3383b90c23-istiod-ca-cert\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:46:44.767034 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:46:44.766964    2558 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b7f764c-be16-4f85-996c-9b3383b90c23-istio-data\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:46:44.767142 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:46:44.767118    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b7f764c-be16-4f85-996c-9b3383b90c23-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "0b7f764c-be16-4f85-996c-9b3383b90c23" (UID: "0b7f764c-be16-4f85-996c-9b3383b90c23"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:46:44.768548 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:46:44.768517    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b7f764c-be16-4f85-996c-9b3383b90c23-istio-token" (OuterVolumeSpecName: "istio-token") pod "0b7f764c-be16-4f85-996c-9b3383b90c23" (UID: "0b7f764c-be16-4f85-996c-9b3383b90c23"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 02:46:44.768754 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:46:44.768734    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b7f764c-be16-4f85-996c-9b3383b90c23-kube-api-access-9ffbc" (OuterVolumeSpecName: "kube-api-access-9ffbc") pod "0b7f764c-be16-4f85-996c-9b3383b90c23" (UID: "0b7f764c-be16-4f85-996c-9b3383b90c23"). InnerVolumeSpecName "kube-api-access-9ffbc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 02:46:44.768816 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:46:44.768735    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0b7f764c-be16-4f85-996c-9b3383b90c23-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "0b7f764c-be16-4f85-996c-9b3383b90c23" (UID: "0b7f764c-be16-4f85-996c-9b3383b90c23"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 02:46:44.768992 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:46:44.768974    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b7f764c-be16-4f85-996c-9b3383b90c23-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "0b7f764c-be16-4f85-996c-9b3383b90c23" (UID: "0b7f764c-be16-4f85-996c-9b3383b90c23"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:46:44.867483 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:46:44.867443    2558 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b7f764c-be16-4f85-996c-9b3383b90c23-credential-socket\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:46:44.867483 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:46:44.867478    2558 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b7f764c-be16-4f85-996c-9b3383b90c23-istio-podinfo\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:46:44.867483 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:46:44.867487    2558 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b7f764c-be16-4f85-996c-9b3383b90c23-istio-envoy\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:46:44.867740 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:46:44.867495    2558 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b7f764c-be16-4f85-996c-9b3383b90c23-workload-certs\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:46:44.867740 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:46:44.867503    2558 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b7f764c-be16-4f85-996c-9b3383b90c23-workload-socket\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:46:44.867740 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:46:44.867511    2558 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b7f764c-be16-4f85-996c-9b3383b90c23-istio-token\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:46:44.867740 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:46:44.867520    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9ffbc\" (UniqueName: \"kubernetes.io/projected/0b7f764c-be16-4f85-996c-9b3383b90c23-kube-api-access-9ffbc\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:46:45.534665 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:46:45.534630    2558 generic.go:356] "Generic (PLEG): container finished" podID="0b7f764c-be16-4f85-996c-9b3383b90c23" containerID="bd6318476479f00951f9bf4c06664b0ecf05535608ac2a2a954e319aea5ff83f" exitCode=0
Sep 16 02:46:45.534861 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:46:45.534674    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-7448c4-openshift-default-7884d9db5b-g56df"
Sep 16 02:46:45.534861 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:46:45.534735    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-7448c4-openshift-default-7884d9db5b-g56df" event={"ID":"0b7f764c-be16-4f85-996c-9b3383b90c23","Type":"ContainerDied","Data":"bd6318476479f00951f9bf4c06664b0ecf05535608ac2a2a954e319aea5ff83f"}
Sep 16 02:46:45.534861 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:46:45.534790    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-7448c4-openshift-default-7884d9db5b-g56df" event={"ID":"0b7f764c-be16-4f85-996c-9b3383b90c23","Type":"ContainerDied","Data":"0ab3e477b8902481c8d5dfdae6450311afb7c1a2a34861bdd1cb4acbde482eff"}
Sep 16 02:46:45.534861 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:46:45.534812    2558 scope.go:117] "RemoveContainer" containerID="bd6318476479f00951f9bf4c06664b0ecf05535608ac2a2a954e319aea5ff83f"
Sep 16 02:46:45.544665 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:46:45.544649    2558 scope.go:117] "RemoveContainer" containerID="f2981c57f7b9b44fbae9667c29b5ed0644608227accf505649d35d500b459bae"
Sep 16 02:46:45.553012 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:46:45.552991    2558 scope.go:117] "RemoveContainer" containerID="bd6318476479f00951f9bf4c06664b0ecf05535608ac2a2a954e319aea5ff83f"
Sep 16 02:46:45.553259 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:46:45.553241    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd6318476479f00951f9bf4c06664b0ecf05535608ac2a2a954e319aea5ff83f\": container with ID starting with bd6318476479f00951f9bf4c06664b0ecf05535608ac2a2a954e319aea5ff83f not found: ID does not exist" containerID="bd6318476479f00951f9bf4c06664b0ecf05535608ac2a2a954e319aea5ff83f"
Sep 16 02:46:45.553305 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:46:45.553268    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd6318476479f00951f9bf4c06664b0ecf05535608ac2a2a954e319aea5ff83f"} err="failed to get container status \"bd6318476479f00951f9bf4c06664b0ecf05535608ac2a2a954e319aea5ff83f\": rpc error: code = NotFound desc = could not find container \"bd6318476479f00951f9bf4c06664b0ecf05535608ac2a2a954e319aea5ff83f\": container with ID starting with bd6318476479f00951f9bf4c06664b0ecf05535608ac2a2a954e319aea5ff83f not found: ID does not exist"
Sep 16 02:46:45.553305 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:46:45.553284    2558 scope.go:117] "RemoveContainer" containerID="f2981c57f7b9b44fbae9667c29b5ed0644608227accf505649d35d500b459bae"
Sep 16 02:46:45.553538 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:46:45.553519    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2981c57f7b9b44fbae9667c29b5ed0644608227accf505649d35d500b459bae\": container with ID starting with f2981c57f7b9b44fbae9667c29b5ed0644608227accf505649d35d500b459bae not found: ID does not exist" containerID="f2981c57f7b9b44fbae9667c29b5ed0644608227accf505649d35d500b459bae"
Sep 16 02:46:45.553642 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:46:45.553540    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2981c57f7b9b44fbae9667c29b5ed0644608227accf505649d35d500b459bae"} err="failed to get container status \"f2981c57f7b9b44fbae9667c29b5ed0644608227accf505649d35d500b459bae\": rpc error: code = NotFound desc = could not find container \"f2981c57f7b9b44fbae9667c29b5ed0644608227accf505649d35d500b459bae\": container with ID starting with f2981c57f7b9b44fbae9667c29b5ed0644608227accf505649d35d500b459bae not found: ID does not exist"
Sep 16 02:46:45.557206 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:46:45.557183    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-7448c4-openshift-default-7884d9db5b-g56df"]
Sep 16 02:46:45.560491 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:46:45.560468    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-7448c4-openshift-default-7884d9db5b-g56df"]
Sep 16 02:46:47.553414 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:46:47.553377    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0b7f764c-be16-4f85-996c-9b3383b90c23" path="/var/lib/kubelet/pods/0b7f764c-be16-4f85-996c-9b3383b90c23/volumes"
Sep 16 02:46:55.255316 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:46:55.255283    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-e21dae-696f5cd9b6-4hl4m"]
Sep 16 02:46:55.255797 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:46:55.255528    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e21dae-696f5cd9b6-4hl4m" podUID="916cd446-8e09-4f5c-ad3f-52ce91b98f6c" containerName="payload-pre-processing" containerID="cri-o://069a44f0cfea7e55e4b7097de0f48c66107990ecd422009112e12ef091543efd" gracePeriod=30
Sep 16 02:46:55.562787 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:46:55.562760    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e21dae-696f5cd9b6-4hl4m"
Sep 16 02:46:55.606495 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:46:55.606456    2558 generic.go:356] "Generic (PLEG): container finished" podID="916cd446-8e09-4f5c-ad3f-52ce91b98f6c" containerID="069a44f0cfea7e55e4b7097de0f48c66107990ecd422009112e12ef091543efd" exitCode=0
Sep 16 02:46:55.606495 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:46:55.606493    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e21dae-696f5cd9b6-4hl4m"
Sep 16 02:46:55.606739 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:46:55.606507    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e21dae-696f5cd9b6-4hl4m" event={"ID":"916cd446-8e09-4f5c-ad3f-52ce91b98f6c","Type":"ContainerDied","Data":"069a44f0cfea7e55e4b7097de0f48c66107990ecd422009112e12ef091543efd"}
Sep 16 02:46:55.606739 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:46:55.606562    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e21dae-696f5cd9b6-4hl4m" event={"ID":"916cd446-8e09-4f5c-ad3f-52ce91b98f6c","Type":"ContainerDied","Data":"a574ed3aaa69da48ed247876aa0ffcc612891596f9add8de34606b8b3f84377b"}
Sep 16 02:46:55.606739 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:46:55.606595    2558 scope.go:117] "RemoveContainer" containerID="069a44f0cfea7e55e4b7097de0f48c66107990ecd422009112e12ef091543efd"
Sep 16 02:46:55.620762 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:46:55.620740    2558 scope.go:117] "RemoveContainer" containerID="069a44f0cfea7e55e4b7097de0f48c66107990ecd422009112e12ef091543efd"
Sep 16 02:46:55.621183 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:46:55.621150    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"069a44f0cfea7e55e4b7097de0f48c66107990ecd422009112e12ef091543efd\": container with ID starting with 069a44f0cfea7e55e4b7097de0f48c66107990ecd422009112e12ef091543efd not found: ID does not exist" containerID="069a44f0cfea7e55e4b7097de0f48c66107990ecd422009112e12ef091543efd"
Sep 16 02:46:55.621262 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:46:55.621189    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"069a44f0cfea7e55e4b7097de0f48c66107990ecd422009112e12ef091543efd"} err="failed to get container status \"069a44f0cfea7e55e4b7097de0f48c66107990ecd422009112e12ef091543efd\": rpc error: code = NotFound desc = could not find container \"069a44f0cfea7e55e4b7097de0f48c66107990ecd422009112e12ef091543efd\": container with ID starting with 069a44f0cfea7e55e4b7097de0f48c66107990ecd422009112e12ef091543efd not found: ID does not exist"
Sep 16 02:46:55.686081 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:46:55.686057    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/916cd446-8e09-4f5c-ad3f-52ce91b98f6c-plugins-config-volume\") pod \"916cd446-8e09-4f5c-ad3f-52ce91b98f6c\" (UID: \"916cd446-8e09-4f5c-ad3f-52ce91b98f6c\") "
Sep 16 02:46:55.686189 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:46:55.686113    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zplrb\" (UniqueName: \"kubernetes.io/projected/916cd446-8e09-4f5c-ad3f-52ce91b98f6c-kube-api-access-zplrb\") pod \"916cd446-8e09-4f5c-ad3f-52ce91b98f6c\" (UID: \"916cd446-8e09-4f5c-ad3f-52ce91b98f6c\") "
Sep 16 02:46:55.686472 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:46:55.686447    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/916cd446-8e09-4f5c-ad3f-52ce91b98f6c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "916cd446-8e09-4f5c-ad3f-52ce91b98f6c" (UID: "916cd446-8e09-4f5c-ad3f-52ce91b98f6c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 02:46:55.686725 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:46:55.686681    2558 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/916cd446-8e09-4f5c-ad3f-52ce91b98f6c-plugins-config-volume\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:46:55.688002 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:46:55.687984    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/916cd446-8e09-4f5c-ad3f-52ce91b98f6c-kube-api-access-zplrb" (OuterVolumeSpecName: "kube-api-access-zplrb") pod "916cd446-8e09-4f5c-ad3f-52ce91b98f6c" (UID: "916cd446-8e09-4f5c-ad3f-52ce91b98f6c"). InnerVolumeSpecName "kube-api-access-zplrb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 02:46:55.788265 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:46:55.788178    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zplrb\" (UniqueName: \"kubernetes.io/projected/916cd446-8e09-4f5c-ad3f-52ce91b98f6c-kube-api-access-zplrb\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:46:55.929082 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:46:55.929036    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-e21dae-696f5cd9b6-4hl4m"]
Sep 16 02:46:55.934621 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:46:55.934569    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-e21dae-696f5cd9b6-4hl4m"]
Sep 16 02:46:56.614609 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:46:56.614574    2558 generic.go:356] "Generic (PLEG): container finished" podID="34139e40-51de-48b1-b460-a6b6a20e82b7" containerID="72902ba8343f4bf161d4fba2aaf3124ac47958df8eefd4096df84dc3f7b224bd" exitCode=0
Sep 16 02:46:56.615010 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:46:56.614645    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-5b136e-openshift-default-696f88d9c6-zwnvx" event={"ID":"34139e40-51de-48b1-b460-a6b6a20e82b7","Type":"ContainerDied","Data":"72902ba8343f4bf161d4fba2aaf3124ac47958df8eefd4096df84dc3f7b224bd"}
Sep 16 02:46:56.615068 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:46:56.615054    2558 scope.go:117] "RemoveContainer" containerID="72902ba8343f4bf161d4fba2aaf3124ac47958df8eefd4096df84dc3f7b224bd"
Sep 16 02:46:56.616178 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:46:56.616152    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 16 02:46:56.616251 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:46:56.616218    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 16 02:46:56.616289 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:46:56.616249    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 16 02:46:57.550247 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:46:57.550214    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="916cd446-8e09-4f5c-ad3f-52ce91b98f6c" path="/var/lib/kubelet/pods/916cd446-8e09-4f5c-ad3f-52ce91b98f6c/volumes"
Sep 16 02:46:57.621608 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:46:57.621580    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-5b136e-openshift-default-696f88d9c6-zwnvx" event={"ID":"34139e40-51de-48b1-b460-a6b6a20e82b7","Type":"ContainerStarted","Data":"6037f35b4cb0dc53c51af6b55977cffdcc46a6a954d3f661948547baeaf1b346"}
Sep 16 02:46:58.513065 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:46:58.513024    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-5b136e-openshift-default-696f88d9c6-zwnvx"
Sep 16 02:46:58.513278 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:46:58.513252    2558 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-5b136e-openshift-default-696f88d9c6-zwnvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Sep 16 02:46:58.513331 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:46:58.513304    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-5b136e-openshift-default-696f88d9c6-zwnvx" podUID="34139e40-51de-48b1-b460-a6b6a20e82b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Sep 16 02:46:59.512890 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:46:59.512846    2558 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-5b136e-openshift-default-696f88d9c6-zwnvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Sep 16 02:46:59.513283 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:46:59.512907    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-5b136e-openshift-default-696f88d9c6-zwnvx" podUID="34139e40-51de-48b1-b460-a6b6a20e82b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Sep 16 02:47:00.513437 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:00.513389    2558 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-5b136e-openshift-default-696f88d9c6-zwnvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Sep 16 02:47:00.514176 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:00.514132    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-5b136e-openshift-default-696f88d9c6-zwnvx" podUID="34139e40-51de-48b1-b460-a6b6a20e82b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Sep 16 02:47:01.513963 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:01.513920    2558 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-5b136e-openshift-default-696f88d9c6-zwnvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Sep 16 02:47:01.514479 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:01.514007    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-5b136e-openshift-default-696f88d9c6-zwnvx" podUID="34139e40-51de-48b1-b460-a6b6a20e82b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Sep 16 02:47:02.280763 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:02.279105    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-e21dae-openshift-default-78c8949998-cn6ks"]
Sep 16 02:47:02.280763 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:02.279354    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-e21dae-openshift-default-78c8949998-cn6ks" podUID="eb94df1b-32d7-4c01-b581-8e09b4766544" containerName="istio-proxy" containerID="cri-o://c7d1f443539af7a4bb910d77ae256cfc5d6cc7d380cb310c0d2c5a54316947b0" gracePeriod=30
Sep 16 02:47:02.513859 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:02.513819    2558 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-5b136e-openshift-default-696f88d9c6-zwnvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Sep 16 02:47:02.514032 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:02.513894    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-5b136e-openshift-default-696f88d9c6-zwnvx" podUID="34139e40-51de-48b1-b460-a6b6a20e82b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Sep 16 02:47:03.517302 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:03.517262    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-5b136e-openshift-default-696f88d9c6-zwnvx"
Sep 16 02:47:03.517737 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:03.517562    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-5b136e-openshift-default-696f88d9c6-zwnvx"
Sep 16 02:47:03.518540 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:03.518516    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-5b136e-openshift-default-696f88d9c6-zwnvx"
Sep 16 02:47:05.279053 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:05.279015    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-5b136e-t8l55"]
Sep 16 02:47:05.279520 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:05.279436    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="916cd446-8e09-4f5c-ad3f-52ce91b98f6c" containerName="payload-pre-processing"
Sep 16 02:47:05.279520 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:05.279447    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="916cd446-8e09-4f5c-ad3f-52ce91b98f6c" containerName="payload-pre-processing"
Sep 16 02:47:05.279520 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:05.279465    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b7f764c-be16-4f85-996c-9b3383b90c23" containerName="istio-proxy"
Sep 16 02:47:05.279520 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:05.279471    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b7f764c-be16-4f85-996c-9b3383b90c23" containerName="istio-proxy"
Sep 16 02:47:05.279520 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:05.279479    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b7f764c-be16-4f85-996c-9b3383b90c23" containerName="istio-proxy"
Sep 16 02:47:05.279520 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:05.279484    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b7f764c-be16-4f85-996c-9b3383b90c23" containerName="istio-proxy"
Sep 16 02:47:05.279758 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:05.279552    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="916cd446-8e09-4f5c-ad3f-52ce91b98f6c" containerName="payload-pre-processing"
Sep 16 02:47:05.279758 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:05.279563    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b7f764c-be16-4f85-996c-9b3383b90c23" containerName="istio-proxy"
Sep 16 02:47:05.279758 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:05.279570    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b7f764c-be16-4f85-996c-9b3383b90c23" containerName="istio-proxy"
Sep 16 02:47:05.283512 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:05.283489    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-5b136e-t8l55"
Sep 16 02:47:05.286232 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:05.286206    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 16 02:47:05.286784 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:05.286764    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jpd4f\""
Sep 16 02:47:05.289150 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:05.289120    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-5b136e-t8l55"]
Sep 16 02:47:05.400053 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:05.400014    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/15c29ab2-70df-4eeb-92f7-42b9648082c2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-5b136e-t8l55\" (UID: \"15c29ab2-70df-4eeb-92f7-42b9648082c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-5b136e-t8l55"
Sep 16 02:47:05.500943 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:05.500905    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/15c29ab2-70df-4eeb-92f7-42b9648082c2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-5b136e-t8l55\" (UID: \"15c29ab2-70df-4eeb-92f7-42b9648082c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-5b136e-t8l55"
Sep 16 02:47:05.501520 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:05.501500    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/15c29ab2-70df-4eeb-92f7-42b9648082c2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-5b136e-t8l55\" (UID: \"15c29ab2-70df-4eeb-92f7-42b9648082c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-5b136e-t8l55"
Sep 16 02:47:05.597835 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:05.597746    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-5b136e-t8l55"
Sep 16 02:47:05.871530 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:05.871502    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-5b136e-t8l55"]
Sep 16 02:47:05.873730 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:47:05.873691    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15c29ab2_70df_4eeb_92f7_42b9648082c2.slice/crio-8fb5d4455c4f9d5d4d410ee220c5a790f6be377b75d657bc2016b3810cbb27d5 WatchSource:0}: Error finding container 8fb5d4455c4f9d5d4d410ee220c5a790f6be377b75d657bc2016b3810cbb27d5: Status 404 returned error can't find the container with id 8fb5d4455c4f9d5d4d410ee220c5a790f6be377b75d657bc2016b3810cbb27d5
Sep 16 02:47:06.676896 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:06.676858    2558 generic.go:356] "Generic (PLEG): container finished" podID="15c29ab2-70df-4eeb-92f7-42b9648082c2" containerID="5e06720532521b2a7ed37bc2d1ca59c2bbc2fb20ccb9c8588a5ea8f99b96e634" exitCode=0
Sep 16 02:47:06.677282 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:06.676938    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-5b136e-t8l55" event={"ID":"15c29ab2-70df-4eeb-92f7-42b9648082c2","Type":"ContainerDied","Data":"5e06720532521b2a7ed37bc2d1ca59c2bbc2fb20ccb9c8588a5ea8f99b96e634"}
Sep 16 02:47:06.677282 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:06.676992    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-5b136e-t8l55" event={"ID":"15c29ab2-70df-4eeb-92f7-42b9648082c2","Type":"ContainerStarted","Data":"8fb5d4455c4f9d5d4d410ee220c5a790f6be377b75d657bc2016b3810cbb27d5"}
Sep 16 02:47:07.552785 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:07.552015    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-e21dae-openshift-default-78c8949998-cn6ks"
Sep 16 02:47:07.684350 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:07.684314    2558 generic.go:356] "Generic (PLEG): container finished" podID="eb94df1b-32d7-4c01-b581-8e09b4766544" containerID="c7d1f443539af7a4bb910d77ae256cfc5d6cc7d380cb310c0d2c5a54316947b0" exitCode=0
Sep 16 02:47:07.684744 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:07.684371    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-e21dae-openshift-default-78c8949998-cn6ks"
Sep 16 02:47:07.684744 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:07.684398    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e21dae-openshift-default-78c8949998-cn6ks" event={"ID":"eb94df1b-32d7-4c01-b581-8e09b4766544","Type":"ContainerDied","Data":"c7d1f443539af7a4bb910d77ae256cfc5d6cc7d380cb310c0d2c5a54316947b0"}
Sep 16 02:47:07.684744 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:07.684454    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e21dae-openshift-default-78c8949998-cn6ks" event={"ID":"eb94df1b-32d7-4c01-b581-8e09b4766544","Type":"ContainerDied","Data":"f0e8e774e756e46bca4642cad2ae385393b078719d4818a26d91e2d2750f62bc"}
Sep 16 02:47:07.684744 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:07.684478    2558 scope.go:117] "RemoveContainer" containerID="c7d1f443539af7a4bb910d77ae256cfc5d6cc7d380cb310c0d2c5a54316947b0"
Sep 16 02:47:07.693764 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:07.693738    2558 scope.go:117] "RemoveContainer" containerID="129862a294c189e5bae9216779b427b84a51c3c767493c7be7a1df0624b82ff0"
Sep 16 02:47:07.703345 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:07.703317    2558 scope.go:117] "RemoveContainer" containerID="c7d1f443539af7a4bb910d77ae256cfc5d6cc7d380cb310c0d2c5a54316947b0"
Sep 16 02:47:07.703685 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:47:07.703662    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7d1f443539af7a4bb910d77ae256cfc5d6cc7d380cb310c0d2c5a54316947b0\": container with ID starting with c7d1f443539af7a4bb910d77ae256cfc5d6cc7d380cb310c0d2c5a54316947b0 not found: ID does not exist" containerID="c7d1f443539af7a4bb910d77ae256cfc5d6cc7d380cb310c0d2c5a54316947b0"
Sep 16 02:47:07.703777 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:07.703710    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7d1f443539af7a4bb910d77ae256cfc5d6cc7d380cb310c0d2c5a54316947b0"} err="failed to get container status \"c7d1f443539af7a4bb910d77ae256cfc5d6cc7d380cb310c0d2c5a54316947b0\": rpc error: code = NotFound desc = could not find container \"c7d1f443539af7a4bb910d77ae256cfc5d6cc7d380cb310c0d2c5a54316947b0\": container with ID starting with c7d1f443539af7a4bb910d77ae256cfc5d6cc7d380cb310c0d2c5a54316947b0 not found: ID does not exist"
Sep 16 02:47:07.703777 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:07.703734    2558 scope.go:117] "RemoveContainer" containerID="129862a294c189e5bae9216779b427b84a51c3c767493c7be7a1df0624b82ff0"
Sep 16 02:47:07.704051 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:47:07.704036    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"129862a294c189e5bae9216779b427b84a51c3c767493c7be7a1df0624b82ff0\": container with ID starting with 129862a294c189e5bae9216779b427b84a51c3c767493c7be7a1df0624b82ff0 not found: ID does not exist" containerID="129862a294c189e5bae9216779b427b84a51c3c767493c7be7a1df0624b82ff0"
Sep 16 02:47:07.704095 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:07.704055    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"129862a294c189e5bae9216779b427b84a51c3c767493c7be7a1df0624b82ff0"} err="failed to get container status \"129862a294c189e5bae9216779b427b84a51c3c767493c7be7a1df0624b82ff0\": rpc error: code = NotFound desc = could not find container \"129862a294c189e5bae9216779b427b84a51c3c767493c7be7a1df0624b82ff0\": container with ID starting with 129862a294c189e5bae9216779b427b84a51c3c767493c7be7a1df0624b82ff0 not found: ID does not exist"
Sep 16 02:47:07.724444 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:07.724413    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb94df1b-32d7-4c01-b581-8e09b4766544-credential-socket\") pod \"eb94df1b-32d7-4c01-b581-8e09b4766544\" (UID: \"eb94df1b-32d7-4c01-b581-8e09b4766544\") "
Sep 16 02:47:07.724594 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:07.724457    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb94df1b-32d7-4c01-b581-8e09b4766544-istiod-ca-cert\") pod \"eb94df1b-32d7-4c01-b581-8e09b4766544\" (UID: \"eb94df1b-32d7-4c01-b581-8e09b4766544\") "
Sep 16 02:47:07.724594 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:07.724495    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb94df1b-32d7-4c01-b581-8e09b4766544-istio-envoy\") pod \"eb94df1b-32d7-4c01-b581-8e09b4766544\" (UID: \"eb94df1b-32d7-4c01-b581-8e09b4766544\") "
Sep 16 02:47:07.724594 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:07.724548    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb94df1b-32d7-4c01-b581-8e09b4766544-istio-data\") pod \"eb94df1b-32d7-4c01-b581-8e09b4766544\" (UID: \"eb94df1b-32d7-4c01-b581-8e09b4766544\") "
Sep 16 02:47:07.724759 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:07.724600    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb94df1b-32d7-4c01-b581-8e09b4766544-istio-podinfo\") pod \"eb94df1b-32d7-4c01-b581-8e09b4766544\" (UID: \"eb94df1b-32d7-4c01-b581-8e09b4766544\") "
Sep 16 02:47:07.724759 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:07.724626    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xc2xg\" (UniqueName: \"kubernetes.io/projected/eb94df1b-32d7-4c01-b581-8e09b4766544-kube-api-access-xc2xg\") pod \"eb94df1b-32d7-4c01-b581-8e09b4766544\" (UID: \"eb94df1b-32d7-4c01-b581-8e09b4766544\") "
Sep 16 02:47:07.724759 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:07.724664    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb94df1b-32d7-4c01-b581-8e09b4766544-istio-token\") pod \"eb94df1b-32d7-4c01-b581-8e09b4766544\" (UID: \"eb94df1b-32d7-4c01-b581-8e09b4766544\") "
Sep 16 02:47:07.724759 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:07.724707    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb94df1b-32d7-4c01-b581-8e09b4766544-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "eb94df1b-32d7-4c01-b581-8e09b4766544" (UID: "eb94df1b-32d7-4c01-b581-8e09b4766544"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:47:07.724759 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:07.724720    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb94df1b-32d7-4c01-b581-8e09b4766544-workload-certs\") pod \"eb94df1b-32d7-4c01-b581-8e09b4766544\" (UID: \"eb94df1b-32d7-4c01-b581-8e09b4766544\") "
Sep 16 02:47:07.725022 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:07.724867    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb94df1b-32d7-4c01-b581-8e09b4766544-workload-socket\") pod \"eb94df1b-32d7-4c01-b581-8e09b4766544\" (UID: \"eb94df1b-32d7-4c01-b581-8e09b4766544\") "
Sep 16 02:47:07.725081 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:07.725038    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb94df1b-32d7-4c01-b581-8e09b4766544-istio-data" (OuterVolumeSpecName: "istio-data") pod "eb94df1b-32d7-4c01-b581-8e09b4766544" (UID: "eb94df1b-32d7-4c01-b581-8e09b4766544"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:47:07.725140 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:07.725079    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb94df1b-32d7-4c01-b581-8e09b4766544-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "eb94df1b-32d7-4c01-b581-8e09b4766544" (UID: "eb94df1b-32d7-4c01-b581-8e09b4766544"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 02:47:07.725271 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:07.725243    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb94df1b-32d7-4c01-b581-8e09b4766544-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "eb94df1b-32d7-4c01-b581-8e09b4766544" (UID: "eb94df1b-32d7-4c01-b581-8e09b4766544"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:47:07.725271 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:07.725263    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb94df1b-32d7-4c01-b581-8e09b4766544-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "eb94df1b-32d7-4c01-b581-8e09b4766544" (UID: "eb94df1b-32d7-4c01-b581-8e09b4766544"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:47:07.725957 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:07.725482    2558 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb94df1b-32d7-4c01-b581-8e09b4766544-credential-socket\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:47:07.725957 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:07.725502    2558 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb94df1b-32d7-4c01-b581-8e09b4766544-istiod-ca-cert\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:47:07.725957 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:07.725511    2558 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb94df1b-32d7-4c01-b581-8e09b4766544-istio-data\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:47:07.725957 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:07.725519    2558 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb94df1b-32d7-4c01-b581-8e09b4766544-workload-certs\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:47:07.725957 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:07.725530    2558 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb94df1b-32d7-4c01-b581-8e09b4766544-workload-socket\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:47:07.728232 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:07.727515    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb94df1b-32d7-4c01-b581-8e09b4766544-istio-token" (OuterVolumeSpecName: "istio-token") pod "eb94df1b-32d7-4c01-b581-8e09b4766544" (UID: "eb94df1b-32d7-4c01-b581-8e09b4766544"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 02:47:07.728232 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:07.727575    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/eb94df1b-32d7-4c01-b581-8e09b4766544-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "eb94df1b-32d7-4c01-b581-8e09b4766544" (UID: "eb94df1b-32d7-4c01-b581-8e09b4766544"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 02:47:07.728232 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:07.727581    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb94df1b-32d7-4c01-b581-8e09b4766544-kube-api-access-xc2xg" (OuterVolumeSpecName: "kube-api-access-xc2xg") pod "eb94df1b-32d7-4c01-b581-8e09b4766544" (UID: "eb94df1b-32d7-4c01-b581-8e09b4766544"). InnerVolumeSpecName "kube-api-access-xc2xg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 02:47:07.728232 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:07.727583    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb94df1b-32d7-4c01-b581-8e09b4766544-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "eb94df1b-32d7-4c01-b581-8e09b4766544" (UID: "eb94df1b-32d7-4c01-b581-8e09b4766544"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:47:07.814566 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:07.814535    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-5b136e-t8l55"
Sep 16 02:47:07.826804 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:07.826781    2558 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb94df1b-32d7-4c01-b581-8e09b4766544-istio-envoy\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:47:07.826804 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:07.826804    2558 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb94df1b-32d7-4c01-b581-8e09b4766544-istio-podinfo\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:47:07.826948 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:07.826813    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xc2xg\" (UniqueName: \"kubernetes.io/projected/eb94df1b-32d7-4c01-b581-8e09b4766544-kube-api-access-xc2xg\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:47:07.826948 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:07.826823    2558 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb94df1b-32d7-4c01-b581-8e09b4766544-istio-token\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:47:07.927822 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:07.927780    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/15c29ab2-70df-4eeb-92f7-42b9648082c2-maas-api-service-ca\") pod \"15c29ab2-70df-4eeb-92f7-42b9648082c2\" (UID: \"15c29ab2-70df-4eeb-92f7-42b9648082c2\") "
Sep 16 02:47:07.928164 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:07.928138    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15c29ab2-70df-4eeb-92f7-42b9648082c2-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "15c29ab2-70df-4eeb-92f7-42b9648082c2" (UID: "15c29ab2-70df-4eeb-92f7-42b9648082c2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 02:47:08.004029 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:08.003996    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-e21dae-openshift-default-78c8949998-cn6ks"]
Sep 16 02:47:08.009169 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:08.009133    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-e21dae-openshift-default-78c8949998-cn6ks"]
Sep 16 02:47:08.029248 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:08.029218    2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/15c29ab2-70df-4eeb-92f7-42b9648082c2-maas-api-service-ca\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:47:08.690627 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:08.690597    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-5b136e-t8l55"
Sep 16 02:47:08.691077 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:08.690593    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-5b136e-t8l55" event={"ID":"15c29ab2-70df-4eeb-92f7-42b9648082c2","Type":"ContainerDied","Data":"8fb5d4455c4f9d5d4d410ee220c5a790f6be377b75d657bc2016b3810cbb27d5"}
Sep 16 02:47:08.691077 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:08.690763    2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8fb5d4455c4f9d5d4d410ee220c5a790f6be377b75d657bc2016b3810cbb27d5"
Sep 16 02:47:09.550171 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:09.550126    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eb94df1b-32d7-4c01-b581-8e09b4766544" path="/var/lib/kubelet/pods/eb94df1b-32d7-4c01-b581-8e09b4766544/volumes"
Sep 16 02:47:13.362679 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:13.362641    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-bd0fb8-6c4f6f65f-sd7gz"]
Sep 16 02:47:13.363175 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:13.362970    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-bd0fb8-6c4f6f65f-sd7gz" podUID="4d8288a9-39c6-41bf-aa3e-392304de982d" containerName="payload-processing" containerID="cri-o://62a258f67803eb84215061f1abfcc7bcdf981d22ac59bc0fd4fba0aa2de69d0c" gracePeriod=30
Sep 16 02:47:13.369002 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:13.368969    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-bd0fb8-55f956944b-tl98x"]
Sep 16 02:47:13.369233 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:13.369200    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-bd0fb8-55f956944b-tl98x" podUID="c298d7e3-7484-4e0b-814e-6b4151d40a16" containerName="payload-pre-processing" containerID="cri-o://fa2eb16e42820247631269ab1a30bea94feba395f440ad0745fc6f2e61a1e222" gracePeriod=30
Sep 16 02:47:13.675001 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:13.674954    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-bd0fb8-6c4f6f65f-sd7gz"
Sep 16 02:47:13.688267 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:13.688227    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-grz5l\" (UniqueName: \"kubernetes.io/projected/4d8288a9-39c6-41bf-aa3e-392304de982d-kube-api-access-grz5l\") pod \"4d8288a9-39c6-41bf-aa3e-392304de982d\" (UID: \"4d8288a9-39c6-41bf-aa3e-392304de982d\") "
Sep 16 02:47:13.688471 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:13.688341    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d8288a9-39c6-41bf-aa3e-392304de982d-plugins-config-volume\") pod \"4d8288a9-39c6-41bf-aa3e-392304de982d\" (UID: \"4d8288a9-39c6-41bf-aa3e-392304de982d\") "
Sep 16 02:47:13.688752 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:13.688721    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d8288a9-39c6-41bf-aa3e-392304de982d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4d8288a9-39c6-41bf-aa3e-392304de982d" (UID: "4d8288a9-39c6-41bf-aa3e-392304de982d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 02:47:13.691515 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:13.691478    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d8288a9-39c6-41bf-aa3e-392304de982d-kube-api-access-grz5l" (OuterVolumeSpecName: "kube-api-access-grz5l") pod "4d8288a9-39c6-41bf-aa3e-392304de982d" (UID: "4d8288a9-39c6-41bf-aa3e-392304de982d"). InnerVolumeSpecName "kube-api-access-grz5l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 02:47:13.717640 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:13.717609    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-bd0fb8-55f956944b-tl98x"
Sep 16 02:47:13.720361 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:13.720325    2558 generic.go:356] "Generic (PLEG): container finished" podID="4d8288a9-39c6-41bf-aa3e-392304de982d" containerID="62a258f67803eb84215061f1abfcc7bcdf981d22ac59bc0fd4fba0aa2de69d0c" exitCode=0
Sep 16 02:47:13.720520 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:13.720379    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-bd0fb8-6c4f6f65f-sd7gz"
Sep 16 02:47:13.720520 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:13.720437    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-bd0fb8-6c4f6f65f-sd7gz" event={"ID":"4d8288a9-39c6-41bf-aa3e-392304de982d","Type":"ContainerDied","Data":"62a258f67803eb84215061f1abfcc7bcdf981d22ac59bc0fd4fba0aa2de69d0c"}
Sep 16 02:47:13.720520 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:13.720492    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-bd0fb8-6c4f6f65f-sd7gz" event={"ID":"4d8288a9-39c6-41bf-aa3e-392304de982d","Type":"ContainerDied","Data":"01fc67e322f9f4aed209b4dee2f8cbd7742a8b742b9cbf9daee5a6c091c6a187"}
Sep 16 02:47:13.720721 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:13.720522    2558 scope.go:117] "RemoveContainer" containerID="62a258f67803eb84215061f1abfcc7bcdf981d22ac59bc0fd4fba0aa2de69d0c"
Sep 16 02:47:13.722061 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:13.721960    2558 generic.go:356] "Generic (PLEG): container finished" podID="c298d7e3-7484-4e0b-814e-6b4151d40a16" containerID="fa2eb16e42820247631269ab1a30bea94feba395f440ad0745fc6f2e61a1e222" exitCode=0
Sep 16 02:47:13.722061 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:13.722013    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-bd0fb8-55f956944b-tl98x" event={"ID":"c298d7e3-7484-4e0b-814e-6b4151d40a16","Type":"ContainerDied","Data":"fa2eb16e42820247631269ab1a30bea94feba395f440ad0745fc6f2e61a1e222"}
Sep 16 02:47:13.722061 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:13.722052    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-bd0fb8-55f956944b-tl98x" event={"ID":"c298d7e3-7484-4e0b-814e-6b4151d40a16","Type":"ContainerDied","Data":"293dc7fe6063488d4276ca6d03aae112a8e8720537c0a42f2020c03017d4ac24"}
Sep 16 02:47:13.722248 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:13.722112    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-bd0fb8-55f956944b-tl98x"
Sep 16 02:47:13.733094 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:13.732758    2558 scope.go:117] "RemoveContainer" containerID="62a258f67803eb84215061f1abfcc7bcdf981d22ac59bc0fd4fba0aa2de69d0c"
Sep 16 02:47:13.733601 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:47:13.733567    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62a258f67803eb84215061f1abfcc7bcdf981d22ac59bc0fd4fba0aa2de69d0c\": container with ID starting with 62a258f67803eb84215061f1abfcc7bcdf981d22ac59bc0fd4fba0aa2de69d0c not found: ID does not exist" containerID="62a258f67803eb84215061f1abfcc7bcdf981d22ac59bc0fd4fba0aa2de69d0c"
Sep 16 02:47:13.733735 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:13.733613    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62a258f67803eb84215061f1abfcc7bcdf981d22ac59bc0fd4fba0aa2de69d0c"} err="failed to get container status \"62a258f67803eb84215061f1abfcc7bcdf981d22ac59bc0fd4fba0aa2de69d0c\": rpc error: code = NotFound desc = could not find container \"62a258f67803eb84215061f1abfcc7bcdf981d22ac59bc0fd4fba0aa2de69d0c\": container with ID starting with 62a258f67803eb84215061f1abfcc7bcdf981d22ac59bc0fd4fba0aa2de69d0c not found: ID does not exist"
Sep 16 02:47:13.733735 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:13.733640    2558 scope.go:117] "RemoveContainer" containerID="fa2eb16e42820247631269ab1a30bea94feba395f440ad0745fc6f2e61a1e222"
Sep 16 02:47:13.743907 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:13.743885    2558 scope.go:117] "RemoveContainer" containerID="fa2eb16e42820247631269ab1a30bea94feba395f440ad0745fc6f2e61a1e222"
Sep 16 02:47:13.744255 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:47:13.744234    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa2eb16e42820247631269ab1a30bea94feba395f440ad0745fc6f2e61a1e222\": container with ID starting with fa2eb16e42820247631269ab1a30bea94feba395f440ad0745fc6f2e61a1e222 not found: ID does not exist" containerID="fa2eb16e42820247631269ab1a30bea94feba395f440ad0745fc6f2e61a1e222"
Sep 16 02:47:13.744320 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:13.744263    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa2eb16e42820247631269ab1a30bea94feba395f440ad0745fc6f2e61a1e222"} err="failed to get container status \"fa2eb16e42820247631269ab1a30bea94feba395f440ad0745fc6f2e61a1e222\": rpc error: code = NotFound desc = could not find container \"fa2eb16e42820247631269ab1a30bea94feba395f440ad0745fc6f2e61a1e222\": container with ID starting with fa2eb16e42820247631269ab1a30bea94feba395f440ad0745fc6f2e61a1e222 not found: ID does not exist"
Sep 16 02:47:13.748890 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:13.748860    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-bd0fb8-6c4f6f65f-sd7gz"]
Sep 16 02:47:13.751858 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:13.751831    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-bd0fb8-6c4f6f65f-sd7gz"]
Sep 16 02:47:13.789198 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:13.789163    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m6d2b\" (UniqueName: \"kubernetes.io/projected/c298d7e3-7484-4e0b-814e-6b4151d40a16-kube-api-access-m6d2b\") pod \"c298d7e3-7484-4e0b-814e-6b4151d40a16\" (UID: \"c298d7e3-7484-4e0b-814e-6b4151d40a16\") "
Sep 16 02:47:13.789399 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:13.789212    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c298d7e3-7484-4e0b-814e-6b4151d40a16-plugins-config-volume\") pod \"c298d7e3-7484-4e0b-814e-6b4151d40a16\" (UID: \"c298d7e3-7484-4e0b-814e-6b4151d40a16\") "
Sep 16 02:47:13.789651 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:13.789629    2558 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d8288a9-39c6-41bf-aa3e-392304de982d-plugins-config-volume\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:47:13.789766 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:13.789640    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c298d7e3-7484-4e0b-814e-6b4151d40a16-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c298d7e3-7484-4e0b-814e-6b4151d40a16" (UID: "c298d7e3-7484-4e0b-814e-6b4151d40a16"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 02:47:13.789766 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:13.789656    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-grz5l\" (UniqueName: \"kubernetes.io/projected/4d8288a9-39c6-41bf-aa3e-392304de982d-kube-api-access-grz5l\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:47:13.791475 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:13.791453    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c298d7e3-7484-4e0b-814e-6b4151d40a16-kube-api-access-m6d2b" (OuterVolumeSpecName: "kube-api-access-m6d2b") pod "c298d7e3-7484-4e0b-814e-6b4151d40a16" (UID: "c298d7e3-7484-4e0b-814e-6b4151d40a16"). InnerVolumeSpecName "kube-api-access-m6d2b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 02:47:13.891216 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:13.891157    2558 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c298d7e3-7484-4e0b-814e-6b4151d40a16-plugins-config-volume\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:47:13.891216 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:13.891208    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m6d2b\" (UniqueName: \"kubernetes.io/projected/c298d7e3-7484-4e0b-814e-6b4151d40a16-kube-api-access-m6d2b\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:47:14.044208 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:14.044170    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-bd0fb8-55f956944b-tl98x"]
Sep 16 02:47:14.048193 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:14.048161    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-bd0fb8-55f956944b-tl98x"]
Sep 16 02:47:15.550517 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:15.550483    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4d8288a9-39c6-41bf-aa3e-392304de982d" path="/var/lib/kubelet/pods/4d8288a9-39c6-41bf-aa3e-392304de982d/volumes"
Sep 16 02:47:15.550915 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:15.550884    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c298d7e3-7484-4e0b-814e-6b4151d40a16" path="/var/lib/kubelet/pods/c298d7e3-7484-4e0b-814e-6b4151d40a16/volumes"
Sep 16 02:47:16.544145 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:16.544102    2558 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-processing-5d7dbbd56b-pltkr" secret="" err="secret \"payload-processing-dockercfg-zzx6j\" not found"
Sep 16 02:47:18.042659 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:18.042617    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-5b136e-t8l55"]
Sep 16 02:47:18.045266 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:18.045238    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-5b136e-t8l55"]
Sep 16 02:47:19.550350 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:19.550315    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="15c29ab2-70df-4eeb-92f7-42b9648082c2" path="/var/lib/kubelet/pods/15c29ab2-70df-4eeb-92f7-42b9648082c2/volumes"
Sep 16 02:47:20.835748 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:20.835693    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-bd0fb8-openshift-default-84f4588456-hdfcv"]
Sep 16 02:47:20.836208 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:20.835983    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-bd0fb8-openshift-default-84f4588456-hdfcv" podUID="f4a618df-1dcc-42ed-ac72-e519ee2c5515" containerName="istio-proxy" containerID="cri-o://d547325cdc13aeac9bfe39a340d29152b7f4a03b1e7289fea7e446d97aebbb71" gracePeriod=30
Sep 16 02:47:21.190977 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:21.190935    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-8e10f1-pw9qt"]
Sep 16 02:47:21.191344 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:21.191332    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="15c29ab2-70df-4eeb-92f7-42b9648082c2" containerName="revoke-keys"
Sep 16 02:47:21.191390 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:21.191346    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="15c29ab2-70df-4eeb-92f7-42b9648082c2" containerName="revoke-keys"
Sep 16 02:47:21.191390 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:21.191356    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c298d7e3-7484-4e0b-814e-6b4151d40a16" containerName="payload-pre-processing"
Sep 16 02:47:21.191390 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:21.191362    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="c298d7e3-7484-4e0b-814e-6b4151d40a16" containerName="payload-pre-processing"
Sep 16 02:47:21.191390 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:21.191368    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb94df1b-32d7-4c01-b581-8e09b4766544" containerName="istio-proxy"
Sep 16 02:47:21.191390 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:21.191374    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb94df1b-32d7-4c01-b581-8e09b4766544" containerName="istio-proxy"
Sep 16 02:47:21.191390 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:21.191382    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d8288a9-39c6-41bf-aa3e-392304de982d" containerName="payload-processing"
Sep 16 02:47:21.191390 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:21.191387    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d8288a9-39c6-41bf-aa3e-392304de982d" containerName="payload-processing"
Sep 16 02:47:21.191609 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:21.191394    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb94df1b-32d7-4c01-b581-8e09b4766544" containerName="istio-proxy"
Sep 16 02:47:21.191609 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:21.191399    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb94df1b-32d7-4c01-b581-8e09b4766544" containerName="istio-proxy"
Sep 16 02:47:21.191609 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:21.191463    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="eb94df1b-32d7-4c01-b581-8e09b4766544" containerName="istio-proxy"
Sep 16 02:47:21.191609 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:21.191470    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="15c29ab2-70df-4eeb-92f7-42b9648082c2" containerName="revoke-keys"
Sep 16 02:47:21.191609 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:21.191479    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="c298d7e3-7484-4e0b-814e-6b4151d40a16" containerName="payload-pre-processing"
Sep 16 02:47:21.191609 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:21.191488    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="4d8288a9-39c6-41bf-aa3e-392304de982d" containerName="payload-processing"
Sep 16 02:47:21.196381 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:21.196359    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-8e10f1-pw9qt"
Sep 16 02:47:21.198745 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:21.198721    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jpd4f\""
Sep 16 02:47:21.198745 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:21.198735    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 16 02:47:21.201300 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:21.201272    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-8e10f1-pw9qt"]
Sep 16 02:47:21.272979 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:21.272941    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8defe3a-a5ad-46f7-bd65-d849e529d9b4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-8e10f1-pw9qt\" (UID: \"c8defe3a-a5ad-46f7-bd65-d849e529d9b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-8e10f1-pw9qt"
Sep 16 02:47:21.373727 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:21.373657    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8defe3a-a5ad-46f7-bd65-d849e529d9b4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-8e10f1-pw9qt\" (UID: \"c8defe3a-a5ad-46f7-bd65-d849e529d9b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-8e10f1-pw9qt"
Sep 16 02:47:21.374378 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:21.374357    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8defe3a-a5ad-46f7-bd65-d849e529d9b4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-8e10f1-pw9qt\" (UID: \"c8defe3a-a5ad-46f7-bd65-d849e529d9b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-8e10f1-pw9qt"
Sep 16 02:47:21.509505 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:21.509423    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-8e10f1-pw9qt"
Sep 16 02:47:21.784377 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:21.784352    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-8e10f1-pw9qt"]
Sep 16 02:47:21.786436 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:47:21.786410    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8defe3a_a5ad_46f7_bd65_d849e529d9b4.slice/crio-2c307834b36f80c472ccef19b665e9aaf6a7c0994d735f366cb57179ac5befe2 WatchSource:0}: Error finding container 2c307834b36f80c472ccef19b665e9aaf6a7c0994d735f366cb57179ac5befe2: Status 404 returned error can't find the container with id 2c307834b36f80c472ccef19b665e9aaf6a7c0994d735f366cb57179ac5befe2
Sep 16 02:47:22.697172 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:22.697092    2558 patch_prober.go:28] interesting pod/e2e-shared-a-w1-bd0fb8-openshift-default-84f4588456-hdfcv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 16 02:47:22.697172 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:22.697147    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w1-bd0fb8-openshift-default-84f4588456-hdfcv" podUID="f4a618df-1dcc-42ed-ac72-e519ee2c5515" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 02:47:22.775604 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:22.775572    2558 generic.go:356] "Generic (PLEG): container finished" podID="c8defe3a-a5ad-46f7-bd65-d849e529d9b4" containerID="e5dab622da112e4fb7a1bd21f1a030532a4feb2b970e6d366ac2b83d63369e48" exitCode=0
Sep 16 02:47:22.775779 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:22.775656    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-8e10f1-pw9qt" event={"ID":"c8defe3a-a5ad-46f7-bd65-d849e529d9b4","Type":"ContainerDied","Data":"e5dab622da112e4fb7a1bd21f1a030532a4feb2b970e6d366ac2b83d63369e48"}
Sep 16 02:47:22.775779 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:22.775724    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-8e10f1-pw9qt" event={"ID":"c8defe3a-a5ad-46f7-bd65-d849e529d9b4","Type":"ContainerStarted","Data":"2c307834b36f80c472ccef19b665e9aaf6a7c0994d735f366cb57179ac5befe2"}
Sep 16 02:47:23.916525 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:23.916501    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-8e10f1-pw9qt"
Sep 16 02:47:24.003327 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:24.003298    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8defe3a-a5ad-46f7-bd65-d849e529d9b4-maas-api-service-ca\") pod \"c8defe3a-a5ad-46f7-bd65-d849e529d9b4\" (UID: \"c8defe3a-a5ad-46f7-bd65-d849e529d9b4\") "
Sep 16 02:47:24.003658 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:24.003630    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8defe3a-a5ad-46f7-bd65-d849e529d9b4-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c8defe3a-a5ad-46f7-bd65-d849e529d9b4" (UID: "c8defe3a-a5ad-46f7-bd65-d849e529d9b4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 02:47:24.104302 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:24.104208    2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8defe3a-a5ad-46f7-bd65-d849e529d9b4-maas-api-service-ca\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:47:24.763817 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:24.763780    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-5b136e-openshift-default-696f88d9c6-zwnvx"]
Sep 16 02:47:24.764081 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:24.764039    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-cleanup-5b136e-openshift-default-696f88d9c6-zwnvx" podUID="34139e40-51de-48b1-b460-a6b6a20e82b7" containerName="istio-proxy" containerID="cri-o://6037f35b4cb0dc53c51af6b55977cffdcc46a6a954d3f661948547baeaf1b346" gracePeriod=30
Sep 16 02:47:24.788522 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:24.788496    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-8e10f1-pw9qt"
Sep 16 02:47:24.788680 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:24.788534    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-8e10f1-pw9qt" event={"ID":"c8defe3a-a5ad-46f7-bd65-d849e529d9b4","Type":"ContainerDied","Data":"2c307834b36f80c472ccef19b665e9aaf6a7c0994d735f366cb57179ac5befe2"}
Sep 16 02:47:24.788680 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:24.788587    2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2c307834b36f80c472ccef19b665e9aaf6a7c0994d735f366cb57179ac5befe2"
Sep 16 02:47:26.081837 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:26.081808    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-bd0fb8-openshift-default-84f4588456-hdfcv"
Sep 16 02:47:26.225322 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:26.225286    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4a618df-1dcc-42ed-ac72-e519ee2c5515-istio-data\") pod \"f4a618df-1dcc-42ed-ac72-e519ee2c5515\" (UID: \"f4a618df-1dcc-42ed-ac72-e519ee2c5515\") "
Sep 16 02:47:26.225501 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:26.225340    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4a618df-1dcc-42ed-ac72-e519ee2c5515-credential-socket\") pod \"f4a618df-1dcc-42ed-ac72-e519ee2c5515\" (UID: \"f4a618df-1dcc-42ed-ac72-e519ee2c5515\") "
Sep 16 02:47:26.225501 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:26.225384    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4a618df-1dcc-42ed-ac72-e519ee2c5515-istiod-ca-cert\") pod \"f4a618df-1dcc-42ed-ac72-e519ee2c5515\" (UID: \"f4a618df-1dcc-42ed-ac72-e519ee2c5515\") "
Sep 16 02:47:26.225501 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:26.225410    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4a618df-1dcc-42ed-ac72-e519ee2c5515-istio-envoy\") pod \"f4a618df-1dcc-42ed-ac72-e519ee2c5515\" (UID: \"f4a618df-1dcc-42ed-ac72-e519ee2c5515\") "
Sep 16 02:47:26.225501 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:26.225477    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4a618df-1dcc-42ed-ac72-e519ee2c5515-workload-socket\") pod \"f4a618df-1dcc-42ed-ac72-e519ee2c5515\" (UID: \"f4a618df-1dcc-42ed-ac72-e519ee2c5515\") "
Sep 16 02:47:26.225793 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:26.225515    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4a618df-1dcc-42ed-ac72-e519ee2c5515-workload-certs\") pod \"f4a618df-1dcc-42ed-ac72-e519ee2c5515\" (UID: \"f4a618df-1dcc-42ed-ac72-e519ee2c5515\") "
Sep 16 02:47:26.225793 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:26.225555    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nqblf\" (UniqueName: \"kubernetes.io/projected/f4a618df-1dcc-42ed-ac72-e519ee2c5515-kube-api-access-nqblf\") pod \"f4a618df-1dcc-42ed-ac72-e519ee2c5515\" (UID: \"f4a618df-1dcc-42ed-ac72-e519ee2c5515\") "
Sep 16 02:47:26.225793 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:26.225597    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4a618df-1dcc-42ed-ac72-e519ee2c5515-istio-token\") pod \"f4a618df-1dcc-42ed-ac72-e519ee2c5515\" (UID: \"f4a618df-1dcc-42ed-ac72-e519ee2c5515\") "
Sep 16 02:47:26.225793 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:26.225631    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4a618df-1dcc-42ed-ac72-e519ee2c5515-istio-podinfo\") pod \"f4a618df-1dcc-42ed-ac72-e519ee2c5515\" (UID: \"f4a618df-1dcc-42ed-ac72-e519ee2c5515\") "
Sep 16 02:47:26.225793 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:26.225643    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4a618df-1dcc-42ed-ac72-e519ee2c5515-istio-data" (OuterVolumeSpecName: "istio-data") pod "f4a618df-1dcc-42ed-ac72-e519ee2c5515" (UID: "f4a618df-1dcc-42ed-ac72-e519ee2c5515"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:47:26.225793 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:26.225683    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4a618df-1dcc-42ed-ac72-e519ee2c5515-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "f4a618df-1dcc-42ed-ac72-e519ee2c5515" (UID: "f4a618df-1dcc-42ed-ac72-e519ee2c5515"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:47:26.226075 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:26.225912    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4a618df-1dcc-42ed-ac72-e519ee2c5515-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "f4a618df-1dcc-42ed-ac72-e519ee2c5515" (UID: "f4a618df-1dcc-42ed-ac72-e519ee2c5515"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:47:26.226075 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:26.225970    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4a618df-1dcc-42ed-ac72-e519ee2c5515-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "f4a618df-1dcc-42ed-ac72-e519ee2c5515" (UID: "f4a618df-1dcc-42ed-ac72-e519ee2c5515"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 02:47:26.226075 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:26.225983    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4a618df-1dcc-42ed-ac72-e519ee2c5515-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "f4a618df-1dcc-42ed-ac72-e519ee2c5515" (UID: "f4a618df-1dcc-42ed-ac72-e519ee2c5515"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:47:26.226075 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:26.226052    2558 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4a618df-1dcc-42ed-ac72-e519ee2c5515-credential-socket\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:47:26.226262 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:26.226074    2558 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4a618df-1dcc-42ed-ac72-e519ee2c5515-workload-socket\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:47:26.226262 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:26.226089    2558 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4a618df-1dcc-42ed-ac72-e519ee2c5515-istio-data\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:47:26.227869 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:26.227835    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4a618df-1dcc-42ed-ac72-e519ee2c5515-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "f4a618df-1dcc-42ed-ac72-e519ee2c5515" (UID: "f4a618df-1dcc-42ed-ac72-e519ee2c5515"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:47:26.227869 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:26.227852    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4a618df-1dcc-42ed-ac72-e519ee2c5515-kube-api-access-nqblf" (OuterVolumeSpecName: "kube-api-access-nqblf") pod "f4a618df-1dcc-42ed-ac72-e519ee2c5515" (UID: "f4a618df-1dcc-42ed-ac72-e519ee2c5515"). InnerVolumeSpecName "kube-api-access-nqblf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 02:47:26.228015 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:26.227859    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f4a618df-1dcc-42ed-ac72-e519ee2c5515-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "f4a618df-1dcc-42ed-ac72-e519ee2c5515" (UID: "f4a618df-1dcc-42ed-ac72-e519ee2c5515"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 02:47:26.228015 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:26.227908    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4a618df-1dcc-42ed-ac72-e519ee2c5515-istio-token" (OuterVolumeSpecName: "istio-token") pod "f4a618df-1dcc-42ed-ac72-e519ee2c5515" (UID: "f4a618df-1dcc-42ed-ac72-e519ee2c5515"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 02:47:26.327088 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:26.327055    2558 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4a618df-1dcc-42ed-ac72-e519ee2c5515-istiod-ca-cert\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:47:26.327088 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:26.327083    2558 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4a618df-1dcc-42ed-ac72-e519ee2c5515-istio-envoy\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:47:26.327088 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:26.327095    2558 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4a618df-1dcc-42ed-ac72-e519ee2c5515-workload-certs\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:47:26.327317 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:26.327108    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nqblf\" (UniqueName: \"kubernetes.io/projected/f4a618df-1dcc-42ed-ac72-e519ee2c5515-kube-api-access-nqblf\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:47:26.327317 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:26.327117    2558 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4a618df-1dcc-42ed-ac72-e519ee2c5515-istio-token\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:47:26.327317 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:26.327125    2558 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4a618df-1dcc-42ed-ac72-e519ee2c5515-istio-podinfo\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:47:26.801083 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:26.801040    2558 generic.go:356] "Generic (PLEG): container finished" podID="f4a618df-1dcc-42ed-ac72-e519ee2c5515" containerID="d547325cdc13aeac9bfe39a340d29152b7f4a03b1e7289fea7e446d97aebbb71" exitCode=0
Sep 16 02:47:26.801260 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:26.801090    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-bd0fb8-openshift-default-84f4588456-hdfcv" event={"ID":"f4a618df-1dcc-42ed-ac72-e519ee2c5515","Type":"ContainerDied","Data":"d547325cdc13aeac9bfe39a340d29152b7f4a03b1e7289fea7e446d97aebbb71"}
Sep 16 02:47:26.801260 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:26.801112    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-bd0fb8-openshift-default-84f4588456-hdfcv"
Sep 16 02:47:26.801260 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:26.801144    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-bd0fb8-openshift-default-84f4588456-hdfcv" event={"ID":"f4a618df-1dcc-42ed-ac72-e519ee2c5515","Type":"ContainerDied","Data":"53410d63c6a6fc782e4f7eb479fe09ed2a4875e76879b0c939d1e8f1aa37e7b7"}
Sep 16 02:47:26.801260 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:26.801173    2558 scope.go:117] "RemoveContainer" containerID="d547325cdc13aeac9bfe39a340d29152b7f4a03b1e7289fea7e446d97aebbb71"
Sep 16 02:47:26.810804 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:26.810785    2558 scope.go:117] "RemoveContainer" containerID="f0fb1f59a7d7d0d73262d9156bcf6336d94a22a2b413eddef7118743709163cd"
Sep 16 02:47:26.819235 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:26.819211    2558 scope.go:117] "RemoveContainer" containerID="d547325cdc13aeac9bfe39a340d29152b7f4a03b1e7289fea7e446d97aebbb71"
Sep 16 02:47:26.819961 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:47:26.819928    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d547325cdc13aeac9bfe39a340d29152b7f4a03b1e7289fea7e446d97aebbb71\": container with ID starting with d547325cdc13aeac9bfe39a340d29152b7f4a03b1e7289fea7e446d97aebbb71 not found: ID does not exist" containerID="d547325cdc13aeac9bfe39a340d29152b7f4a03b1e7289fea7e446d97aebbb71"
Sep 16 02:47:26.820056 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:26.819973    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d547325cdc13aeac9bfe39a340d29152b7f4a03b1e7289fea7e446d97aebbb71"} err="failed to get container status \"d547325cdc13aeac9bfe39a340d29152b7f4a03b1e7289fea7e446d97aebbb71\": rpc error: code = NotFound desc = could not find container \"d547325cdc13aeac9bfe39a340d29152b7f4a03b1e7289fea7e446d97aebbb71\": container with ID starting with d547325cdc13aeac9bfe39a340d29152b7f4a03b1e7289fea7e446d97aebbb71 not found: ID does not exist"
Sep 16 02:47:26.820056 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:26.819996    2558 scope.go:117] "RemoveContainer" containerID="f0fb1f59a7d7d0d73262d9156bcf6336d94a22a2b413eddef7118743709163cd"
Sep 16 02:47:26.820271 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:47:26.820252    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0fb1f59a7d7d0d73262d9156bcf6336d94a22a2b413eddef7118743709163cd\": container with ID starting with f0fb1f59a7d7d0d73262d9156bcf6336d94a22a2b413eddef7118743709163cd not found: ID does not exist" containerID="f0fb1f59a7d7d0d73262d9156bcf6336d94a22a2b413eddef7118743709163cd"
Sep 16 02:47:26.820319 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:26.820279    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0fb1f59a7d7d0d73262d9156bcf6336d94a22a2b413eddef7118743709163cd"} err="failed to get container status \"f0fb1f59a7d7d0d73262d9156bcf6336d94a22a2b413eddef7118743709163cd\": rpc error: code = NotFound desc = could not find container \"f0fb1f59a7d7d0d73262d9156bcf6336d94a22a2b413eddef7118743709163cd\": container with ID starting with f0fb1f59a7d7d0d73262d9156bcf6336d94a22a2b413eddef7118743709163cd not found: ID does not exist"
Sep 16 02:47:26.821895 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:26.821878    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-bd0fb8-openshift-default-84f4588456-hdfcv"]
Sep 16 02:47:26.825445 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:26.825423    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-bd0fb8-openshift-default-84f4588456-hdfcv"]
Sep 16 02:47:27.550019 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:27.549987    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f4a618df-1dcc-42ed-ac72-e519ee2c5515" path="/var/lib/kubelet/pods/f4a618df-1dcc-42ed-ac72-e519ee2c5515/volumes"
Sep 16 02:47:28.495760 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:28.495723    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fd614e-w5pjr"]
Sep 16 02:47:28.496148 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:28.496135    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f4a618df-1dcc-42ed-ac72-e519ee2c5515" containerName="istio-proxy"
Sep 16 02:47:28.496194 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:28.496150    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="f4a618df-1dcc-42ed-ac72-e519ee2c5515" containerName="istio-proxy"
Sep 16 02:47:28.496194 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:28.496160    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c8defe3a-a5ad-46f7-bd65-d849e529d9b4" containerName="revoke-keys"
Sep 16 02:47:28.496194 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:28.496166    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="c8defe3a-a5ad-46f7-bd65-d849e529d9b4" containerName="revoke-keys"
Sep 16 02:47:28.496284 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:28.496224    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="c8defe3a-a5ad-46f7-bd65-d849e529d9b4" containerName="revoke-keys"
Sep 16 02:47:28.496284 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:28.496233    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="eb94df1b-32d7-4c01-b581-8e09b4766544" containerName="istio-proxy"
Sep 16 02:47:28.496284 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:28.496241    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="f4a618df-1dcc-42ed-ac72-e519ee2c5515" containerName="istio-proxy"
Sep 16 02:47:28.500472 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:28.500455    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fd614e-w5pjr"
Sep 16 02:47:28.502778 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:28.502755    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jpd4f\""
Sep 16 02:47:28.502778 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:28.502771    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 16 02:47:28.507773 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:28.507691    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fd614e-w5pjr"]
Sep 16 02:47:28.651266 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:28.651220    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d9e4180-2d9b-44e3-8980-080f593b0af1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-fd614e-w5pjr\" (UID: \"9d9e4180-2d9b-44e3-8980-080f593b0af1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fd614e-w5pjr"
Sep 16 02:47:28.752340 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:28.752247    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d9e4180-2d9b-44e3-8980-080f593b0af1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-fd614e-w5pjr\" (UID: \"9d9e4180-2d9b-44e3-8980-080f593b0af1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fd614e-w5pjr"
Sep 16 02:47:28.752926 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:28.752904    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d9e4180-2d9b-44e3-8980-080f593b0af1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-fd614e-w5pjr\" (UID: \"9d9e4180-2d9b-44e3-8980-080f593b0af1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fd614e-w5pjr"
Sep 16 02:47:28.812340 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:28.812308    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fd614e-w5pjr"
Sep 16 02:47:29.081479 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:29.081453    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fd614e-w5pjr"]
Sep 16 02:47:29.084168 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:47:29.084139    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d9e4180_2d9b_44e3_8980_080f593b0af1.slice/crio-5305005f2703eb1c9898323b6c0b82dc0cd3f9de0e94aa56a4fe304b6b549ff1 WatchSource:0}: Error finding container 5305005f2703eb1c9898323b6c0b82dc0cd3f9de0e94aa56a4fe304b6b549ff1: Status 404 returned error can't find the container with id 5305005f2703eb1c9898323b6c0b82dc0cd3f9de0e94aa56a4fe304b6b549ff1
Sep 16 02:47:29.821979 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:29.821937    2558 generic.go:356] "Generic (PLEG): container finished" podID="9d9e4180-2d9b-44e3-8980-080f593b0af1" containerID="072938f8ca5c1485fe40b0436b010970ea1eb2ab407387b2a9d38579f2ef4940" exitCode=0
Sep 16 02:47:29.822360 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:29.821996    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fd614e-w5pjr" event={"ID":"9d9e4180-2d9b-44e3-8980-080f593b0af1","Type":"ContainerDied","Data":"072938f8ca5c1485fe40b0436b010970ea1eb2ab407387b2a9d38579f2ef4940"}
Sep 16 02:47:29.822360 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:29.822054    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fd614e-w5pjr" event={"ID":"9d9e4180-2d9b-44e3-8980-080f593b0af1","Type":"ContainerStarted","Data":"5305005f2703eb1c9898323b6c0b82dc0cd3f9de0e94aa56a4fe304b6b549ff1"}
Sep 16 02:47:30.007789 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:30.007764    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-5b136e-openshift-default-696f88d9c6-zwnvx"
Sep 16 02:47:30.166669 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:30.166626    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34139e40-51de-48b1-b460-a6b6a20e82b7-credential-socket\") pod \"34139e40-51de-48b1-b460-a6b6a20e82b7\" (UID: \"34139e40-51de-48b1-b460-a6b6a20e82b7\") "
Sep 16 02:47:30.166888 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:30.166677    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tlzgz\" (UniqueName: \"kubernetes.io/projected/34139e40-51de-48b1-b460-a6b6a20e82b7-kube-api-access-tlzgz\") pod \"34139e40-51de-48b1-b460-a6b6a20e82b7\" (UID: \"34139e40-51de-48b1-b460-a6b6a20e82b7\") "
Sep 16 02:47:30.166888 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:30.166728    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34139e40-51de-48b1-b460-a6b6a20e82b7-istio-podinfo\") pod \"34139e40-51de-48b1-b460-a6b6a20e82b7\" (UID: \"34139e40-51de-48b1-b460-a6b6a20e82b7\") "
Sep 16 02:47:30.166888 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:30.166774    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34139e40-51de-48b1-b460-a6b6a20e82b7-istio-envoy\") pod \"34139e40-51de-48b1-b460-a6b6a20e82b7\" (UID: \"34139e40-51de-48b1-b460-a6b6a20e82b7\") "
Sep 16 02:47:30.166888 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:30.166801    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34139e40-51de-48b1-b460-a6b6a20e82b7-istio-token\") pod \"34139e40-51de-48b1-b460-a6b6a20e82b7\" (UID: \"34139e40-51de-48b1-b460-a6b6a20e82b7\") "
Sep 16 02:47:30.166888 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:30.166836    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34139e40-51de-48b1-b460-a6b6a20e82b7-workload-socket\") pod \"34139e40-51de-48b1-b460-a6b6a20e82b7\" (UID: \"34139e40-51de-48b1-b460-a6b6a20e82b7\") "
Sep 16 02:47:30.167149 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:30.166958    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34139e40-51de-48b1-b460-a6b6a20e82b7-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "34139e40-51de-48b1-b460-a6b6a20e82b7" (UID: "34139e40-51de-48b1-b460-a6b6a20e82b7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:47:30.167149 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:30.166980    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34139e40-51de-48b1-b460-a6b6a20e82b7-istio-data\") pod \"34139e40-51de-48b1-b460-a6b6a20e82b7\" (UID: \"34139e40-51de-48b1-b460-a6b6a20e82b7\") "
Sep 16 02:47:30.167149 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:30.167006    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34139e40-51de-48b1-b460-a6b6a20e82b7-istiod-ca-cert\") pod \"34139e40-51de-48b1-b460-a6b6a20e82b7\" (UID: \"34139e40-51de-48b1-b460-a6b6a20e82b7\") "
Sep 16 02:47:30.167149 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:30.167064    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34139e40-51de-48b1-b460-a6b6a20e82b7-workload-certs\") pod \"34139e40-51de-48b1-b460-a6b6a20e82b7\" (UID: \"34139e40-51de-48b1-b460-a6b6a20e82b7\") "
Sep 16 02:47:30.167351 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:30.167187    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34139e40-51de-48b1-b460-a6b6a20e82b7-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "34139e40-51de-48b1-b460-a6b6a20e82b7" (UID: "34139e40-51de-48b1-b460-a6b6a20e82b7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:47:30.167351 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:30.167303    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34139e40-51de-48b1-b460-a6b6a20e82b7-istio-data" (OuterVolumeSpecName: "istio-data") pod "34139e40-51de-48b1-b460-a6b6a20e82b7" (UID: "34139e40-51de-48b1-b460-a6b6a20e82b7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:47:30.167461 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:30.167389    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34139e40-51de-48b1-b460-a6b6a20e82b7-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "34139e40-51de-48b1-b460-a6b6a20e82b7" (UID: "34139e40-51de-48b1-b460-a6b6a20e82b7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:47:30.167461 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:30.167434    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34139e40-51de-48b1-b460-a6b6a20e82b7-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "34139e40-51de-48b1-b460-a6b6a20e82b7" (UID: "34139e40-51de-48b1-b460-a6b6a20e82b7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 02:47:30.167565 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:30.167526    2558 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34139e40-51de-48b1-b460-a6b6a20e82b7-istio-data\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:47:30.167565 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:30.167542    2558 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34139e40-51de-48b1-b460-a6b6a20e82b7-istiod-ca-cert\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:47:30.167565 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:30.167554    2558 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34139e40-51de-48b1-b460-a6b6a20e82b7-workload-certs\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:47:30.167748 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:30.167566    2558 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34139e40-51de-48b1-b460-a6b6a20e82b7-credential-socket\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:47:30.167748 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:30.167579    2558 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34139e40-51de-48b1-b460-a6b6a20e82b7-workload-socket\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:47:30.169118 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:30.169089    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34139e40-51de-48b1-b460-a6b6a20e82b7-istio-token" (OuterVolumeSpecName: "istio-token") pod "34139e40-51de-48b1-b460-a6b6a20e82b7" (UID: "34139e40-51de-48b1-b460-a6b6a20e82b7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 02:47:30.169118 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:30.169109    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34139e40-51de-48b1-b460-a6b6a20e82b7-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "34139e40-51de-48b1-b460-a6b6a20e82b7" (UID: "34139e40-51de-48b1-b460-a6b6a20e82b7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:47:30.169252 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:30.169128    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34139e40-51de-48b1-b460-a6b6a20e82b7-kube-api-access-tlzgz" (OuterVolumeSpecName: "kube-api-access-tlzgz") pod "34139e40-51de-48b1-b460-a6b6a20e82b7" (UID: "34139e40-51de-48b1-b460-a6b6a20e82b7"). InnerVolumeSpecName "kube-api-access-tlzgz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 02:47:30.169252 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:30.169181    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/34139e40-51de-48b1-b460-a6b6a20e82b7-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "34139e40-51de-48b1-b460-a6b6a20e82b7" (UID: "34139e40-51de-48b1-b460-a6b6a20e82b7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 02:47:30.268419 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:30.268325    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tlzgz\" (UniqueName: \"kubernetes.io/projected/34139e40-51de-48b1-b460-a6b6a20e82b7-kube-api-access-tlzgz\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:47:30.268419 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:30.268355    2558 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34139e40-51de-48b1-b460-a6b6a20e82b7-istio-podinfo\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:47:30.268419 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:30.268365    2558 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34139e40-51de-48b1-b460-a6b6a20e82b7-istio-envoy\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:47:30.268419 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:30.268373    2558 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34139e40-51de-48b1-b460-a6b6a20e82b7-istio-token\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:47:30.829862 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:30.829827    2558 generic.go:356] "Generic (PLEG): container finished" podID="34139e40-51de-48b1-b460-a6b6a20e82b7" containerID="6037f35b4cb0dc53c51af6b55977cffdcc46a6a954d3f661948547baeaf1b346" exitCode=0
Sep 16 02:47:30.830277 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:30.829876    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-5b136e-openshift-default-696f88d9c6-zwnvx"
Sep 16 02:47:30.830277 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:30.829909    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-5b136e-openshift-default-696f88d9c6-zwnvx" event={"ID":"34139e40-51de-48b1-b460-a6b6a20e82b7","Type":"ContainerDied","Data":"6037f35b4cb0dc53c51af6b55977cffdcc46a6a954d3f661948547baeaf1b346"}
Sep 16 02:47:30.830277 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:30.829961    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-5b136e-openshift-default-696f88d9c6-zwnvx" event={"ID":"34139e40-51de-48b1-b460-a6b6a20e82b7","Type":"ContainerDied","Data":"007534df8e3a77d2e6592e6934dc61915f13ec41c66c55fb14db003830b486c2"}
Sep 16 02:47:30.830277 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:30.829983    2558 scope.go:117] "RemoveContainer" containerID="6037f35b4cb0dc53c51af6b55977cffdcc46a6a954d3f661948547baeaf1b346"
Sep 16 02:47:30.839689 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:30.839668    2558 scope.go:117] "RemoveContainer" containerID="72902ba8343f4bf161d4fba2aaf3124ac47958df8eefd4096df84dc3f7b224bd"
Sep 16 02:47:30.849535 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:30.849506    2558 scope.go:117] "RemoveContainer" containerID="6037f35b4cb0dc53c51af6b55977cffdcc46a6a954d3f661948547baeaf1b346"
Sep 16 02:47:30.849992 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:47:30.849963    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6037f35b4cb0dc53c51af6b55977cffdcc46a6a954d3f661948547baeaf1b346\": container with ID starting with 6037f35b4cb0dc53c51af6b55977cffdcc46a6a954d3f661948547baeaf1b346 not found: ID does not exist" containerID="6037f35b4cb0dc53c51af6b55977cffdcc46a6a954d3f661948547baeaf1b346"
Sep 16 02:47:30.850080 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:30.850005    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6037f35b4cb0dc53c51af6b55977cffdcc46a6a954d3f661948547baeaf1b346"} err="failed to get container status \"6037f35b4cb0dc53c51af6b55977cffdcc46a6a954d3f661948547baeaf1b346\": rpc error: code = NotFound desc = could not find container \"6037f35b4cb0dc53c51af6b55977cffdcc46a6a954d3f661948547baeaf1b346\": container with ID starting with 6037f35b4cb0dc53c51af6b55977cffdcc46a6a954d3f661948547baeaf1b346 not found: ID does not exist"
Sep 16 02:47:30.850080 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:30.850031    2558 scope.go:117] "RemoveContainer" containerID="72902ba8343f4bf161d4fba2aaf3124ac47958df8eefd4096df84dc3f7b224bd"
Sep 16 02:47:30.850343 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:47:30.850322    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72902ba8343f4bf161d4fba2aaf3124ac47958df8eefd4096df84dc3f7b224bd\": container with ID starting with 72902ba8343f4bf161d4fba2aaf3124ac47958df8eefd4096df84dc3f7b224bd not found: ID does not exist" containerID="72902ba8343f4bf161d4fba2aaf3124ac47958df8eefd4096df84dc3f7b224bd"
Sep 16 02:47:30.850415 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:30.850352    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72902ba8343f4bf161d4fba2aaf3124ac47958df8eefd4096df84dc3f7b224bd"} err="failed to get container status \"72902ba8343f4bf161d4fba2aaf3124ac47958df8eefd4096df84dc3f7b224bd\": rpc error: code = NotFound desc = could not find container \"72902ba8343f4bf161d4fba2aaf3124ac47958df8eefd4096df84dc3f7b224bd\": container with ID starting with 72902ba8343f4bf161d4fba2aaf3124ac47958df8eefd4096df84dc3f7b224bd not found: ID does not exist"
Sep 16 02:47:30.852437 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:30.852416    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-5b136e-openshift-default-696f88d9c6-zwnvx"]
Sep 16 02:47:30.857412 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:30.857385    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-5b136e-openshift-default-696f88d9c6-zwnvx"]
Sep 16 02:47:30.962566 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:30.962539    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fd614e-w5pjr"
Sep 16 02:47:31.076212 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:31.076173    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d9e4180-2d9b-44e3-8980-080f593b0af1-maas-api-service-ca\") pod \"9d9e4180-2d9b-44e3-8980-080f593b0af1\" (UID: \"9d9e4180-2d9b-44e3-8980-080f593b0af1\") "
Sep 16 02:47:31.076533 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:31.076505    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d9e4180-2d9b-44e3-8980-080f593b0af1-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "9d9e4180-2d9b-44e3-8980-080f593b0af1" (UID: "9d9e4180-2d9b-44e3-8980-080f593b0af1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 02:47:31.178021 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:31.177988    2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d9e4180-2d9b-44e3-8980-080f593b0af1-maas-api-service-ca\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:47:31.550328 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:31.550242    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="34139e40-51de-48b1-b460-a6b6a20e82b7" path="/var/lib/kubelet/pods/34139e40-51de-48b1-b460-a6b6a20e82b7/volumes"
Sep 16 02:47:31.837193 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:31.837106    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fd614e-w5pjr" event={"ID":"9d9e4180-2d9b-44e3-8980-080f593b0af1","Type":"ContainerDied","Data":"5305005f2703eb1c9898323b6c0b82dc0cd3f9de0e94aa56a4fe304b6b549ff1"}
Sep 16 02:47:31.837193 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:31.837164    2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5305005f2703eb1c9898323b6c0b82dc0cd3f9de0e94aa56a4fe304b6b549ff1"
Sep 16 02:47:31.837193 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:31.837116    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fd614e-w5pjr"
Sep 16 02:47:33.244070 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:33.244036    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-8e10f1-pw9qt"]
Sep 16 02:47:33.246766 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:33.246745    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-8e10f1-pw9qt"]
Sep 16 02:47:33.550612 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:33.550511    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c8defe3a-a5ad-46f7-bd65-d849e529d9b4" path="/var/lib/kubelet/pods/c8defe3a-a5ad-46f7-bd65-d849e529d9b4/volumes"
Sep 16 02:47:40.060588 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:40.060554    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-8e10f1-openshift-default-7596f6d7c6-88psf"]
Sep 16 02:47:40.061154 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:40.060841    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-8e10f1-openshift-default-7596f6d7c6-88psf" podUID="de5a022a-4dfa-4881-b370-ef83e1129056" containerName="istio-proxy" containerID="cri-o://bf63b47f4de688b06683f339141b276decb756f82e02b1504102a7f13fd5985c" gracePeriod=30
Sep 16 02:47:44.353001 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:44.352960    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fd614e-w5pjr"]
Sep 16 02:47:44.355521 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:44.355494    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fd614e-w5pjr"]
Sep 16 02:47:45.312253 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:45.312226    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-8e10f1-openshift-default-7596f6d7c6-88psf"
Sep 16 02:47:45.320179 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:45.320115    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de5a022a-4dfa-4881-b370-ef83e1129056-workload-socket\") pod \"de5a022a-4dfa-4881-b370-ef83e1129056\" (UID: \"de5a022a-4dfa-4881-b370-ef83e1129056\") "
Sep 16 02:47:45.320179 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:45.320148    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de5a022a-4dfa-4881-b370-ef83e1129056-workload-certs\") pod \"de5a022a-4dfa-4881-b370-ef83e1129056\" (UID: \"de5a022a-4dfa-4881-b370-ef83e1129056\") "
Sep 16 02:47:45.320383 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:45.320189    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de5a022a-4dfa-4881-b370-ef83e1129056-istio-token\") pod \"de5a022a-4dfa-4881-b370-ef83e1129056\" (UID: \"de5a022a-4dfa-4881-b370-ef83e1129056\") "
Sep 16 02:47:45.320383 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:45.320230    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de5a022a-4dfa-4881-b370-ef83e1129056-istio-envoy\") pod \"de5a022a-4dfa-4881-b370-ef83e1129056\" (UID: \"de5a022a-4dfa-4881-b370-ef83e1129056\") "
Sep 16 02:47:45.320383 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:45.320256    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jz25x\" (UniqueName: \"kubernetes.io/projected/de5a022a-4dfa-4881-b370-ef83e1129056-kube-api-access-jz25x\") pod \"de5a022a-4dfa-4881-b370-ef83e1129056\" (UID: \"de5a022a-4dfa-4881-b370-ef83e1129056\") "
Sep 16 02:47:45.320383 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:45.320370    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de5a022a-4dfa-4881-b370-ef83e1129056-credential-socket\") pod \"de5a022a-4dfa-4881-b370-ef83e1129056\" (UID: \"de5a022a-4dfa-4881-b370-ef83e1129056\") "
Sep 16 02:47:45.320600 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:45.320411    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de5a022a-4dfa-4881-b370-ef83e1129056-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "de5a022a-4dfa-4881-b370-ef83e1129056" (UID: "de5a022a-4dfa-4881-b370-ef83e1129056"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:47:45.320600 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:45.320465    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de5a022a-4dfa-4881-b370-ef83e1129056-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "de5a022a-4dfa-4881-b370-ef83e1129056" (UID: "de5a022a-4dfa-4881-b370-ef83e1129056"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:47:45.320600 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:45.320495    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de5a022a-4dfa-4881-b370-ef83e1129056-istiod-ca-cert\") pod \"de5a022a-4dfa-4881-b370-ef83e1129056\" (UID: \"de5a022a-4dfa-4881-b370-ef83e1129056\") "
Sep 16 02:47:45.320600 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:45.320528    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de5a022a-4dfa-4881-b370-ef83e1129056-istio-podinfo\") pod \"de5a022a-4dfa-4881-b370-ef83e1129056\" (UID: \"de5a022a-4dfa-4881-b370-ef83e1129056\") "
Sep 16 02:47:45.320600 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:45.320556    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de5a022a-4dfa-4881-b370-ef83e1129056-istio-data\") pod \"de5a022a-4dfa-4881-b370-ef83e1129056\" (UID: \"de5a022a-4dfa-4881-b370-ef83e1129056\") "
Sep 16 02:47:45.320887 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:45.320602    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de5a022a-4dfa-4881-b370-ef83e1129056-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "de5a022a-4dfa-4881-b370-ef83e1129056" (UID: "de5a022a-4dfa-4881-b370-ef83e1129056"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:47:45.320960 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:45.320906    2558 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de5a022a-4dfa-4881-b370-ef83e1129056-workload-socket\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:47:45.320960 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:45.320912    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de5a022a-4dfa-4881-b370-ef83e1129056-istio-data" (OuterVolumeSpecName: "istio-data") pod "de5a022a-4dfa-4881-b370-ef83e1129056" (UID: "de5a022a-4dfa-4881-b370-ef83e1129056"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:47:45.320960 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:45.320927    2558 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de5a022a-4dfa-4881-b370-ef83e1129056-workload-certs\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:47:45.320960 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:45.320914    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de5a022a-4dfa-4881-b370-ef83e1129056-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "de5a022a-4dfa-4881-b370-ef83e1129056" (UID: "de5a022a-4dfa-4881-b370-ef83e1129056"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 02:47:45.320960 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:45.320940    2558 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de5a022a-4dfa-4881-b370-ef83e1129056-credential-socket\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:47:45.322668 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:45.322641    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/de5a022a-4dfa-4881-b370-ef83e1129056-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "de5a022a-4dfa-4881-b370-ef83e1129056" (UID: "de5a022a-4dfa-4881-b370-ef83e1129056"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 02:47:45.322668 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:45.322655    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de5a022a-4dfa-4881-b370-ef83e1129056-istio-token" (OuterVolumeSpecName: "istio-token") pod "de5a022a-4dfa-4881-b370-ef83e1129056" (UID: "de5a022a-4dfa-4881-b370-ef83e1129056"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 02:47:45.322941 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:45.322912    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de5a022a-4dfa-4881-b370-ef83e1129056-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "de5a022a-4dfa-4881-b370-ef83e1129056" (UID: "de5a022a-4dfa-4881-b370-ef83e1129056"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:47:45.323026 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:45.322921    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de5a022a-4dfa-4881-b370-ef83e1129056-kube-api-access-jz25x" (OuterVolumeSpecName: "kube-api-access-jz25x") pod "de5a022a-4dfa-4881-b370-ef83e1129056" (UID: "de5a022a-4dfa-4881-b370-ef83e1129056"). InnerVolumeSpecName "kube-api-access-jz25x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 02:47:45.421520 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:45.421478    2558 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de5a022a-4dfa-4881-b370-ef83e1129056-istiod-ca-cert\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:47:45.421520 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:45.421515    2558 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de5a022a-4dfa-4881-b370-ef83e1129056-istio-podinfo\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:47:45.421520 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:45.421524    2558 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de5a022a-4dfa-4881-b370-ef83e1129056-istio-data\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:47:45.421520 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:45.421532    2558 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de5a022a-4dfa-4881-b370-ef83e1129056-istio-token\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:47:45.422033 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:45.421539    2558 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de5a022a-4dfa-4881-b370-ef83e1129056-istio-envoy\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:47:45.422033 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:45.421547    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jz25x\" (UniqueName: \"kubernetes.io/projected/de5a022a-4dfa-4881-b370-ef83e1129056-kube-api-access-jz25x\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:47:45.550967 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:45.550923    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9d9e4180-2d9b-44e3-8980-080f593b0af1" path="/var/lib/kubelet/pods/9d9e4180-2d9b-44e3-8980-080f593b0af1/volumes"
Sep 16 02:47:45.913749 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:45.913716    2558 generic.go:356] "Generic (PLEG): container finished" podID="de5a022a-4dfa-4881-b370-ef83e1129056" containerID="bf63b47f4de688b06683f339141b276decb756f82e02b1504102a7f13fd5985c" exitCode=0
Sep 16 02:47:45.913920 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:45.913765    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-8e10f1-openshift-default-7596f6d7c6-88psf"
Sep 16 02:47:45.913920 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:45.913785    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-8e10f1-openshift-default-7596f6d7c6-88psf" event={"ID":"de5a022a-4dfa-4881-b370-ef83e1129056","Type":"ContainerDied","Data":"bf63b47f4de688b06683f339141b276decb756f82e02b1504102a7f13fd5985c"}
Sep 16 02:47:45.913920 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:45.913840    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-8e10f1-openshift-default-7596f6d7c6-88psf" event={"ID":"de5a022a-4dfa-4881-b370-ef83e1129056","Type":"ContainerDied","Data":"889ba7bddf2705902f9abe77589305821e8c5245a77e177ed51ef26e5f614c44"}
Sep 16 02:47:45.913920 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:45.913863    2558 scope.go:117] "RemoveContainer" containerID="bf63b47f4de688b06683f339141b276decb756f82e02b1504102a7f13fd5985c"
Sep 16 02:47:45.922896 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:45.922870    2558 scope.go:117] "RemoveContainer" containerID="4265b8c81d8c3a9475be8a47d2efd65c89590dad5678176f49fe345e3435b643"
Sep 16 02:47:45.931392 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:45.931368    2558 scope.go:117] "RemoveContainer" containerID="bf63b47f4de688b06683f339141b276decb756f82e02b1504102a7f13fd5985c"
Sep 16 02:47:45.931656 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:45.931633    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-8e10f1-openshift-default-7596f6d7c6-88psf"]
Sep 16 02:47:45.931836 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:47:45.931724    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf63b47f4de688b06683f339141b276decb756f82e02b1504102a7f13fd5985c\": container with ID starting with bf63b47f4de688b06683f339141b276decb756f82e02b1504102a7f13fd5985c not found: ID does not exist" containerID="bf63b47f4de688b06683f339141b276decb756f82e02b1504102a7f13fd5985c"
Sep 16 02:47:45.931836 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:45.931758    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf63b47f4de688b06683f339141b276decb756f82e02b1504102a7f13fd5985c"} err="failed to get container status \"bf63b47f4de688b06683f339141b276decb756f82e02b1504102a7f13fd5985c\": rpc error: code = NotFound desc = could not find container \"bf63b47f4de688b06683f339141b276decb756f82e02b1504102a7f13fd5985c\": container with ID starting with bf63b47f4de688b06683f339141b276decb756f82e02b1504102a7f13fd5985c not found: ID does not exist"
Sep 16 02:47:45.931836 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:45.931782    2558 scope.go:117] "RemoveContainer" containerID="4265b8c81d8c3a9475be8a47d2efd65c89590dad5678176f49fe345e3435b643"
Sep 16 02:47:45.932248 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:47:45.932213    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4265b8c81d8c3a9475be8a47d2efd65c89590dad5678176f49fe345e3435b643\": container with ID starting with 4265b8c81d8c3a9475be8a47d2efd65c89590dad5678176f49fe345e3435b643 not found: ID does not exist" containerID="4265b8c81d8c3a9475be8a47d2efd65c89590dad5678176f49fe345e3435b643"
Sep 16 02:47:45.932362 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:45.932247    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4265b8c81d8c3a9475be8a47d2efd65c89590dad5678176f49fe345e3435b643"} err="failed to get container status \"4265b8c81d8c3a9475be8a47d2efd65c89590dad5678176f49fe345e3435b643\": rpc error: code = NotFound desc = could not find container \"4265b8c81d8c3a9475be8a47d2efd65c89590dad5678176f49fe345e3435b643\": container with ID starting with 4265b8c81d8c3a9475be8a47d2efd65c89590dad5678176f49fe345e3435b643 not found: ID does not exist"
Sep 16 02:47:45.934529 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:45.934508    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-8e10f1-openshift-default-7596f6d7c6-88psf"]
Sep 16 02:47:47.550225 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:47:47.550184    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="de5a022a-4dfa-4881-b370-ef83e1129056" path="/var/lib/kubelet/pods/de5a022a-4dfa-4881-b370-ef83e1129056/volumes"
Sep 16 02:48:22.543931 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:48:22.543902    2558 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-processing-5d7dbbd56b-pltkr" secret="" err="secret \"payload-processing-dockercfg-zzx6j\" not found"
Sep 16 02:48:47.051448 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:48:47.051414    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-fd614e-openshift-default-649576fb58-sg4bp"]
Sep 16 02:48:47.052031 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:48:47.051676    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-fd614e-openshift-default-649576fb58-sg4bp" podUID="aa9e2461-89e4-4edc-8637-8f8934e91072" containerName="istio-proxy" containerID="cri-o://98890aeae1f27d25a7fc5e7fb62f762e6ec102a5df04bc7fd5df7337a0125b4c" gracePeriod=30
Sep 16 02:48:47.426685 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:48:47.426646    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d3d202-c452k"]
Sep 16 02:48:47.427214 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:48:47.427198    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34139e40-51de-48b1-b460-a6b6a20e82b7" containerName="istio-proxy"
Sep 16 02:48:47.427260 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:48:47.427218    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="34139e40-51de-48b1-b460-a6b6a20e82b7" containerName="istio-proxy"
Sep 16 02:48:47.427260 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:48:47.427234    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f4a618df-1dcc-42ed-ac72-e519ee2c5515" containerName="istio-proxy"
Sep 16 02:48:47.427260 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:48:47.427242    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="f4a618df-1dcc-42ed-ac72-e519ee2c5515" containerName="istio-proxy"
Sep 16 02:48:47.427260 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:48:47.427257    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de5a022a-4dfa-4881-b370-ef83e1129056" containerName="istio-proxy"
Sep 16 02:48:47.427385 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:48:47.427266    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="de5a022a-4dfa-4881-b370-ef83e1129056" containerName="istio-proxy"
Sep 16 02:48:47.427385 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:48:47.427278    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34139e40-51de-48b1-b460-a6b6a20e82b7" containerName="istio-proxy"
Sep 16 02:48:47.427385 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:48:47.427287    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="34139e40-51de-48b1-b460-a6b6a20e82b7" containerName="istio-proxy"
Sep 16 02:48:47.427385 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:48:47.427308    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de5a022a-4dfa-4881-b370-ef83e1129056" containerName="istio-proxy"
Sep 16 02:48:47.427385 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:48:47.427316    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="de5a022a-4dfa-4881-b370-ef83e1129056" containerName="istio-proxy"
Sep 16 02:48:47.427385 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:48:47.427330    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d9e4180-2d9b-44e3-8980-080f593b0af1" containerName="revoke-keys"
Sep 16 02:48:47.427385 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:48:47.427338    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d9e4180-2d9b-44e3-8980-080f593b0af1" containerName="revoke-keys"
Sep 16 02:48:47.427597 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:48:47.427412    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="9d9e4180-2d9b-44e3-8980-080f593b0af1" containerName="revoke-keys"
Sep 16 02:48:47.427597 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:48:47.427423    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="de5a022a-4dfa-4881-b370-ef83e1129056" containerName="istio-proxy"
Sep 16 02:48:47.427597 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:48:47.427432    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="34139e40-51de-48b1-b460-a6b6a20e82b7" containerName="istio-proxy"
Sep 16 02:48:47.427597 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:48:47.427441    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="de5a022a-4dfa-4881-b370-ef83e1129056" containerName="istio-proxy"
Sep 16 02:48:47.427597 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:48:47.427451    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="f4a618df-1dcc-42ed-ac72-e519ee2c5515" containerName="istio-proxy"
Sep 16 02:48:47.427597 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:48:47.427460    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="34139e40-51de-48b1-b460-a6b6a20e82b7" containerName="istio-proxy"
Sep 16 02:48:47.430919 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:48:47.430895    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d3d202-c452k"
Sep 16 02:48:47.433314 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:48:47.433290    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 16 02:48:47.433613 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:48:47.433584    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jpd4f\""
Sep 16 02:48:47.436113 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:48:47.436086    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d3d202-c452k"]
Sep 16 02:48:47.531158 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:48:47.531120    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a924eb26-efff-4c6e-85bc-8191ca817c87-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-d3d202-c452k\" (UID: \"a924eb26-efff-4c6e-85bc-8191ca817c87\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d3d202-c452k"
Sep 16 02:48:47.632809 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:48:47.632772    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a924eb26-efff-4c6e-85bc-8191ca817c87-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-d3d202-c452k\" (UID: \"a924eb26-efff-4c6e-85bc-8191ca817c87\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d3d202-c452k"
Sep 16 02:48:47.633444 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:48:47.633419    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a924eb26-efff-4c6e-85bc-8191ca817c87-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-d3d202-c452k\" (UID: \"a924eb26-efff-4c6e-85bc-8191ca817c87\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d3d202-c452k"
Sep 16 02:48:47.744377 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:48:47.744283    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d3d202-c452k"
Sep 16 02:48:48.013422 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:48:48.013395    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d3d202-c452k"]
Sep 16 02:48:48.015222 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:48:48.015189    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda924eb26_efff_4c6e_85bc_8191ca817c87.slice/crio-c260da9522c6a0aee34d35c14930429732fc043efac6c92980414f238755813c WatchSource:0}: Error finding container c260da9522c6a0aee34d35c14930429732fc043efac6c92980414f238755813c: Status 404 returned error can't find the container with id c260da9522c6a0aee34d35c14930429732fc043efac6c92980414f238755813c
Sep 16 02:48:48.244121 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:48:48.244084    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d3d202-c452k" event={"ID":"a924eb26-efff-4c6e-85bc-8191ca817c87","Type":"ContainerStarted","Data":"c260da9522c6a0aee34d35c14930429732fc043efac6c92980414f238755813c"}
Sep 16 02:48:49.250741 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:48:49.250679    2558 generic.go:356] "Generic (PLEG): container finished" podID="a924eb26-efff-4c6e-85bc-8191ca817c87" containerID="3742d8457bc59f2ac2fc693624ff12654d2aad25759f9a3cc93c6b2f17b0b872" exitCode=0
Sep 16 02:48:49.251164 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:48:49.250763    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d3d202-c452k" event={"ID":"a924eb26-efff-4c6e-85bc-8191ca817c87","Type":"ContainerDied","Data":"3742d8457bc59f2ac2fc693624ff12654d2aad25759f9a3cc93c6b2f17b0b872"}
Sep 16 02:48:50.386125 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:48:50.386100    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d3d202-c452k"
Sep 16 02:48:50.561071 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:48:50.560974    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a924eb26-efff-4c6e-85bc-8191ca817c87-maas-api-service-ca\") pod \"a924eb26-efff-4c6e-85bc-8191ca817c87\" (UID: \"a924eb26-efff-4c6e-85bc-8191ca817c87\") "
Sep 16 02:48:50.561375 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:48:50.561349    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a924eb26-efff-4c6e-85bc-8191ca817c87-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a924eb26-efff-4c6e-85bc-8191ca817c87" (UID: "a924eb26-efff-4c6e-85bc-8191ca817c87"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 02:48:50.662747 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:48:50.662710    2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a924eb26-efff-4c6e-85bc-8191ca817c87-maas-api-service-ca\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:48:51.262400 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:48:51.262363    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d3d202-c452k" event={"ID":"a924eb26-efff-4c6e-85bc-8191ca817c87","Type":"ContainerDied","Data":"c260da9522c6a0aee34d35c14930429732fc043efac6c92980414f238755813c"}
Sep 16 02:48:51.262575 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:48:51.262418    2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c260da9522c6a0aee34d35c14930429732fc043efac6c92980414f238755813c"
Sep 16 02:48:51.262575 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:48:51.262376    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d3d202-c452k"
Sep 16 02:48:52.270826 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:48:52.270796    2558 generic.go:356] "Generic (PLEG): container finished" podID="aa9e2461-89e4-4edc-8637-8f8934e91072" containerID="98890aeae1f27d25a7fc5e7fb62f762e6ec102a5df04bc7fd5df7337a0125b4c" exitCode=0
Sep 16 02:48:52.271321 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:48:52.270842    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-fd614e-openshift-default-649576fb58-sg4bp" event={"ID":"aa9e2461-89e4-4edc-8637-8f8934e91072","Type":"ContainerDied","Data":"98890aeae1f27d25a7fc5e7fb62f762e6ec102a5df04bc7fd5df7337a0125b4c"}
Sep 16 02:48:52.271321 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:48:52.270899    2558 scope.go:117] "RemoveContainer" containerID="9b437313db209b820f8eb8427639104959de2e41a51e41dfdf1d5f7c00daeb71"
Sep 16 02:48:52.303593 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:48:52.303574    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-fd614e-openshift-default-649576fb58-sg4bp"
Sep 16 02:48:52.377302 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:48:52.377218    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa9e2461-89e4-4edc-8637-8f8934e91072-workload-certs\") pod \"aa9e2461-89e4-4edc-8637-8f8934e91072\" (UID: \"aa9e2461-89e4-4edc-8637-8f8934e91072\") "
Sep 16 02:48:52.377302 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:48:52.377292    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa9e2461-89e4-4edc-8637-8f8934e91072-istio-data\") pod \"aa9e2461-89e4-4edc-8637-8f8934e91072\" (UID: \"aa9e2461-89e4-4edc-8637-8f8934e91072\") "
Sep 16 02:48:52.377534 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:48:52.377324    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa9e2461-89e4-4edc-8637-8f8934e91072-credential-socket\") pod \"aa9e2461-89e4-4edc-8637-8f8934e91072\" (UID: \"aa9e2461-89e4-4edc-8637-8f8934e91072\") "
Sep 16 02:48:52.377534 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:48:52.377360    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa9e2461-89e4-4edc-8637-8f8934e91072-istiod-ca-cert\") pod \"aa9e2461-89e4-4edc-8637-8f8934e91072\" (UID: \"aa9e2461-89e4-4edc-8637-8f8934e91072\") "
Sep 16 02:48:52.377534 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:48:52.377395    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa9e2461-89e4-4edc-8637-8f8934e91072-workload-socket\") pod \"aa9e2461-89e4-4edc-8637-8f8934e91072\" (UID: \"aa9e2461-89e4-4edc-8637-8f8934e91072\") "
Sep 16 02:48:52.377534 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:48:52.377476    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa9e2461-89e4-4edc-8637-8f8934e91072-istio-podinfo\") pod \"aa9e2461-89e4-4edc-8637-8f8934e91072\" (UID: \"aa9e2461-89e4-4edc-8637-8f8934e91072\") "
Sep 16 02:48:52.377534 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:48:52.377484    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa9e2461-89e4-4edc-8637-8f8934e91072-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "aa9e2461-89e4-4edc-8637-8f8934e91072" (UID: "aa9e2461-89e4-4edc-8637-8f8934e91072"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:48:52.377534 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:48:52.377507    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa9e2461-89e4-4edc-8637-8f8934e91072-istio-token\") pod \"aa9e2461-89e4-4edc-8637-8f8934e91072\" (UID: \"aa9e2461-89e4-4edc-8637-8f8934e91072\") "
Sep 16 02:48:52.377903 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:48:52.377551    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jvczb\" (UniqueName: \"kubernetes.io/projected/aa9e2461-89e4-4edc-8637-8f8934e91072-kube-api-access-jvczb\") pod \"aa9e2461-89e4-4edc-8637-8f8934e91072\" (UID: \"aa9e2461-89e4-4edc-8637-8f8934e91072\") "
Sep 16 02:48:52.377903 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:48:52.377578    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa9e2461-89e4-4edc-8637-8f8934e91072-istio-envoy\") pod \"aa9e2461-89e4-4edc-8637-8f8934e91072\" (UID: \"aa9e2461-89e4-4edc-8637-8f8934e91072\") "
Sep 16 02:48:52.377903 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:48:52.377592    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa9e2461-89e4-4edc-8637-8f8934e91072-istio-data" (OuterVolumeSpecName: "istio-data") pod "aa9e2461-89e4-4edc-8637-8f8934e91072" (UID: "aa9e2461-89e4-4edc-8637-8f8934e91072"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:48:52.377903 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:48:52.377614    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa9e2461-89e4-4edc-8637-8f8934e91072-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "aa9e2461-89e4-4edc-8637-8f8934e91072" (UID: "aa9e2461-89e4-4edc-8637-8f8934e91072"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:48:52.377903 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:48:52.377730    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa9e2461-89e4-4edc-8637-8f8934e91072-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "aa9e2461-89e4-4edc-8637-8f8934e91072" (UID: "aa9e2461-89e4-4edc-8637-8f8934e91072"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:48:52.377903 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:48:52.377798    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa9e2461-89e4-4edc-8637-8f8934e91072-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "aa9e2461-89e4-4edc-8637-8f8934e91072" (UID: "aa9e2461-89e4-4edc-8637-8f8934e91072"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 02:48:52.378241 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:48:52.377960    2558 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa9e2461-89e4-4edc-8637-8f8934e91072-workload-certs\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:48:52.378241 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:48:52.377979    2558 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa9e2461-89e4-4edc-8637-8f8934e91072-istio-data\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:48:52.378241 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:48:52.377992    2558 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa9e2461-89e4-4edc-8637-8f8934e91072-credential-socket\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:48:52.378241 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:48:52.378005    2558 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa9e2461-89e4-4edc-8637-8f8934e91072-istiod-ca-cert\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:48:52.378241 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:48:52.378017    2558 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa9e2461-89e4-4edc-8637-8f8934e91072-workload-socket\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:48:52.379587 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:48:52.379561    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/aa9e2461-89e4-4edc-8637-8f8934e91072-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "aa9e2461-89e4-4edc-8637-8f8934e91072" (UID: "aa9e2461-89e4-4edc-8637-8f8934e91072"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 02:48:52.379772 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:48:52.379691    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa9e2461-89e4-4edc-8637-8f8934e91072-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "aa9e2461-89e4-4edc-8637-8f8934e91072" (UID: "aa9e2461-89e4-4edc-8637-8f8934e91072"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:48:52.379772 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:48:52.379719    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa9e2461-89e4-4edc-8637-8f8934e91072-istio-token" (OuterVolumeSpecName: "istio-token") pod "aa9e2461-89e4-4edc-8637-8f8934e91072" (UID: "aa9e2461-89e4-4edc-8637-8f8934e91072"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 02:48:52.380021 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:48:52.379997    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa9e2461-89e4-4edc-8637-8f8934e91072-kube-api-access-jvczb" (OuterVolumeSpecName: "kube-api-access-jvczb") pod "aa9e2461-89e4-4edc-8637-8f8934e91072" (UID: "aa9e2461-89e4-4edc-8637-8f8934e91072"). InnerVolumeSpecName "kube-api-access-jvczb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 02:48:52.479475 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:48:52.479441    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jvczb\" (UniqueName: \"kubernetes.io/projected/aa9e2461-89e4-4edc-8637-8f8934e91072-kube-api-access-jvczb\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:48:52.479475 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:48:52.479476    2558 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa9e2461-89e4-4edc-8637-8f8934e91072-istio-envoy\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:48:52.479669 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:48:52.479490    2558 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa9e2461-89e4-4edc-8637-8f8934e91072-istio-podinfo\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:48:52.479669 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:48:52.479503    2558 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa9e2461-89e4-4edc-8637-8f8934e91072-istio-token\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:48:53.277171 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:48:53.277138    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-fd614e-openshift-default-649576fb58-sg4bp" event={"ID":"aa9e2461-89e4-4edc-8637-8f8934e91072","Type":"ContainerDied","Data":"39a1c9d7defcce857ab07a45c976db27ceeea84ee8ecf434890aaab8a42e3fc9"}
Sep 16 02:48:53.277171 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:48:53.277170    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-fd614e-openshift-default-649576fb58-sg4bp"
Sep 16 02:48:53.277756 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:48:53.277199    2558 scope.go:117] "RemoveContainer" containerID="98890aeae1f27d25a7fc5e7fb62f762e6ec102a5df04bc7fd5df7337a0125b4c"
Sep 16 02:48:53.300138 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:48:53.300108    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-fd614e-openshift-default-649576fb58-sg4bp"]
Sep 16 02:48:53.304742 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:48:53.304717    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-fd614e-openshift-default-649576fb58-sg4bp"]
Sep 16 02:48:53.550735 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:48:53.550645    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aa9e2461-89e4-4edc-8637-8f8934e91072" path="/var/lib/kubelet/pods/aa9e2461-89e4-4edc-8637-8f8934e91072/volumes"
Sep 16 02:48:58.056163 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:48:58.056125    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d3d202-c452k"]
Sep 16 02:48:58.058564 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:48:58.058543    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d3d202-c452k"]
Sep 16 02:48:59.550112 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:48:59.550078    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a924eb26-efff-4c6e-85bc-8191ca817c87" path="/var/lib/kubelet/pods/a924eb26-efff-4c6e-85bc-8191ca817c87/volumes"
Sep 16 02:49:04.436550 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:49:04.436510    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-d3d202-openshift-default-6745f5bd6-gvffh"]
Sep 16 02:49:04.437092 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:49:04.436817    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-d3d202-openshift-default-6745f5bd6-gvffh" podUID="cdd07dc7-673e-46cc-b248-4838609577c1" containerName="istio-proxy" containerID="cri-o://dcbb49801615e5ebc2a3831e48668a17d3acb46f51081fa2dc3b6b4c3da5f4f2" gracePeriod=30
Sep 16 02:49:07.984059 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:49:07.984022    2558 scope.go:117] "RemoveContainer" containerID="d2bb20ae412d51cc9513cca0fa74941c4a056ebffca4f95b5ee82ef93b6dec84"
Sep 16 02:49:09.684385 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:49:09.684355    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-d3d202-openshift-default-6745f5bd6-gvffh"
Sep 16 02:49:09.743802 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:49:09.743682    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wznjd\" (UniqueName: \"kubernetes.io/projected/cdd07dc7-673e-46cc-b248-4838609577c1-kube-api-access-wznjd\") pod \"cdd07dc7-673e-46cc-b248-4838609577c1\" (UID: \"cdd07dc7-673e-46cc-b248-4838609577c1\") "
Sep 16 02:49:09.743802 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:49:09.743769    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cdd07dc7-673e-46cc-b248-4838609577c1-istio-podinfo\") pod \"cdd07dc7-673e-46cc-b248-4838609577c1\" (UID: \"cdd07dc7-673e-46cc-b248-4838609577c1\") "
Sep 16 02:49:09.744041 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:49:09.743932    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdd07dc7-673e-46cc-b248-4838609577c1-workload-socket\") pod \"cdd07dc7-673e-46cc-b248-4838609577c1\" (UID: \"cdd07dc7-673e-46cc-b248-4838609577c1\") "
Sep 16 02:49:09.744041 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:49:09.744012    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cdd07dc7-673e-46cc-b248-4838609577c1-istiod-ca-cert\") pod \"cdd07dc7-673e-46cc-b248-4838609577c1\" (UID: \"cdd07dc7-673e-46cc-b248-4838609577c1\") "
Sep 16 02:49:09.744158 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:49:09.744106    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdd07dc7-673e-46cc-b248-4838609577c1-credential-socket\") pod \"cdd07dc7-673e-46cc-b248-4838609577c1\" (UID: \"cdd07dc7-673e-46cc-b248-4838609577c1\") "
Sep 16 02:49:09.744158 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:49:09.744142    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cdd07dc7-673e-46cc-b248-4838609577c1-istio-envoy\") pod \"cdd07dc7-673e-46cc-b248-4838609577c1\" (UID: \"cdd07dc7-673e-46cc-b248-4838609577c1\") "
Sep 16 02:49:09.744158 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:49:09.744142    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdd07dc7-673e-46cc-b248-4838609577c1-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "cdd07dc7-673e-46cc-b248-4838609577c1" (UID: "cdd07dc7-673e-46cc-b248-4838609577c1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:49:09.744324 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:49:09.744174    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cdd07dc7-673e-46cc-b248-4838609577c1-workload-certs\") pod \"cdd07dc7-673e-46cc-b248-4838609577c1\" (UID: \"cdd07dc7-673e-46cc-b248-4838609577c1\") "
Sep 16 02:49:09.744324 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:49:09.744200    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cdd07dc7-673e-46cc-b248-4838609577c1-istio-token\") pod \"cdd07dc7-673e-46cc-b248-4838609577c1\" (UID: \"cdd07dc7-673e-46cc-b248-4838609577c1\") "
Sep 16 02:49:09.744324 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:49:09.744235    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cdd07dc7-673e-46cc-b248-4838609577c1-istio-data\") pod \"cdd07dc7-673e-46cc-b248-4838609577c1\" (UID: \"cdd07dc7-673e-46cc-b248-4838609577c1\") "
Sep 16 02:49:09.744516 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:49:09.744381    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdd07dc7-673e-46cc-b248-4838609577c1-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "cdd07dc7-673e-46cc-b248-4838609577c1" (UID: "cdd07dc7-673e-46cc-b248-4838609577c1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:49:09.744516 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:49:09.744435    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdd07dc7-673e-46cc-b248-4838609577c1-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "cdd07dc7-673e-46cc-b248-4838609577c1" (UID: "cdd07dc7-673e-46cc-b248-4838609577c1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:49:09.744516 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:49:09.744461    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cdd07dc7-673e-46cc-b248-4838609577c1-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "cdd07dc7-673e-46cc-b248-4838609577c1" (UID: "cdd07dc7-673e-46cc-b248-4838609577c1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 02:49:09.744682 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:49:09.744562    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdd07dc7-673e-46cc-b248-4838609577c1-istio-data" (OuterVolumeSpecName: "istio-data") pod "cdd07dc7-673e-46cc-b248-4838609577c1" (UID: "cdd07dc7-673e-46cc-b248-4838609577c1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:49:09.744682 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:49:09.744635    2558 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdd07dc7-673e-46cc-b248-4838609577c1-credential-socket\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:49:09.744682 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:49:09.744651    2558 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cdd07dc7-673e-46cc-b248-4838609577c1-workload-certs\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:49:09.744682 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:49:09.744665    2558 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cdd07dc7-673e-46cc-b248-4838609577c1-istio-data\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:49:09.744682 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:49:09.744677    2558 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdd07dc7-673e-46cc-b248-4838609577c1-workload-socket\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:49:09.744918 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:49:09.744689    2558 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cdd07dc7-673e-46cc-b248-4838609577c1-istiod-ca-cert\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:49:09.745999 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:49:09.745970    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cdd07dc7-673e-46cc-b248-4838609577c1-kube-api-access-wznjd" (OuterVolumeSpecName: "kube-api-access-wznjd") pod "cdd07dc7-673e-46cc-b248-4838609577c1" (UID: "cdd07dc7-673e-46cc-b248-4838609577c1"). InnerVolumeSpecName "kube-api-access-wznjd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 02:49:09.746092 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:49:09.746061    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdd07dc7-673e-46cc-b248-4838609577c1-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "cdd07dc7-673e-46cc-b248-4838609577c1" (UID: "cdd07dc7-673e-46cc-b248-4838609577c1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:49:09.746344 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:49:09.746324    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cdd07dc7-673e-46cc-b248-4838609577c1-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "cdd07dc7-673e-46cc-b248-4838609577c1" (UID: "cdd07dc7-673e-46cc-b248-4838609577c1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 02:49:09.746566 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:49:09.746550    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cdd07dc7-673e-46cc-b248-4838609577c1-istio-token" (OuterVolumeSpecName: "istio-token") pod "cdd07dc7-673e-46cc-b248-4838609577c1" (UID: "cdd07dc7-673e-46cc-b248-4838609577c1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 02:49:09.845340 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:49:09.845303    2558 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cdd07dc7-673e-46cc-b248-4838609577c1-istio-token\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:49:09.845340 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:49:09.845336    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wznjd\" (UniqueName: \"kubernetes.io/projected/cdd07dc7-673e-46cc-b248-4838609577c1-kube-api-access-wznjd\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:49:09.845546 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:49:09.845353    2558 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cdd07dc7-673e-46cc-b248-4838609577c1-istio-podinfo\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:49:09.845546 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:49:09.845362    2558 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cdd07dc7-673e-46cc-b248-4838609577c1-istio-envoy\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:49:10.365542 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:49:10.365506    2558 generic.go:356] "Generic (PLEG): container finished" podID="cdd07dc7-673e-46cc-b248-4838609577c1" containerID="dcbb49801615e5ebc2a3831e48668a17d3acb46f51081fa2dc3b6b4c3da5f4f2" exitCode=0
Sep 16 02:49:10.365765 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:49:10.365559    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-d3d202-openshift-default-6745f5bd6-gvffh"
Sep 16 02:49:10.365765 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:49:10.365598    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-d3d202-openshift-default-6745f5bd6-gvffh" event={"ID":"cdd07dc7-673e-46cc-b248-4838609577c1","Type":"ContainerDied","Data":"dcbb49801615e5ebc2a3831e48668a17d3acb46f51081fa2dc3b6b4c3da5f4f2"}
Sep 16 02:49:10.365765 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:49:10.365656    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-d3d202-openshift-default-6745f5bd6-gvffh" event={"ID":"cdd07dc7-673e-46cc-b248-4838609577c1","Type":"ContainerDied","Data":"3a929b051d3759aa0dc65b16eaf94920f01581d6dedb7712b9c1dda6df3eba33"}
Sep 16 02:49:10.365765 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:49:10.365681    2558 scope.go:117] "RemoveContainer" containerID="dcbb49801615e5ebc2a3831e48668a17d3acb46f51081fa2dc3b6b4c3da5f4f2"
Sep 16 02:49:10.375175 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:49:10.375157    2558 scope.go:117] "RemoveContainer" containerID="f52358d4482f3fe79785a205eceb5a13bd47b795c73f8bc2e3eac96c043a0cbf"
Sep 16 02:49:10.382913 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:49:10.382894    2558 scope.go:117] "RemoveContainer" containerID="dcbb49801615e5ebc2a3831e48668a17d3acb46f51081fa2dc3b6b4c3da5f4f2"
Sep 16 02:49:10.383180 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:49:10.383161    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dcbb49801615e5ebc2a3831e48668a17d3acb46f51081fa2dc3b6b4c3da5f4f2\": container with ID starting with dcbb49801615e5ebc2a3831e48668a17d3acb46f51081fa2dc3b6b4c3da5f4f2 not found: ID does not exist" containerID="dcbb49801615e5ebc2a3831e48668a17d3acb46f51081fa2dc3b6b4c3da5f4f2"
Sep 16 02:49:10.383253 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:49:10.383195    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dcbb49801615e5ebc2a3831e48668a17d3acb46f51081fa2dc3b6b4c3da5f4f2"} err="failed to get container status \"dcbb49801615e5ebc2a3831e48668a17d3acb46f51081fa2dc3b6b4c3da5f4f2\": rpc error: code = NotFound desc = could not find container \"dcbb49801615e5ebc2a3831e48668a17d3acb46f51081fa2dc3b6b4c3da5f4f2\": container with ID starting with dcbb49801615e5ebc2a3831e48668a17d3acb46f51081fa2dc3b6b4c3da5f4f2 not found: ID does not exist"
Sep 16 02:49:10.383253 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:49:10.383217    2558 scope.go:117] "RemoveContainer" containerID="f52358d4482f3fe79785a205eceb5a13bd47b795c73f8bc2e3eac96c043a0cbf"
Sep 16 02:49:10.383478 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:49:10.383463    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f52358d4482f3fe79785a205eceb5a13bd47b795c73f8bc2e3eac96c043a0cbf\": container with ID starting with f52358d4482f3fe79785a205eceb5a13bd47b795c73f8bc2e3eac96c043a0cbf not found: ID does not exist" containerID="f52358d4482f3fe79785a205eceb5a13bd47b795c73f8bc2e3eac96c043a0cbf"
Sep 16 02:49:10.383520 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:49:10.383484    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f52358d4482f3fe79785a205eceb5a13bd47b795c73f8bc2e3eac96c043a0cbf"} err="failed to get container status \"f52358d4482f3fe79785a205eceb5a13bd47b795c73f8bc2e3eac96c043a0cbf\": rpc error: code = NotFound desc = could not find container \"f52358d4482f3fe79785a205eceb5a13bd47b795c73f8bc2e3eac96c043a0cbf\": container with ID starting with f52358d4482f3fe79785a205eceb5a13bd47b795c73f8bc2e3eac96c043a0cbf not found: ID does not exist"
Sep 16 02:49:10.386316 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:49:10.386295    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-d3d202-openshift-default-6745f5bd6-gvffh"]
Sep 16 02:49:10.389783 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:49:10.389765    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-d3d202-openshift-default-6745f5bd6-gvffh"]
Sep 16 02:49:11.550411 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:49:11.550373    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cdd07dc7-673e-46cc-b248-4838609577c1" path="/var/lib/kubelet/pods/cdd07dc7-673e-46cc-b248-4838609577c1/volumes"
Sep 16 02:49:15.227514 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:49:15.227478    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-24aabe-29825445-trdjt"]
Sep 16 02:49:15.231653 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:49:15.231623    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-24aabe-29825445-trdjt"]
Sep 16 02:49:15.258074 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:49:15.258043    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-24aabe-7bc4694f6b-k4j76"]
Sep 16 02:49:15.258340 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:49:15.258292    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-24aabe-7bc4694f6b-k4j76" podUID="0e7244d4-fbe0-4519-805f-3eb191354cfa" containerName="payload-pre-processing" containerID="cri-o://93e52de39ec76e9288a404c12d16eb6d4315b31e58285c6fc2b1df4f3d51b271" gracePeriod=30
Sep 16 02:49:15.393472 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:49:15.393432    2558 generic.go:356] "Generic (PLEG): container finished" podID="0e7244d4-fbe0-4519-805f-3eb191354cfa" containerID="93e52de39ec76e9288a404c12d16eb6d4315b31e58285c6fc2b1df4f3d51b271" exitCode=0
Sep 16 02:49:15.393646 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:49:15.393483    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-24aabe-7bc4694f6b-k4j76" event={"ID":"0e7244d4-fbe0-4519-805f-3eb191354cfa","Type":"ContainerDied","Data":"93e52de39ec76e9288a404c12d16eb6d4315b31e58285c6fc2b1df4f3d51b271"}
Sep 16 02:49:15.517326 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:49:15.517296    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-24aabe-7bc4694f6b-k4j76"
Sep 16 02:49:15.549694 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:49:15.549661    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="111ee69d-f9d4-421b-a6e3-ad5f828b2eaf" path="/var/lib/kubelet/pods/111ee69d-f9d4-421b-a6e3-ad5f828b2eaf/volumes"
Sep 16 02:49:15.603770 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:49:15.603732    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l7j88\" (UniqueName: \"kubernetes.io/projected/0e7244d4-fbe0-4519-805f-3eb191354cfa-kube-api-access-l7j88\") pod \"0e7244d4-fbe0-4519-805f-3eb191354cfa\" (UID: \"0e7244d4-fbe0-4519-805f-3eb191354cfa\") "
Sep 16 02:49:15.603920 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:49:15.603802    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e7244d4-fbe0-4519-805f-3eb191354cfa-plugins-config-volume\") pod \"0e7244d4-fbe0-4519-805f-3eb191354cfa\" (UID: \"0e7244d4-fbe0-4519-805f-3eb191354cfa\") "
Sep 16 02:49:15.604215 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:49:15.604178    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e7244d4-fbe0-4519-805f-3eb191354cfa-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0e7244d4-fbe0-4519-805f-3eb191354cfa" (UID: "0e7244d4-fbe0-4519-805f-3eb191354cfa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 02:49:15.605801 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:49:15.605778    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e7244d4-fbe0-4519-805f-3eb191354cfa-kube-api-access-l7j88" (OuterVolumeSpecName: "kube-api-access-l7j88") pod "0e7244d4-fbe0-4519-805f-3eb191354cfa" (UID: "0e7244d4-fbe0-4519-805f-3eb191354cfa"). InnerVolumeSpecName "kube-api-access-l7j88". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 02:49:15.705519 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:49:15.705482    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l7j88\" (UniqueName: \"kubernetes.io/projected/0e7244d4-fbe0-4519-805f-3eb191354cfa-kube-api-access-l7j88\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:49:15.705519 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:49:15.705514    2558 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e7244d4-fbe0-4519-805f-3eb191354cfa-plugins-config-volume\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:49:16.399896 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:49:16.399864    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-24aabe-7bc4694f6b-k4j76"
Sep 16 02:49:16.400373 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:49:16.399897    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-24aabe-7bc4694f6b-k4j76" event={"ID":"0e7244d4-fbe0-4519-805f-3eb191354cfa","Type":"ContainerDied","Data":"68190c736cec01421ea48b267c799d096e2fe9c872e49e56642bc6b6163b3f70"}
Sep 16 02:49:16.400373 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:49:16.399989    2558 scope.go:117] "RemoveContainer" containerID="93e52de39ec76e9288a404c12d16eb6d4315b31e58285c6fc2b1df4f3d51b271"
Sep 16 02:49:16.420178 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:49:16.420148    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-24aabe-7bc4694f6b-k4j76"]
Sep 16 02:49:16.423966 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:49:16.423939    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-24aabe-7bc4694f6b-k4j76"]
Sep 16 02:49:17.549436 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:49:17.549404    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0e7244d4-fbe0-4519-805f-3eb191354cfa" path="/var/lib/kubelet/pods/0e7244d4-fbe0-4519-805f-3eb191354cfa/volumes"
Sep 16 02:49:32.915374 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:49:32.915327    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-a53a44-29825445-j986r"]
Sep 16 02:49:32.916978 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:49:32.916953    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-a53a44-29825445-j986r"]
Sep 16 02:49:33.549883 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:49:33.549847    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="34d8f74b-ffb7-40b1-9d88-cc958b9db01a" path="/var/lib/kubelet/pods/34d8f74b-ffb7-40b1-9d88-cc958b9db01a/volumes"
Sep 16 02:49:36.543855 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:49:36.543818    2558 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-processing-5d7dbbd56b-pltkr" secret="" err="secret \"payload-processing-dockercfg-zzx6j\" not found"
Sep 16 02:49:39.549961 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:49:39.549933    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-a53a44-openshift-default-75f7864f96-gh2ct"]
Sep 16 02:49:39.550375 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:49:39.550119    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-a53a44-openshift-default-75f7864f96-gh2ct" podUID="1e8ae5f7-3d2a-4725-81c3-a2ae2fdaaaa2" containerName="istio-proxy" containerID="cri-o://8ce56a48cc53fc4f138764b7304ac5ef6a829e3271279499037c1777bbfab333" gracePeriod=30
Sep 16 02:49:44.801390 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:49:44.801362    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-a53a44-openshift-default-75f7864f96-gh2ct"
Sep 16 02:49:44.875191 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:49:44.875123    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e8ae5f7-3d2a-4725-81c3-a2ae2fdaaaa2-workload-socket\") pod \"1e8ae5f7-3d2a-4725-81c3-a2ae2fdaaaa2\" (UID: \"1e8ae5f7-3d2a-4725-81c3-a2ae2fdaaaa2\") "
Sep 16 02:49:44.875191 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:49:44.875156    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e8ae5f7-3d2a-4725-81c3-a2ae2fdaaaa2-istiod-ca-cert\") pod \"1e8ae5f7-3d2a-4725-81c3-a2ae2fdaaaa2\" (UID: \"1e8ae5f7-3d2a-4725-81c3-a2ae2fdaaaa2\") "
Sep 16 02:49:44.875191 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:49:44.875190    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4qgqj\" (UniqueName: \"kubernetes.io/projected/1e8ae5f7-3d2a-4725-81c3-a2ae2fdaaaa2-kube-api-access-4qgqj\") pod \"1e8ae5f7-3d2a-4725-81c3-a2ae2fdaaaa2\" (UID: \"1e8ae5f7-3d2a-4725-81c3-a2ae2fdaaaa2\") "
Sep 16 02:49:44.875408 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:49:44.875212    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e8ae5f7-3d2a-4725-81c3-a2ae2fdaaaa2-workload-certs\") pod \"1e8ae5f7-3d2a-4725-81c3-a2ae2fdaaaa2\" (UID: \"1e8ae5f7-3d2a-4725-81c3-a2ae2fdaaaa2\") "
Sep 16 02:49:44.875408 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:49:44.875333    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e8ae5f7-3d2a-4725-81c3-a2ae2fdaaaa2-istio-envoy\") pod \"1e8ae5f7-3d2a-4725-81c3-a2ae2fdaaaa2\" (UID: \"1e8ae5f7-3d2a-4725-81c3-a2ae2fdaaaa2\") "
Sep 16 02:49:44.875408 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:49:44.875370    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e8ae5f7-3d2a-4725-81c3-a2ae2fdaaaa2-istio-token\") pod \"1e8ae5f7-3d2a-4725-81c3-a2ae2fdaaaa2\" (UID: \"1e8ae5f7-3d2a-4725-81c3-a2ae2fdaaaa2\") "
Sep 16 02:49:44.875408 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:49:44.875399    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e8ae5f7-3d2a-4725-81c3-a2ae2fdaaaa2-istio-data\") pod \"1e8ae5f7-3d2a-4725-81c3-a2ae2fdaaaa2\" (UID: \"1e8ae5f7-3d2a-4725-81c3-a2ae2fdaaaa2\") "
Sep 16 02:49:44.875622 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:49:44.875454    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e8ae5f7-3d2a-4725-81c3-a2ae2fdaaaa2-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "1e8ae5f7-3d2a-4725-81c3-a2ae2fdaaaa2" (UID: "1e8ae5f7-3d2a-4725-81c3-a2ae2fdaaaa2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:49:44.875622 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:49:44.875461    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e8ae5f7-3d2a-4725-81c3-a2ae2fdaaaa2-istio-podinfo\") pod \"1e8ae5f7-3d2a-4725-81c3-a2ae2fdaaaa2\" (UID: \"1e8ae5f7-3d2a-4725-81c3-a2ae2fdaaaa2\") "
Sep 16 02:49:44.875622 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:49:44.875468    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e8ae5f7-3d2a-4725-81c3-a2ae2fdaaaa2-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "1e8ae5f7-3d2a-4725-81c3-a2ae2fdaaaa2" (UID: "1e8ae5f7-3d2a-4725-81c3-a2ae2fdaaaa2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:49:44.875622 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:49:44.875573    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e8ae5f7-3d2a-4725-81c3-a2ae2fdaaaa2-credential-socket\") pod \"1e8ae5f7-3d2a-4725-81c3-a2ae2fdaaaa2\" (UID: \"1e8ae5f7-3d2a-4725-81c3-a2ae2fdaaaa2\") "
Sep 16 02:49:44.875622 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:49:44.875586    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e8ae5f7-3d2a-4725-81c3-a2ae2fdaaaa2-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "1e8ae5f7-3d2a-4725-81c3-a2ae2fdaaaa2" (UID: "1e8ae5f7-3d2a-4725-81c3-a2ae2fdaaaa2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 02:49:44.875900 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:49:44.875770    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e8ae5f7-3d2a-4725-81c3-a2ae2fdaaaa2-istio-data" (OuterVolumeSpecName: "istio-data") pod "1e8ae5f7-3d2a-4725-81c3-a2ae2fdaaaa2" (UID: "1e8ae5f7-3d2a-4725-81c3-a2ae2fdaaaa2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:49:44.875998 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:49:44.875972    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e8ae5f7-3d2a-4725-81c3-a2ae2fdaaaa2-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "1e8ae5f7-3d2a-4725-81c3-a2ae2fdaaaa2" (UID: "1e8ae5f7-3d2a-4725-81c3-a2ae2fdaaaa2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:49:44.876065 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:49:44.875999    2558 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e8ae5f7-3d2a-4725-81c3-a2ae2fdaaaa2-workload-certs\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:49:44.876065 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:49:44.876017    2558 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e8ae5f7-3d2a-4725-81c3-a2ae2fdaaaa2-istio-data\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:49:44.876065 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:49:44.876030    2558 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e8ae5f7-3d2a-4725-81c3-a2ae2fdaaaa2-workload-socket\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:49:44.876065 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:49:44.876042    2558 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e8ae5f7-3d2a-4725-81c3-a2ae2fdaaaa2-istiod-ca-cert\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:49:44.877504 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:49:44.877475    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1e8ae5f7-3d2a-4725-81c3-a2ae2fdaaaa2-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "1e8ae5f7-3d2a-4725-81c3-a2ae2fdaaaa2" (UID: "1e8ae5f7-3d2a-4725-81c3-a2ae2fdaaaa2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 02:49:44.877504 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:49:44.877493    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e8ae5f7-3d2a-4725-81c3-a2ae2fdaaaa2-istio-token" (OuterVolumeSpecName: "istio-token") pod "1e8ae5f7-3d2a-4725-81c3-a2ae2fdaaaa2" (UID: "1e8ae5f7-3d2a-4725-81c3-a2ae2fdaaaa2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 02:49:44.877654 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:49:44.877518    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e8ae5f7-3d2a-4725-81c3-a2ae2fdaaaa2-kube-api-access-4qgqj" (OuterVolumeSpecName: "kube-api-access-4qgqj") pod "1e8ae5f7-3d2a-4725-81c3-a2ae2fdaaaa2" (UID: "1e8ae5f7-3d2a-4725-81c3-a2ae2fdaaaa2"). InnerVolumeSpecName "kube-api-access-4qgqj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 02:49:44.877654 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:49:44.877541    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e8ae5f7-3d2a-4725-81c3-a2ae2fdaaaa2-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "1e8ae5f7-3d2a-4725-81c3-a2ae2fdaaaa2" (UID: "1e8ae5f7-3d2a-4725-81c3-a2ae2fdaaaa2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:49:44.976766 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:49:44.976726    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4qgqj\" (UniqueName: \"kubernetes.io/projected/1e8ae5f7-3d2a-4725-81c3-a2ae2fdaaaa2-kube-api-access-4qgqj\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:49:44.976766 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:49:44.976763    2558 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e8ae5f7-3d2a-4725-81c3-a2ae2fdaaaa2-istio-envoy\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:49:44.976766 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:49:44.976775    2558 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e8ae5f7-3d2a-4725-81c3-a2ae2fdaaaa2-istio-token\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:49:44.977006 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:49:44.976783    2558 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e8ae5f7-3d2a-4725-81c3-a2ae2fdaaaa2-istio-podinfo\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:49:44.977006 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:49:44.976791    2558 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e8ae5f7-3d2a-4725-81c3-a2ae2fdaaaa2-credential-socket\") on node \"ip-10-0-140-34.ec2.internal\" DevicePath \"\""
Sep 16 02:49:45.553245 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:49:45.553214    2558 generic.go:356] "Generic (PLEG): container finished" podID="1e8ae5f7-3d2a-4725-81c3-a2ae2fdaaaa2" containerID="8ce56a48cc53fc4f138764b7304ac5ef6a829e3271279499037c1777bbfab333" exitCode=0
Sep 16 02:49:45.553408 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:49:45.553267    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-a53a44-openshift-default-75f7864f96-gh2ct"
Sep 16 02:49:45.553408 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:49:45.553286    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-a53a44-openshift-default-75f7864f96-gh2ct" event={"ID":"1e8ae5f7-3d2a-4725-81c3-a2ae2fdaaaa2","Type":"ContainerDied","Data":"8ce56a48cc53fc4f138764b7304ac5ef6a829e3271279499037c1777bbfab333"}
Sep 16 02:49:45.553408 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:49:45.553324    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-a53a44-openshift-default-75f7864f96-gh2ct" event={"ID":"1e8ae5f7-3d2a-4725-81c3-a2ae2fdaaaa2","Type":"ContainerDied","Data":"1ad9a48c5fbc142a3522c8896d94c17c72be7badb6f68b57845e1f734b4f9192"}
Sep 16 02:49:45.553408 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:49:45.553345    2558 scope.go:117] "RemoveContainer" containerID="8ce56a48cc53fc4f138764b7304ac5ef6a829e3271279499037c1777bbfab333"
Sep 16 02:49:45.563545 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:49:45.563525    2558 scope.go:117] "RemoveContainer" containerID="2abbcfbc772a22c012be5618e2976bf43fc84c1176e7305eb471f6b44ca9298a"
Sep 16 02:49:45.571613 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:49:45.571544    2558 scope.go:117] "RemoveContainer" containerID="8ce56a48cc53fc4f138764b7304ac5ef6a829e3271279499037c1777bbfab333"
Sep 16 02:49:45.571881 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:49:45.571861    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ce56a48cc53fc4f138764b7304ac5ef6a829e3271279499037c1777bbfab333\": container with ID starting with 8ce56a48cc53fc4f138764b7304ac5ef6a829e3271279499037c1777bbfab333 not found: ID does not exist" containerID="8ce56a48cc53fc4f138764b7304ac5ef6a829e3271279499037c1777bbfab333"
Sep 16 02:49:45.571948 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:49:45.571889    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ce56a48cc53fc4f138764b7304ac5ef6a829e3271279499037c1777bbfab333"} err="failed to get container status \"8ce56a48cc53fc4f138764b7304ac5ef6a829e3271279499037c1777bbfab333\": rpc error: code = NotFound desc = could not find container \"8ce56a48cc53fc4f138764b7304ac5ef6a829e3271279499037c1777bbfab333\": container with ID starting with 8ce56a48cc53fc4f138764b7304ac5ef6a829e3271279499037c1777bbfab333 not found: ID does not exist"
Sep 16 02:49:45.571948 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:49:45.571907    2558 scope.go:117] "RemoveContainer" containerID="2abbcfbc772a22c012be5618e2976bf43fc84c1176e7305eb471f6b44ca9298a"
Sep 16 02:49:45.572182 ip-10-0-140-34 kubenswrapper[2558]: E0916 02:49:45.572161    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2abbcfbc772a22c012be5618e2976bf43fc84c1176e7305eb471f6b44ca9298a\": container with ID starting with 2abbcfbc772a22c012be5618e2976bf43fc84c1176e7305eb471f6b44ca9298a not found: ID does not exist" containerID="2abbcfbc772a22c012be5618e2976bf43fc84c1176e7305eb471f6b44ca9298a"
Sep 16 02:49:45.572228 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:49:45.572192    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2abbcfbc772a22c012be5618e2976bf43fc84c1176e7305eb471f6b44ca9298a"} err="failed to get container status \"2abbcfbc772a22c012be5618e2976bf43fc84c1176e7305eb471f6b44ca9298a\": rpc error: code = NotFound desc = could not find container \"2abbcfbc772a22c012be5618e2976bf43fc84c1176e7305eb471f6b44ca9298a\": container with ID starting with 2abbcfbc772a22c012be5618e2976bf43fc84c1176e7305eb471f6b44ca9298a not found: ID does not exist"
Sep 16 02:49:45.573455 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:49:45.573436    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-a53a44-openshift-default-75f7864f96-gh2ct"]
Sep 16 02:49:45.576767 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:49:45.576749    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-a53a44-openshift-default-75f7864f96-gh2ct"]
Sep 16 02:49:47.549799 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:49:47.549766    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1e8ae5f7-3d2a-4725-81c3-a2ae2fdaaaa2" path="/var/lib/kubelet/pods/1e8ae5f7-3d2a-4725-81c3-a2ae2fdaaaa2/volumes"
Sep 16 02:50:41.544157 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:50:41.544051    2558 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-processing-5d7dbbd56b-pltkr" secret="" err="secret \"payload-processing-dockercfg-zzx6j\" not found"
Sep 16 02:51:08.030749 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:51:08.030687    2558 scope.go:117] "RemoveContainer" containerID="bfb87fe09448a28336c3a98c966f9e0336268200b3f68f0d595b5da1c9ff28f6"
Sep 16 02:51:08.038994 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:51:08.038974    2558 scope.go:117] "RemoveContainer" containerID="ad67f4280e9de0539fe86156c28e4c397291a773e39d05ddfe1bac5809f0698f"
Sep 16 02:51:56.543951 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:51:56.543916    2558 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-processing-5d7dbbd56b-pltkr" secret="" err="secret \"payload-processing-dockercfg-zzx6j\" not found"
Sep 16 02:53:08.067111 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:53:08.067079    2558 scope.go:117] "RemoveContainer" containerID="5e06720532521b2a7ed37bc2d1ca59c2bbc2fb20ccb9c8588a5ea8f99b96e634"
Sep 16 02:53:08.075454 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:53:08.075434    2558 scope.go:117] "RemoveContainer" containerID="319df266619bf085f84e853987889086c22266e8d72194a67741c5353180b828"
Sep 16 02:53:17.546891 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:53:17.546852    2558 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-processing-5d7dbbd56b-pltkr" secret="" err="secret \"payload-processing-dockercfg-zzx6j\" not found"
Sep 16 02:54:08.097536 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:54:08.097503    2558 scope.go:117] "RemoveContainer" containerID="072938f8ca5c1485fe40b0436b010970ea1eb2ab407387b2a9d38579f2ef4940"
Sep 16 02:54:08.105682 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:54:08.105660    2558 scope.go:117] "RemoveContainer" containerID="e5dab622da112e4fb7a1bd21f1a030532a4feb2b970e6d366ac2b83d63369e48"
Sep 16 02:54:34.544175 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:54:34.544139    2558 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-processing-5d7dbbd56b-pltkr" secret="" err="secret \"payload-processing-dockercfg-zzx6j\" not found"
Sep 16 02:55:08.126756 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:55:08.126662    2558 scope.go:117] "RemoveContainer" containerID="3742d8457bc59f2ac2fc693624ff12654d2aad25759f9a3cc93c6b2f17b0b872"
Sep 16 02:55:35.544235 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:55:35.544194    2558 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-processing-5d7dbbd56b-pltkr" secret="" err="secret \"payload-processing-dockercfg-zzx6j\" not found"
Sep 16 02:56:48.544826 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:56:48.544784    2558 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-processing-5d7dbbd56b-pltkr" secret="" err="secret \"payload-processing-dockercfg-zzx6j\" not found"
Sep 16 02:58:05.544615 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:58:05.544575    2558 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-processing-5d7dbbd56b-pltkr" secret="" err="secret \"payload-processing-dockercfg-zzx6j\" not found"
Sep 16 02:58:45.846189 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:58:45.846161    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-f227r_f5cc1e46-3424-49f8-8f2c-ec44086643d9/global-pull-secret-syncer/0.log"
Sep 16 02:58:46.013945 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:58:46.013911    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-gmvp8_28767adf-af94-4851-947e-f0f2b21ef658/konnectivity-agent/0.log"
Sep 16 02:58:46.082605 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:58:46.082579    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-140-34.ec2.internal_74fa655d1549fe38db1b28b2b10dd892/haproxy/0.log"
Sep 16 02:58:52.189507 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:58:52.189475    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a2de3f62-ec4e-49fc-8d7c-42c62f64dd94/alertmanager/0.log"
Sep 16 02:58:52.209969 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:58:52.209943    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a2de3f62-ec4e-49fc-8d7c-42c62f64dd94/config-reloader/0.log"
Sep 16 02:58:52.231712 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:58:52.231616    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a2de3f62-ec4e-49fc-8d7c-42c62f64dd94/kube-rbac-proxy-web/0.log"
Sep 16 02:58:52.258882 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:58:52.258862    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a2de3f62-ec4e-49fc-8d7c-42c62f64dd94/kube-rbac-proxy/0.log"
Sep 16 02:58:52.281324 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:58:52.281302    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a2de3f62-ec4e-49fc-8d7c-42c62f64dd94/kube-rbac-proxy-metric/0.log"
Sep 16 02:58:52.301457 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:58:52.301435    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a2de3f62-ec4e-49fc-8d7c-42c62f64dd94/prom-label-proxy/0.log"
Sep 16 02:58:52.321443 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:58:52.321419    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a2de3f62-ec4e-49fc-8d7c-42c62f64dd94/init-config-reloader/0.log"
Sep 16 02:58:52.469485 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:58:52.469453    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-59787c5d8d-sth5n_c6abece5-0093-4c2e-b2e3-e0fd93abc7f1/metrics-server/0.log"
Sep 16 02:58:52.682423 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:58:52.682390    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-qvr4r_b8b6adb3-45cc-4eab-a2f7-b1435a4ac409/node-exporter/0.log"
Sep 16 02:58:52.702955 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:58:52.702935    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-qvr4r_b8b6adb3-45cc-4eab-a2f7-b1435a4ac409/kube-rbac-proxy/0.log"
Sep 16 02:58:52.721961 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:58:52.721939    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-qvr4r_b8b6adb3-45cc-4eab-a2f7-b1435a4ac409/init-textfile/0.log"
Sep 16 02:58:52.748516 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:58:52.748489    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-s686b_9e294a9b-44a5-499a-8c28-c88e4ef85b7e/kube-rbac-proxy-main/0.log"
Sep 16 02:58:52.774784 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:58:52.774710    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-s686b_9e294a9b-44a5-499a-8c28-c88e4ef85b7e/kube-rbac-proxy-self/0.log"
Sep 16 02:58:52.793752 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:58:52.793686    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-s686b_9e294a9b-44a5-499a-8c28-c88e4ef85b7e/openshift-state-metrics/0.log"
Sep 16 02:58:52.846072 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:58:52.846039    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_f4ed152f-8d15-4a7a-af3f-9e234b4cc1b8/prometheus/0.log"
Sep 16 02:58:52.865955 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:58:52.865921    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_f4ed152f-8d15-4a7a-af3f-9e234b4cc1b8/config-reloader/0.log"
Sep 16 02:58:52.885842 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:58:52.885815    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_f4ed152f-8d15-4a7a-af3f-9e234b4cc1b8/thanos-sidecar/0.log"
Sep 16 02:58:52.910456 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:58:52.910433    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_f4ed152f-8d15-4a7a-af3f-9e234b4cc1b8/kube-rbac-proxy-web/0.log"
Sep 16 02:58:52.929529 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:58:52.929505    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_f4ed152f-8d15-4a7a-af3f-9e234b4cc1b8/kube-rbac-proxy/0.log"
Sep 16 02:58:52.948627 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:58:52.948595    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_f4ed152f-8d15-4a7a-af3f-9e234b4cc1b8/kube-rbac-proxy-thanos/0.log"
Sep 16 02:58:52.968268 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:58:52.968246    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_f4ed152f-8d15-4a7a-af3f-9e234b4cc1b8/init-config-reloader/0.log"
Sep 16 02:58:56.569654 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:58:56.569621    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-x7gcl/perf-node-gather-daemonset-4rzcv"]
Sep 16 02:58:56.570044 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:58:56.569984    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa9e2461-89e4-4edc-8637-8f8934e91072" containerName="istio-proxy"
Sep 16 02:58:56.570044 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:58:56.569996    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa9e2461-89e4-4edc-8637-8f8934e91072" containerName="istio-proxy"
Sep 16 02:58:56.570044 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:58:56.570016    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a924eb26-efff-4c6e-85bc-8191ca817c87" containerName="revoke-keys"
Sep 16 02:58:56.570044 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:58:56.570022    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="a924eb26-efff-4c6e-85bc-8191ca817c87" containerName="revoke-keys"
Sep 16 02:58:56.570044 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:58:56.570028    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e7244d4-fbe0-4519-805f-3eb191354cfa" containerName="payload-pre-processing"
Sep 16 02:58:56.570044 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:58:56.570034    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e7244d4-fbe0-4519-805f-3eb191354cfa" containerName="payload-pre-processing"
Sep 16 02:58:56.570044 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:58:56.570046    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1e8ae5f7-3d2a-4725-81c3-a2ae2fdaaaa2" containerName="istio-proxy"
Sep 16 02:58:56.570334 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:58:56.570052    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="1e8ae5f7-3d2a-4725-81c3-a2ae2fdaaaa2" containerName="istio-proxy"
Sep 16 02:58:56.570334 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:58:56.570096    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cdd07dc7-673e-46cc-b248-4838609577c1" containerName="istio-proxy"
Sep 16 02:58:56.570334 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:58:56.570104    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="cdd07dc7-673e-46cc-b248-4838609577c1" containerName="istio-proxy"
Sep 16 02:58:56.570334 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:58:56.570116    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1e8ae5f7-3d2a-4725-81c3-a2ae2fdaaaa2" containerName="istio-proxy"
Sep 16 02:58:56.570334 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:58:56.570124    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="1e8ae5f7-3d2a-4725-81c3-a2ae2fdaaaa2" containerName="istio-proxy"
Sep 16 02:58:56.570334 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:58:56.570191    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="1e8ae5f7-3d2a-4725-81c3-a2ae2fdaaaa2" containerName="istio-proxy"
Sep 16 02:58:56.570334 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:58:56.570203    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="cdd07dc7-673e-46cc-b248-4838609577c1" containerName="istio-proxy"
Sep 16 02:58:56.570334 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:58:56.570209    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="cdd07dc7-673e-46cc-b248-4838609577c1" containerName="istio-proxy"
Sep 16 02:58:56.570334 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:58:56.570215    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="a924eb26-efff-4c6e-85bc-8191ca817c87" containerName="revoke-keys"
Sep 16 02:58:56.570334 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:58:56.570222    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="0e7244d4-fbe0-4519-805f-3eb191354cfa" containerName="payload-pre-processing"
Sep 16 02:58:56.570334 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:58:56.570227    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="aa9e2461-89e4-4edc-8637-8f8934e91072" containerName="istio-proxy"
Sep 16 02:58:56.570334 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:58:56.570234    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="1e8ae5f7-3d2a-4725-81c3-a2ae2fdaaaa2" containerName="istio-proxy"
Sep 16 02:58:56.573332 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:58:56.573317    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-x7gcl/perf-node-gather-daemonset-4rzcv"
Sep 16 02:58:56.575321 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:58:56.575301    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-x7gcl\"/\"kube-root-ca.crt\""
Sep 16 02:58:56.576009 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:58:56.575995    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-x7gcl\"/\"default-dockercfg-zwm8t\""
Sep 16 02:58:56.576128 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:58:56.576108    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-x7gcl\"/\"openshift-service-ca.crt\""
Sep 16 02:58:56.582798 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:58:56.582775    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-x7gcl/perf-node-gather-daemonset-4rzcv"]
Sep 16 02:58:56.632842 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:58:56.632801    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5537c2f4-7895-4caf-ad3c-b20013bb009b-proc\") pod \"perf-node-gather-daemonset-4rzcv\" (UID: \"5537c2f4-7895-4caf-ad3c-b20013bb009b\") " pod="openshift-must-gather-x7gcl/perf-node-gather-daemonset-4rzcv"
Sep 16 02:58:56.633019 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:58:56.632874    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfl9s\" (UniqueName: \"kubernetes.io/projected/5537c2f4-7895-4caf-ad3c-b20013bb009b-kube-api-access-mfl9s\") pod \"perf-node-gather-daemonset-4rzcv\" (UID: \"5537c2f4-7895-4caf-ad3c-b20013bb009b\") " pod="openshift-must-gather-x7gcl/perf-node-gather-daemonset-4rzcv"
Sep 16 02:58:56.633019 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:58:56.632942    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5537c2f4-7895-4caf-ad3c-b20013bb009b-sys\") pod \"perf-node-gather-daemonset-4rzcv\" (UID: \"5537c2f4-7895-4caf-ad3c-b20013bb009b\") " pod="openshift-must-gather-x7gcl/perf-node-gather-daemonset-4rzcv"
Sep 16 02:58:56.633153 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:58:56.633071    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5537c2f4-7895-4caf-ad3c-b20013bb009b-lib-modules\") pod \"perf-node-gather-daemonset-4rzcv\" (UID: \"5537c2f4-7895-4caf-ad3c-b20013bb009b\") " pod="openshift-must-gather-x7gcl/perf-node-gather-daemonset-4rzcv"
Sep 16 02:58:56.633224 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:58:56.633203    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5537c2f4-7895-4caf-ad3c-b20013bb009b-podres\") pod \"perf-node-gather-daemonset-4rzcv\" (UID: \"5537c2f4-7895-4caf-ad3c-b20013bb009b\") " pod="openshift-must-gather-x7gcl/perf-node-gather-daemonset-4rzcv"
Sep 16 02:58:56.734663 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:58:56.734630    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5537c2f4-7895-4caf-ad3c-b20013bb009b-podres\") pod \"perf-node-gather-daemonset-4rzcv\" (UID: \"5537c2f4-7895-4caf-ad3c-b20013bb009b\") " pod="openshift-must-gather-x7gcl/perf-node-gather-daemonset-4rzcv"
Sep 16 02:58:56.734863 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:58:56.734681    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5537c2f4-7895-4caf-ad3c-b20013bb009b-proc\") pod \"perf-node-gather-daemonset-4rzcv\" (UID: \"5537c2f4-7895-4caf-ad3c-b20013bb009b\") " pod="openshift-must-gather-x7gcl/perf-node-gather-daemonset-4rzcv"
Sep 16 02:58:56.734863 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:58:56.734746    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfl9s\" (UniqueName: \"kubernetes.io/projected/5537c2f4-7895-4caf-ad3c-b20013bb009b-kube-api-access-mfl9s\") pod \"perf-node-gather-daemonset-4rzcv\" (UID: \"5537c2f4-7895-4caf-ad3c-b20013bb009b\") " pod="openshift-must-gather-x7gcl/perf-node-gather-daemonset-4rzcv"
Sep 16 02:58:56.734863 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:58:56.734778    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5537c2f4-7895-4caf-ad3c-b20013bb009b-sys\") pod \"perf-node-gather-daemonset-4rzcv\" (UID: \"5537c2f4-7895-4caf-ad3c-b20013bb009b\") " pod="openshift-must-gather-x7gcl/perf-node-gather-daemonset-4rzcv"
Sep 16 02:58:56.734863 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:58:56.734802    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5537c2f4-7895-4caf-ad3c-b20013bb009b-lib-modules\") pod \"perf-node-gather-daemonset-4rzcv\" (UID: \"5537c2f4-7895-4caf-ad3c-b20013bb009b\") " pod="openshift-must-gather-x7gcl/perf-node-gather-daemonset-4rzcv"
Sep 16 02:58:56.734863 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:58:56.734851    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5537c2f4-7895-4caf-ad3c-b20013bb009b-proc\") pod \"perf-node-gather-daemonset-4rzcv\" (UID: \"5537c2f4-7895-4caf-ad3c-b20013bb009b\") " pod="openshift-must-gather-x7gcl/perf-node-gather-daemonset-4rzcv"
Sep 16 02:58:56.735135 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:58:56.734849    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5537c2f4-7895-4caf-ad3c-b20013bb009b-podres\") pod \"perf-node-gather-daemonset-4rzcv\" (UID: \"5537c2f4-7895-4caf-ad3c-b20013bb009b\") " pod="openshift-must-gather-x7gcl/perf-node-gather-daemonset-4rzcv"
Sep 16 02:58:56.735135 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:58:56.734875    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5537c2f4-7895-4caf-ad3c-b20013bb009b-sys\") pod \"perf-node-gather-daemonset-4rzcv\" (UID: \"5537c2f4-7895-4caf-ad3c-b20013bb009b\") " pod="openshift-must-gather-x7gcl/perf-node-gather-daemonset-4rzcv"
Sep 16 02:58:56.735135 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:58:56.734947    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5537c2f4-7895-4caf-ad3c-b20013bb009b-lib-modules\") pod \"perf-node-gather-daemonset-4rzcv\" (UID: \"5537c2f4-7895-4caf-ad3c-b20013bb009b\") " pod="openshift-must-gather-x7gcl/perf-node-gather-daemonset-4rzcv"
Sep 16 02:58:56.742725 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:58:56.742272    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfl9s\" (UniqueName: \"kubernetes.io/projected/5537c2f4-7895-4caf-ad3c-b20013bb009b-kube-api-access-mfl9s\") pod \"perf-node-gather-daemonset-4rzcv\" (UID: \"5537c2f4-7895-4caf-ad3c-b20013bb009b\") " pod="openshift-must-gather-x7gcl/perf-node-gather-daemonset-4rzcv"
Sep 16 02:58:56.884758 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:58:56.884645    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-x7gcl/perf-node-gather-daemonset-4rzcv"
Sep 16 02:58:57.154266 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:58:57.154241    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-x7gcl/perf-node-gather-daemonset-4rzcv"]
Sep 16 02:58:57.156522 ip-10-0-140-34 kubenswrapper[2558]: W0916 02:58:57.156494    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod5537c2f4_7895_4caf_ad3c_b20013bb009b.slice/crio-35712067d0dd74768e5f68e8de940bc856662f63e2a44113f4760361aa85b53d WatchSource:0}: Error finding container 35712067d0dd74768e5f68e8de940bc856662f63e2a44113f4760361aa85b53d: Status 404 returned error can't find the container with id 35712067d0dd74768e5f68e8de940bc856662f63e2a44113f4760361aa85b53d
Sep 16 02:58:57.162114 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:58:57.162084    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-x7gcl/perf-node-gather-daemonset-4rzcv" event={"ID":"5537c2f4-7895-4caf-ad3c-b20013bb009b","Type":"ContainerStarted","Data":"35712067d0dd74768e5f68e8de940bc856662f63e2a44113f4760361aa85b53d"}
Sep 16 02:58:58.168332 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:58:58.168295    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-x7gcl/perf-node-gather-daemonset-4rzcv" event={"ID":"5537c2f4-7895-4caf-ad3c-b20013bb009b","Type":"ContainerStarted","Data":"b78968984b4d95547425699d96e77ed4c950bad0af2902f2390e86e3c9fa66d6"}
Sep 16 02:58:58.168727 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:58:58.168361    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-x7gcl/perf-node-gather-daemonset-4rzcv"
Sep 16 02:58:58.184365 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:58:58.184304    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-x7gcl/perf-node-gather-daemonset-4rzcv" podStartSLOduration=2.184286374 podStartE2EDuration="2.184286374s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 02:58:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:58:58.18318569 +0000 UTC m=+2751.101674939" watchObservedRunningTime="2026-09-16 02:58:58.184286374 +0000 UTC m=+2751.102775622"
Sep 16 02:58:58.192804 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:58:58.192778    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-q8tpf_dd03566d-e8e3-4aad-b3b2-0a06e43e5eea/dns/0.log"
Sep 16 02:58:58.213154 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:58:58.213127    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-q8tpf_dd03566d-e8e3-4aad-b3b2-0a06e43e5eea/kube-rbac-proxy/0.log"
Sep 16 02:58:58.336364 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:58:58.336338    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-vrf75_e0800020-f9de-4dfa-9a7b-d7c88ef0561f/dns-node-resolver/0.log"
Sep 16 02:58:58.903510 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:58:58.903478    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-vn8jb_9562b5b9-856c-4a81-89d3-8faed08f792c/node-ca/0.log"
Sep 16 02:58:59.858824 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:58:59.858794    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-z9zbw_d128e57b-91ba-4ed8-ab92-837ead863c2d/istio-proxy/0.log"
Sep 16 02:59:00.484682 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:59:00.484653    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-6nqh9_dfc5e005-4a80-42db-b611-e4c01f9bbb76/discovery/0.log"
Sep 16 02:59:00.558608 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:59:00.558576    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-tc8md_98d6db2c-d022-45af-89d5-5ba2eb8518b7/istio-proxy/0.log"
Sep 16 02:59:00.562587 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:59:00.562553    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-tc8md_98d6db2c-d022-45af-89d5-5ba2eb8518b7/istio-proxy/1.log"
Sep 16 02:59:00.601517 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:59:00.601491    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5d7dbbd56b-pltkr_4189cfcd-c3ba-4bad-813f-5a6851cb5317/payload-processing/0.log"
Sep 16 02:59:01.540857 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:59:01.540822    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-b7cxx_504ae285-7f73-4ebf-b39b-192c08b6e922/serve-healthcheck-canary/0.log"
Sep 16 02:59:02.153712 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:59:02.153665    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-qnjpw_d0cba7f9-30a5-4c62-bc04-976491dc83e4/kube-rbac-proxy/0.log"
Sep 16 02:59:02.172371 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:59:02.172349    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-qnjpw_d0cba7f9-30a5-4c62-bc04-976491dc83e4/exporter/0.log"
Sep 16 02:59:02.192122 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:59:02.192096    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-qnjpw_d0cba7f9-30a5-4c62-bc04-976491dc83e4/extractor/0.log"
Sep 16 02:59:04.182947 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:59:04.182918    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-x7gcl/perf-node-gather-daemonset-4rzcv"
Sep 16 02:59:04.206686 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:59:04.206653    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-ff88c4765-6bc9x_4a3a2d7f-ef64-4b21-b1d6-0bb19ff6eb0e/manager/0.log"
Sep 16 02:59:04.662133 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:59:04.662099    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-zcg5h_09e730e6-3b3f-488c-99d9-d285e5ff195e/manager/0.log"
Sep 16 02:59:04.849775 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:59:04.849745    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-74fbd865d6-7sp4h_f13e89aa-f62e-48e3-9f2d-137e6cc6276e/manager/0.log"
Sep 16 02:59:04.868518 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:59:04.868497    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-74fbd865d6-7sp4h_f13e89aa-f62e-48e3-9f2d-137e6cc6276e/copy-manifests/0.log"
Sep 16 02:59:06.318510 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:59:06.318425    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-z9zbw_d128e57b-91ba-4ed8-ab92-837ead863c2d/istio-proxy/0.log"
Sep 16 02:59:06.413887 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:59:06.413859    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-79c8b99f94-b6hpk_2cacabd8-aaf3-4b2b-948b-bd75714a4bf6/manager/0.log"
Sep 16 02:59:06.940368 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:59:06.940334    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-6nqh9_dfc5e005-4a80-42db-b611-e4c01f9bbb76/discovery/0.log"
Sep 16 02:59:06.985383 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:59:06.985345    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-tc8md_98d6db2c-d022-45af-89d5-5ba2eb8518b7/istio-proxy/0.log"
Sep 16 02:59:06.989315 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:59:06.989293    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-tc8md_98d6db2c-d022-45af-89d5-5ba2eb8518b7/istio-proxy/1.log"
Sep 16 02:59:07.013102 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:59:07.013078    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5d7dbbd56b-pltkr_4189cfcd-c3ba-4bad-813f-5a6851cb5317/payload-processing/0.log"
Sep 16 02:59:08.529267 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:59:08.529233    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-z9zbw_d128e57b-91ba-4ed8-ab92-837ead863c2d/istio-proxy/0.log"
Sep 16 02:59:09.162201 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:59:09.162168    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-6nqh9_dfc5e005-4a80-42db-b611-e4c01f9bbb76/discovery/0.log"
Sep 16 02:59:09.223475 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:59:09.223439    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-tc8md_98d6db2c-d022-45af-89d5-5ba2eb8518b7/istio-proxy/0.log"
Sep 16 02:59:09.227751 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:59:09.227726    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-tc8md_98d6db2c-d022-45af-89d5-5ba2eb8518b7/istio-proxy/1.log"
Sep 16 02:59:09.252931 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:59:09.252902    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5d7dbbd56b-pltkr_4189cfcd-c3ba-4bad-813f-5a6851cb5317/payload-processing/0.log"
Sep 16 02:59:12.712078 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:59:12.712051    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-84qhm_58f0674c-b5b4-4d0e-8d14-3b18a232d365/kube-multus/0.log"
Sep 16 02:59:13.039557 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:59:13.039477    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-h7czj_a5be83aa-c2b7-4808-a32d-80a845f3435a/kube-multus-additional-cni-plugins/0.log"
Sep 16 02:59:13.058963 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:59:13.058937    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-h7czj_a5be83aa-c2b7-4808-a32d-80a845f3435a/egress-router-binary-copy/0.log"
Sep 16 02:59:13.078646 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:59:13.078614    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-h7czj_a5be83aa-c2b7-4808-a32d-80a845f3435a/cni-plugins/0.log"
Sep 16 02:59:13.098901 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:59:13.098878    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-h7czj_a5be83aa-c2b7-4808-a32d-80a845f3435a/bond-cni-plugin/0.log"
Sep 16 02:59:13.117947 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:59:13.117922    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-h7czj_a5be83aa-c2b7-4808-a32d-80a845f3435a/routeoverride-cni/0.log"
Sep 16 02:59:13.136621 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:59:13.136592    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-h7czj_a5be83aa-c2b7-4808-a32d-80a845f3435a/whereabouts-cni-bincopy/0.log"
Sep 16 02:59:13.155220 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:59:13.155197    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-h7czj_a5be83aa-c2b7-4808-a32d-80a845f3435a/whereabouts-cni/0.log"
Sep 16 02:59:13.314247 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:59:13.314172    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-ck2vw_25e85d58-77e2-4428-b099-a5075363d56f/network-metrics-daemon/0.log"
Sep 16 02:59:13.329759 ip-10-0-140-34 kubenswrapper[2558]: I0916 02:59:13.329733    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-ck2vw_25e85d58-77e2-4428-b099-a5075363d56f/kube-rbac-proxy/0.log"